KR20220122334A - Apparatus for analysing and providing soft keyboard and method thereof - Google Patents

Abstract

A keyboard analysis system disclosed in the present document includes: a mobile device; and a decoding module. The present invention displays a designated character through a display, obtains a touch input corresponding to the designated character through the display, and sends acquired positional coordinates of the touch input and information about the designated character to the decoding module. The decoding module may be configured to analyze the received information using deep learning and generate an invisible soft keyboard on the display based on an analysis result. Therefore, the keyboard analysis system can provide an optimized keyboard by reflecting a user's body structure or a typing habit.

Description

Apparatus for analyzing and providing a soft keyboard and a method therefor

Various embodiments provided in this document relate to techniques for analyzing and providing a soft keyboard.

Since the interaction between a human and a computer through keyboard input is the most intuitive and convenient, it is still most commonly used with the invention of the computer. With the development of information and communication technology (ICT) technology, mobile devices have become common, and a bulky, heavy, and less portable physical keyboard has been replaced with a soft keyboard (or virtual keyboard). The soft keyboard has the same keyboard layout as the physical keyboard, and since it is provided through the touch display of the device, it does not require a separate connection or installation, so it has an advantage in portability.

The soft keyboard can ensure portability of the mobile device, but the size of the soft keyboard is inevitably limited due to the limited size of the touch display. The user has to press about 30 keys using only the thumb or index finger, and since there is no boundary space between the keys and there is no physical feedback, the use of the soft keyboard increases the incidence of typos compared to the use of the physical keyboard.

In addition, since the soft keyboard displayed to obtain a user's typing input occupies a certain space of the touch display, it may result in interference with display of other graphic user interfaces (GUIs).

Various embodiments disclosed in this document are intended to provide a keyboard analysis system for solving the above-described problems and methods related thereto.

A keyboard analysis system according to an embodiment disclosed in this document includes a mobile device, and a decoding module, wherein the mobile device displays a designated character through a display, and on the designated character through the display Obtaining a corresponding touch input, and transmitting the position coordinates of the obtained touch input and information about the designated character to the decoding module, the decoding module, using deep learning, the received information Analyze and generate an invisible soft keyboard on the display based on the analysis result.

The operating method of the keyboard analysis system according to an embodiment disclosed in this document is an operation of displaying a specified character through a display, an operation of obtaining a touch input corresponding to the specified character through the display, and deep learning using , analyzing the obtained location coordinates of the touch input and information on the designated character, and generating an invisible soft keyboard on the display based on the analysis result.

A non-transitory computer-readable storage medium according to an embodiment disclosed in this document, when executed by one or more processors, displays a designated character through a display, and receives a touch input corresponding to the designated character through the display keyboard analysis to obtain, using deep learning, to analyze the position coordinates of the acquired touch input and information about the designated character, and to generate an invisible soft keyboard on the display based on the analysis result It may store one or more programs containing instructions that cause the system to be stored.

According to the embodiments disclosed in this document, the keyboard analysis system may provide a keyboard optimized by reflecting the user's body structure or typing habit.

In addition, according to the embodiments disclosed in this document, the keyboard analysis system may correct typos in units of characters while allowing the user to type faster.

In addition, according to the embodiments disclosed in this document, the keyboard analysis system may provide a keyboard that guarantees the user's autonomy in the existing mobile device without additionally purchasing the mobile device.

In addition, according to the embodiments disclosed in this document, the keyboard analysis system can reduce typos while maximizing the utilization of the display screen by providing a transparent keyboard that is not displayed on the display.

In addition, various effects directly or indirectly identified through this document may be provided.

1 illustrates a display screen for obtaining input data of a user according to various embodiments of the present disclosure;
2 illustrates a distribution of user input data obtained according to various embodiments.
3 illustrates an architecture of a keyboard analysis system in accordance with various embodiments.
4 illustrates an algorithm structure of a long-term decoder according to various embodiments.
5 illustrates a soft keyboard provided through a keyboard analysis system according to various embodiments of the present disclosure;
6 illustrates a table comparing typographical errors between keyboards according to various embodiments of the present disclosure;
7 illustrates an ablation study result of a soft keyboard provided through a keyboard analysis system according to various embodiments of the present disclosure;
8 illustrates a table comparing typing speeds between keyboards according to various embodiments.
9 is a graph illustrating a user experience result of a soft keyboard provided through a keyboard analysis system according to various embodiments of the present disclosure;
10 illustrates an example of using a soft keyboard provided through a keyboard analysis system according to various embodiments of the present disclosure;
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that various modifications, equivalents, and/or alternatives of the embodiments of the present invention are included.

In this document, the singular form of a noun corresponding to an item may include one or more items, unless the context clearly dictates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C” and “A, Each of the phrases "at least one of B, or C" may include any one of, or all possible combinations of, items listed together in the corresponding one of the phrases. Terms such as "first", "second", or "first" or "second" may simply be used to distinguish an element from other elements in question, and may refer elements to other aspects (e.g., importance or order) is not limited. that one (e.g. first) component is "coupled" or "connected" to another (e.g. second) component with or without the terms "functionally" or "communicatively" When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

Each component (eg, a module or a program) of components described in this document may include a singular or a plurality of entities. According to various embodiments, one or more components or operations among the corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

As used herein, the term “module” or “unit” may include a unit implemented in hardware, software, or firmware, and includes terms such as, for example, logic, logic block, component, or circuit; They can be used interchangeably. A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (eg, a program or an application) including one or more instructions stored in a storage medium (eg, memory) readable by a machine. For example, the processor of the device may call at least one of the one or more instructions stored from the storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

1 to 2 describe an operation for acquiring input data of a user according to various embodiments of the present disclosure. The screen 100 illustrated in FIG. 1 is only an interface generated to check the distribution of a user's typing input, and does not represent an example of a transparent soft keyboard according to various embodiments.

Since the soft keyboard provided by the keyboard analysis system is in a transparent form that is not visible on the display, a location touched by a user to input a designated character may be different for each size of the display, a user's body structure, or a typing habit.

For example, referring to FIG. 1 , a screen 100 may represent a display screen of a mobile device having portability, such as a smart phone, a tablet PC, or a wearable device. The screen 100 may include an output field 110 in which a sentence to be typed by the user is displayed, and an input field 120 in which the user types (touch input) a sentence displayed in the output field 110 . The mobile device may store the location coordinates (eg, x y coordinates) of the sensed touch input on the input field 120 and information about the size (eg, height and width) of the input field 120 . Even when a plurality of users type the same sentence, distributions of touch inputs sensed on the input field 120 may be different from each other as shown in FIG. 2 . In addition, since a key for inputting a character is not visible, the initial point of starting typing is different for each user, and even if the same user starts typing again after a predetermined interval, the initial point may be changed.

The keyboard analysis system according to various embodiments may reduce typos and increase typing speed by reflecting a display size, a user's body structure, or a typing habit while providing a transparent soft keyboard that can make the most of a display screen.

3 illustrates an architecture of a keyboard analysis system in accordance with various embodiments. Referring to FIG. 3 , the keyboard analysis system 300 may include a mobile device 310 , a messenger module 320 , and a decoding module 330 .

The mobile device 310 can include an input interface 312 and an output interface 314 . The input interface 312 may refer to a configuration (eg, a touch circuit) configured to sense a touch input on the display in order to perform a soft keyboard function. The output interface 314 may refer to a configuration that provides a visual user interface (UI) such as a display. The mobile device 310 may generate the soft keyboard generated by the decoding module 330 on the display in a transparent form. The mobile device 310 transmits the touch input obtained through the soft keyboard to the decoding module 330 through the messenger module 320, and displays a character corresponding to the touch input recognized through the decoding module 330 on the display. can be printed on

The messenger module 320 may serve as an interface between the mobile device 310 and the decoding module 330 . For example, the messenger module 320 may process information obtained by the mobile device 310 to be suitable for the decoding module 330 . Conversely, the messenger module 320 may process the information generated by the decoding module 330 to be suitable for the mobile device 310 . The messenger module 320 may be, for example, a separate entity from the mobile device 310 and the decoding module 330 , or may be a program (eg, an application) stored in the mobile device 310 .

The messenger module 320 may include a pre-processing module 322 , a transmission module 324 , and an input buffer 326 according to functions. The pre-processing module 322 may pair the touch information acquired by the mobile device 310 to a designated character. In this document, 'touch information' may include location coordinates of a touch input sensed on the display (or input field 120 of FIG. 1 ) of the mobile device 310 and the size of the display or input field 120 . can The pre-processing module 322 may perform tokenization so that the paired information is suitable for the decoding module 330 . Tokenization can be understood as the operation of dividing a sequence of characters into specified units (eg words or characters). Additionally, the preprocessing module 322 may further perform at least one of cleaning or normalization before or after tokenization. As another example, the preprocessing module 322 may convert the information received from the decoding module 330 into an input form suitable for the mobile device 310 . The transmission module 324 may be used by the messenger module 320 to transmit/receive data to and from the mobile device 310 and the decoding module 330 . The input buffer 326 may store data received from the mobile device 310 or the decoding module 330 . For example, as will be described below, the long-term decoder 334 of the decoding module 330 may estimate the meaning of a specific touch input only when a specified number of touch inputs or more are acquired, so that the input buffer 326 is A series of data can be stored.

The decoding module 330 learns the touch information acquired by the mobile device 310 and the characters corresponding thereto by using deep learning, and corresponds to the touch input acquired by the mobile device 310 after learning. character (or sentence) can be recognized. The decoding module 330 may be a server that transmits and receives data to and from the mobile device 310 and the messenger module 320 . For another example, the decoding module 330 may be programmed in the mobile device 310 together with the messenger module 320 .

The decoding module 330 may include a short-term decoder 332 and a long-term decoder 334 according to the number of characters for decoding. The short-term decoder 332 may be set to learn and analyze less than a specified number (eg, 9) of characters. The short term decoder 332 may include, for example, a multi-layer perceptron (MLP). Since the long-term decoder 334 needs to understand the front and back contexts in order to recognize the current character, it is necessary to learn and analyze more than a specified number of characters. Accordingly, the decoding module 330 can quickly and simply analyze the input to the soft keyboard by using the short-term decoder 332 until a specified number of characters are input. On the other hand, since the long-term decoder 334 estimates the character of the corresponding touch input by decoding all previous touch inputs as well as the current touch input, it is possible to more precisely estimate the meaning and correct typos.

The long-term decoder 334 includes a geometric decoder 410 that analyzes a character corresponding to a touch input based on geometric information and a semantic decoder 410 that analyzes a character corresponding to a touch input based on semantic information. A decoder 420 may be included. The geometric decoder 410 may convert the input vector into a character sequence by using the position coordinates of the touch inputs obtained through the display. Also, the geometric decoder 410 may determine a candidate character for the current input coordinates using the relative distance between the input location coordinates. The semantic decoder 420 may serve as a character language model (CLM) for determining errors by semantically analyzing the character sequence determined by the geometric decoder 410 and correcting typos by reflecting context. Also, the semantic decoder 420 may determine a final character from among candidate characters by using a semantic relationship between the current input and other inputs. The algorithm structures for the geometric decoder 410 and the semantic decoder 420 are described below in FIG. 4 .

4 illustrates an algorithmic structure 400 of a long term decoder in accordance with various embodiments. Referring to FIG. 4 , the long-term decoder 334 of the decoding module 300 includes a bi-directional gated recurrent unit (BiGRU) as a geometric decoder and a transformer model as a semantic decoder. have.

Since the keyboard analysis system 300 needs to understand the positional relationship between the touch inputs ( t ₁ , t ₂ , t ₃ ,..., t _n ) on the transparent soft keyboard, it is necessary to determine the unidirectional recurrent hidden state (recurrent hidden state). state) and a bidirectional GRU using a repeating hidden state in the opposite direction can provide higher stability. The bidirectional GRU may determine a specific character according to the input coordinate of the touch input and the size of the display.

The softmax function may determine confidence for a character determined by the bidirectional GRU. For example, when the user inputs the word 'RUN', if the character 'RUN' is determined by the bidirectional GRU, the confidence for each touch input may be high. On the other hand, when the character 'RJN' is determined by the bidirectional GRU, the confidence for 'J' may be determined to be low by the softmax function.

The confidence masking module may blank a position where the confidence is less than a specified threshold value. For example, in the previous example, if the word 'RJN' is determined by the bidirectional GRU and the confidence for the position of 'J' is determined to be low, the confidence masking is to blank the position corresponding to 'J' such as 'R_N' can be processed

The transformer module may analyze sequence data using a self-attention mechanism. The self-attention mechanism uses query(Q), key(K), and value(V) parameters, and when analyzing the input, consider the current input as well as the previous inputs as a whole, It can be understood as a mechanism that considers related inputs more intensively. The transformer module can improve the performance of natural language processing by grasping the meaning existing in a sentence from multiple perspectives by applying self-attention as a multi-head. The description of the transformer module can be referred to through the paper “Attention is all you need” published in 2017. For example, the transformer module may determine a character (eg, 'U') of a blanked and delivered position through the entire context of a word or sentence.

According to embodiments, the transformer module of the keyboard analysis system 300 may use only an encoder without using a decoder. The existing transformer module assumes that the input and output lengths may be different, but the keyboard analysis system 300 that matches the position coordinates of the touch input and the character does not need to use a decoder because the input and output lengths are the same. .

According to embodiments, the keyboard analysis system 300 may use various natural language processing (NLP) models to better enable the semantic decoder to generate multi-head self-attention within a character sequence. For example, the keyboard analysis system 300 generates efficient backpropagation using intermediate layer losses among the auxiliary losses for CLM, and applies layer normalization. can be changed to before self-attention to induce a stable gradient and fast convergence.

5 illustrates a soft keyboard provided through a keyboard analysis system in accordance with various embodiments. The screens 510 , 520 , 530 , and 540 illustrated in FIG. 5 are for visually explaining the analysis result of the decoding module 330 , and are not output through the display of the mobile device 310 .

Referring to FIG. 5 , the keyboard analysis system 300 may estimate a character corresponding to a current touch input by analyzing the input character sequence. For example, when the user inputs 'have to stand' through the soft keyboard of the mobile device 310 and inputs up to , the keyboard analysis system 300 converts the sentence to 'have to stand up' in consideration of contextual meaning. , and it may be determined that the character corresponding to the current touch input is 'u'. The keyboard analysis system 300 generates a transparent key representing 'u' on a partial area of the display like the screen 510, but considers the probability that 'u' can be input. area can be determined. The keyboard analysis system 300 may generate transparent keys for other inputable characters other than 'u' on the remaining area of the display. The arrangement of the transparent keys may be determined according to a basic standard (eg, QWERTY) of the soft keyboard. In the same principle, when 'was at a recor' is input, the keyboard analysis system 300 may generate a transparent key indicating 'd' on a partial area of the display like the screen 520 . In the same principle, when 'he is able t' is input, the keyboard analysis system 300 may generate a transparent key indicating 'o' on a partial area of the display like the screen 530 . In the same principle, when 'he is able to' is input, the keyboard analysis system 300 may generate a transparent key indicating '(space)' on a partial area of the display like the screen 520 .

Through the above method, the keyboard analysis system 300 can correct typos even when typing without restriction of finger positions, thereby increasing the typing speed of users.

6 illustrates a table comparing typographical errors between keyboards according to various embodiments of the present disclosure;

6, the graph is a soft keyboard (trident module invisible (TMI) keyboard) generated based on the algorithm of the keyboard analysis system 300 of FIG. 4 and existing models (eg, GRU, LSTM (long-short) term memory), a transformer, and an I-keyboard) can be used to compare results of keyboards generated. MacK, Common, and Test may refer to a data set for collecting touch input data.

I-keyboard may mean a model in which GRU is implemented with two modules. CER (character error rate) indicates the error rate in units of characters, and WER (word error rate) indicates the error rate in units of words. A higher CER or WER may mean that errors increase. According to the graph, if the layer (eg network size) is deepened, the performance of GRU, LSTM, and I-keyboard decreases, while the performance of transformer and TMI keyboard does not decrease significantly. Although the transformer performs natural language processing using the relationship between input sequences through multi-head attention, it cannot use the geometric relationship between typing inputs, so performance may be lower than that of the TMI keyboard.

7 illustrates an ablation study result of a soft keyboard provided through a keyboard analysis system according to various embodiments of the present disclosure;

Referring to FIG. 7 , the graph shows a soft keyboard (TMI keyboard) according to the keyboard analysis system 300 and ACC ( accuracy) and the result of comparing WER. A masked language model (MLM) may mean a masked input.

8 illustrates a table comparing typing speeds between keyboards according to various embodiments.

Referring to FIG. 8 , the graph compares WPM (words per minute) results between a soft keyboard (TMI keyboard) and other keyboards (eg, KeyScretch, Invisible keyboard, and I-keyboard) according to the keyboard analysis system 300 . will be. The soft keyboard according to the keyboard analysis system 300 may provide a high typing speed as shown in the graph of FIG. 8 while increasing accuracy and reducing an error rate.

9 is a graph illustrating a user experience result of a soft keyboard provided through a keyboard analysis system according to various embodiments of the present disclosure;

Referring to FIG. 9 , the graph represents user satisfaction with the use of the soft keyboard provided through the keyboard analysis system 300 . The graph is based on 5 points on the Likert scale, showing how hard it was mentally and physically, how much time it took (temporal demand), comfort (comfortability), and overall satisfaction (overall satisfaction). represents a response to

10 illustrates an example of using a soft keyboard provided through a keyboard analysis system according to various embodiments of the present disclosure;

Referring to FIG. 10 , the user of the mobile device 310 may input a sentence (eg, 'then') using the transparent soft keyboard even when the application screen (eg, a chat window) is displayed on the entire display area. . Through this, the keyboard analysis system 300 may remove the area occupied by the soft keyboard on the display screen and maximize screen utilization.

Claims (10)

In the keyboard analysis system,
mobile device; and
a decoding module,
the mobile device,
Display a specified character through a display, obtain a touch input corresponding to the specified character through the display, and transmit the position coordinates of the obtained touch input and information about the specified character to the decoding module, and ,
The decoding module,
A keyboard analysis system configured to analyze the received information using deep learning and generate an invisible soft keyboard on the display based on the analysis result. The method according to claim 1,
The arrangement of the soft keyboard is changed based on a character displayed on the display or a previous character sequence. The method according to claim 1, The decoding module,
a short-term decoder configured to analyze less than a specified number of said touch inputs; and
a long-term decoder configured to analyze more than the specified number of touch inputs;
The long-term decoder is
a geometric decoder that analyzes a character corresponding to the touch input based on geometric information; and
and a semantic decoder for analyzing a character corresponding to the touch input based on semantic information of a character sequence. 4. The method of claim 3,
The geometric decoder includes a bi-directional gated recurrent unit (GRU),
wherein the semantic decoder comprises a transformer model. The method according to claim 4, wherein the long-term decoder,
calculating confidence for the touch input using the bidirectional GRU and softmax function;
If the confidence is less than a specified threshold, the keyboard analysis system, configured to process a position corresponding to the touch input as a blank (blank). The method according to claim 4, The transformer model,
A keyboard analysis system comprising an encoder and no decoder. The method according to claim 1,
The keyboard analysis system further comprising a messenger module configured to tokenize and/or normalize the location coordinates of the touch input and information about the designated character. The method according to claim 7, wherein the messenger module,
Corresponding to the application stored on the mobile device, keyboard analysis system. In the operating method of the keyboard analysis system,
displaying the specified character through the display;
acquiring a touch input corresponding to the designated character through the display;
analyzing the position coordinates of the acquired touch input and information about the designated character using deep learning; and
and generating an invisible soft keyboard on the display based on the analysis result. A non-transitory computer-readable storage medium comprising:
When executed by one or more processors,
Display the specified character through the display,
acquiring a touch input corresponding to the designated character through the display;
using deep learning to analyze the location coordinates of the acquired touch input and information about the designated character, and
A non-transitory computer-readable storage medium storing one or more programs comprising instructions for causing a keyboard analysis system to generate an invisible soft keyboard on the display based on the analysis result.

Images