KR102079985B1 - Method And Device For Processing Touch Input - Google Patents

Abstract

Disclosed is a touch input processing method for a displayed keyboard interface. The touch input processing method includes: receiving a touch input for the keyboard interface including a plurality of keys; Obtaining Gaussian mixture probability information for the touch input; Obtaining prediction probability information based on a preceding touch input of the touch input; And mapping the touch input to a key value using the Gaussian mixture probability information and the prediction probability information.

Description

Touch input processing method and device {Method And Device For Processing Touch Input}

The present invention relates to a touch input processing method and device / device, and more particularly, to generate a probabilistic model by analyzing touch input data on a displayed soft keyboard interface, and to generate a user's touch input according to the generated probabilistic model. It's about how to map the keys to your intentions.

Due to the development of display panel manufacturing technology and electronic device manufacturing technology, display devices of various sizes are becoming popular. Such devices may include a touch sensor to recognize a user's touch on the display panel and perform an operation accordingly. Recently, various portable devices, for example, a mobile phone, a notebook, a PDA, a tablet PC, and the like, provide such a touch recognition function.

Touch-sensitive display devices can display a soft keyboard interface for user's character input. However, a typing error occurs due to a user's touch input method, a hand shape, a user's posture, a size of a display panel, and the like. Accordingly, touch input error correction methods based on language models have been developed. For example, the device may predict the character to be input according to the input characters, and correct the touch input error accordingly.

The error correction method according to language analysis / prediction requires large size data for various words, which are combinations of characters, and has a weakness in incompatibility because different models are required for each language. Therefore, the present specification is to propose a method that can reduce the typing error of the user while requiring less language-dependent constraints.

In order to solve the above technical problem, a touch input processing method for a displayed keyboard interface is disclosed.

A touch input processing method according to an embodiment of the present invention comprises the steps of: receiving a touch input for the keyboard interface including a plurality of keys; Obtaining Gaussian mixture probability information for the touch input; Obtaining prediction probability information based on a preceding touch input of the touch input; And mapping the touch input to a key value using the Gaussian mixture probability information and the prediction probability information, wherein the Gaussian mixture probability information is based on a Gaussian mixture model for the plurality of pre-processed touch inputs. And a mapping probability of the touch input to at least one key, and the prediction probability information indicates a key input probability of the current touch input predicted by the preceding touch input.

In the touch input processing method according to an embodiment of the present invention, the mapping of the touch input to a key value using the Gaussian mixture probability information and the prediction probability information may include mapping the Gaussian mixture probability information and the prediction probability information. Using to obtain first final probability information; And mapping the touch input to the key value based on the first final probability information.

In the touch input processing method according to an embodiment of the present invention, the obtaining of the predicted probability information includes, when there is a preceding touch input of the touch input, repeats second final probability information on the preceding touch input in a cyclic neural network. And outputting the predicted probability information by processing, wherein the cyclic neural network delivers information on the preceding cyclic neural network processing through cyclic weights, and the cyclic neural network model learns a touch input sequence according to a time sequence. to be.

In the touch input processing method according to an embodiment of the present invention, the cyclic weight is reset based on a specific number of touch input sequences, word units, or a specific user input.

In the touch input processing method according to the embodiment of the present invention, obtaining first final probability information by using the Gaussian mixed probability information and the prediction probability information comprises: a first probability vector corresponding to the Gaussian mixed probability information; And an element-wise product of a second probability vector corresponding to the prediction probability information.

In the touch input processing method according to an embodiment of the present invention, at least one of the first probability vector and the second probability vector may be weighted.

In the touch input processing method according to an embodiment of the present invention, the Gaussian mixture probability information or the prediction probability information may be adjusted to use only a probability for at least one key adjacent to the touch input.

In the touch input processing method according to an embodiment of the present invention, obtaining the Gaussian mixture probability information for the touch input includes: generating a Gaussian model for each key region; Generating a Gaussian mixture model for the plurality of key regions of the key interface; And obtaining the Gaussian mixture probability information for the touch input from the Gaussian mixture model.

In order to solve the above technical problem, a touch input processing device is disclosed.

A touch input processing device according to an embodiment of the present invention includes a memory unit for storing touch input data for an application for processing touch input and a displayed keyboard interface; And a processing unit for driving the application to process the touch input, wherein the processing unit receives a touch input for the keyboard interface including a plurality of keys, and receives Gaussian mixture probability information for the touch input. Acquires, obtains predictive probability information based on a preceding touch input of the touch input, and maps the touch input to a key value using the Gaussian mixed probability information and the predicted probability information, wherein the Gaussian mixed probability information is And a mapping probability of the touch input for at least one key, based on a Gaussian mixture model for the plurality of pre-processed touch inputs, wherein the prediction probability information is a current touch input predicted by the preceding touch input. Represents a key input probability for.

According to an embodiment of the present invention, a typo may be effectively corrected when a user inputs a touch. In particular, the present invention mathematically models the touch distribution and uses the distribution model to infer user intentions to map the received touch input to key values. Therefore, the present invention does not depend on the language of the keyboard interface, and thus can be applied to keyboard interfaces for various languages more universally. In addition, since the present invention does not depend on a language, a typo can be corrected by one modeling of one keyboard shape, and a plurality of modelings are not required on one keyboard shape.

The present invention can further infer and map touch inputs closer to the user's intention by using the distribution of touches and, in addition, predictive touch input by a cyclic neural network. In addition to modeling using touch distribution, by using a cyclic neural network based on a language model, it is possible to improve the accurate reasoning of touch input.

In addition, the effect on the present invention will be further described in the following description.

1 illustrates a display device and a soft keyboard interface provided by the display device according to an embodiment of the present invention.
2 illustrates touch input data and error occurrences for the keyboard interface.
3 illustrates two key areas among key areas included in a keyboard interface.
4 illustrates touch input data for two key areas and two key areas included in a keyboard interface.
5 illustrates a method of initializing touch input data according to an embodiment of the present invention.
6 illustrates a Gaussian model generation method for each key region according to an embodiment of the present invention.
7 illustrates a Gaussian mixture model generation method for a keyboard interface according to an embodiment of the present invention.
8 illustrates a GMM region determination method according to an embodiment of the present invention.
9 illustrates a mapping method of a touch input using a GMM and a high reliability region according to an embodiment of the present invention.
10 illustrates a mapping method of a touch input using a GMM and a high reliability region according to an embodiment of the present invention.
11 illustrates a key mapping method of a received touch input according to an embodiment of the present invention.
12 is a conceptual diagram of a recurrent neural network.
13 illustrates a Gaussian mixture distribution according to an embodiment of the present invention.
14 illustrates a probability vector according to an embodiment of the present invention.
15 illustrates a touch input processing method according to an embodiment of the present invention.
16 shows a learning process of a cyclic neural network according to the embodiment of the present invention.
17 illustrates a touch input processing method according to another embodiment of the present invention.
18 illustrates Gaussian mixture probability information and prediction probability information according to an embodiment of the present invention.
19 illustrates a touch input processing device according to an embodiment of the invention.
20 illustrates a touch input processing method according to an embodiment of the present invention.

The terminology used herein is a general term that has been widely used as far as possible in consideration of functions in the present specification, but may vary according to the intention of a person skilled in the art, convention or the emergence of a new technology. In addition, in certain cases, there is a term arbitrarily selected by the applicant, in which case the meaning will be described in the description of the corresponding embodiment. Therefore, it is to be understood that the terminology used herein is to be interpreted based on the actual meaning and contents of the present specification, not on the terms of simple terms.

As used herein, the display device and the touch input processing method of the display device mean that the display device includes various electronic devices. For example, a mobile phone, a personal digital assistant (PDA), a notebook computer, a tablet PC, an MP3 player, a CD player, It includes various electronic devices capable of displaying visual information such as a DVD player, a head mounted display (HMD), a smart watch, a watch phone, a television, a kiosk, a TV, and the like. Hereinafter, for convenience of description, the display device may be referred to as a 'device'.

The invention will now be described with reference to a device for carrying out the method of the invention. However, the present invention may be implemented and performed by an application or software. In this case, the device is a device for driving an application or software for implementing / implementing the present invention, and the device is to implement the present invention according to the driving of the application or software. Therefore, the following description of the present invention with reference to the device, the description of the invention should be considered as both a description of the application / software for implementing the present invention. The description of the method of the following specification and claims should be considered as describing the operation of the application / software in accordance with an embodiment of the present invention. Thus, the following description of the present invention naturally applies to software / applications coded to perform the method of the present invention separately from the device. The operations of the devices described below may all be understood as operations of the software / application.

1 illustrates a display device and a soft keyboard interface provided by the display device according to an embodiment of the present invention.

1 shows a display device 1010 according to an embodiment of the invention. Although the display device 1010 is shown as an embodiment of a tablet PC, the display device 1010 of the present invention is defined by the configuration and operation of the device, not the type or shape of the device.

The device 1010 may display the soft keyboard interface 1020. Soft keyboard interface 1020 represents a keyboard interface that includes at least one key region provided by being displayed without a physical mechanical configuration. The soft keyboard interface 1020 may be referred to herein as a virtual keyboard interface, a keyboard interface, or a soft keyboard.

The key area refers to an area corresponding to one displayed key. However, an area recognized as a corresponding key value may be represented as a key area instead of the displayed area. As an embodiment, even when the keys of the displayed keyboard interface are spaced from each other, that is, even if there is a gap between the displayed keys, the reference area for recognizing the key value may not coincide with this gap. Therefore, in the present specification, the key area may indicate a displayed key layout or may indicate a default recognition area recognized by the corresponding key.

The keyboard interface 1020 may be provided in various forms. In FIG. 1, a keyboard interface 1020 having a key arrangement generally provided is illustrated as an example, but the shape of the keyboard interface 1020, the number of keys included, the arrangement of keys, the layout of the keyboard, and the like may vary according to embodiments. Can be. In addition, although FIG. 1 illustrates an embodiment in which the keyboard interface 1020 is displayed at the bottom of the display, the display position, orientation, and size of the keyboard interface 1020 are not limited thereto. The keyboard interface 1020 includes a plurality of key areas. And basically, the device maps touch inputs for key areas to the corresponding key values. For example, a touch input with coordinates in the key region of A is mapped to a key value of A.

2 illustrates touch input data and error occurrences for the keyboard interface.

When the keyboard interface is displayed as shown in FIG. 1, the user applies an input using a body part or an input tool to the display panel on which the keyboard interface is displayed. As means for sensing user input to the display, the device may include a touch sensor. The user input recognition method for the display of the touch sensor is a well-known technology and thus will not be described in detail.

The touch input may be dataized by the predetermined protocol of the touch sensor and the processor. That is, when a user applies a touch input to the display, the device may recognize the touch input as a touch sensor and the recognized touch input may be processed as touch input data. The touch input data may include coordinate values (x, y). In an embodiment, the touch input data may further include touch start coordinates, touch end coordinates, touch input time, touch duration, touch area, additional sensor information of a touch time, environment information of the device, and the like. In an embodiment, the touch input data may include additional sensor information, and may include tilt information and acceleration information of the device when the touch input is stored.

As in FIG. 2, the displayed keyboard interface includes a plurality of key areas. In the present specification, the key area may indicate an area corresponding to one displayed key. The device may recognize the touch input by mapping the touch input to the key value of the key region including the touch input coordinates. For example, when the location of the received touch input is located within the key region 2010 of 'ㅕ', the device may map the received touch input to a key value of 'ㅕ'. However, as shown in FIG. 2, an input for a key area not intended by the user may occur at the boundary of the key areas.

In FIG. 2, the dots represent stored touch input data. That is, the coordinate values of the touch input data are displayed as dots. In FIG. 2, the touch inputs marked with X are touch inputs recognized near the boundary of the key areas. As shown, in the case of specific keys, it can be seen that the distribution of touch input for the keys is biased in a specific direction. In FIG. 2, it can be seen that touch inputs to the key region 2010 of 'ㅕ' are biased to the right. Therefore, for the key area 2020 of the right 'ㅑ' key area 2010, the touch inputs of the left boundary of the key area 2020 correspond to the user's intention to view the touch input of the 'ㅕ' as a touch input. . Accordingly, the touch inputs of the left boundary of the key region 2020 of the 'ㅑ' may be determined as a typo of the user, and the typos may be corrected by mapping them to the key value of the 'ㅕ' rather than the key value of the 'ㅑ'.

The present invention thus proposes a method of mapping the touch inputs to the boundary of the key areas to key values suitable for the user's intention through mathematical modeling. Hereinafter, a method of mapping a touch input for a key region of the keyboard interface to a key value according to a user's intention will be described in detail.

3 illustrates two key areas among key areas included in a keyboard interface.

3 illustrates a first key region 3010 for inputting a character A and a second key region 3020 for inputting a character S. Referring to FIG. The characters A and S are selected as embodiments, and the first key region 3010 and the second key region 3020 may be mapped to key numbers corresponding to arbitrary numbers and characters according to the configuration of the keyboard interface.

4 illustrates touch input data for two key areas and two key areas included in a keyboard interface.

As in FIG. 4, user input data for the keyboard interface may be collected. The user input data may include touch coordinates. The device may store more than a preset number of touch input data for a preset period of time, and perform touch input processing by analyzing and modeling the stored touch input data. The analysis and modeling of such data may be set to be performed when the user does not use the device or when the device is connected to the charger.

The device may perform data analysis and modeling when the touch input data for the entire keyboard interface is greater than or equal to a preset number. For example, when 3000 or more touch coordinate values of the touch input data are collected, the device may perform data analysis and modeling. However, in this case, the total number of data is 3000 or more, but the number of touch coordinates may be biased in a specific key area. Thus, the device of the present invention can consider not only the number of global coordinate values but also the number of data for each key region. For example, when the number of recently received touch inputs for the keyboard interface is 3000 or more, and the number of received touch inputs for each key area is 25 or more, the device may perform data analysis and modeling.

When the device stores touch input data, the device may randomize the storage order. The present invention performs touch input processing by analyzing / modeling only touch positions without performing language based touch input guessing. Therefore, even if the touch input data is not stored in order, it does not affect the analysis / modeling performance of the data. By randomizing the storage order, exposure of personal information such as passwords and specific word sequences can be minimized. In addition, when storing the touch input data, an input to a personal information input interface such as a password field may not be stored. Touch input data is used locally only by the device and is not transmitted externally. However, if necessary, the touch input data may be transmitted to a server linked with the device. In this case, the server may be a server in which a program that manages / controls an app driven to implement the present invention in a device is stored and driven.

As shown in FIG. 4, the user input for the first key area 3010 and the second key area 3020 and the user input for S may have a distribution biased to the right. In this case, the touch inputs 4010 that are adjacent to the left side of the second key area 3020, that is, the boundary of the first key area 3010, are not the inputs to the second key area 3020, but the first key area 3010. Viewing as input to is consistent with the user's intent. Thus, mapping of inputs to the boundaries of these adjacent key regions can have a decisive effect on reducing typing errors. Hereinafter, a method of mapping such key inputs to key values suitable for user intention will be described in detail.

5 illustrates a method of initializing touch input data according to an embodiment of the present invention.

First, the present invention sets the base areas 5010 and 5020 in the key areas 3010 and 3020 of the keyboard interface, and starts modeling the touch input data in the base area. The device may map touch input data in the base area to key values of the corresponding area, and this mapping may be referred to as labeling. Labeling is the act of assigning the correct answer, which means mapping the received input to the correct result. In the present invention, the operation of mapping the received touch input to a correct key value may correspond to labeling. In the embodiment of FIG. 5, the device maps touch inputs in the first base area 5010 to a key value of 'A' and maps touch inputs in the second base area 5020 to a key value of 'S'. Can be.

The first base area 5010 is positioned at the center of the displayed first key area 3010 and may have an area of a certain percentage of the key area. In an embodiment, the first base area 5010 with respect to the first key area 3010 occupies 80% of the area of the first key area 3010, and the shape is the same as that of the first key area 3010. It can be set to a shape. In an embodiment, the area, shape, and position of each base area with respect to the respective key areas may be the same, or may be different.

The reason for using the base area is as follows. In FIG. 5, when modeling all touch inputs included in the key area to model the second key area 3020, the touch inputs 4010 intended for the input of the first key area are the second key area 3020. Modeled as input to. Therefore, since it is difficult to perform key value mapping according to user intention to the boundary area, the present invention first performs modeling based on the touch input of the base area, and then performs additional modeling using touch inputs other than the base area. Suggest a method.

The base area represents an area that can be estimated with a high probability that the key area will not be out of the user's intention even if it is recognized as a touch input to the corresponding key area.

In an embodiment, the device may delete the touch input data when the number of touch input data included in the base area in each key area is equal to or less than a preset number. That is, when the number of touch input data included in the base area is less than the preset number, since the number of samples for proper Gaussian modeling is small, the Gaussian modeling for the corresponding key area may not be performed. In this case, if only the touch input data of the base area is deleted, the touch data remaining in the key area may affect the generation of the GMM for the entire keyboard interface. Therefore, when the number of touch input data for the base area is less than the preset number, the device may delete data for the key area as well as the base area, and generate the GMM using the deleted data.

6 illustrates a Gaussian model generation method for each key region according to an embodiment of the present invention.

As shown in FIG. 6, the present invention maps the touch input data 6010 in the first base area 5010 of the first key area 3010 to 'A'. That is, the touch input data 6010 in the first base area 5010 is mapped to a key value corresponding to the first key area 3010. In addition, the present invention can learn a Gaussian distribution using the mapped data. Training of the Gaussian distribution may be referred to as Gaussian model generation.

Gaussian probabilistic models are probabilistic models that represent a distribution of aggregated observations around a mean. The mathematical description of the Gaussian model itself is not detailed.

The present invention can generate a Gaussian model for each of the key areas included in the keyboard interface. For example, after the Gaussian distribution learning for the first key region, the present invention uses the touch input data 6020 in the second base region 5020 of the second key region 3020 to learn the second Gaussian distribution. This operation can be performed for all key areas included in the keyboard interface. However, Gaussian modeling may be skipped for key regions where the number of sample data is less than a certain number.

Gaussian distribution can be learned / generated based on Maximum Likelihood. The learned parameters may be used as parameters of the GMM to be described later.

As shown in FIG. 6, since the present invention learns a Gaussian distribution of individual key regions based on the base region, the Gaussian distribution of the first region 3010 and the Gaussian distribution of the second region 3020 are shifted to the right, respectively. You can expect it. That is, the mean of the Gaussian distribution (model) of the first region 3010 may be modeled to be close to the position of the touch input data 6010 within the first base region 5010. Similarly, the mean of the Gaussian distribution (model) of the second region 3020 will be modeled close to the location of the touch input data 6020 within the second base region 5020.

7 illustrates a Gaussian mixture model generation method for a keyboard interface according to an embodiment of the present invention.

6, the present invention now generates a Gaussian Mixture Model (GMM) for the entire keyboard interface. The present invention models the GMM using all of the touch input data not used in the process up to FIG. 6 as shown in FIG. 7. In this case, the present invention can model the GMM using an Expectation Maximization (EM) technique. The GMM may indicate the probability of which key value the touch input for each key region of the keyboard interface should be mapped to.

Gaussian probabilistic models are probabilistic models that represent a distribution of aggregated observations around a mean. Therefore, there is a limitation that can express only a unimodal form in which data is grouped into a group centered on an average. Accordingly, the present invention uses GMM to represent a plurality of probability distributions for a plurality of keys included in a keyboard interface.

The expectation maximization technique is a method used to estimate the probability model with hidden random variables. That is, the EM algorithm is an iterative algorithm that finds parameters with maximum likelihood or maximum a posteriori (MAP) in a probability model that depends on unobserved latent variables. The EM algorithm alternates the expected value (E) step of calculating the expected value of the log likelihood with the estimated value of the parameter and the maximized (M) step of obtaining the variable value that maximizes the expected value. The variable value calculated in the maximization step is then used as an estimate for the next expected value step.

The GMM is generated using a plurality of Gaussian models for each key region generated in the process up to FIG. 6. As an example, the GMM includes parameters of weight, mean, and covariance, wherein the double weight, mean, uses the parameters obtained for each Gaussian distribution, and the covariance is 1 / n. Can be initialized. n represents the number of keys modeled by GMM.

However, in the present invention, the parameter is optimized by repeatedly performing the EM technique in the modeling process of the GMM, and a value of the parameter may be out of an appropriate range and may be optimized to a local minimum value. For example, the position of the average of the Gaussian model with respect to region A in FIG. 7 may invade the key region of S, or the covariance may be too wide. Thus, the present invention may limit the width at which the mean and covariance can vary for each key region. For example, the present invention may limit an area where an average of the Gaussian model for each key area may exist to an internal area such as the base area described above. You can also set limits on the possible ranges of covariance.

Using the learned GMM, the likelihood of the Gaussian distribution with respect to the coordinates of the received touch input may be calculated. Then, by comparing the probabilities of all Gaussian distributions with respect to one coordinate, the key value having the highest probabilities can be estimated as the user's intention. However, instead of comparing the probabilities of Gaussian distributions for all keys in order to reduce the amount of computation, the present invention may perform the comparison of the probabilities only for the input key and the adjacent keys. As an example of a typical alphabetic keyboard layout, the present invention provides a key in which a user input is located, such as performing a communicative comparison on S, A, D, or S, A, D, W, X for an input to S. You can also perform a communicative comparison on an area and two to eight adjacent key areas.

Hereinafter, a method of mapping a received touch input to a key value by comparing the probabilities of the plurality of Gaussian distributions included in the GMM will be described in more detail.

8 illustrates a GMM region determination method according to an embodiment of the present invention.

Fig. 8 shows a method for determining a GMM area by taking three key areas A, S, and D as an example. The three layout areas 8010 of FIG. 8A represent three key areas of the displayed keyboard interface. 8 shows a GMM for three key regions 8010 and a method for configuring the GMM regions accordingly.

8 (b) shows a GMM for three key areas A, S, and D. FIG. The GMM includes Gaussian distributions for each key region. In FIG. 8B, the GMM includes a Gaussian distribution 8020 for A, a Gaussian distribution 8030 for S, and a Gaussian distribution 8040 for D. In FIG. The present invention can map the coordinates to key values of the Gaussian model if the probability of a specific coordinate is based on each Gaussian model of the GMM or more.

In the embodiment of FIG. 8, it may be assumed that the coordinates of the specific touch input 8050 correspond to the position of x. The x position has the highest communist value for the Gaussian model 8040 of D in the GMM, which is greater than the threshold that can map the x coordinate to the key value of D. Accordingly, although the device exists in the key area of S of the key area 8010 where the touch input 8050 is displayed, the device may map the touch input 8050 to a key value of D instead of S.

As shown in FIG. 8 (c), the present invention may configure a GMM region by collecting coordinates whose communicative value is greater than or equal to a threshold value from the GMM. In FIG. 8C, the GMM region is a first region 8060 mapped to a key value of A, a second region 8070 mapped to a key value of S, and a third region 8080 mapped to a key value of D. It includes.

The threshold value can be determined / adjusted to reflect the user's intent. For example, increasing the threshold reduces the size of the GMM area that can correct typos, and only corrects typos for touch inputs that are close to the average of each Gaussian model. Therefore, the accuracy of the typo correction can be increased, but the frequency of the typo correction can be reduced. Lowering the threshold can correct typos for touch inputs that are relatively far from the boundary. Instead, there is a risk that the area can be recognized as a typo by the intended touch input. Therefore, the accuracy of the typo correction may be lowered, but the frequency of the typo correction may be high. The present invention is characterized by setting an appropriate threshold value in consideration of such a tradeoff.

The present invention is intended to correct typing errors that are not intended by the user, and determining and correcting the intended typing as an error is the most avoiding factor. Therefore, in addition to performing key mapping with only GMM or placing a threshold, a high-reliability area may be set in a key area, and touch input to this area may always be mapped to a key value for that area. have.

9 illustrates a mapping method of a touch input using a GMM and a high reliability region according to an embodiment of the present invention.

FIG. 9 (a) shows three key areas of the keyboard interface displayed as shown in FIG. 8 (a).

9 (b) further shows a high reliability region within each key region. The high reliability area refers to an area that should be determined as a key input to the display area when a touch input is received in the area. As shown in FIG. 9B, the touch input in which the coordinate value is recognized into the high reliability region 9010 of the A key region is recognized as the A key regardless of the GMM.

FIG. 9C is a diagram illustrating the GMM region described above with reference to FIG. 8, and overlapping descriptions thereof will be omitted.

9 (d) shows final key mapping regions for mapping a touch input to a key value. As shown in FIG. 9 (d), once the high reliability region is mapped to the corresponding key value irrespective of the GMM, the high reliability region additionally sets regions beyond the threshold in the GMM. The high reliability region and the remaining regions of the GMM region may be mapped according to the key values of the displayed key region. The finally determined regions may be referred to as key mapping regions. The key mapping area of A, the key mapping area of S, and the key mapping area of D are determined as shown in FIG. 9 (d). The entire keyboard interface including the plurality of key mapping regions may be referred to as a calibrated key mapping interface.

The high reliability region is located in the central region within the key region, and the size can be changed according to the setting. For example, when the user's touch input pattern, that is, the average value of the GMMs, is generally far from the center of each key area, the size of the high reliability area may be reduced to increase the error correction capability. Alternatively, when the user's touch input pattern is generally located close to the center of each key area, the size of the high reliability area may be increased to reduce incorrect typographical correction and to improve typo correcting accuracy. The location of the high reliability region may also be adjusted according to the user's touch input pattern to improve the error correction capability and accuracy. For example, in the key region of 'D' of FIG. 9 according to the user's touch input pattern, the high reliability region may be moved to the left to improve both error correction capability and accuracy.

10 illustrates a mapping method of a touch input using a GMM and a high reliability region according to an embodiment of the present invention.

FIG. 10 shows the method shown in FIG. 9 with respect to the x plane.

Fig. 10 (a) shows the displayed layout area of Fig. 9 (a), Fig. 10 (b) shows the high reliability area of Fig. 9 (b), and Fig. 10 (c) shows the GMM area of Fig. 9 (c). 10 (d) and 9 (d) show the final determined key mapping regions in the x plane, respectively.

As shown in FIG. 10, in determining the final region, the present invention considers the GMM region, but does not modify the high reliability region. That is, the present invention configures the final key mapping region so that the GMM region for 'D' of FIG. 10 (c) does not exceed the high reliability region for 'S' of FIG. 10 (b).

In addition, the present invention configures a key mapping area according to the GMM area for an area greater than or equal to a specific threshold in the GMM, and follows the mapping of the displayed key area for the high reliability area and the outside of the GMM area.

11 illustrates a key mapping method of a received touch input according to an embodiment of the present invention.

As described above with reference to FIGS. 8 to 10, the present invention may configure a key mapping area for the entire keyboard interface and perform key mapping according to a position in the key mapping area of the coordinates of the received touch input. However, in this embodiment, the amount of pre-computation may increase. Therefore, as another embodiment, the present invention may process the received touch input for each coordinate without configuring the entire key mapping area. 11 is a flow chart illustrating this method.

The device may receive a touch input for the displayed keyboard interface in operation S11010. The received touch input includes coordinate information.

The device may first determine whether the coordinates of the received touch input are located within the high reliability area (S11020). If the touch input coordinates exist in the high reliability region, the device may map the received touch input to a key value of the high reliability region (S11030).

If the coordinates of the received touch input are not located in the high reliability region, the device may determine whether the GMM corresponding to the coordinates of the touch input is equal to or greater than the threshold (S11040). If the GMM corresponding to the touch input coordinates is equal to or larger than the threshold, the device may map the received touch input to a key value of the corresponding GMM region (S11050). In other words, the device may map the received touch input to a key value of the Gaussian model if the communicability of the received touch input coordinates is equal to or greater than a threshold value for one of the plurality of Gaussian models included in the GMM. have.

If the coordinates of the received touch input are outside the high reliability region and less than the communicative threshold of the GMM, the device may map the touch input to a key value of a key region where the touch input is located (S11060).

Hereinafter, the touch input processing method according to the embodiment of the present invention will be described again.

As described above, the touch input processing method according to the present invention may be performed by a device or by an application / software driven by the device.

The device may store touch input data for the keyboard interface. The displayed keyboard interface may include at least one key area, which may be referred to as a layout area. The touch input data may include at least one of coordinates of the touch input, an area of the touch input, a duration, and additional sensor information at the time of the touch input. The device may randomly store the storage order of the touch input data.

The device may generate a Gaussian model for each of the key regions. The device may generate a Gaussian model / distribution using the stored touch input data. The device may generate a Gaussian model using only touch input data in the base area of each key area. The device may generate a Gaussian model for all or some of the key areas included in the keyboard interface.

The device may use the generated Gaussian model to generate a Gaussian mixture model (GMM) for the keyboard interface. The device may generate a Gaussian mixture model by using touch input data in the base area and touch input data other than the base area together. The description of FIGS. 3 to 8 applies to the Gaussian model generation of the device and the method of generating the Gaussian mixed model using the same, and overlapping descriptions will not be repeated.

The device may use a Gaussian mixture model to map incoming touch inputs to key values. The device may map the received touch input to a key value of a key region where the received touch input is located or a key value of an adjacent key region of the key region where the received touch input is located. The key value may represent a character, a number, a symbol, or the like displayed on the keyboard interface, or a digital value corresponding to the character, number, symbol, or the like.

The device may perform key value mapping of the received touch input as described with reference to FIGS. 9 through 11. The device may map the touch input based on the key mapping area reflecting the high reliability area and the GMM area. Alternatively, the device may map the touch input based on the coordinate of the touch input as in the method of FIG. 11.

The device of the present invention is characterized in that a touch input is mapped to a key value of a key region where a touch input is located or a key mapping region where a touch input is located to a key value of an adjacent key region. That is, the input of a specific key region may be mapped to the key value of an adjacent key region according to the distribution of the received touch input without changing the layout of the displayed key regions.

The device may first determine whether the location of the received touch input is a high reliability area of a particular key area. If the received touch input is located in the high reliability region, the device may map the received touch input to a key value of a key region including the corresponding high reliability region. When the received touch input is located outside the high reliability region, the device may map the received touch input to a key value of a key region where the received touch input is located or a key value of an adjacent key region based on the Gaussian mixture model.

The method of mapping the received touch input based on the mixed Gaussian model is as described with reference to FIGS. 8 to 11. If the communicative value of the touch input is equal to or greater than the threshold value of the GMM, the device may map the received touch input to a key value of the GMM that is greater than or equal to the threshold value. The key value of the GMM that is greater than or equal to the threshold value may be the key value of the Gaussian model whose communicative value of the corresponding coordinate is greater than or equal to the threshold value among the plurality of Gaussian models represented by the GMM. When the communicative value of the received touch input is less than a predetermined threshold value of the mixed Gaussian model, the device may map the received touch input to a key value of a key region in which the received touch input is located.

As described above, the size, shape, position, etc. of at least one of the base region and the high reliability region may be variable. According to the embodiment, the base region and the high reliability region may be set identically.

In addition to the above-described method, a method of processing touch input using a cyclic neural network will be described.

In the present specification, the above-described GMM may be referred to as a Gaussian mixture model or Gaussian mixture distribution.

12 is a conceptual diagram of a recurrent neural network.

Cyclic neural networks are a type of deep learning model. Cyclic neural networks are used for continuous signals (testing, speech) and can be used for machine translation and speech recognition. Cyclic neural networks are effective models for ordered patterns.

A cyclic neural network is a kind of artificial neural network, in which a connection between units has a cyclic structure. The cyclic structure of the cyclic neural network allows the state to be stored inside the neural network to model time-varying dynamic features. Thus, cyclic artificial neural networks are advantageous for processing data with time-varying features.

As shown in FIG. 12, the cyclic neural network module may receive an input X_t and output an output h_t. Only the circulatory nerve module that receives X_t and outputs h_t is affected by the operation on the outputs of h_0 to h_t-1 from the inputs of X_0 to X_t-1. The cyclic neural network can connect the neural network of the past with the current neural network when processing new data through the recurrent weight. Thus, past experiences in data processing can be reflected in new data processing. Past experience with data processing can be reflected in the computation of cyclic neural networks as weights.

Tasks to which cyclic neural networks are mainly applied include speech recognition, machine translation, and video classification. These tasks train the model using the correct label with the data. However, since the correct answer label does not exist in the case of touch input, it is difficult to apply the same learning as the existing task. Therefore, the present invention describes a method of processing a touch sequence through cyclic neural network learning of unsupervised learning.

Hereinafter, after the touch input is processed using the Gaussian mixture model described above, a method of further processing the touch input using the cyclic neural network model will be described.

13 illustrates a Gaussian mixture distribution according to an embodiment of the present invention.

As described above with reference to FIGS. 2 through 11, the device generates a Gaussian mixture probability distribution by GMM modeling the touch distribution. The expansion rule vector for the touch input may be generated, which is summarized in FIG. 13.

As shown in FIG. 13A, a keyboard interface is displayed, and examples of a, s, and d are described in FIG. 13.

As shown in FIG. 13 (b), the Gaussian mixture distribution for a, s, and d is modeled. Modeling of Gaussian mixed distribution is as described above. The Gaussian mixture distribution of FIG. 13 (b) can be thought of as a map representing the mapping probability for each key value at the coordinates on the keyboard. The probabilities for each key are modeled in a Gaussian form, and a plurality of Gaussian models are collected to form a Gaussian mixture distribution.

As shown in FIG. 13C, probability values for specific touch coordinates are obtained. FIG. 13C shows that the probability values of the respective keys for the specific touch coordinates are 0.2 for a, 0.8 for s, and 0.1 for d. In this case, the vector combining the probability values for each key may be referred to as the probability vector for this coordinate. In the embodiment of FIG. 13, the probability vector for the touch coordinates is [0.2, 0.8, 0.1].

In the case of Gaussian mixture modeling for touch input, a Gaussian mixture probability distribution is determined by learning an input pattern for each key. If a specific coordinate is input thereafter, the device may obtain a probability vector corresponding to the coordinate and map the coordinate to a key having the largest value in the probability vector. In the example of FIG. 13, the device may map the touch input to s.

The probability vector may include as many values as the total number of keys, or may contain fewer values than the total number of keys. The probability vector is as long as the number of keys, but zero values may be given to keys whose modeling sample is small.

14 illustrates a probability vector according to an embodiment of the present invention.

In Figure 14, the vertical length represents the probability value for each key. On the horizontal axis, the probability vector may have a length corresponding to the total number of keys included in the keyboard interface. In the following, the probability vector may be referred to as probability information. Hereinafter, based on the GMM, a probability vector obtained as shown in FIGS. 13 to 14 may be referred to as a Gaussian mixed probability vector or Gaussian mixed probability information.

As shown in FIGS. 13 and 14, a process of generating a probability vector by substituting touch coordinates into a GMM obtained based on a plurality of pre-processed touch inputs may be referred to as GMM processing. That is, GMM processing refers to a process of obtaining a probability vector of specific coordinates for a Gaussian mixed probability distribution.

15 illustrates a touch input processing method according to an embodiment of the present invention.

In FIG. 15, the present invention recognizes a touch input and obtains a probability vector based on a Gaussian mixture model. As described above, the device obtains a Gaussian mixture probability vector corresponding to the touch input using a pre-processed Gaussian mixture model. The obtained probability vector is then processed by the cyclic neural network.

The device receives four touch inputs. The device uses the touch input of X_1 as shown in FIG. 15 (a), the touch input of X_2 as shown in FIG. 15 (b), the touch input of X_3 as shown in FIG. 15 (c), and the touch input of X_4. Receive.

The device may obtain a Gaussian mixture probability vector based on the GMM pre-modeled for each touch input. The device may output a probability vector for each touch input as described above. The device may obtain a probability vector G_4 for a probability vector G_1 for X_1 and a probability vector G_2 for X_2 and a probability vector G_3 for X_3. The device may input the probability vector into the cyclic neural network.

The cyclic neural network receives W_i as an input and outputs W_0, and transmits W_r as a cyclic weight / weight to the cyclic neural network of the next operation. The cyclic neural network can adjust the size of the model by adjusting the number of layers and the number of nodes of these three parameters.In addition, the basic units are long-term memory (LSTM) and GRU ( It can be replaced by a unit such as a gated recurrent unit.

Given a touch sequence (X_1, X_2, X_3, ...) as input, the device uses a Gaussian mixed model (G (·)), a touch model, to determine the probability vectors G_1, G_2, G_3, for each key. And then enter the probability vector into the cyclic neural network. Each time a touch occurs, the cyclic neural network generates a predictive vector (P_t) as its output, which is a past touch sequence (X_t, X_ (t-1), X_ (t-2) ...) The mapping key value at time t can be calculated by the Hadamard product of P_t and G_t. The prediction value vector may be referred to as prediction probability information.

In the example of FIG. 15, the device receives touch inputs of X_1, X_2, X_3, and X_4, and obtains a probability distribution of G_1, G2, G3, and G_4 for each touch input. The device sequentially inputs the probability distribution into the cyclic neural network. The cyclic neural network receives G_1 as input W_i and obtains P_1 as output W_o. The cyclic neural network transfers the state value according to the G1 operation as W_r to the cyclic neural network in the next operation. The device may perform this process for X1, X2, and X3, and output Q4 as a touch prediction value for X4.

In Fig. 15, the Gaussian mixture probability vector for X_4 has the highest W and the next highest E. Thus, when processing touch input based only on GMM, the device may map the touch input of X_4 to W. However, in the present invention, further performing cyclic neural network processing, the final probability value is output as E.

In FIG. 15, the device calculates the predicted vector as the next output, P3, based on the touch inputs of X_1, X_2, and X_3. That is, based on the inputs of L, O, and V, the device predicts the next touch as E. The device calculates a probability value of X_4 and a predicted value vector P_3 to obtain a final probability vector for X_4, and the touch key value for X_4 is E in the embodiment of FIG. 13. In an embodiment, the present invention multiplies P_3 by G_4 to obtain a probability vector Q_4 and outputs the most probable key based on Q4.

16 shows a learning process of a cyclic neural network according to the embodiment of the present invention.

As described above, in order for the cyclic neural network to work, that is, to predict the next key well, the cyclic neural network model must be trained.

When X_1 is input, the device GMM processes X_1 to obtain a Gaussian mixed probability vector G_1. The cyclic neural network processes G_1 and outputs a predicted probability vector P_1. The prediction probability vector may correspond to a probability vector obtained by processing G1 into a cyclic neural network.

When X_2 is input, the device may learn the cyclic neural network with the correct answer to X_1 in addition to performing the above-described GMM processing and cyclic neural network processing. That is, the present invention can learn the cyclic neural network by inputting the mixed Gaussian modeling result of X_2 of 16 (b) as the correct answer to X_1 of 16 (a). Similarly, the present invention can learn the cyclic neural network by inputting the mixed Gaussian modeling result for X_3 of 16 (c) as the correct answer for X_2 of 16 (b). In addition, the present invention can learn the cyclic neural network by inputting the mixed Gaussian modeling result of X_4 of 16 (d) as the correct answer to X_3 of 16 (c).

The cyclic neural network may change the weights of parameters W_i, W_o, W_r or parameters in the cyclic neural network based on the difference between the predicted value and the correct answer value. That is, in 16 (a), the parameters of the cyclic neural network are changed based on the difference between P_1 and W_1, and this process may be referred to as learning of the cyclic neural network. As the learning is repeated, the parameters may be changed in the direction of increasing the probability of correct answer.

17 illustrates a touch input processing method according to another embodiment of the present invention.

As shown in FIG. 17, the touch input processing method of the present invention receives (1) a Gaussian mixture probability vector G_n by receiving a touch input, and (2) obtains a prediction probability vector P_n through cyclic neural network processing. , (3) A final probability vector Q_n may be obtained using a Gaussian mixed probability vector and a predicted probability vector. The touch input may be mapped to a key having the highest probability in the final probability vector Q_n. Therefore, the key having the highest value in the final probability vector may be referred to as the distribution Q_n as the key mapping value.

In Fig. 17, in the horizontal direction, each of Figs. 17 (1), (2) and (3) shows the processing steps for each touch input. In the longitudinal direction, FIG. 17A illustrates processing for the first touch input, FIG. 17B illustrates processing for the second touch input, FIG. 17C illustrates processing for the third touch input, and FIG. ) Represent the processing for the fourth touch input, respectively.

Each step of the touch input processing method of FIG. 17 will be described in more detail below.

(1) Acquisition of Gaussian Mixed Probability Vector (G_n)

When the device receives the touch input, the device may GMM process the touch input to obtain a Gaussian mixture probability vector. The Gaussian mixture probability vector may indicate which key's probability distribution belongs to which touch input coordinate, and how strong is in the probability distribution of each key. In the embodiment of FIG. 17A, the Gaussian mixture probability vector G_1 for the first touch input indicates that the probability that the touch input is L is the highest.

The letter indicated by the largest key in the Gaussian mixture probability vector may be a GMM-inferred keycode. A method of obtaining a Gaussian mixture probability vector is as described above with reference to FIGS. 1 to 14. GMM processing may be referred to as touch modeling.

(2) obtaining the predictive probability vector P_n

The cyclic neural network model may be referred to as a sequence model. The sequence model may correspond to a cyclic neural network model that learns a touch input over time. The trained cyclic neural network model may be compressed and inserted into an application, and the inserted cyclic neural network model may be referred to as a sequence model. Sequence modeling may also be referred to as language modeling.

The sequence model may receive and model a probability vector for a preceding key and output a predicted probability vector indicating a probability of a next key to be input. The sequence model, that is, the cyclic neural network, may receive the previous key mapping value as an input and output a prediction vector for the current key by reflecting the cyclic weight of the operation of the preceding key.

In the embodiment of FIG. 17B, the cyclic neural network receives the probability vector Q_1 for the touch input of X_1 and applies the cyclic weight Wr for the cyclic neural network processing up to X_1, thereby predicting the probability for X_1. Outputs the vector, P_2. The predictive probability vector P_2 indicates the predicted value / probability of X_2 based on the X_1 input. In FIG. 17B, the predictive probability vector P_2 obtained by performing cyclic neural network processing of the X_1 touch input indicates that the probability that the X_2 touch input is “O” is highest.

(3) final probability vector acquisition and key mapping

The present invention can obtain the final probability vector for the current key input by using the preceding final probability vector that is the processing result of the preceding key input and the Gaussian mixed probability vector for the current key input. For example, in FIG. 17C, the device performs cyclic neural network processing on the preceding final probability vector Q_2, which is a result of processing of the preceding key input for the second touch input X_2, to obtain a predicted mixed probability vector P3. The device obtains a final probability vector Q_3 for X_3 using a Gaussian mixed probability vector G_3 and a predicted mixed probability vector P_3 that are the result of GMM processing the third touch input X_3. In the final probability vector Q_3 for the third touch, since V has the highest probability value, the third touch X_3 is mapped to V.

The present invention obtains the final probability vector by summing the Gaussian mixed probability vector and the predicted probability vector. As a summing method, an element-wise product of a predicted probability vector and a GMM probability vector for all keys may be used, and weights for each element may be adjusted or sum-to-one vectors may be used. Preprocessing, such as normalization and softening, can be adjusted to ensure that the resulting values are well deduced.

(4) state / state reset

In FIG. 17, the cyclic neural network feeds back information on the last operation with the cyclic weight W_r. That is, in FIG. 17B, information on neural network processing for Q1 is applied / reflected to neural network processing of Q_2 as W_r. However, the sequence input in the present invention is a set of words. If the sequence is long and the cyclic weight continues, the prediction accuracy may be lowered. In addition, as the sequence lengthens, the speed of the application may be reduced, thereby limiting the number / length of touch sequences that transmit cyclic weights. In an embodiment, the sequence length may be limited to 10 touch inputs.

Since the present invention predicts the next word based on the word, upon recognizing the end of the word input such as the space bar or the enter, the state of the cyclic neural network can be reset. Resetting the state of the cyclic neural network means stopping the transfer of weights to the cyclic neural network processing of the preceding key. When the state of the cyclic neural network is reset, the cyclic neural network may perform the current operation without weighting the preceding operation. In Figure 17 (a), for the first touch input, the cyclic neural network starts processing with the state reset.

As an embodiment, even when a spacebar input or a deletion input such as a backspace is received, the present invention can reset the state of a cyclic neural network. For example, if you try to type "love" and type "b" instead of "v", the user will enter "l", "o", "b", "<= (backspace)", "v" , touch input is performed in the order of "o". In this case, since the present invention must predict words after "v", it is reasonable to perform a state reset in case of backspace input.

The present invention can reset the state of the cyclic neural network when a user input indicative of interruption of continuous input of a sentence or a word is recognized. In addition to spacebar or enter input, user input indicating a break may include various non-character inputs (eg,?,!, ",:, Etc.) In addition, word input may be interrupted for more than a certain period of time. Even if it is, the present invention can reset the state of the circulatory neural network.

18 illustrates Gaussian mixture probability information and prediction probability information according to an embodiment of the present invention.

FIG. 18A illustrates a Gaussian mixture probability vector obtained by GMM processing a touch input, and FIG. 18B illustrates a predicted probability vector obtained by performing cyclic neural network processing of a preceding touch key.

Probability vector values by GMM processing are like the one-hot vector, the probability values are biased to specific keys. As shown in FIG. 18A, the Gaussian mixture distribution for the touch input shows a high probability value for the key in which the touch input is located and the surrounding keys, and a very low probability value for the other keys. However, as shown in FIG. 18B, the prediction probability vector obtained through the sequence model has a more gentle distribution of probability values. Therefore, multiplying the Gaussian mixture probability vector by the prediction probability vector may result in a result reflecting only the characteristics of the Gaussian mixture probability vector. Therefore, a process of adjusting vector values may be performed to improve performance.

As an embodiment, the present invention may smooth the Gaussian mixed probability vector value that is the result of GMM processing. Smoothing means adjusting so that the difference between probability values corresponding to each key is reduced. To this end, the present invention may apply a softmax function to the vector value, add a smoothing factor to the softmax function, and adjust the smoothing factor. This smoothing process may be expressed as Equation 1 below.

In Equation 1, T may correspond to a smoothing factor, and the weight of the GMM prediction value may be adjusted as the T value is adjusted.

In another embodiment, weights may be assigned to at least one of a Gaussian mixed probability vector or a predicted probability vector. For example, the present invention may obtain a final probability vector by multiplying a Gaussian mixture probability vector by a positive number, multiplying the prediction probability vector by a positive number b, and then multiplying the two vectors by elementwise. If a = b = 1, no weight is assigned.

In another embodiment, the present invention may perform optimization by smoothing the Gaussian mixture probability vector value through scaling and weighting at least one of the smoothed mixed Gaussian mixture probability vector and the prediction probability vector.

In addition to the above description, in order to reduce the recognition error for the touch input, the following additional processing may be performed.

1.Control using the Mahalanobis Distance

The Mahalanobis distance is a metric representing the distance between a particular data point and a probability distribution. The Mahalanobis distance may indicate how many times the standard data point P is a standard deviation from the mean value of the particular probability distribution D. When the above-described GMM processing and cyclic neural network modeling are applied, the Mahalanobis distance can be calculated for each probability vector value as a result of inference. If the calculated distance value is too large, the resulting probability vector is too far from the model's mean, so it can be considered that the predicted value of the model is not accurate. Therefore, in this case, a method of returning to the previous value without using the predicted value may be used.

2. How to reduce the operation of GMM processing

The soft keyboard interface has a characteristic (a characteristic of a hard real-time application) that the user is very uncomfortable if the target character does not appear on the screen within a certain time. Therefore, several implementation techniques can be used in the keyboard reasoning process to reduce computation. As one of these, a method of reducing the amount of computation that yields a predictive probability vector through GMM processing may be used.

In the case of GMM processing, the key that the user wishes to enter will be located near the key corresponding to the user input coordinate. Therefore, the final mapping of non-adjacent keys to user input coordinates is very likely a result of incorrect mapping of user input. In addition, when GMM modeling for a key input, the probability value is very low for keys other than the key around the input coordinate. Therefore, in the process of processing the mixed Gaussian modeling and the predicted value, the operation may be performed only on the probability values for the keys adjacent to the user input coordinates, thereby reducing the total amount of computation and improving the computation speed. In an embodiment of the present invention, the probability value may be adjusted to 0 for keys other than the keys adjacent to the user input coordinates.

3. Correction of predictive vector

In the above-described embodiments, the sum of the probabilities for each key included in the final probability vector and the final probability vector, which is the result of adding the Gaussian mixed probability vector and the predicted probability vector, may not be guaranteed. Therefore, an operation of multiplying each element by a predetermined ratio may be further performed such that the sum of the elements of the final probability vector is one. In the present invention, the final probability vector value can be used as a cyclic neural network input for the next key input. Since the sum of the elements of the predictive vector is corrected to be 1, the cyclic neural network of the present invention can output a more accurate result.

4. Correction of cyclic neural modeling results

Unlike the GMM probability vector based on the Gaussian distribution, the predictive probability vector represents the probability distribution for the next letters to be input, so the difference between the probability values of each letter is not very large. Therefore, in the present invention, only the probability value for the upper specific number of keys can be maintained for the result inferred by the sequence model, and the probability for the remaining keys can be adjusted to an average value. The reason for adjusting the inference value is that the key with high probability in the sequence model is maintained, while the key with high probability of Gaussian distribution is too summed by the predicted value of the sequence model, so that the structural characteristics of the keyboard are ignored. To prevent the consequences.

19 illustrates a touch input processing device according to an embodiment of the invention.

In FIG. 19, the touch input processing device 19000 may include a display unit 19010, a communication unit 1902, a processing unit 19030, a sensor unit 19040, and a memory unit 19050. As indicated by a dotted line in FIG. 19, the display unit 19010, the communication unit 1902, and the sensor unit 19040 may be selectively included. The touch input processing apparatus 19000 may perform an operation of the present invention by using an internal / externally connected display unit 19010, a sensor unit 19040, and a communication unit 19020.

The display unit 19010 may display visual information on the display screen. Here, the visual information may represent a still image, a moving picture, an application execution screen, various interfaces, or various visually expressible information including the same, which may be displayed by the display unit 19010. The display unit 19010 may output various visual information on the display screen based on the control command of the processing unit 19030. In particular, the display unit 19010 of the present invention can display a soft keyboard interface as shown in FIG. 1, and display GUIs according to other embodiments of the present invention.

The communication unit 19020 may perform wired or wireless communication with an external device of the device. The communication unit 19020 may not be provided depending on the configuration of the device, or may be configured of a plurality of communication chipsets. The communication unit 19030 may perform communication using various communication protocols such as 3G, 4G (LTE), 5G, WIFI, Bluetooth, and NFC.

The sensor unit 19040 may sense a user input or an environment of the device by using at least one sensor mounted on the device. In an embodiment, the at least one sensor is a variety of sensors, such as touch sensors, fingerprint sensors, motion sensors, pressure sensors, camera (image) sensors, tilt sensors, gyroscope sensors, gyroscope sensors, angular velocity sensors, illuminance sensors, and angle sensors. It may include. The above-described sensors may be included in the device as a separate element or may be integrated in at least one or more elements and included in the device. In particular, the sensor unit 19040 of the present invention includes a touch sensor capable of sensing a touch input to the display unit 19010. The touch sensor may sense a touch input and transmit data of a promised type, such as a coordinate value, to the processing unit 19030. The touch input herein includes both contact and non-contact touch inputs (eg, hovering inputs) to the display unit 19010. In addition, the touch input may contact or not touch the display unit 19010 using touch input means (for example, a stylus pen or a touch pen), as well as an input for directly contacting or non-contacting the display unit 19010 with a part of a user's body. It includes all of the touch input.

The memory unit 19050 may store data including various information. The memory unit 19050 collectively refers to volatile and nonvolatile memories. In the present invention, the memory unit 19050 may store the received touch input data. In addition, in the present invention, the memory unit may store an application / software for driving the above-described method of the present invention.

The processing unit 19030 may control at least one other unit included in the device. The control unit 19030 may process data inside the device. In addition, the control unit 19030 may control at least one unit included in the device based on the detected user input. Processing unit 19030 can implement the method of the present invention by running applications / software for performing the method of the present invention.

20 illustrates a touch input processing method according to an embodiment of the present invention.

The device receives a touch input for a keyboard interface (S20010). The touch input may include touch coordinates. The keyboard interface includes a plurality of keys.

The device obtains Gaussian mixture probability information on the touch input (S20020). The Gaussian mixture probability information may correspond to a Gaussian mixture probability vector. Gaussian mixture probability information is generated based on Gaussian mixture models for a plurality of pre-processed touch inputs. That is, as described with reference to FIGS. 1 to 11 and 13 to 14, the device may model GMMs of a plurality of touch inputs and obtain Gaussian mixture probability information about specific coordinates from the GMM model. The process of obtaining Gaussian mixture probability information on the touch input may be referred to as GMM processing as described above.

Acquiring Gaussian mixture probability information for a touch input of a device comprises: generating a Gaussian model for each of the key regions, generating a Gaussian mixture model for the plurality of key regions of the key interface, and from the Gaussian mixture model The method may further include obtaining the Gaussian mixture probability information on the touch input. This may be performed by generating a probability vector instead of the final key mapping value in the touch input processing method described with reference to FIG. 11.

The device may acquire prediction probability information based on the preceding touch input (S22030). The prediction probability information may correspond to the prediction probability vector. The prediction probability information indicates a key input probability for the current touch input predicted by the preceding touch input. The acquisition of the predictive probability information may be performed by outputting the predictive probability information by performing cyclic neural network processing on the final probability information on the preceding touch input when the preceding touch input of the received touch input exists. The cyclic neural network conveys information about the preceding cyclic neural network processing through cyclic weights. The cyclic neural network is a neural network model that learns a touch input sequence according to a time sequence. The cyclic weight may be reset based on a specific number of touch input sequences, word units, or specific user input.

The device may map the touch input to the key value using the Gaussian mixture probability information and the prediction probability information (S22040). The key mapping of the device may include obtaining final probability information using Gaussian mixed probability information and prediction information, and mapping a touch input to a key value based on the final probability information. The final probability information corresponds to a final probability vector, and the touch input may be mapped to a key value having the highest probability in the final probability vector. Acquisition of the final probability information may be performed by element-wise product of the probability vector corresponding to the Gaussian mixture probability information and the probability vector corresponding to the prediction probability information. At least one of the Gaussian mixed probability vector or the predicted probability vector may be weighted. Gaussian mixture probability information or prediction probability information may be adjusted to use only the probabilities for at least one key adjacent to the touch input.

GMM processing is performed by applying a touch input to a Gaussian mixture probability distribution obtained by GMM modeling.

19000: touch input processing device
19010: display unit
19020: communication unit
19030: Processing Unit
19040: sensor unit
19050 memory units

Claims (16)

In the touch input processing method for the displayed keyboard interface,
Receiving a touch input for the keyboard interface including a plurality of keys;
Obtaining Gaussian mixture probability information for the touch input;
Obtaining prediction probability information based on a preceding touch input of the touch input; And
Mapping the touch input to a key value using the Gaussian mixture probability information and the prediction probability information;
The Gaussian mixture probability information indicates a mapping probability of the touch input to at least one key based on a Gaussian mixture model for the plurality of pre-processed touch inputs,
The prediction probability information indicates a key input probability for the current touch input predicted by the preceding touch input,
Obtaining the prediction probability information,
And if the preceding touch input of the touch input is present, outputting the predicted probability information by cyclic neural network processing of the second final probability information for the preceding touch input. The method of claim 1,
The mapping of the touch input to a key value using the Gaussian mixture probability information and the prediction probability information may include:
Obtaining first final probability information using the Gaussian mixture probability information and the prediction probability information; And
And mapping the touch input to the key value based on the first final probability information. The method of claim 1,
The cyclic neural network delivers information on the preceding cyclic neural network processing through cyclic weights,
The cyclic neural network is a neural network model that learns a touch input sequence in a time sequence. The method of claim 3, wherein
Wherein the cyclic weight is reset based on a specific number of touch input sequences, word units, or a specific user input. In the touch input processing method for the displayed keyboard interface,
Receiving a touch input for the keyboard interface including a plurality of keys;
Obtaining Gaussian mixture probability information for the touch input;
Obtaining prediction probability information based on a preceding touch input of the touch input; And
Mapping the touch input to a key value using the Gaussian mixture probability information and the prediction probability information;
The Gaussian mixture probability information indicates a mapping probability of the touch input to at least one key based on a Gaussian mixture model for the plurality of pre-processed touch inputs,
The prediction probability information indicates a key input probability for the current touch input predicted by the preceding touch input,
The mapping of the touch input to a key value using the Gaussian mixture probability information and the prediction probability information may include:
Obtaining first final probability information using the Gaussian mixture probability information and the prediction probability information; And
Mapping the touch input to the key value based on the first final probability information,
Acquiring first final probability information by using the Gaussian mixed probability information and the prediction probability information,
And an element-wise product of a first probability vector corresponding to the Gaussian mixed probability information and a second probability vector corresponding to the prediction probability information. The method of claim 5, wherein
And at least one of the first probability vector and the second probability vector is weighted. The method of claim 1,
Wherein the Gaussian mixture probability information or the prediction probability information is adjusted to use only a probability for at least one key adjacent to the touch input. The method of claim 1,
Acquiring Gaussian mixture probability information on the touch input includes:
Generating a Gaussian model for each key region;
Generating a Gaussian mixture model for the plurality of key regions of the key interface; And
Obtaining the Gaussian mixture probability information for the touch input from the Gaussian mixture model. A memory unit that stores touch input data for an application that processes touch input and a displayed keyboard interface; And
A touch input processing device comprising a processing unit for driving the application to process the touch input,
The processing unit,
Receive a touch input to the keyboard interface including a plurality of keys,
Obtaining Gaussian mixture probability information on the touch input,
Obtaining prediction probability information based on a preceding touch input of the touch input,
The touch input is mapped to a key value by using the Gaussian mixture probability information and the prediction probability information.
The Gaussian mixture probability information indicates a mapping probability of the touch input to at least one key based on a Gaussian mixture model for the plurality of pre-processed touch inputs,
The prediction probability information indicates a key input probability for the current touch input predicted by the preceding touch input,
Acquiring the prediction probability information,
And if there is a preceding touch input of the touch input, performing the output of the predicted probability information by cyclic neural network processing of the second final probability information for the preceding touch input. The method of claim 9,
The mapping of the touch input to a key value using the Gaussian mixed probability information and the predicted probability information is
Obtain first final probability information using the Gaussian mixture probability information and the prediction probability information,
And mapping the touch input to the key value based on the first final probability information. The method of claim 9,
The cyclic neural network delivers information on the preceding cyclic neural network processing through cyclic weights,
Wherein the cyclic neural network is a neural network model that has learned a touch input sequence in time order. The method of claim 11,
And the cyclic weight is reset based on a specific number of touch input sequences, word units, or specific user input. A memory unit that stores touch input data for an application that processes touch input and a displayed keyboard interface; And
A touch input processing device comprising a processing unit for driving the application to process the touch input,
The processing unit,
Receive a touch input to the keyboard interface including a plurality of keys,
Obtaining Gaussian mixture probability information on the touch input,
Obtaining prediction probability information based on a preceding touch input of the touch input,
The touch input is mapped to a key value by using the Gaussian mixture probability information and the prediction probability information.
The Gaussian mixture probability information indicates a mapping probability of the touch input to at least one key based on a Gaussian mixture model for the plurality of pre-processed touch inputs,
The prediction probability information indicates a key input probability for the current touch input predicted by the preceding touch input,
The mapping of the touch input to a key value using the Gaussian mixed probability information and the predicted probability information is
Obtain first final probability information using the Gaussian mixture probability information and the prediction probability information,
Is performed by mapping the touch input to the key value based on the first final probability information,
Acquiring the first final probability information,
And an element-wise product of a first probability vector corresponding to the Gaussian mixed probability information and a second probability vector corresponding to the prediction probability information. The method of claim 13,
And at least one of the first probability vector and the second probability vector is weighted. The method of claim 9,
Wherein the Gaussian mixed probability information or the predicted probability information is adjusted to use only a probability for at least one key adjacent to the touch input. The method of claim 9,
Acquisition of Gaussian mixture probability information on the touch input includes:
A touch performed by generating a Gaussian model for each of the key regions, generating a Gaussian mixture model for the plurality of key regions of the key interface, and obtaining the Gaussian mixture probability information for the touch input from the Gaussian mixture model. Input processing device.

Images