KR20160097352A - System and method for inputting images or labels into electronic devices - Google Patents

Abstract

A system and method for inputting an image / label into an electronic device is disclosed. A system and method are provided for predicting images / labels associated with text entry by a user. In a first aspect, a system is provided that includes means for receiving text input by a user and prediction means trained for sections of text associated with the image / label. The predictor means receives the textual input by the user, determines the relevance of the textual input by the user to the sections of text associated with the image / label, and determines the relevance of the textual input to the user based on the sections of text associated with the image / Lt; RTI ID = 0.0 > image / label < / RTI > The system and method of the present invention reduces the burden of inputting images / labels.

Description

FIELD OF THE INVENTION [0001] The present invention relates to systems and methods for inputting images or labels into electronic devices,

The present invention relates to a system and method for inputting images / labels into an electronic device. Specifically, the present invention relates to a system and method for presenting images / labels to be input to a device based on text entered by a user.

In text creation and messaging environments, it has become popular for users to include images in word-based text. For example, entering a text-based representation of an image known as an emoticon to express emotions such as :-) or ;-p [common in the West] or (^ _ ^) [common in Asia It is common. More recently, small letter-sized images called emojis have become popular. Stickers were also popularized. A sticker is a detailed example of a character expressing an action or an emotion, which is a mixture of a cartoon and an emotion.

As of October 2010, the Unicode (6.0) standard has assigned 722 code points as the description of the emotion (eg U + 1F60D: smiley face with heart-shaped eyes and U + 1F692: fire engine box). It is customary to design each set of images used to render each of these Unicode characters so that messaging services (e.g., Facebook, Watts apps) can be sent and received. In addition, Android (4.1+) and iOS (5+) both provide a representation of these characters by default as part of the default font.

Although it is popular to enter emoji, it is still difficult to do because the user has to find the appropriate emoji, and even if they know the proper emoji, they have to search a large number of possible emoji to find what they want to input.

Keyboard and messaging clients have attempted to reduce the problem by including an image selection panel in which the images are organized into multiple categories that can be scrolled. Although the images are grouped into related categories, the user must still search for images of the category to search for the desired image. In addition, some emoticons may not be easily classified and it is more difficult for the user to determine in which category the emoticons should be searched.

There are known solutions that attempt to further reduce the burden of inputting emoticons. For example, multiple messaging clients will automatically replace certain shorthand text with images. For example, Facebook Messenger will convert the emoticon :-) into a picture of a smiling face, and translate the shorthand text sequence (y) into a thumb-up picture when the message is sent .

가 후보 입력으로서 사용자에게 제의된다. In addition, the Google Android Jellybean keyboard will offer an emotional candidate when the user types the correct word for the emoticon's technique, such as when the 'snowflake' is typed,

Is presented to the user as a candidate input.

These known solutions to reduce the burden of the input of the still input still require the user to provide a shorthand text frame that identifies the anomaly or to type the exact description of the anomaly. Although the known system eliminates the requirement to scroll the screens of the images, they still require that the user explicitly and correctly identify the images they want to enter.

It is an object of the present invention to address the above-mentioned problems and to reduce the burden of image (e.g., emoji, emoticon or sticker) and label input in a messaging / text writing environment.

The invention provides a system according to independent claims 1 and 2, a method according to independent claims 32, 33, 34, 54 and 55, and a program according to independent claim 56.

Optional features of the invention reside in the context of the dependent claims.

The present invention will now be described with reference to the accompanying drawings.
Figures 1A and 1B illustrate a system for generating image / label predictions in accordance with a first system type of the present invention.
Figures 2a-2c are schematic diagrams of an alternative image / label language model according to the present invention for use in the systems of Figures 1a and 1b.
Figure 3 is a schematic diagram of an n-gram map that includes text sections associated with an image / label (in this example, emotion) for use in the language model of Figures 2b and 2c.
Figure 4 is a schematic diagram of an n-gram map that includes text sections associated with an image / label (in this example, emotion), for use in the image / label language model of Figures 2b and 2c, The image / label was associated with sections of text not immediately preceding the identified image / label.
Figure 5 illustrates a system for generating image / label predictions in accordance with a second system type of the present invention.
Figure 6 illustrates a system for generating image / label predictions in accordance with a third system type of the present invention.
Figures 7 to 11 illustrate different embodiments of a user interface according to the present invention.
12 to 16 show flowcharts according to the inventions of the present invention.

The system of the present invention is configured to generate an associated image / label prediction for user input text. In general, the system of the present invention includes predictive means trained for sections of text associated with an image / label. The prediction means is configured to receive text input by a user and to predict the relevance of the image / label to the user input text.

Image prediction may relate to any kind of image, including photographs, logos, drawings, icons, emoticons or emoticons, stickers or any other images that may be associated with sections of text. In a preferred embodiment of the present invention, the image is an image.

The label prediction can be associated with any label associated with the body of the text, which is used to identify or classify the body of the text. The label may thus refer to the author of the text, the company / person who created the sections of text, or any other relevant label. In a preferred embodiment of the present invention, the label is a hashtag when used, for example, in a Twitter feed.

The present invention provides three alternative ways of generating image / label prediction to address the problem of reducing the burden of image / label input to the electronic device. Specifically, the solution is to use a language model to generate image / label predictions and to use a search engine to generate image / label predictions from a plurality of statistical models , And using a classifier to generate image / label predictions. Alternative solutions (i.e., alternative prediction means) will be described in this order.

The system according to the first solution can be implemented as shown in FIGS. 1A and 1B, and FIGS. 1A and 1B show a block diagram of a high-level text prediction architecture according to the present invention. The system includes a prediction engine 100 configured to generate an associated image / label prediction 50 for user input text.

1A, the prediction engine 100 includes an image / label language model 10 for generating image / label prediction 50 and optionally word prediction (s) The image / label language model 10 may be a generic image / label language model, such as a language model based on English, or an application-specific image / label language model, such as an SMS message or an email message A language model trained for, or any other suitable type of language model. 1B, the prediction engine 100 may include any number of additional language models, which may be a text-only language model or an image / label language model in accordance with the present invention.

1B, if the prediction engine 100 includes one or more additional language models, such as a further language model 20, then the prediction engine 100 may determine that each of the language models 10, To generate a final image / label prediction 50 and / or a final word prediction 60 that can be provided to the user interface for display and user selection, combining image / label prediction and / Language model 30 (Multi-LM). The final image / label prediction 50 is preferably the entire set of probable predictions (i.e., the designated number). The system can present only the most likely image / label prediction 50 to the user.

The use of Multi-LM 30 to combine word predictions from a plurality of language models is described in line 12, line 1 to page 12, page 11 of WO 2010/112841, and WO 2010/112841, ≪ / RTI >

For example, if the additional language model 20 is a standard word-based language model, as described in detail in WO 2010/112842 and in particular in connection with Figures 2A-2D of WO 2010/112842 , A standard word based language model can be used with the image / label based language model 10 so that the prediction engine 100 can generate image / label predictions 50 from the image / Based language model 20 from word-based prediction. If desired, the image / word-based language model 10 may also generate word predictions used by the Multi-LM 30 to generate a final set of word predictions 60 (Figures 2A-2C Lt; / RTI > as described below). The Multi-LM 30 need not output the final image / label prediction 50 because the additional language model 20 of this embodiment can only predict words. The word-based language model 20 may be replaced by any suitable language model for generating word prediction, which may be morphemes or word-segment-based languages, as described in detail in UK Patent Application No. 1321927.4 Model, the entirety of which is incorporated herein by reference.

If the additional language model 20 is a further image / label language model, then the Multi-LM 30 generates the final image / label prediction 50 from image / label prediction from both language models 10, Lt; / RTI >

The Multi-LM 30 may also send the user input text to the token (s), as described in the first paragraph on page 21 of WO 2010/112842 and as described in more detail below in connection with the language model embodiment of the present invention. Can be used for tokenizing.

2A-2C illustrate an embodiment of an image / label language model 10 that receives user input text and includes an image / label prediction 50 (and optionally a word / term) Prediction 60) of the image / label language model.

There are two possible inputs to a given language model: the current word input 11 and the context input 12. The language model may use any or both of the possible inputs. The current word input 11 includes information that the system has about the word the system is trying to predict, e.g., the word the user attempts to input (e.g., if the user enters "I am working on ge" The current word input (11) is 'ge'). This may be a sequence of multiple character keystrokes, a separate character keystroke, a character determined from the continuous touch gesture across the touchscreen keypad, or a mix of input forms. Context input 12 includes a sequence of words entered there by the user immediately before the current word (e.g., "I am working") and the sequence is transmitted to Multi-LM 30 or individual tokenizer Tokens "). If the system is generating a prediction for the nth word, the context input 12 will include the previous n-1 words selected by the user and entered into the system. The n-1 words of the context may contain a single word or a sequence of words, or may not contain any words if the current word input is related to a word starting a sentence.

The language model may include an input model (taking current word input 11 as input) and a context model (taking context input 12 as input).

2a, the language model includes a trie 13 (an example of an input model) and a word-based n < RTI ID = 0.0 > -gram map 14 (an example of a context model). The first part of this language model corresponds to what is described in detail in WO 2010/112841 and specifically corresponds to what has been described in connection with Figs. 2A to 2D of WO 2010/112841. The language model of Figure 2a of the present invention also includes an intersection 15 for calculating the final set of word predictions 60 from the predictions generated by tree 13 and n-gram map 14. [ can do. As described in detail in line 14 of page 16 of page < RTI ID = 0.0 > 2010/112841, < / RTI > line 14, tree 13 may be a tree (see Figure 3 of WO 2010/112841) or a direct current word- (See FIG. 4A of WO 2010/112841), which is queried with a query 112. Alternatively, the tree 13 may be constructed as described in detail on line 16 to page 16, at page 17 of WO 2010/112841, incorporated by reference herein (and shown in Figures 4b and 4c) It can be a probabilistic trie queried with the KeyPressVector generated from the current input. The language model may also include any number of filters for generating a final set of word predictions 60, as described in previous applications.

If desired, the intersection 15 of the language model 10 of FIGS. 2A and 2C may be configured to employ a back-off approach, if the candidate predicted by the tree is also not predicted by the n-gram map (Not just those candidates generated by both, as described in WO 2010/112841). Each time the system has to back off for a retrieved context, the intersection mechanism 15 may apply a probability to a 'back off' penalty (which may be a fixed penalty, for example, by multiplying a fixed value). In this embodiment, the context model (e.g., n-gram map) may include a unigram probability to which the backoff penalty is applied.

2a includes a word-to-image / label mapping map 40 that maps each word of the language model 10 to one or more associated images / labels, such as word prediction 60, ', The language model outputs an image of the pizza as an image prediction 50 (e.g., pizza imager).

Figure 2B illustrates a second image / label language model 10 according to the first solution of the present invention. The image / label language model 10 is configured to generate an image / label prediction 50 and optionally a word prediction 60 based on the context 12 alone. In this embodiment, the image / label language model only receives the context input 12, which contains one or more words used to search the n-gram map 14 '. The n-gram map 14 'of FIG. 2b is trained in a manner different from the n-gram map of FIG. 2a, and the image / label language model 10 is trained without using the word- / Label prediction 50. < / RTI > In the absence of the context 12, the language model 10 may output the most probable image / label 50 associated with the most probable word 60 used to start the sentence. In certain situations, predicting the image / label based on context alone, such as predicting the emotion, may be appropriate. In other situations, it may be more appropriate to use the current word input, for example, in predicting a label (such as a hash tag), since the user may have partially typed the label before being predicted ).

An example of the n-gram map 14 'of the second embodiment is schematically illustrated in FIGS. 3 and 4, and for the purpose of illustration, an image was selected for the image / label.

"을 포함하는 경우, 이모지

는 그의 앞의 컨텍스트만 따를 것이며, 예컨대 n-gram이 4개의 "happy about this

"의 깊이를 갖는 경우 그러하다. 따라서 언어 모델은, 언어 모델로 공급된 컨텍스트(12)가 "happy about this"를 포함하는 경우, 시퀀스의 다음 부분이므로, 이모지

를 예측할 것이다. n-gram 맵은 워드들의 시퀀스 및 이모지와 연관된 확률을 포함하는데, 이모지 및 워드는 확률을 할당하기 위해 마구잡이로 취급된다. 따라서, 그 트레이닝 데이터 내의 특정 컨텍스트가 주어진다면, 트레이닝 데이터 내의 출현 빈도에 기초하여 확률이 할당될 수 있다. The n-gram map 14 'of FIG. 3 has been trained for source data including images / labels embedded in sections of text. For example, the language model may be trained for data from tweeters, and tweets were filtered to collect tweets that include the emoji. In the n-gram map 14 'of FIG. 3, the image (used only as an example for an image / label) is treated like a word to produce a language model, that is, an n-gram context map, . For example, if the source data is sentence "I am not happy about this

Quot ;, "

Will follow only the context of its predecessor, for example, if the n-gram has four "happy about this

"Therefore, the language model is the next part of the sequence if the context 12 supplied in the language model contains" happy about this "

. The n-gram map contains the sequence associated with the word and the probability associated with the word, and the word and word are treated at random to assign a probability. Thus, given a particular context in the training data, probabilities can be assigned based on the frequency of occurrence in the training data.

"에 대한 "I am"과 같이 보통 이미지/라벨 바로 앞에 있을 텍스트를 입력하지 않았더라도, 언어 모델은 관련/적절한 이미지/라벨을 예측할 수 있다. 이 언어 모델(10)을 트레이닝하도록, 이미지/라벨이 소스 텍스트(예컨대, 필터링된 트위터 트위트들) 내에서 식별되고, 각각의 식별된 이미지/라벨은 그 소스 텍스트 내의 텍스트의 섹션들과 연관된다. 트위트들의 예를 사용하여, 특정 트위트의 이모지는 그 트위트로부터의 모든 n-gram과 연관된다. 예를 들어, 트위트 "I'm not happy about this

"에 대한 트레이닝은, 관련 이모지를 갖는 다음 n-gram을 생성할 것이다:The n-gram map of Figure 4 was trained by associating the identified image / label in the source text with a section of text that is not immediately preceding the identified image / label. By training the language model in this way, if the user did not enter the text describing the associated image / label and "I am

The language model can predict the relevant / appropriate image / label, even if the normal image / label immediately preceding the label, such as "I am" Is identified in the source text (e.g., filtered tweeter tweets), and each identified image / label is associated with sections of text in its source text. Using the example of tweets, For example, tweets such as "I'm not happy about this "

"Will produce the next n-gram with the relevant emotions:

○ I'm not happy

○ not happy about

○ happy about this

○ I'm not

○ not happy

Etc.

One way to generate the exception prediction from such a non-direct context n-gram map 14 'is to use an n-gram map 14' that most closely matches the word sequence of the user input text, ) ≪ / RTI > If the user input text is W ₁ W ₂ W ₃ and W ₄ , the predicted emotion is emotion attached to the sequences W ₁ W ₂ W ₃ and W ₄ . An alternative way to generate the exception prediction from the non-direct context n-gram map 14 'is to predict the emotions for each word of the user input text, for example if the word sequence of the user input text is W ₁ W ₂ W ₃ W ₄ or the like case, the first emoji e ₁ with respect to W _1, the second emoji e ₂ with respect to W ₁ W ₂ (W _1, W ₂ is whether aunt for word sequence W ₁ W ₂ means to predict), an e ₃ with respect to W ₁ W ₂ W _3, and e ₄ to predict with respect to _{W 1 W 2 W 3 W 4} . Emoji prediction set (e _1, e _2, e _3, e _4), weighted with which the average is used to establish the aunt not predicted 50, that is, land the most frequent prediction aunt most as possible aunt sure that the Will be output. By taking the weighted average of the immediate prediction set, it may be possible to increase the context reach of the immediate prediction.

Because of the number of different sections of text that can be associated with each emotion, the model can preferably be pruned in two ways. The first is to prune based on the frequency of occurrence, for example, to prune n-grams with a frequency count that is less than a fixed number of occurrences (e.g., certain n-grams and associated aids are less than 10 in training data Remove the n-grams and related implants if they are visible).

을 예측하는 확률은

의 유니그램 확률보다 훨씬 더 크지 않을 것인데, 트레이닝 또한 어떠한 특정 치우침 없이 이 [EMOJI]에 관하여 많은 다른 n-gram의 형태에 마주쳤을 것이기 때문이다. 따라서 n-gram "about this

"은 가지치기될 수 있다. 2가지 가지치기 방법의 조합도 또한 가능하며, 임의의 다른 적합한 가지치기 방법도 마찬가지로 가능하다.The second pruning method is pruning based on the probability difference from the unigram probability. As an example, after the context "about this"

The probability to predict

, The training would also have encountered many different n-gram forms about this [EMOJI] without any specific bias. Therefore, n-gram "about this

"A combination of two pruning methods is also possible, and any other suitable pruning method is likewise possible.

"를 예측할 것이다. 도 4의 n-gram 맵에 관련하여, 언어 모델이 다이렉트 및 비-다이렉트 컨텍스트에 대해 트레이닝되었기 때문에, 언어 모델은 훨씬 더 자주 이모지 예측을 생성한다. 2B, the language model 10 receives a sequence of one or more words (context 12) from the Multi-LM 30 and stores a sequence of one or more words into words (words) stored in the n-gram map 14 '≪ / RTI > With respect to the n-gram map of Fig. 3, the emotions are only predicted if the emotions immediately follow a sequence of one or more words. For example, from the context sequence " not happy about this "

"With respect to the n-gram map of FIG. 4, because the language model has been trained for direct and non-direct contexts, the language model generates the emotion prediction much more frequently.

As shown in FIG. 2B, the language model may selectively output one or more word predictions 60 with image / label prediction (s) 50. The language model compares the input sequence of one or more words (context 12) with the stored sequence of words (with attached images). For example, for display of the next word 60 on the user interface for user selection, or for direct input of the next word into the system, one of a plurality of words, And outputs the next word in the stored sequence following the sequence of words above.

A third embodiment of the language model 10 is illustrated in Fig. 2C. As with the language model of Figure 2a, the language model 10 of Figure 2c includes a tree 13 and an n-gram map 14 'for generating word predictions from the current input 11 and the context input 12, respectively (15) for generating one or more final word prediction (s) (60). The n-gram map 14 'of the third embodiment is the same as the n-gram map of the second embodiment, i.e. it includes an image / label embedded in sections of text or attached to sections of text. Thus, the same n-gram map 14 'may be used to generate word prediction 60 as well as image / label prediction 50.

As will be appreciated from the above, the system of the first solution predicts the image / label based on the user input text and, optionally, the word / word based on the user input text.

Although the image / label language model 10 of the first solution has been described in relation to a language model containing a trained n-gram map, this is done by way of example only and any other suitably trained language model may be used .

A second solution for reducing the burden of image / label input is to generate image / label predictions for user input, similar to that detailed in UK patent application 1223450.6, which is hereby incorporated by reference in its entirety Search engines.

5 shows a block diagram of a high-level system architecture of the system of the present invention. The search engine 100 'uses an image / label database 70 which preferably includes a one-to-one mapping of the statistical model to the image / label, i.e. the image / label database stores each image / label (e.g., Tag), and each image / label statistical model is trained for sections of text associated with the image / label. A language model is a non-limiting example of a statistical model, where the language model is a probability distribution that represents the statistical probability of sequences of words occurring in a natural language. Unlike the language model 10 of the first solution, the language model according to this solution does not have an image / label in the language model, which is a text-only language model mapped to a particular image / label.

To generate the image / label prediction (s) 50, the search engine 100 'uses the image / label database 70 and user input text 12', and optionally one or more other evidence sources 12 ''), For example, an image / label input history for a given user of the system. To trigger a search, the search engine receives user input text 12 '.

As further described below, the image / label database 70 associates individual images / labels with the same number of statistical models and optionally with alternative statistical models (not shown) that are not language based (e.g., A model that estimates user relevance given the prior input of a particular image / label.

The search engine 100 'may provide the user input text proof 12' to the image (s) in order to generate an estimate of the likelihood associated with the image / label for each image / / Label < / RTI > The search engine outputs the most likely or p most influential images / labels as image / label predictions 50 that may be selectively presented to the user.

Assuming that the image / label, c, is associated with the associated image / label statistical model M, the estimate of the probability of observing the user input text, e, P is:

There are a number of techniques that can be applied by the search engine to calculate the necessary estimates as follows:

● naive Bayesian modeling

● Maximum entropy modeling

● Statistical language modeling

The first two approaches are based on extracting a set of features and training a generative model (in this case, it extracts the features from the text associated with the image / label and applies the image / label statistical model Quot;), statistical language modeling attempts to model a sequential distribution over words in user input text. To provide a working example, the first approach is described, but all of them are applicable.

The set of features is extracted from the user input text, preferably by using any suitable feature extraction mechanism that is part of the search engine 100 '. To generate the relevant estimates, these features are assumed to be generated independently by the associated image / label statistical model.

An estimate of the probability of a given feature associated with a particular image / label is stored in the image / label statistical model. Specifically, the image / label statistical model is trained for the text associated with the image / label by extracting features from the text associated with the image / label and analyzing the frequency of these features within the text.

There are a variety of methods used in the art for generating these features from text. E.g:

• "Bag-of-words" words: Features are a set of unique words used in text.

● Unigram: Features are simply words of text. In this model, words appearing more than once are proportionally given a larger weight.

● Term combination: Features may contain a combination of words that represent a continuous n-gram or non-local sentence relationship.

Syntactic: Features may contain syntax information such as a part tag or a higher level parse tree element.

● Latent topics / clusters: Features can be a basic "subject" or a set / cluster of words that can represent a theme within the text.

Preferred features are typically individual words or short n-grams. Individual word features can be used to invert a sequence into words (such as when words represent both word and morpheme and / or additional notation items such as punctuation), unwanted words (e.g., 'stopwords' Words that do not have a syntax value). In some cases, the features may also be case-normalized, i.e., transformed into sub-cases. The n-gram features are created by concatenating adjacent words into atomic objects. For example, given the text sequence "Dear special friends", the individual word features would be "Dear", "special" and "friends" while the Biagram (2-gram) feature would be "Dear_special" and "special_friends "would.

The feature generation mechanism of the search engine 100 'is preferably to weight features extracted from the user input text 12 to exaggerate the importance of those that are more likely to have a priori useful information. For example, for a word feature, it is usually done using a kind of heuristic technique that encapsulates the scarcity of words in normal English (for example, word frequency - inverse document frequency, TFiDF (term frequency-inverse document frequency)), and the unusual word is more likely to represent the associated image / label statistical model than a normal word. TFiDF is defined as:

는 단어 t가 사용자 입력 텍스트에서 발견되는 횟수이고,

는 모든 이미지/라벨 통계적 모델들에 걸쳐 t가 발견되는 이미지/라벨 통계적 모델의 수이다. From here,

Is the number of times the word t is found in the user input text,

Is the number of image / label statistical models for which t is found across all image / label statistical models.

The D feature of the user input text 12 'can be represented by a D-dimensional vector of real values. Normalization can then be achieved by converting each vector to a unit length by the search engine 100 '. The disadvantageous result of the independence of features is that different lengths of user input text samples are described by different numbers of events which can result in spurious differences in the range of values returned by different system queries , It may be desirable to normalize the feature vector.

은, 사용자에 의한 텍스트 입력, e로부터 추출된 독립 특징들,

에 대한 곱으로서 계산된다:If the image / label, c, is associated under the associated image / label statistical model M, the user input text, the probability of observing e,

Text input by the user, independent features extracted from e,

Lt; / RTI >

를 이미지/라벨 데이터베이스(70)에 질의하도록 구성된다. 데이터베이스는 그 특징을 포함하는 모든 이미지/라벨 통계적 모델의 리스트 및 각각의 이미지/라벨 통계적 모델에 대하여 그 특징과 연관된 확률 추정치를 반환한다. 이미지/라벨 통계적 모델, M 하에 이미지/라벨, c가 주어진다면, 사용자 입력 텍스트, e를 관찰할 확률,

은, 이들 특징

를 포함하는 모든 이미지/라벨 통계적 모델 M에 대해 사용자 입력 증거 e의 모든 특징들

에 대한 확률 추정치의 곱으로서 계산된다.The search engine 100 '

To the image / label database (70). The database returns a list of all image / label statistical models that include that feature and a probability estimate associated with that feature for each image / label statistical model. Image / label statistical model, given an image / label, c under M, the user input text, the probability of observing e,

, These features

All the features of the user input evidence e for all image / label statistical models M

Lt; / RTI >

임) , 다시 쓰여질 수 있다:This expression is obtained by taking g _i , which is each unique feature found for the number of times (n _i ) given in the user input text e (12 ')

, Can be rewritten:

Assuming that search engine 100 'includes a TFiDF weighting, n _i can be replaced with its corresponding weight, w _i . The weight vector w is a vector containing TiFDF scores for all features extracted from the user input text. The weight vector is preferably normalized to have a unit length:

And when you convert to log:

는 2개의 벡터의 내적(dot product)으로 다시 쓰여질 수 있으며, 하나는 가중치를 나타내고 다른 하나는 로그 확률을 나타낸다:

Can be rewritten as a dot product of two vectors, one representing the weight and the other representing the logarithmic probability:

가 필요하다. 검색 엔진(100')은 소스 텍스트 내의 특징들의 빈도를 분석함으로써 트레이닝된 이미지/라벨 통계적 모델로부터 이 추정치를 취한다. To calculate the above, an estimate of the image / label dependent feature likelihood

. The search engine 100 'takes this estimate from the trained image / label statistical model by analyzing the frequency of features in the source text.

은 제로일 것이다. 트레이닝 코퍼스(corpus)가 희박한(sparse) 경우, 사용자 입력 텍스트 내의 모든 특징이 이미지/라벨 통계적 모델에 대한 트레이닝 코퍼스에서 발견되었을 가능성이 낮을 것이다. 따라서, 발견된 특징들의 확률 질량의 일부를 발견되지 않은 특징들에 재할당하도록 일종의 스무딩(smoothing)이 사용될 수 있다. 빈도 기반의 확률을 스무딩하기 위한 널리 일반화된 많은 기술들, 예컨대 라플라스(Laplace) 스무딩이 존재한다. However, according to this approach, if the probability estimate for any feature of the user input text is zero (e.g., because the word does not exist in the language model)

Will be zero. If the training corpus is sparse, it is unlikely that all the features in the user input text have been found in the training corpus for the image / label statistical model. Thus, a sort of smoothing may be used to reassign some of the probability mass of found features to undiscovered features. There are a number of widely generalized techniques for smoothing frequency based probabilities, such as Laplace smoothing.

을 이미지/라벨 데이터베이스(70)의 각각의 이미지/라벨 통계적 모델에 질의함으로써, 사용자 입력 텍스트가 주어진다면 어느 이미지/라벨(50)이 가장 관련되는지 결정할 수 있다.Thus, the search engine 100 'is able to determine which image / label statistical model provides the largest probability estimate (since the image / label statistical model is mapped to the corresponding image / label) field

By querying each image / label statistical model of the image / label database 70, it is possible to determine which image / label 50 is most relevant given the user input text.

As noted above, the search engine 100 'may include additional types of evidence, such as evidence specifically related to a given user, such as previously generated language, previously entered images / labels, or social context / demographics (e.g., Since the type of emotion used in public may vary depending on the country / culture / age).

In addition, the search engine may take into account the prior probability of image / label relevance, e.g., a measure of likelihood that an image / label will be associated when there is no specific evidence related to an individual user or situation. This prior probability can be modeled using a comprehensive analysis of common usage patterns across all images / labels. There are a number of additional sources of information that can be considered, such as recent recency (how recently the image / label was entered by the user) may be important, specifically up-to-date ) Especially when the image / label is particularly relevant or when the image / label is used in a twitter feed followed by a large number of followers.

If a plurality of evidence sources 12 ', 12 " are considered, the search engine 100' generates an estimate for each image / label if each evidence source is given. For each image / label, the search engine is configured to combine the estimates for the evidence sources to produce a global estimate for the image / label. To this end, the search engine 100 'can be configured to treat each of the evidence sources as independent, i.e., to treat the user's image / label input history as independent of the text input.

을 계산하기 위해, 특정 이미지/라벨, c가 주어진다면, 증거, E는 비중첩적이고 상호 독립적인 세트들, [e₁,...,e_n]로 분리되는 것으로 가정되며, 이들은 타겟 이미지/라벨 c 및 연관된 모델 M_c을 조건으로 하여, 일부 분포로부터 독립적으로 생성된다. 이 독립성 가정은 다음과 같이 쓰일 수 있다: Evidence, probability of seeing E,

, E is assumed to be separated into non-overlapping and mutually independent sets, [e ₁ , ..., e _n ], given a particular image / label, c, Is generated independently from some distributions, subject to label c and associated model M _c . This independence assumption can be written as:

은 독립적인 증거 소스 e_i에 대한 확률 추정치들의 곱으로서 검색 엔진(100')에 의해 계산된다. 따라서 검색 엔진(100')은 개별 증거 추정치들을 개별적으로 계산하도록 구성된다. Therefore,

Is calculated by the search engine 100 'as the product of probability estimates for the independent evidence sources e _i. Thus, search engine 100 'is configured to separately calculate individual evidence estimates.

에 미치는 소스의 전체적인 영향은 줄어든다. There is a statistical model, M, for each image / label associated with each evidence source, and the relative impact of the individual evidence sources is determined by the distribution that allows the system to specify the bounds on the amount of information produced by each source Can be controlled by the search engine 100 'by a hyper-parameter per smoothing hyper-parameter. This can be interpreted as confidence in each evidence source. An aggressive smoothing factor (a uniform distribution, in the case of limitation, in which case the evidence source is essentially ignored) for the evidence source relative to the other evidence sources, Lt; RTI ID = 0.0 > a < / RTI > As the smoothing increases, the distribution becomes more even and the probability

The overall effect of the source on the source is reduced.

As described above, in one example, the statistical model may be a language model, so that there are a plurality of language models associated with the plurality of images / labels, which include n-gram word sequences. In this embodiment, the language model may be used to generate word predictions based on user input text (e.g., by comparing a sequence of user input text with a sequence of stored words to predict a next word based on a stored sequence) . Thus, the system can generate image / label predictions through search engines as well as word predictions through individual language models. Alternatively, the system may include one or more language models (e.g., a word-based language model, a morpheme-based language model, etc.) in addition to the statistical model of the search engine to generate text predictions.

In order to increase the processing speed, the search engine 100 'may be configured to discard all features f _i that have a lower TFiDF value than a certain threshold. The features with low TFiDF weights will generally have a minimal impact on the overall probability estimate. In addition, the low TFiDF word ('abbreviation') also tends to have a fairly uniform occurrence distribution across the content corpus, meaning that its effect on probability estimates will also be fairly uniform across the classes. By reducing the number of features that search engine 100 ' uses to query image / label database 70, the processing speed is increased.

에 의해 위 경계지어지는 후보 이미지/라벨 세트를 획득하도록 모든 특징들

에 걸쳐 유니온(union)을 결정할 수 있다. 그 다음, 검색 엔진은 이 제한된 후보 이미지/라벨 세트에 관련하여 증거를 '채점(score)'한다. k는 이미지/라벨의 원래 수에 비교하여 작을 것이므로, 이는 상당한 성능 개선을 제공한다. 예를 들어 Apache Lucene(http://lucene.apache.org/)을 사용함으로써 또는 k 개의 가장 가까운 이웃 접근(http://en.wikipedia.org/Nearest_neighbor_search#k-nearest_neighbor) 등을 사용함으로써, 상위 k개의 이미지/라벨을 조회하기 위한 임의의 다른 적합한 해결책이 채용될 수 있다. k에 대한 값은 디바이스 능력 대 정확도 요건 및 계산 복잡도(예를 들어, 특징들의 수 등)에 따라 좌우될 것이다. 이미지/라벨 입력의 부담을 감소시키기 위한 세 번째 해결책은, 사용자 입력 텍스트에 기초하여 관련 이미지/라벨 예측을 생성하도록 분류기를 사용한다. Alternatively or additionally, the search engine may be configured to retrieve the top k images / labels. The top k image / label lookups act as a first pass to reduce the number of candidate images / labels, which can then be ranked using procedures that require more resources. For each feature of the user input text, f, having a TFiDF t (normalized to be in the range [0,1]), the search engine is configured to look for a kt image / label with the highest probability association associated with f, here, the image / label set is denoted _as Cf. Then, the search engine

To obtain a candidate image / label set that is bounded above by all features

The union can be determined. The search engine then 'scores' the evidence in relation to this limited candidate image / label set. Since k will be small compared to the original number of images / labels, this provides a significant performance improvement. For example, by using Apache Lucene (http://lucene.apache.org/) or by using k nearest neighbor approaches (http://en.wikipedia.org/Nearest_neighbor_search#k-nearest_neighbor) Any other suitable solution for querying k images / labels may be employed. The value for k will depend on the device capability versus accuracy requirements and computational complexity (e.g., number of features, etc.). A third solution to reduce the burden of image / label input uses a classifier to generate an associated image / label prediction based on the user input text.

Figure 6 illustrates a system according to a third embodiment of the present invention including a classifier 100 " to generate an image / label prediction 50 associated with a user input text 12 '. A classifier 100 " for generating text predictions is described in detail in WO 2011/042710, which is hereby incorporated by reference in its entirety. In machine learning and statistics, classification is based on a training set of data containing observations (or instances) whose category membership is known, so that new observations can be made to any of the set of categories (sub-population) ). ≪ / RTI > The classifier 100 " is a feature that maps input data to a category and implements the classification. In the present invention, the classifier 100 " is configured to map user input text to an image / label.

The classifier 100 " is trained for textual data pre-labeled with images / labels and performs real-time image / label prediction 50 on the sections of text 12 entered by the user into the system.

A plurality of text sources 80 are used to train the classifier 100 ". Each of the plurality of text sources 80 includes all of the sections of text associated with a particular image / label found in the source data. For an autonomous approach to creating text sources, any text in a sentence containing a particular image / label may be taken as being text associated with that image / label, or any text before the image / E.g., a twitter feed and its associated hash tag or sentence, and its associated anomaly.

Thus, each text source of a plurality of text sources 80 is mapped or associated with a particular image / label.

The user input text 12 'is input to the feature vector generator 90 of the system. The feature vector generator 90 is configured to transform the user input text 12 'into a feature vector that is ready for classification. Feature vector generator 90 is as described above for the search engine system. The feature vector generator 90 is also used to generate the feature vectors used to train the classifier (from a plurality of text sources) via the classifier trainer 95.

는 현재 소스 텍스트에서 단어 t가 발견되는 횟수이고,

는 텍스트 소스들의 전체 콜렉션에 걸쳐 t가 발견되는 소스 텍스트의 수이다. 그 다음, 각각의 벡터는 특징 벡터 생성기(90)에 의해 단위 길이로 정규화된다. The value D of the vector space is dominated by the total number of features used in the model, typically over 10,000 for real-world classification problems. The feature vector generator 90 may weight each cell according to a value related to the frequency of occurrence of that word in a given text section normalized by the inverse of its occurrence frequency TFiDF over the entire body of the text, To a vector,

Is the number of times the word t is found in the current source text,

Is the number of source texts in which t is found over the entire collection of text sources. Each vector is then normalized to a unit length by a feature vector generator 90.

The feature vector generator 90 is configured to divide the user input text 12 'into features (typically individual words or short phrases) and generate feature vectors from the features. The feature vector is a vector of D-dimensional real values, R ^D , where each dimension represents a particular feature used to represent the text. The feature vector is passed to the classifier 100 " (using the feature vector to generate the image / label prediction).

The classifier 100 'is trained by the training module 95 using the feature vectors generated by the feature vector generator 90 from the text sources 80. The trained classifier 100 " includes an image / label prediction 50 that takes as input the feature vector generated from the text input 12 'by the user and includes an image / label prediction set mapped to a probability value as an output, . The image / label prediction 50 draws from the space of the image / label prediction mapped to / associated with a plurality of text sources.

In a preferred embodiment, the classifier 100 'is either a linear classifier (which makes a classification decision based on the value of the linear combination of features) or a batch perceptron (or classifier) if the weight vector is updated simultaneously in the direction of all instances mis- batch perceptron) principle, but any suitable classifier may be used. In one embodiment, a timed aggregate perceptron (TAP) classifier is used. The TAP classifier is basically a binary (two class) classification model. One-to-one approach is used in which a TAP classifier is trained for each image / label against all other images / labels in order to handle a multi-class problem, i.e. multiple images / labels. The training of the classifier is described in more detail in WO 2011/042710, page 10, line 26 to page 12, line 8, which is hereby incorporated by reference.

The classifier training module 95 performs the training process as already mentioned. The training module 95 calculates a weight vector for each class, i.e., a weight vector for each image / label.

과 짝지은 차원수(dimensionality) D의 N 샘플 벡터의 세트가 주어진다면, 분류기 트레이닝 절차는 최적화된 가중치 벡터

를 반환한다. 이미지/라벨이 새로운 사용자 입력 텍스트 샘플,

에 대하여 관련되는지 여부의 예측,

은 다음에 의해 결정될 수 있다:Target label

Given a set of N sample vectors with a matched dimensionality D, the categorizer training procedure may use an optimized weight vector

. Image / label This new user-entered text sample,

, ≪ / RTI >

Can be determined by: < RTI ID = 0.0 >

(1)

(One)

를 따라 놓이지만, 바이어스를 조정하도록 임계치가 도입될 수 있다. Where the sign function converts any real number to +/- 1 based on its sign. The default decision boundary is the unbiased hyperplane,

But a threshold may be introduced to adjust the bias.

로 표현되는 새로운 처음 보는 사용자 입력 텍스트 섹션이 주어진다고 하면, 다음 신뢰성 벡터

가 생성될 것이다(여기에서, 단순화를 위해 M=3임):The modified form of the classification expression (1) is used without sign function to yield a confidence value for each image / label, resulting in an M-dimensional vector of confidence values, where M is the number of images / labels. Thus, for example,

Given a new first-seen user input text section expressed as < RTI ID = 0.0 >

(Here, for simplicity, M = 3): < RTI ID = 0.0 >

Assuming an even probability for all images / labels, the image / label confidence values generated by the classifier 100 " are used to generate an image / label prediction set (the inverse with the highest value (highest confidence) Is matched to the most likely image / label).

If the image / label is provided with prior probabilities, e.g., a measure of likelihood that the image / label will be relevant without any specific evidence relating to the individual user or situation, or a prior probability based on the user's image / label input history, And may further include a weighting module. The weighting module (not shown) may use the vector of reliability values generated by the classifier to weight the prior probabilities for each image / label to provide a weighted set of image / have.

The weighting module may be configured to adhere to an absolute probability assigned to the image / label prediction set so as to not falsely distort subsequent comparisons. Thus, the weighting module may be configured to leave the image / label prediction from the most likely prediction component unchanged, and downscale the probability from the less likely image / label proportionally.

The image / label prediction 100 " output by the classifier 100 " (or weighting module) may be displayed on the user interface for user selection.

As can be appreciated from the above, the classifier 100 " is required to produce an inner product with each image / label vector of the input vector to produce the image / Therefore, the larger the number of images / labels, the greater the number of dot products that the classifier has to calculate.

To reduce the number of classes, images / labels may be grouped together and all emotions associated with a particular emotion (such as happiness) may be grouped into a class or may be grouped into a single class, This is the case with all of the related emotions. In that case, the classifier will predict the class, for example the emotion (sadness, happiness, etc.) of the class and the n most likely emotional predictions may be displayed to the user for user selection. However, this may require the user to select from a larger panel of images. To reduce the processing power, a more coarse-class class is used to find the correct category of the target, while still predicting the most relevant target, and a finer prediction may only occur for that rough category , Which reduces the number of inner products that the classifier should take.

Alternatively, a first feature set may be extracted from the user input text to produce an initial set of image / label predictions, and a second feature set may be extracted from the initial set of image / Lt; RTI ID = 0.0 > text < / RTI > To conserve processing power, the first feature set may be fewer in number than the second feature set.

If the system is required to process a large number of images / labels, then the use of the search engine 100 'may be preferable to the sorter 100 ", where the search engine calculates a probability estimate for a large number of images / Because it calculates the probability associated with the image / label by a different mechanism that can better cope with the decision.

The system of the present invention can be employed in a wide range of electronic devices. By way of non-limiting example, the system may be used for messaging, text creation, email creation, tweeting, etc. on mobile phones, PDA devices, tablets, or computers.

The invention also relates to a user interface for an electronic device, wherein the user interface displays the predicted image / label (50) for user selection and input. The image / label prediction 50 may be generated by any of the systems described above. As described in more detail below, the user interface preferably displays one or more word / word predictions 60 for user selection, in addition to displaying one or more image / label predictions 50.

7 to 11, a user interface according to an embodiment of the present invention will be described. Figures 7 to 11 illustrate the display of an image on the user interface for user selection and input, by way of example only. However, the present invention is not limited to the display and input of imagery, but is applicable to any image / label prediction 50.

In the first embodiment of the user interface, as illustrated in FIG. 7, the user interface includes one or more (three in this example) most likely user text predictions (i.e., 'The', 'I' One candidate prediction button (three candidate prediction buttons in this example) for displaying a plurality of candidate prediction buttons " What " The user interface 150 also includes a virtual button 155 for displaying the currently most relevant image / label prediction 60 (in the preferred embodiment, an image and, in the particular example illustrated, a beer ember). The processing circuitry of the device may be configured such that a tab on a touch screen device towards a virtual button 155 displaying a first user input, e. The second user input (different from the first user input), e.g., a long push or directional swipe towards button 155, may be used to select a menu to another action, e.g., And return the carriage.

In a second embodiment of the user interface 150 illustrated in FIG. 8, an image (e. G., Imagery) prediction 50 mapped to the word prediction 60 (e. G. , Along with a matching word prediction 161, will be presented as a prediction 160 on a prediction window. Thus, the candidate prediction button displays the two most relevant word prediction (in the case of an example of a user interface with three candidate buttons) and the image most suitable for the most relevant word prediction (e.g., emotion). Alternatively, the image / label prediction presented as prediction 160 on the prediction window is the most likely image / label prediction (as determined by any of the systems described above), and thus need not correspond to the word prediction of the prediction window. For consistency of the layout, the image (e.g., imagery) prediction 160 is always displayed on the right side of the prediction window, making it easier to locate the image. An alternative image (e. G., Emotion) prediction 60 may be made available by long depression of an image (e. The image button 155 reflects this prediction and also presents an image related to the recently typed word. The first gesture (e.g., tab) on the image (in the illustrated example, the outer cover) button 155 will insert the image displayed by the button and the second gesture on the button (e.g., long press or swipe) And will display emotions related to recently typed words for user selection.

In a third embodiment of the user interface 150 illustrated in FIG. 9, an image / label (e.g., in the illustrated example, an emotion) candidate prediction button 165 ) Appears permanently on the prediction window. Is presented on this candidate button 165 when there is an emotion associated with the current word candidate (in this example, 'food', 'and', 'is') or a recently typed word (eg, 'cat' . The emotion displayed on the button 165 may be inserted via the first gesture (e.g., tab) on the button 165 or on the button 155 and through the second gesture on the button 155 or button 165 (E. G., Long press or swipe) alternatives are available.

In a preferred embodiment, an image / label window (e.g., an imagery window) that displays an alternative related image (e.g., an image) can be accessed by pressing and holding the image / label candidate prediction button 165. To access all of the images (not just those suggested as the most likely images), the user presses and holds the image candidate prediction button 165, slides his finger toward the image window icon, and releases it. The Emoji window icon will be on the far left of the pop-up to access it with a 'blind-directional swipe'. The rest of the pop-up is filled with extended emotion predictions.

In an alternative user interface, an image / label (e.g., an image) may be displayed with its matching word on the candidate button 170 of the prediction window, as illustrated in FIG. The word may be inserted by a first user gesture on the candidate button 170 (e.g., by tapping the button 170) and the image / label (e.g., (For example, by depressing the button 170 for a long time). Also, if desired, a standard aide key 155 may be provided to allow the user to insert a predicted emotion (which may not necessarily match the predicted word) or to allow the user to search for alternate emoticons. Lt; RTI ID = 0.0 > embodiment < / RTI >

Figure 11 illustrates how an image (e.g., an emulsion) can be imaged as described in detail in earlier applications WO2013 / 107998, the entirety of which is incorporated herein by reference, and as illustrated in Figure 1 of WO2013 / Is displayed and can be inserted into the continuous touch input. In the user interface of Fig. 11, the prediction window includes a word prediction button 175 ' heart ' and an associated prediction image, e.g., [heart emoji]. To insert the text prediction " heart ", the user moves to the word prediction window and removes his finger from contact with the user interface at a location on the word prediction button 175. [ Alternatively, the word prediction is inserted whenever the user lifts his or her finger from the user interface, unless the user's finger is lifted from the outer button. For example, the processing circuitry may be used when the prediction engine predicts and displays the word for user selection and input, and when the user lifts his finger from the user interface while on the last character of the word, or even on the intermediate word Word. ≪ / RTI > To insert the predicted emoticon, the user closes the contact with the touch screen interface at the instant candidate button 165. In addition, the processing circuitry for the user interface may allow the user to select a pop-up panel of alternate emoticons for user selection if the user has terminated the continuous touch gesture on the epilating button 165 and remains on the epilating button 165 for a specified period of time. (Not shown).

The user interface has been described as including various " buttons ". The word ' button ' is used to describe the area on the user interface where the image / label / word is to be displayed, such that the image / label / word to be displayed is displayed on the area displaying the image / Or by a gesture from above.

With the described user interface, the user can insert relevant images / labels (including emoticons) with minimal effort.

The method of the present invention will now be described with reference to Figures 12-16, which is a schematic flow diagram of a method according to the present invention.

Referring to FIG. 12, the present invention provides a method for generating prediction means for predicting an image / label associated with a user input text. As described above in connection with the various systems of the present invention, the method includes receiving (400) text having one or more images / labels embedded within sections of text, identifying embedded images / labels within the text 410) and associating (420) the identified image / label with sections of text. The prediction means is then trained for the sections of text associated with the image / label. As described above, when the prediction means is a language model 10, the language model 10 may be implemented as an image / label by including an image / label in, for example, an n-gram word / image sequence or an n-gram word sequence To the text containing the image / label. When the prediction means is a search engine 100 'that includes a plurality of statistical models, each statistical model may be mapped to a given image / label and trained for the text associated with that image / label. When the prediction means is a classifier 100 " trained for a plurality of text sources, each text source includes sections of text associated with a given image / label.

In a second method of the present invention, a method is provided for predicting an image / label associated with text entry into a system by a user using prediction means, as illustrated in Figure 13, And are trained on sections of text. The method includes receiving (500) a text input by a user at a prediction means, determining 510 the relevance of a textual input by a user to sections of text associated with the image / label, (520) the relevance of the image / label to the text entry by the user based on the sections of the image / label. As described above with respect to the system description, when the prediction means is a search engine 100 ', the search engine 100' extracts features from the user input text and queries these features into the image / Thereby determining the relevance of the user input text. By querying the database 70, the search engine 100 'can determine which image / label statistical model is most relevant, and therefore the image / label prediction 50 (50), because each statistical model is mapped to a particular image / Can be generated. Again, when the prediction means is a classifier 100 ", as described above with respect to the system, the classifier 100 " includes a feature vector representing the user input text and a section of the text associated with that image / By generating an inner product of a feature vector representing an image / label (generated from the source text that contains the image / label).

In a third method of the present invention, as illustrated in FIG. 14, a method is provided for predicting an image / label associated with text entry into the system by a user using prediction means, Is trained for text including an image / label, and the prediction means is trained by identifying an image / label in the text and associating the identified image / label with sections of the text. The method includes receiving (600) a text input by a user at the prediction means, comparing the text input by the user to sections of text associated with the image / label (610), comparing the section of text associated with the identified image / (620) the relevance of the image / label to the text entry by the user based on the image / As described above with reference to the system description, when the prediction means is a language model 10, the language model is an image / label in the n-gram word / image sequence of the n-gram map 14 ' 14 ') < / RTI > The language model predicts the associated image / label 50 by comparing the user input text with the stored n-gram sequence and outputting the associated image / label that is part of the stored n-gram or attached to the stored n-gram do. Alternatively, the language model includes a word-to-image correspondence map 40 trained for a word-based n-gram map 14 and sections of text associated with the image (i.e., words). The language model compares the word sequence with the stored n-gram of the map 14 and then maps the predicted word to the image using the corresponding map 40 to predict the next word in the sequence of user input words .

The third and fourth methods of the invention relate to user interaction with a touch screen user interface of a device comprising one or more of the systems described above to generate image / label prediction 50. Specifically, a third method of the present invention provides a method of inputting data into an electronic device including a touch screen user interface having a keyboard, the user interface including a virtual interface configured to display the predicted image / Image / label button. The method includes inputting (700) a character sequence through a continuous gesture across the keyboard. In response to the user gesture over the image / label virtual button, the method includes inputting image / label 720 as data. The gesture may include breaking the contact with the user interface at the image / label virtual button.

The fourth method includes inputting a word / word on a touch screen user interface including a virtual button configured to display the predicted word / word and / or predicted image / label and the image / To a method for selecting between the inputs of a label. The method includes inputting (800) a predicted word / word in response to receiving a first gesture type over a / button on the button, inputting the predicted image / label in response to a second gesture type across the / button on the button 810 < / RTI >

As is apparent from the above description, the present invention solves the above-mentioned problem by providing a system and method for predicting emotions / stickers based on user input text. The present invention can increase the speed of the immediate input by suggesting one or more related immediate prediction which obtains the user from having to scroll through different aids to identify what the user wants.

In addition, the system and method of the present invention can also be used as contextual means in which predictions and emotions of emotions based on the next word predictions / corrections can be predicted and presented to the user, even if the user does not know that relevant or appropriate emotions exist, And provides the ability to detect abnormal emissaries.

The system and method of the present invention thus provides efficient emotional selection and input to electronic devices. Rather than having to scroll through the possible emoticons, the user can insert the relevant epilating by the tab of the virtual key that displays the predicted emoticons.

While examples have been provided with respect to emotions, the present invention is equally applicable to the insertion of any image / label associated with a user input text as described above.

The present invention also provides a computer program product comprising a computer readable medium having stored thereon a computer program for causing a processor to perform one or more of the methods according to the present invention.

The computer program product may be a data carrier in which a processor external to the data carrier, i. E. A computer program means, for causing a processor of the electronic device to perform the method according to the present invention is stored. The computer program product may also be downloadable from, for example, a data carrier, or from a supplier via the Internet or other available network, downloaded as an app to a mobile device (e.g., a mobile phone) A device, or a processor for executing computer program means once downloaded.

It is to be understood that this disclosure is made only by way of example, and alternatives and modifications may be made to the embodiments described without departing from the scope of the invention as defined in the claims.

Claims (56)

A system configured to predict an image / label associated with text entry by a user,
Means for receiving text input by a user; And
Comprising prediction means trained for sections of text associated with an image / label,
Wherein,
Receive text input by the user;
Determining relevance of textual input by the user to sections of text associated with the image / label;
And to predict the relevance of the image / label to the text entry by the user based on the sections of text associated with the image / label. A system configured to predict an image / label associated with text entry by a user,
Means for receiving text input by a user; And
Wherein the means for predicting is adapted to identify the image / label in the text and to correlate the identified image / label with the sections of the text to identify the image / However,
Wherein,
Receive text input by the user;
Compare the textual input by the user with sections of text associated with the image / label;
And to predict the relevance of the image / label to textual input by the user based on the sections of text associated with the identified image / label. 3. The system of claim 2, wherein the prediction means is a language model trained for text including a plurality of images / labels embedded with the text. The system of claim 1, wherein the prediction means is a search engine that includes a plurality of statistical models corresponding to a plurality of images / labels, each of the plurality of statistical models being associated with a corresponding image / ≪ / RTI > are trained for sections of text. The system according to claim 1, wherein the predicting means is a classifier trained for a plurality of text sources, each textual source comprising sections of text associated with a particular image / label. The system of claim 1, 4, or 5, wherein the system comprises a feature generation mechanism configured to extract a set of features from a text input by the user. 7. The system of claim 6, wherein the feature generation mechanism is configured to generate a feature vector from the features, wherein the feature vector represents text input by the user. 8. The system of claim 7, wherein the feature generation mechanism is configured to weight features by their term-frequency-inverse document frequency. The system of claim 7 or claim 8, wherein the feature generation mechanism is configured to extract features from each of the plurality of text sources and generate a feature vector for each image / label. The system of claim 9, wherein the classifier is trained for feature vectors for images / labels. 11. The computer-readable medium of claim 10, wherein the classifier comprises a feature vector representing a text associated with the image / label and a textual input by the user to determine whether the image / And generates a dot product of the feature vector representing the feature vector. 8. The method of claim 7 or claim 8, wherein the feature generation mechanism is configured to extract features from sections of text associated with an image / label and to train a corresponding statistical model for the extracted features system. 13. The system of claim 12, wherein the search engine is configured to query each statistical model for each feature of text input by the user to determine the presence and frequency of the feature. The system of claim 4 or 5, wherein the system further comprises a model that includes a prior probability for the images / labels based on prior use or previous input of images / labels by the user . The system of any one of claims 1 to 14, wherein the identified image / label is associated with sections of text that are not immediately preceding the identified image / label. The system according to any one of claims 1 to 15, wherein the text input does not correspond to a description of the image / label. The system of claim 15 or claim 16, wherein the language model comprises an n-gram map comprising word sequences associated with an image / label. 18. The apparatus of claim 17, wherein the prediction means comprises a prediction engine,
Wherein the prediction engine comprises:
A language model trained for text including a plurality of images / labels embedded with the text; And
Means for generating a sequence of one or more words from text input by the user,
Wherein the prediction engine comprises:
Receive a sequence of one or more words;
Compare the sequence of one or more words with a stored sequence of one or more words associated with the image / label;
And to predict the associated image / label for the sequence of one or more words based on a stored sequence of one or more words associated with the image / label. 19. The system of claim 18, wherein the prediction engine is configured to predict a next word in the sequence of one or more words based on a stored sequence of one or more words associated with the image / label. 3. The method of claim 2, wherein the predicting means comprises: a word-based language model comprising stored sequences of words; a map trained on words associated with images / labels, maps images to words, Means for generating a sequence of one or more words from a text input,
Compare the sequence of one or more words with a stored sequence of words to predict a next word in the sequence of one or more words;
And to predict an image associated with the next word using the map. The image processing apparatus according to any one of claims 1 to 20,
Further comprising a word-based language model comprising stored sequences of words,
Wherein,
Generate a sequence of one or more words from text input by the user;
Compare the sequence of one or more words with a stored sequence of words in the word-based language model;
And to predict the next word in the sequence based on the stored sequence of words. The system according to any one of claims 1 to 21, wherein the image is an emoji, an emoticon or a sticker. The system of any one of claims 1 to 22, wherein the label is a hash tag. 24. The system of any one of claims 1 to 23, wherein the prediction engine is configured to output the image / label if it is determined to be associated with text input by the user. In an electronic device,
A system according to any one of the preceding claims, And
And a user interface configured to receive user input and display the predicted image / label. 26. The system of claim 25,
And a virtual image / label button configured to display the predicted image / label for user selection. The system of claim 26, wherein the user interface further comprises a word / word virtual button configured to display predicted words / words for user selection. 29. The system of claim 26 or 27, wherein the user interface is configured to accept a user text input as a continuous gesture across a keypad, and wherein the interface is responsive to the gesture over the image / And the predicted image / label is input. 28. The method according to any one of claims 25 to 27,
Further comprising a processing circuit, the processing circuit comprising:
A first user input on the user interface and a second user input on the user interface as inputs, the first user input being different from the second user input in at least one aspect;
Displaying the predicted image / label on the display as user input data in response to receiving the first user input toward the virtual button;
In response to receiving the second user input toward the virtual button, display an alternative image / label prediction on the display for user selection. 26. The system of claim 25,
Further comprising a virtual button configured to display predicted word / word and / or predicted image / label for user selection, wherein said word / word corresponds to said image / label. 32. The method of claim 30, wherein the processing circuitry is further configured to distinguish between two gesture types across the virtual button on the virtual button, wherein in response to receiving the first gesture type over the virtual button / Wherein the predicted image / label is entered in response to a second gesture type across the virtual button / over the virtual button. A method for generating prediction means for predicting an image / label associated with text input by a user,
Receiving text containing one or more images / labels embedded within sections of text;
Identifying an image / label embedded within the text;
Associating the identified image / label with sections of the text; And
Training the prediction means for sections of text associated with the image / label. A method of predicting an image / label associated with text input to a system by a user using prediction means, the prediction means being trained for sections of text associated with an image / label,
Receiving the text input by the user at the prediction means;
Determining a relevance of textual input by the user to sections of text associated with the image / label; And
Predicting the relevance of the image / label to textual input by the user based on the sections of text associated with the image / label. A method for predicting an image / label associated with text entry into a system by a user using prediction means, the prediction means being trained for text containing an image / label embedded in the text, Identifying an image / label within the text and associating the identified image / label with sections of the text, the method comprising:
Receiving the text input by the user at the prediction means;
Comparing text input by the user with sections of text associated with the image / label; And
And predicting the relevance of the image / label to the text entry by the user based on the sections of text associated with the identified image / label. 35. The method of claim 34, wherein the means for predicting is a language model and the method comprises training the language model for text comprising a plurality of images / labels embedded within the text. 34. The method of claim 33 or claim 34, wherein the predicting means is a search engine comprising a plurality of statistical models corresponding to a plurality of images / labels, the method comprising the steps of: Training each of the plurality of statistical models with respect to the sections. 34. The method of claim 33, wherein the means for predicting is a classifier, the method comprising training the classifier for a plurality of textual sources, each textual source comprising sections of text associated with a particular image / / RTI > 37. The method of claim 36 or claim 37, further comprising extracting a set of features from a text input by the user using a feature generation mechanism. 39. The method of claim 38, further comprising generating a feature vector from features using the feature generation mechanism, wherein the feature vector represents text input by the user. 41. The method of claim 39, further comprising: determining word frequency versus inverse document frequency for each feature; and weighting features of the feature vector with the determined values. 39. The method of claim 39, further comprising extracting features from each of the plurality of text sources using the feature generation mechanism and generating feature vectors for each image / label Way. 45. The method of claim 41, further comprising training the classifier for feature vectors for images / labels. 42. The method of claim 41,
To determine whether the image / label is associated, if the text input by the user is given, by using a classifier representing a feature vector representing a text associated with the image / label and a feature vector representing a text input by the user ≪ / RTI > 39. The method of claim 38 or claim 39, further comprising: extracting features from sections of text associated with the image / label using the feature generation mechanism; and training a corresponding statistical model for the extracted features ≪ / RTI > 45. The method of claim 44, further comprising querying each statistical model of each feature of text input by the user using the search engine to determine the presence of a feature and its frequency. 45. The method of any one of claims 33 to 45, wherein the identified image / label is associated with sections of the text that are not immediately preceding the identified image / label. 47. The method of any one of claims 33 to 47, wherein the textual input does not correspond to a description of the image / label. 47. The method of claim 46 or claim 47, wherein the language model comprises an n-gram map comprising word sequences associated with an image / label,
Generating a sequence of one or more words from a text input by the user;
In a prediction engine including the language model, receiving a sequence of the one or more words;
Comparing the sequence of one or more words with a stored sequence of one or more words associated with an image / label; And
Predicting an associated image / label for the sequence of one or more words based on a stored sequence of one or more words associated with the image / label. 49. The method of claim 48, further comprising predicting, using the prediction engine, the next word in the sequence of one or more words based on a stored sequence of one or more words associated with the image / label. 34. The computer-readable medium of claim 33, wherein the means for predicting comprises a word-based language model having stored sequences of words, a map trained for words associated with images and maps images to appropriate words,
Generating a sequence of the one or more words from the input text; And
Comparing the sequence of one or more words with a stored sequence, predicting a next word in the sequence of one or more words, and using the map to identify an image associated with the next word. The method of any one of claims 33 to 49, wherein the predicting means further comprises a word-based language model comprising stored sequences of words,
Generating a sequence of one or more words from a text input by the user;
Comparing the sequence of one or more words with a stored sequence of words in the word-based language model; and
And predicting a next word in the sequence based on a stored sequence of the words. 51. The method of any one of claims 32- 51, wherein the image is an emotion, emoticon, or sticker. The method of any one of claims 32 to 52, wherein the label is a hash tag. A method of inputting data into an electronic device including a touch screen user interface having a keyboard, the user interface comprising a virtual image / label button configured to display a predicted image / label for user selection, ,
Inputting a character sequence through a continuous gesture across the keyboard; And
Responsive to the gesture across the image / label virtual button, entering the image / label as data. A selection between a word / word entry on the touch screen user interface including a virtual button configured to display a predicted word / word and / or a predicted image / label and an input of an image / label corresponding to the word / word In the method,
Inputting the predicted word / word in response to receiving a first gesture type over the button / over the button; And
Entering the predicted image / label in response to a second gesture type over the / button. A computer program for causing a processor to perform the method of any of claims 32 to 55.

Images