US8359282B2 - Supervised semantic indexing and its extensions - Google Patents

Supervised semantic indexing and its extensions Download PDF

Info

Publication number
US8359282B2
US8359282B2 US12562840 US56284009A US8359282B2 US 8359282 B2 US8359282 B2 US 8359282B2 US 12562840 US12562840 US 12562840 US 56284009 A US56284009 A US 56284009A US 8359282 B2 US8359282 B2 US 8359282B2
Authority
US
Grant status
Grant
Patent type
Prior art keywords
words
vector
weight
block
query
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US12562840
Other versions
US20100185659A1 (en )
Inventor
Bing Bai
Jason Weston
Ronan Collorbert
David Grangier
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Laboratories America Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Grant date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30Information retrieval; Database structures therefor ; File system structures therefor
    • G06F17/3061Information retrieval; Database structures therefor ; File system structures therefor of unstructured textual data
    • G06F17/30634Querying
    • G06F17/30657Query processing
    • G06F17/3066Query translation
    • G06F17/30663Selection or weighting of terms from queries, including natural language queries
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30Information retrieval; Database structures therefor ; File system structures therefor
    • G06F17/3061Information retrieval; Database structures therefor ; File system structures therefor of unstructured textual data
    • G06F17/30613Indexing
    • G06F17/30616Selection or weighting of terms for indexing

Abstract

A system and method for determining a similarity between a document and a query includes providing a frequently used dictionary and an infrequently used dictionary in storage memory. For each word or gram in the infrequently used dictionary, n words or grams are correlated from the frequently used dictionary based on a first score. Features for a vector of the infrequently used words or grams are replaced with features from a vector of the correlated words or grams from the frequently used dictionary when the features from a vector of the correlated words or grams meet a threshold value. A similarity score is determined between weight vectors of a query and one or more documents in a corpus by employing the features from the vector of the correlated words or grams that met the threshold value.

Description

RELATED APPLICATION INFORMATION

This application claims priority to provisional application Ser. No. 61/143,942 filed on Jan. 12, 2009, incorporated herein by reference.

This case is related to application Ser. No. 12/562,802 filed concurrently herewith.

BACKGROUND

1. Technical Field

The present invention relates to informational retrieval technology and more particularly to systems and methods for text document matching and ranking.

2. Description of the Related Art

Ranking text documents given a text-based query is one of the key tasks in information retrieval. Classical vector space models use weighted word counts and cosine similarity. This type of model often performs remarkably well, but suffers from the fact that only exact matches of words between query and target texts contribute to the similarity score. It also lacks the ability to adapt to specific datasets since no learning is involved.

Latent Semantic Indexing (LSI), and related methods such as probabilistic Latent Semantic Indexing (pLSA), and Latent Dirichlet Allocation (LDA) choose a low dimensional feature representation of “latent concepts”, and hence words are no longer independent. A support vector machine with hand-coded features based on the title, body, search engine rankings and the URL has also been implemented, as well as neural network methods based on training of a similar set of features. Other methods learned the weights of orthogonal vector space models on Wikipedia links and showed improvements over the OKAPI method. The same authors also used a class of models for matching images to text. Several authors have proposed interesting nonlinear versions of (unsupervised) LSI using neural networks and showed they outperform LSI or pLSA. However, we note their method is rather slow, thus dictionary size is limited.

SUMMARY

A system and method for determining a similarity between a document and a query includes providing a frequently used dictionary and an infrequently used dictionary in storage memory. For each word or gram in the infrequently used dictionary, n words or grams are correlated from the frequently used dictionary based on a first score. Features for a vector of the infrequently used words or grams are replaced with features from a vector of the correlated words or grams from the frequently used dictionary when the features from a vector of the correlated words or grams meet a threshold value. A similarity score is determined between weight vectors of a query and one or more documents in a corpus by employing the features from the vector of the correlated words or grams that met the threshold value.

A system for determining a similarity between a document and a query includes a memory configured to store a frequently used dictionary and an infrequently used dictionary. A processing device is configured to execute a program to correlate n words or grams from the frequently used dictionary based on a first score for each word or gram in the infrequently used dictionary. The program is further configured to replace features for a vector of the infrequently used words or grams with features from a vector of the correlated words or grams from the frequently used dictionary when the features from a vector of the correlated words or grams meet a threshold value. The processing device is further configured to determine a similarity score between weight vectors of a query and one or more documents in a corpus by employing the features from the vector of the correlated words or grams that met the threshold value.

These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

The disclosure will provide details in the following description of preferred embodiments with reference to the following figures wherein:

FIG. 1 is a block/flow diagram showing a system/method for determining similarity between a query and a document;

FIG. 2 is a block/flow diagram showing a system/method for simplifying dimensions when determining similarity between a query and a document;

FIG. 3 is a block/flow diagram showing a system/method for correlated feature hashing (CFH) in accordance with one embodiment;

FIG. 4 is a block/flow diagram showing a system/method for determining similarity between a query and a document with a higher order polynomial model;

FIG. 5 is a block diagram showing a system for conducting searches and/or retrieving documents in accordance with one embodiment; and

FIG. 6 is a block/flow diagram showing a more detailed system/method for determining similarity between a query and a document in accordance with the present principles.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In accordance with the present principles, a class of models is provided that are discriminatively trained to directly map from word content in a query-document or document-document pair to a ranking score. Like Latent Semantic Indexing (LSI), the present models account for correlations between words (e.g., synonymy, polysemy). However, unlike LSI, the present models are trained with a supervised signal (for example, an expert's judgment on whether a document is relevant to a query, or links among documents) directly on the task of interest, which is one reason for superior results. As the query and target texts are modeled separately, the present approach can easily be generalized to other retrieval tasks as well, such as cross-language retrieval. An empirical study is provided on a retrieval task based on Wikipedia documents, where we obtain state-of-the-art performance using the present method.

Embodiments described herein may be entirely hardware, entirely software or including both hardware and software elements. In a preferred embodiment, the present invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. A computer-usable or computer readable medium may include any apparatus that stores, communicates, propagates, or transports the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. The medium may include a computer-readable medium such as a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk, etc.

Supervised Semantic Indexing: We illustratively define a similarity function between a query and a document as:

f ( q , d ) = q T Wd = i , j w ij q i d j
where q and d are feature vectors of query and document, respectively. Each feature in a feature vector could be an individual word, its weight can be, for instance, a TFIDF value. W is a weight matrix and T is the transpose function. We can see that wij models correlation between the ith feature of query q and jth feature of d.

Sparse W matrix: We consider both memory and speed considerations. Firstly, this method so far assumes that W fits in memory. For example, if the dictionary size D=30000, then this needs, e.g., 3.4 Gb of RAM (assuming oats). The vectors q and d are sparse so the speed of computation of a single query-document pair involves mn computations of wijqidj, where q and d have m and n nonzero terms, respectively. We have found this is reasonable for training, but may be an issue at test time. Alternatively, one can compute v=qTW once, and then compute vd for each document. This is the same speed as a classical vector space model where the query contains D terms, assuming W is dense.

Sparse W matrices: If W was itself a sparse matrix, then computation of f(x) would be considerably faster. If the query has m nonzero terms, and any given column of W has p nonzero terms, then the method is at most mp times slower than a classical vector space model. We can enforce W to be sparse using standard feature selection algorithms; we hence generalize the known “Recursive Feature Elimination” algorithm yielding a simple, intuitive method:

1. First, we train the model with a dense matrix W as before.

2. For each column i of W find the k active elements with the smallest values of |wij|. Constrain these elements to equal zero (make them inactive).

3. Train the model with the constrained W matrix.

4. If W contains more than p nonzero terms in each column go back to 2.

Low rank W matrices: An alternative efficient scheme is to constrain W in the following way:
W=U T V+I

This induces a low dimensional “latent concept” space in a similar way to LSI. However, it differs in several ways: most importantly it is trained with a supervised signal. Further, U and V differ so it does not assume a query and a target document should be embedded in the same way, and the addition of the identity term (I) means this model automatically learns the tradeoff between using the low dimensional space and a classical vector space model. In terms of efficiency however—it is the same: its speed depends on the dimensionality of U and V.

We also highlight several variants:

1. W=I: if q and d are normalized TFIDF vectors, this is equivalent to using the standard cosine similarity with no learning (and no synonymy or polysemy).

2. W=D: where D is a diagonal matrix: one learns a re-weighting of TFIDF using labeled data.

3. W=UTU+I: we constrain the model to be symmetric; the query and target document are treated in the same way.

Polynomial Features and Low Rank Approximation: A ranking function may include: f(q,d)=qTWd where q and d are vectors to be ranked, W is a weight and T is the transpose function. If one were to concatenate the vectors q and d: v=(q,d) and then using a polynomial mapping Φ(•): f(q,d)=wTΦ(v), i.e., take all pairwise terms, one ends up with a ranking function: f(q,d)=qTW1q+qTW2d+dTW3d with three “W” matrices to learn. The first term qTW1q is useless since it is the same for all target documents. Although this term is not affected by the query at all, it corresponds to ranking a document's “importance” in a static way, which can still be useful. However, in the present case this static ranking is learned using a supervised signal.

Furthermore, in general, we could consider Φ(•) to be a polynomial of an order greater than 2. Although this can be computationally challenging, it can again be sped up using a low rank approximation. For example, if we would like the ranking function: f(q,d)=Wijkqidjdk. We can do that with a low rank approximation:

f ( q , d ) = i = 1 n ( Uq ) i ( Vd ) i ( Yd ) i .

This corresponds to embedding the query into an n-dimensional space using the mapping U, embedding the target document twice with two separate functions V and Y also in n-dimensional spaces. Then, the product of all three is taken for each dimension of the embedding space. If there are only two embedding spaces U and V, this reduces to the original ranking function. This can be generalized to more than three embeddings (e.g., more than degree 3).

Convolutions or Sliding Windows: In general, if we have an embedding t(qi) where qi is the ith sliding window in a text, we could embed the document with:

f ( q , d ) = ( i t ( q i ) ) T ( i t ( d i ) ) .

One could choose a function t which is a nonlinear neural network based on embedding words, one could use a multi-layer network with sigmoid functions, or one could apply a similar process using polynomials as described above—but instead apply a polynomial map to n-grams (i.e. within a sliding window). For example, for 2-grams containing two words w1 and w2 one might want to consider the following mapping. If g(wi) is the embedding of the word wi, one uses the mapping:

t ( d i ) = j g ( w 1 ) j g ( w 2 ) j .

This can be generalized to n-grams, and basically consists of only using some of the polynomial terms. Alternatively, one could hash n-grams.

Transductive embedding: Each document could have its own embedding vector Vi. (Could be done for advertisements too, e.g., for Yahoo). Note this embedding may not be a function. In other words, it is not necessary to find function V=f(v). We can add one of the following constraints:

1. Vd+ TVw>Vd− TVw—train word embedding with document embedding together, where w is in d+, but not in d−.

2. Vd+ Tf(d+)>Vd+ Tf(d−) train functional embedding with transductive embedding. Again, note Vd T≠f(d), instead, they have nonlinear relationship.

Reducing the dimensionality of infrequent words: One can introduce capacity control (and lower the memory footprint) by making the representation of each word a variable rather than a fixed dimension. The idea is that infrequent words should have less dimensions because they cannot be trained as well anyway. To make this work, a linear layer maps from a reduced dimension to the full dimension in the model. If a particular word w has dim(w) dimensions then the first dim(w) rows of a d by d matrix M are used to linearly map it to d dimensions, the matrix M is learned along with the representations of each word.

Hashing: Random Hashing of Words: One could embed a document using the average of B “bin” vectors rather than using words. One randomly assigns any given word to n (e.g., n=2 or 3) bins. In this way, rare words are still used but with a smaller number of features, making lower capacity and smaller memory footprint models.

Hashing n-grams: One could also use this trick to, e.g., hash n-grams, which otherwise would have no way of fitting in memory. To use less features, one could hash a 3-gram by making it the sum of three 2-grams e.g. “the cat sat”→g(the,cat)+g(cat,sat)+g(the,sat), where g(•) is the embedding.

Hashing with Prior Knowledge: One does not have to use random hashing functions. One could find the synonyms of rare words, e.g., use a web search with a DICE score (a known measuring score) to pick out the n most related words that have a frequency >F, e.g., F=30,000. We then bin that word into the n bins. In this way, rare words are represented by their synonyms resulting in less capacity models that should be smarter than random hashing. In general, if one has prior knowledge, one might always be able to do better than random hashing.

Referring now to the drawings in which like numerals represent the same or similar elements and initially to FIG. 1, a block/flow diagram for determining similarities between a query and documents is illustratively shown in accordance with the present principles. A class of models can be trained on a supervised signal to provide rankings of a database of documents (d) 11 given a query (q) 12. We learn the relationships among words directly using a weight matrix (W). The present principles provide higher searching quality and faster operation.

Block 11 represents a document d, and block 12 represents a query q. In block 13, a TFIDF vector is built for document d and in block 14, a TFIDF vector is built for query q. In blocks 15 and 16, TFIDF vectors Td and Tq are generated. In block 17, a similarity score is computed between q and d. As shown, the similarity between the query and the document is described as:
f(q,d)=q′Wd=Σ i,j w ij q′ i d j  (1)

The dimension of q and d are of the vocabulary size D (q′ is the transpose of q). For illustrative purposes, in Wikipedia, D>2.5 million. So matrix W can be intractably large. Ways to control the size of W will be described herein.

Training: Suppose we are given a set of tuples R (labeled data), where each tuple contains a query q, a relevant document d+ and an irrelevant (or lower ranked) document d. We choose W such that q′Wd+>q′Wd, expressing that d+ should be ranked higher than d. For that purpose, we employ a known method called margin ranking loss, and minimize:

( q , d + , d - ) max W ( 0 , 1 - q Wd + + q Wd - ) .

This optimization problem is solved through stochastic gradient descent, iteratively, one picks a random tuple and makes a gradient step for that tuple: W−λ(qd+−qd), if 1−qd+−qd>0.

One could exploit the sparsity of q and d when calculating these updates. To train the present model, we choose the (fixed) learning rate λ which minimizes the training error. Stochastic training is highly scalable and is easy to implement for the model. The present method includes a margin ranking perceptron with a particular choice of features. It thus involves a convex optimization problem and is hence related to a ranking SVM, except we have a highly scalable optimizer.

The vector space models for text in accordance with the present principles employ a weight or sparse matrix W (block 17). If W was itself a sparse matrix, then computation of f(•) would be considerably faster, not to mention significant memory savings. If the query has m non-zero terms, and any given row of W has p non-zero terms, then there are at most mp terms in v=q′W (compared to mD terms for a dense W, and m terms for a classical vector space cosine).

The simplest way to do this is to apply a regularizer such as minimizing the L1-norm of W:

min W ( γ W 1 + q in Q L ( W , q , d , Y ( q , d ) ) ) ( 2 )
where L(•) is a function that computes the loss of the current model given a query q in Q and a labeling Y(q,d) of the relevance of the documents d with respect to q. However, in general any sparsity promoting regularizer or feature selection technique could be used.

Training W: We enforce sparsity through feature selection. Here, a “projection” method can be seen as a generalization of the Recursive Feature Elimination (RFE) feature selection method. This method may include the following: 1) Train the model with a dense matrix W as before. 2) For each row i of W, find the k active elements with the smallest values of Wij. Constrain these elements to equal zero, i.e., make them inactive. 3) Train the model with the constrained W matrix. 4) If W contains more than p non-zero terms in each row go back to 2).

This method is motivated by the removal of the smallest weights being linked to the smallest change in the objective function and has been shown to work well in several experimental setups. Overall, we found this scheme to be simple and efficient, while yielding good results. Advantageously, feature selection may be applied to supervised semantic indexing (SSI).

Referring to FIG. 2, a low dimensional simplification method is shown in accordance with the present principles. TFIDF vectors d1, d2, d3, in block 21, for each document (doc1, doc2, doc3, . . . ) go through a linear embedding (e.g., g(d)=Vd) in block 22, and become vectors (Vd1, Vd2, Vd3, . . . ) in embedding space in block 23. A TFIDF vector q in block 24 goes through a linear transform (embedding function) in block 25 (e.g., h(q)=Uq) to become vector Uq in block 26. In block 27, a classical vector space model (q′d) and optimal embedding (q′U′Vd) are combined together. As described in FIG. 2, the query TFIDF vector and the document TFIDF vector are transformed with matrix U and V, respectively in block 22 and 25, before a dot product is conducted. It can be viewed as a special form of Eq. (1), where W=U′V+I (3). Thus, Eq. (1) becomes:
f(q,d)=q′Wd=q′U′Vd+q′d  (4)

The first term on the right-hand side (q′U′Vd) can be viewed as a dot product of q and d after transformation of U and V, respectively. This term contains synonym/polysemy information and thus allows fuzzy matching. The second term (q′d) is a traditional vector space model similarity, which uses exact word matching. By dropping the second term in Eq. (4), we get:
f(q,d)=q′U′Vd  (5).

This form is useful for heterogeneous systems where query features and document features are different, e.g. cross-language retrieval. If we model query and document in the same way, Eq. (2) becomes: W=U′U+I (6).

Eq. (6) has fewer parameters to train than Eq. (2), and thus provides for faster training and is less likely to overfit. This can provide better performance.

Training: When the W matrix is constrained, e.g., W=U′V+I training is performed in a similar way as before, but in this case by making a gradient step to optimize the parameters U and V:
U←U+λV(d + −d )q′, if 1−f(q,d +)+f(q,d )>0
V←V+λUq(d + −d )′, if 1−f(q,d +)+f(q,d )>0

Note this is no longer a convex optimization problem. In our experiments, we initialized the matrices U and V randomly using a normal distribution with mean zero and standard deviation one. In block 27, the classical vector space model (q′d) and optimal embedding (q′UVd) are embedded to give better performance. U and V can model query and documents in different ways, thus U and V are good for heterogeneous systems such as systems that perform cross language retrieval (e.g., query is Japanese, but documents are in English).

Referring to FIG. 3, a block/flow diagram shows a system/method with correlated feature hashing in accordance with one embodiment. Block 31 includes frequent word storage. Block 32 includes infrequent word storage. Block 33 includes a method to compute correlated features. In block 33, for each word “a” in (B) (infrequent words in block 32), calculate its DICE score with all words in (A) (frequent words in block 31). DICE(a,b)=occur(a,b)/occur(a)+occur(b). Then, take the m words in (A) with the largest DICE score as the correlated features of a. Examples of correlated features are stored in block 34. In block 34, a first column includes infrequent words, the other columns are the top 5 words correlated with the words in the first column. In blocks 37, 36, and 35, recomputed features using the correlated words are stored in block 34. Infrequent words in TFIDF vectors in black 37 are replaced with frequent words, based on the correlated hashing features in block 36. In block 36, correlated feature hashing occurs. This includes, for each word count for a word in (B), replacing the word with its correlated features of weight at least 0.2. Note the weight threshold may be adjusted as needed. Word vectors are output in block 35.

To further reduce the number of features, correlated feature hashing may be introduced. With this method, we limit the features to the most frequent ones (e.g., the most frequent 30,000 words). Any other word, say w, is “hashed” into the 5 words (or at least two) that co-occur with w most often. The co-occurrence can be measured by a “DICE” score in block 33, which is the number of co-occurrences of two words divided by the sum of the occurrences of the two words. The same method can be applied to the features of n-grams. N-grams can be hashed into most co-occurred word features.

Training: Correlated feature hashing can be viewed as a preprocessing stage (e.g., block 201 of FIG. 6). Rare words can be removed from word vectors by mapping them to frequent words. Then, the training can be conducted on the resulting vectors. Correlated feature hashing has more meaningful and effective feature reduction then prior art methods.

Correlated feature hashing includes providing a frequently used dictionary 31 and an infrequently used dictionary 32 in storage memory. For each word or gram in the infrequently used dictionary 32, correlate n words or grams from the frequently used dictionary 32 based on a first score (e.g., DICE score). In block 36, features are replaced for a vector of the infrequently used words or grams with features from a vector of the correlated words or grams from the frequently used dictionary 31 when the features from a vector of the correlated words or grams meet a threshold value (e.g., 0.2). A similarity score between weight vectors of a query and one or more documents is achieved by employing the features from the vector of the correlated words or grams that met the threshold value.

Referring to FIG. 4, a block/flow diagram illustratively depicts a high-order polynomial model in accordance with another embodiment. The models as described above are all quadratic feature models. However, the models can be extended by extending Eq. (1) to higher order models: f(q,d)=w·Φk([q,d]) (7) where Φk(•) is a feature map that considers all possible k-degree terms Φk(x1, x2, . . . , xn)=<xi 1 , xi 2 , . . . , xi k : 1≦i1, i2, . . . , ik≦D>.

In query-document ranking, we allow document terms to appear more than once, but limit query terms to appear only once. Also, we only consider the terms with both query terms and document terms. For example, take k=3, we have:
f(q,d)=Σi,j,k w ijk q i d i d j d k  (8)

Again, to have better scalability, it can be approximated with a low dimension approximation:
f(q,d)=ΣjεD h(q), πm-1,k-1 g m(d)  (9)

The process of Eq. (9) is depicted in FIG. 4. In block 41, TFIDF vectors are mapped to multiple embedding spaces in block 43 by going through multiple function mappings in block 42. A query TFIDF vector q in block 44 is transformed by function h(q) in block 45. A similarity between q and d is calculated by Eq. 9 in block 46.

If we set k=3, and embedding functions to be linear, we have f(q,d)=Σi(Uq)i(Vd)i(Yd)i (10).

Training: Again, we could adopt the Stochastic Gradient Descent (SGD) method. The gradient can be easily determined using partial derivatives. For formulation in Eq. (8), wijk←wijk+λ(dj +dk +−dj dk )qi, if 1−f(q,d+)+f(q,d)>0. For formulation in Eq. (10), U←U+λ(Vd+

Figure US08359282-20130122-P00001
Yd+−Vd
Figure US08359282-20130122-P00001
Yd)q′, if 1−f(q,d+)+f(q,d)>0; V←V+λ(Uq
Figure US08359282-20130122-P00001
Yd+−Uq
Figure US08359282-20130122-P00001
Yd)(d+−d), if 1−f(q,d+)+f(q,d)>0; and Y←Y+λ(Uq
Figure US08359282-20130122-P00001
Vd+−Ug
Figure US08359282-20130122-P00001
Yd)(d+−d)′, if 1−f(q,d+)+f(q,d)>0 where operator
Figure US08359282-20130122-P00001
is a component-wise product of vectors, i.e., (A
Figure US08359282-20130122-P00001
B)i=AiBi.

Advantageously, Eq. (8) provides significant improvements in document ranking. Eq. (9) can be viewed as a generalized dot product, which simplifies the superior document ranking capabilities of Eq. (8).

Half-transductive Ranking: Eq. (5) can be viewed as dot product of two functions: f(q,d)=φ1(q)·φ2(d)=(Uq)′(Vd) (10)

where two function (linear) mappings are φ1(•) and φ1(•) and the parameters being features in original feature space. This method is also called functional embedding because new samples can be put into embedding space using function φ(•). Another important category of learning methods is transductive learning. This method assigns a vector viεRm to each object yiεY that will be learned using a supervised or unsupervised signal and does not involve any feature representation of yi. Instead, the embedding only uses information of relative distances between yi, yjεY. In this sense, it can be said to be a nonlinear method. Ranking is then typically achieved by measuring the distances in the embedding space (Euclidean distance or dot product similarity). This method provides a point-to-point correspondence between the input space and the intrinsic space in which the data lie. Such methods have multiplied in recent years and are most commonly rooted either in factor analysis (e.g., principal component analysis) or multidimensional scaling and their variants: e.g., kernel PCA, Isomap, Locally Linear Embedding and Laplacian Eigenmaps. In many cases, transductive learning gives outstanding performance for clustering applications. The drawback of this method is that, the query has to be one of the objects yiεY for this approach to make sense as only these objects are embedded. Out-of-sample extension methods have to be adopted for new samples.

In accordance with the present principles, we can combine the functional embedding and transductive embedding, using the following half transductive model:

f(q,di)=M(φ(q),vi) (11) where φ(q) is the functional embedding for new queries, while vi is the nonlinear embedding to give better performance. Specifically, if we require φ(q) to be a linear function, Eq. (11) becomes:
f(q,d i)=M(Wq,v i)  (12).

Training: We can adopt a gradient-descent based optimization method. For Eq. (11), the derivative of function M and φ can be calculated and used in adjusting parameters. For the transductive part vi, technology like “lookup table” can be used.

Advantageously, the new framework for half-transductive learning is provided. Half of the formulation is functional which handles new sample embedding, while the other half is transductive which provides better performance for existing databases.

Sliding window based embedding function: In general if we have an embedding t(qi) where qi is the ith sliding window in a text, we could embed the document with: f(q,d)=Σi(t(qi))T·Σj(t(dj)) or its normalized version:

f ( q , d ) = 1 m i [ 1 , m ] ( t ( q i ) ) T · 1 n j [ 1 , n ] ( t ( d j ) ) .
One could choose a function which is a nonlinear neural network based on embedding words, one could use a multi-layer network with sigmoid functions, or could apply a similar process as in the section on polynomials above (but instead to apply a polynomial map to n-grams, within a sliding window)). For example, for 2-grams containing two words w1 and w2 one might want to consider the following mapping. If g(w1) is the embedding of the word with wi, one uses the mapping: t(di)=Σjg(w1)jg(w2)j: This can be generalized to n-grams. This basically includes using only some of the polynomial terms.

Reducing the dimensionality of infrequent words: One can introduce capacity control (and lower the memory footprint) by making the representation of each word a variable rather than fixed dimension. The idea is that infrequent words should have fewer dimensions because they cannot be trained as well anyway. To make this work a linear layer maps from a reduced dimension to the full dimension in the model. If a particular word w has dim(w) dimensions, then the first dim(w) rows of a d×d matrix M are used to linearly map the word to d dimensions. The matrix M is learned along with the representations of each word.

Referring to FIG. 5, a system 100 is illustratively depicted for carrying out document ranking, document retrieval and text/document analysis in accordance with the present principles. In accordance with one embodiment, system 100 includes at least one terminal or processing device 104 having a user interface 102. The user can present queries or perform documents searches through interface 102. System 100 includes memory 112, which may itself include documents or items that can be queried by the user. The memory 112 stores a program 114 (or programs), which implement the methods in accordance with the present principles. For example, program 114 may learn to rank documents or items using word features. Such learning may include learning with a basic model in accordance with the present embodiments. This basic model may be employed in text-image ranking, or in text-text ranking. The program 114 may include methods for sparsification of a weight matrix or implement low ranking approximations as described hereinabove. Memory 112 and program 114 may be employed to perform correlated feature hashing (CFH). Other aspects of the present principles that may be implemented by program 114 may include polynomial modeling of word features, half-transductive modeling, sliding window mapping, and flexible dimensionality for word embedding. It should be understood that the many aspects described in accordance with the present principles can be employed together in various combinations. For example, half-transductive modeling may be employed with sliding window mapping, and correlated feature hashing (CFH) may be employed with any of the embodiments described herein.

System 100 may interact with local servers 108 and databases 110 or remote servers 108 and databases 110 through a network 106. These databases 110 and servers 108 may include the documents or items to be searched using a user or machine generated query. It should be understood that the program 114 may be remotely disposed from system 100, e.g., located on a server or the like. Training of models may be performed on system 100 or may be performed on other platforms to create models 115 for implementing the present methods.

Referring to FIG. 6, a method for determining a similarity between a document and a query is illustratively shown. The method may be employed for document ranking, document retrieval or any other analysis that employs similarity scores between a query and documents. In block 201, a preprocessing step may be employed for correlated feature hashing (CFH) as described in FIG. 3.

In block 202, a weight vector for each of a plurality of documents in a corpus of documents is built and stored in memory. In block 203, a weight vector is built for a query input into a document system. The weight vectors may include Term Frequency Inverse Document Frequency (TFIDF) weight vectors, although other feature vectors are also contemplated.

In one embodiment, the weight vectors for the plurality of documents are mapped to multiple embedding spaces to handle models having polynomial functions of n-order in block 204. In another embodiment, functional embedding and transductance embedding are combined to transform weight vectors to model both new queries and improve performance in existing databases in block 205. In block 206, a sliding window may be employed while embedding words or grams to consider other words or grams in proximity thereof.

In one simplifying embodiment, in block 20B, building a weight vector for each of a plurality of documents includes performing a linear embedding to transform the weight vector to embedding space using a matrix U, and in block 209, building a weight vector for a query includes performing a linear embedding to transform the weight vector to embedding space using a matrix V.

In block 210, a weight matrix (W) is generated which distinguishes between relevant documents and lower ranked documents by comparing document/query tuples using a gradient step approach. Other comparison techniques may also be employed.

In block 211, the weight matrix generation may include assigning infrequently occurring words to a reduced dimension representation; and mapping the reduced dimension representation to a full dimension representation when the infrequently occurring words are encountered. The weight matrix may be sparsified through feature selection in block 212. Sparification may include finding active elements in the weight matrix with smallest weight values in block 213 by searching the matrix row by row. The active elements in the weight matrix are constrained with smallest weight values set to equal zero in block 214. In block 215, a model is trained with the weight matrix, and, in block 216, if the weight matrix includes more than p non-zero terms, returning to the finding step (block 213). The result is a sparsely populated weight matrix to improve computation time and reduce memory and computational overhead.

In block 220, a similarity score is determined between weight vectors of the query and documents in a corpus by determining a product of a document weight vector, a query weight vector and the weight matrix. In the simplified embodiment, determining the similarity score includes taking a product between a transformed document weight vector (block 208) and a transformed query weight vector (block 209) to simplify computations.

It should be understood that blocks 204, 205, 206, 211, 212 are all alternative options to decide similarity score f(q,d). Although these techniques can be combined, they may be employed independently or combined in any order.

Having described preferred embodiments of systems and methods for supervised semantic indexing and its extensions (which are intended to be illustrative and not limiting), it is noted that modifications and variations can be made by persons skilled in the art in light of the above teachings. It is therefore to be understood that changes may be made in the particular embodiments disclosed which are within the scope and spirit of the invention as outlined by the appended claims. Having thus described aspects of the invention, with the details and particularity required by the patent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims.

Claims (14)

1. A method for determining a similarity between a document and a query, comprising:
providing a frequently used dictionary and an infrequently used dictionary in storage memory;
for each word or gram in the infrequently used dictionary, correlating n words or grams from the frequently used dictionary based on a first score;
replacing features for a vector of the infrequently used words or grams with features from a vector of the correlated words or grams from the frequently used dictionary when the features from a vector of the correlated words or grams meet a threshold value; and
determining a similarity score between weight vectors of a query and one or more documents in a corpus by employing the features from the vector of the correlated words or grams that met the threshold value.
2. The method as recited in claim 1, wherein the vectors include Term Frequency Inverse Document Frequency (TFIDF) weighted vectors.
3. The method as recited in claim 1, further comprising:
building weight vectors for documents in a corpus of documents stored in memory and for a query input in a document system;
generating a weight matrix which distinguishes between relevant documents and lower ranked documents by comparing document/query tuples using a gradient step approach, wherein determining a similarity score includes determining the similarity score between weight vectors of the query and documents in a corpus by determining a product of a document weight vector, a query weight vector and the weight matrix.
4. The method as recited in claim 3, further comprising sparsifying the weight matrix through feature selection.
5. The method as recited in claim 4, wherein sparsifying includes:
finding active elements in the weight matrix with smallest weight values;
constraining the active elements in the weight matrix with smallest weight values to equal zero;
training a model with the weight matrix; and
if the weight matrix includes more than p non-zero terms, returning to the finding step.
6. The method as recited in claim 3, wherein building a weight vector for each of a plurality of documents includes performing a linear embedding to transform the weight vector to embedding space using a matrix U, and wherein building a weight vector for a query includes performing a linear embedding to transform the weight vector to embedding space using a matrix V, and wherein determining a similarity score includes taking a product between a transformed document weight vector and a query weight vector to simplify computations.
7. The method as recited in claim 2, wherein building a weight vector for each of a plurality of documents in a corpus of document includes mapping the weight vectors for the plurality of documents to multiple embedding spaces to handle models having polynomial functions of n-order.
8. The method as recited in claim 2, further comprising combining functional embedding with transductance embedding to transform weight vectors to model both new queries and improve performance in existing databases.
9. The method as recited in claim 2, wherein building a weight vector includes mapping the weight vector using a sliding window while embedding words or grams to consider other words or grams in proximity thereof.
10. The method as recited in claim 2, wherein generating a weight matrix includes:
assigning infrequently occurring words to a reduced dimension representation; and
mapping the reduced dimension representation to a full dimension representation when the infrequently occurring words are encountered.
11. The method as recited in claim 1, wherein n is five.
12. The method as recited in claim 1, wherein the threshold value is at least 0.2.
13. The method as recited in claim 1, wherein the first score includes a DICE score.
14. A system for determining a similarity between a document and a query, comprising:
a memory configured to store a frequently used dictionary and an infrequently used dictionary;
a processing device configured to execute a program to correlate n words or grams from the frequently used dictionary based on a first score for each word or gram in the infrequently used dictionary;
the program further configured to replace features for a vector of the infrequently used words or grams with features from a vector of the correlated words or grams from the frequently used dictionary when the features from a vector of the correlated words or grams meet a threshold value; and
the processing device further configured to determine a similarity score between weight vectors of a query and one or more documents in a corpus by employing the features from the vector of the correlated words or grams that met the threshold value.
US12562840 2009-01-12 2009-09-18 Supervised semantic indexing and its extensions Active 2031-06-23 US8359282B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US14394209 true 2009-01-12 2009-01-12
US12562840 US8359282B2 (en) 2009-01-12 2009-09-18 Supervised semantic indexing and its extensions

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US12562840 US8359282B2 (en) 2009-01-12 2009-09-18 Supervised semantic indexing and its extensions

Publications (2)

Publication Number Publication Date
US20100185659A1 true US20100185659A1 (en) 2010-07-22
US8359282B2 true US8359282B2 (en) 2013-01-22

Family

ID=42319750

Family Applications (2)

Application Number Title Priority Date Filing Date
US12562840 Active 2031-06-23 US8359282B2 (en) 2009-01-12 2009-09-18 Supervised semantic indexing and its extensions
US12562802 Active 2031-03-06 US8341095B2 (en) 2009-01-12 2009-09-18 Supervised semantic indexing and its extensions

Family Applications After (1)

Application Number Title Priority Date Filing Date
US12562802 Active 2031-03-06 US8341095B2 (en) 2009-01-12 2009-09-18 Supervised semantic indexing and its extensions

Country Status (1)

Country Link
US (2) US8359282B2 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150242387A1 (en) * 2014-02-24 2015-08-27 Nuance Communications, Inc. Automated text annotation for construction of natural language understanding grammars
US9477654B2 (en) 2014-04-01 2016-10-25 Microsoft Corporation Convolutional latent semantic models and their applications
US9519859B2 (en) 2013-09-06 2016-12-13 Microsoft Technology Licensing, Llc Deep structured semantic model produced using click-through data
US9535960B2 (en) 2014-04-14 2017-01-03 Microsoft Corporation Context-sensitive search using a deep learning model

Families Citing this family (51)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9195640B1 (en) * 2009-01-12 2015-11-24 Sri International Method and system for finding content having a desired similarity
JP5119288B2 (en) * 2010-03-15 2013-01-16 シャープ株式会社 The mobile terminal device, an information output system, information output method, program and recording medium
US9298818B1 (en) * 2010-05-28 2016-03-29 Sri International Method and apparatus for performing semantic-based data analysis
US20120078918A1 (en) * 2010-09-28 2012-03-29 Siemens Corporation Information Relation Generation
US8819019B2 (en) * 2010-11-18 2014-08-26 Qualcomm Incorporated Systems and methods for robust pattern classification
US9529908B2 (en) 2010-11-22 2016-12-27 Microsoft Technology Licensing, Llc Tiering of posting lists in search engine index
US8478704B2 (en) 2010-11-22 2013-07-02 Microsoft Corporation Decomposable ranking for efficient precomputing that selects preliminary ranking features comprising static ranking features and dynamic atom-isolated components
US9424351B2 (en) 2010-11-22 2016-08-23 Microsoft Technology Licensing, Llc Hybrid-distribution model for search engine indexes
US8713024B2 (en) 2010-11-22 2014-04-29 Microsoft Corporation Efficient forward ranking in a search engine
US8620907B2 (en) * 2010-11-22 2013-12-31 Microsoft Corporation Matching funnel for large document index
US20120253792A1 (en) * 2011-03-30 2012-10-04 Nec Laboratories America, Inc. Sentiment Classification Based on Supervised Latent N-Gram Analysis
US8892488B2 (en) * 2011-06-01 2014-11-18 Nec Laboratories America, Inc. Document classification with weighted supervised n-gram embedding
US8515964B2 (en) * 2011-07-25 2013-08-20 Yahoo! Inc. Method and system for fast similarity computation in high dimensional space
CN102360372B (en) * 2011-10-09 2013-01-30 北京航空航天大学 Cross-language document similarity detection method
US8799269B2 (en) * 2012-01-03 2014-08-05 International Business Machines Corporation Optimizing map/reduce searches by using synthetic events
JP2015004996A (en) * 2012-02-14 2015-01-08 インターナショナル・ビジネス・マシーンズ・コーポレーションInternational Business Machines Corporation Apparatus for clustering plural documents
JP5890539B2 (en) * 2012-02-22 2016-03-22 ノキア テクノロジーズ オーユー Access to the service based on the prediction
CN103309886B (en) * 2012-03-13 2017-05-10 阿里巴巴集团控股有限公司 A structured information search method and apparatus based trading platform
US8898165B2 (en) 2012-07-02 2014-11-25 International Business Machines Corporation Identification of null sets in a context-based electronic document search
US8903813B2 (en) 2012-07-02 2014-12-02 International Business Machines Corporation Context-based electronic document search using a synthetic event
US9460200B2 (en) 2012-07-02 2016-10-04 International Business Machines Corporation Activity recommendation based on a context-based electronic files search
US9262499B2 (en) 2012-08-08 2016-02-16 International Business Machines Corporation Context-based graphical database
US8676857B1 (en) 2012-08-23 2014-03-18 International Business Machines Corporation Context-based search for a data store related to a graph node
US8959119B2 (en) 2012-08-27 2015-02-17 International Business Machines Corporation Context-based graph-relational intersect derived database
US9619580B2 (en) 2012-09-11 2017-04-11 International Business Machines Corporation Generation of synthetic context objects
US8620958B1 (en) 2012-09-11 2013-12-31 International Business Machines Corporation Dimensionally constrained synthetic context objects database
US9251237B2 (en) 2012-09-11 2016-02-02 International Business Machines Corporation User-specific synthetic context object matching
US9223846B2 (en) 2012-09-18 2015-12-29 International Business Machines Corporation Context-based navigation through a database
US8782777B2 (en) 2012-09-27 2014-07-15 International Business Machines Corporation Use of synthetic context-based objects to secure data stores
US9020954B2 (en) 2012-09-28 2015-04-28 International Business Machines Corporation Ranking supervised hashing
US9741138B2 (en) 2012-10-10 2017-08-22 International Business Machines Corporation Node cluster relationships in a graph database
US8931109B2 (en) 2012-11-19 2015-01-06 International Business Machines Corporation Context-based security screening for accessing data
US9286379B2 (en) * 2012-11-26 2016-03-15 Wal-Mart Stores, Inc. Document quality measurement
US9229932B2 (en) 2013-01-02 2016-01-05 International Business Machines Corporation Conformed dimensional data gravity wells
US8914413B2 (en) 2013-01-02 2014-12-16 International Business Machines Corporation Context-based data gravity wells
US8983981B2 (en) 2013-01-02 2015-03-17 International Business Machines Corporation Conformed dimensional and context-based data gravity wells
US9069752B2 (en) 2013-01-31 2015-06-30 International Business Machines Corporation Measuring and displaying facets in context-based conformed dimensional data gravity wells
US8856946B2 (en) 2013-01-31 2014-10-07 International Business Machines Corporation Security filter for context-based data gravity wells
US9053102B2 (en) 2013-01-31 2015-06-09 International Business Machines Corporation Generation of synthetic context frameworks for dimensionally constrained hierarchical synthetic context-based objects
US20140236577A1 (en) * 2013-02-15 2014-08-21 Nec Laboratories America, Inc. Semantic Representations of Rare Words in a Neural Probabilistic Language Model
US9292506B2 (en) 2013-02-28 2016-03-22 International Business Machines Corporation Dynamic generation of demonstrative aids for a meeting
US9110722B2 (en) 2013-02-28 2015-08-18 International Business Machines Corporation Data processing work allocation
US9600777B2 (en) * 2013-03-11 2017-03-21 Georges Harik Configuring and optimizing computational structure for a machine learning application using a tuple of vectors
US20140279748A1 (en) * 2013-03-15 2014-09-18 Georges Harik Method and program structure for machine learning
US9348794B2 (en) 2013-05-17 2016-05-24 International Business Machines Corporation Population of context-based data gravity wells
US9195608B2 (en) 2013-05-17 2015-11-24 International Business Machines Corporation Stored data analysis
US9262485B2 (en) * 2013-08-13 2016-02-16 International Business Machines Corporation Identifying a sketching matrix used by a linear sketch
JP2015166962A (en) * 2014-03-04 2015-09-24 日本電気株式会社 Information processing device, learning method, and program
JPWO2015145981A1 (en) * 2014-03-28 2017-04-13 日本電気株式会社 Multilingual document similarity learning device, multilingual document similarity determination device, multilingual document similarity learning method, multilingual document similarity determination method, and multilingual document similarity learning program
US9330104B2 (en) 2014-04-30 2016-05-03 International Business Machines Corporation Indexing and searching heterogenous data entities
WO2017180475A1 (en) * 2016-04-15 2017-10-19 3M Innovative Properties Company Query optimizer for combined structured and unstructured data records

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6137911A (en) * 1997-06-16 2000-10-24 The Dialog Corporation Plc Test classification system and method
US20100082511A1 (en) * 2008-09-30 2010-04-01 Microsoft Corporation Joint ranking model for multilingual web search

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8195669B2 (en) * 2008-09-22 2012-06-05 Microsoft Corporation Optimizing ranking of documents using continuous conditional random fields
US8166032B2 (en) * 2009-04-09 2012-04-24 MarketChorus, Inc. System and method for sentiment-based text classification and relevancy ranking

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6137911A (en) * 1997-06-16 2000-10-24 The Dialog Corporation Plc Test classification system and method
US20100082511A1 (en) * 2008-09-30 2010-04-01 Microsoft Corporation Joint ranking model for multilingual web search

Non-Patent Citations (19)

* Cited by examiner, † Cited by third party
Title
Balasubramanian et al. The Isomap algorithm and topological stability. www.sciencemag.org, Science. vol. 295 . Jan. 2002. (3 Pages) http://www.sciencemag.org/cgi/reprint/295/5552/7a.pdf.
Belkin et al. Laplacian Eigenmaps for Dimensionality Reduction and Data Representation. Advances in Neural Information Processing Systems 14 . Dec. 2002, pp. 1-28.
Bengio et al. Out-of-sample extensions for LLE, isomap, MDS, eigenmaps, and spectral clustering. Advances in Neural Information Processing Systems 16. 2004. (8 Pages).
Blei et al. Latent Dirichlet Allocation. Journal of Machine Learning Research. vol. 3. Jan. 2003, pp. 993-1022.
Burges et al. Learning to rank using gradient descent. In ICML 2005, in Proceeding of the 22th International Conference on Machine Learning. Aug. 2005. pp. 89-95.
Collins et al. New ranking algorithms for parsing and tagging: kernels over discrete structures, and the voted perceptron. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics. Jul. 2002. pp. 263-270.
Collobert et al. Fast Semantic Extraction Using a Novel Neural Network Architecture. 45th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. ACL. Jun. 2007. (8 pages).
Deerwester et al. Indexing by latent semantic analysis. Journal of the American Society for Information Science, vol. 41. 1990. pp. 391-407.
Grangier et al. A discriminative kernel-based approach to rank Images from text queries. IEEE Transactions on Pattern Analysis and Machine Intelligence. PAMI. vol. 30, Issue: 8. Aug. 2008. pp. 1-14.
Grangier et al. Inferring document similarity from hyperlinks. In CIKM '05. ACM. Oct. 2005. pp. 359-360.
Herbrich et al. Large margin rank boundaries for ordinal regression. Advances in Large Margin Classifers. Mar. 2000. pp. 115-132.
Hofmann. Probabilistic latent semantic Indexing. Proceedings of the Twenty-Second Annual International SIGIR Conference on Research and Development in Information Retrieval. In SIGIR. Aug. 1999. pp. 50-57.
Joachims. Optimizing search engines using clickthrough data. In ACM SIGKDD '02. International Conference on Knowledge Discovery and Data Mining. Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. 2002. pp. 133-142. (10 Pages).
Keller et al. A Neural network for Text Representation. In International Conference on Artificial Neural Networks. Sep. 2005. (6 Pages).
Roweis et al. Nonlinear dimensionality reduction by locally linear embedding. www.sciencemag.org. Science. vol. 290. Dec. 2000. pp. 2323-2326.
Salakhutdinov et al. Semantic Hashing. In proceedings of the SIGIR Workshop on Information Retrieval and Applications of Graphical Models. vol. 41 No. 2. Dec. 2007. (8 Pages).
Scholkopf et al. Kernel principal component analysis. Advances in kernel methods: support vector learning. 1999. (6 Pages).
Shi et al. Hash kernels. In Twelfth International Conference on Artificial Intelligence and Statistics. Apr. 2009. (8 Pages).
Sun et al. Supervised latent semantic Indexing for document categorization. IEEE Computer Society. In ICDM 2004. ICDM '04: Proceedings of the Fourth IEEE International Conference on Data Mining. Nov. 2004. pp. 535-538.

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9519859B2 (en) 2013-09-06 2016-12-13 Microsoft Technology Licensing, Llc Deep structured semantic model produced using click-through data
US20150242387A1 (en) * 2014-02-24 2015-08-27 Nuance Communications, Inc. Automated text annotation for construction of natural language understanding grammars
US9524289B2 (en) * 2014-02-24 2016-12-20 Nuance Communications, Inc. Automated text annotation for construction of natural language understanding grammars
US9477654B2 (en) 2014-04-01 2016-10-25 Microsoft Corporation Convolutional latent semantic models and their applications
US9535960B2 (en) 2014-04-14 2017-01-03 Microsoft Corporation Context-sensitive search using a deep learning model

Also Published As

Publication number Publication date Type
US20100185659A1 (en) 2010-07-22 application
US8341095B2 (en) 2012-12-25 grant
US20100179933A1 (en) 2010-07-15 application

Similar Documents

Publication Publication Date Title
Berry et al. Using linear algebra for intelligent information retrieval
Sarawagi Information extraction
Khan et al. A review of machine learning algorithms for text-documents classification
Bollegala et al. A web search engine-based approach to measure semantic similarity between words
Song et al. Automatic tag recommendation algorithms for social recommender systems
Lee et al. Information gain and divergence-based feature selection for machine learning-based text categorization
Markov et al. Data mining the Web: uncovering patterns in Web content, structure, and usage
Peng et al. Information extraction from research papers using conditional random fields
Huang Similarity measures for text document clustering
Kowalski Information retrieval systems: theory and implementation
Aggarwal et al. A survey of text clustering algorithms
Ando et al. A framework for learning predictive structures from multiple tasks and unlabeled data
Ramakrishnan et al. Data mining: From serendipity to science
US20090006382A1 (en) System and method for measuring the quality of document sets
Hotho et al. A brief survey of text mining.
Giunchiglia et al. Lightweight ontologies
US20050080776A1 (en) Internet searching using semantic disambiguation and expansion
US20070143101A1 (en) Class description generation for clustering and categorization
US20100185691A1 (en) Scalable semi-structured named entity detection
Boley et al. Partitioning-based clustering for web document categorization
US20120278321A1 (en) Visualization of concepts within a collection of information
Mecca et al. A new algorithm for clustering search results
US20090254512A1 (en) Ad matching by augmenting a search query with knowledge obtained through search engine results
Liu et al. Robust and scalable graph-based semisupervised learning
US20060259481A1 (en) Method of analyzing documents

Legal Events

Date Code Title Description
AS Assignment

Owner name: NEC LABORATORIES AMERICA, INC., NEW JERSEY

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BAI, BING;WESTON, JASON;COLLOBERT, RONAN;AND OTHERS;SIGNING DATES FROM 20090911 TO 20090916;REEL/FRAME:023255/0416

AS Assignment

Owner name: NEC CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NEC LABORATORIES AMERICA, INC.;REEL/FRAME:031998/0667

Effective date: 20140113

FPAY Fee payment

Year of fee payment: 4

AS Assignment

Owner name: NEC CORPORATION, JAPAN

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE 8538896 AND ADD 8583896 PREVIOUSLY RECORDED ON REEL 031998 FRAME 0667. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NEC LABORATORIES AMERICA, INC.;REEL/FRAME:042754/0703

Effective date: 20140113