US20010037324A1 - Multilevel taxonomy based on features derived from training documents classification using fisher values as discrimination values - Google Patents
Multilevel taxonomy based on features derived from training documents classification using fisher values as discrimination values Download PDFInfo
- Publication number
- US20010037324A1 US20010037324A1 US09/777,278 US77727801A US2001037324A1 US 20010037324 A1 US20010037324 A1 US 20010037324A1 US 77727801 A US77727801 A US 77727801A US 2001037324 A1 US2001037324 A1 US 2001037324A1
- Authority
- US
- United States
- Prior art keywords
- node
- nodes
- document
- documents
- terms
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
- G06F16/355—Class or cluster creation or modification
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99931—Database or file accessing
- Y10S707/99932—Access augmentation or optimizing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99931—Database or file accessing
- Y10S707/99933—Query processing, i.e. searching
- Y10S707/99936—Pattern matching access
Definitions
- the present invention relates, generally, to a process, system and article of manufacture for organizing and indexing information items such as documents by topic, and in preferred embodiments, to such a process, system and article which employ a topic hierarchy and involve a determination of discriminating terms and stop terms at each internal node in the topic hierarchy.
- the format with which the accessible information is arranged also affects the level of difficulty in locating specific items of information within the totality. For example, searching through vast amounts of information arranged in a free-form format can be substantially more difficult and time consuming than searching through information arranged in a pre-defined order, such as by topic, date, category, or the like.
- searching through vast amounts of information arranged in a free-form format can be substantially more difficult and time consuming than searching through information arranged in a pre-defined order, such as by topic, date, category, or the like.
- a pre-defined order such as by topic, date, category, or the like.
- the accessible information is placed on-line in the form of free-format text.
- the amount of on-line data in the form of free-format text continues to grow very rapidly.
- Search schemes employed to locate specific items of information among the on-line information content typically depend upon the presence or absence of key words (words included in the user-entered query) in the searchable text. Such search schemes identify those textual information items that include (or omit) the key words.
- key word searching can be problematic, for example, resulting in the identification of numerous text items that contain (or omit) the selected key words, but which are not relevant to the actual subject matter to which the user intended to direct the search.
- Taxonomies can provide a means for designing vastly enhanced searching, browsing and filtering systems. Querying with respect to a topic can be more reliable than depending only on the presence or absence of specific words in documents.
- multicast systems such as PointCast (http://www.pointcast.com) are likely to achieve higher quality by registering a user profile in terms of classes in a taxonomy rather than key words.
- Classifiers can be parametric or non-parametric.
- Two well-known classes of non-parametric classifiers are decision trees, such as CART (as in Breiman et al, Classification and Regression Trees, 1984, which is incorporated herein by reference) and C4.5 (as in Quinlan, C4.5: Programs for Machine Learning, 1993, which is incorporated herein by reference), and neural networks (as in Hush and Horne, Progress in Supervised Neural Networks, 1993, Lippmann, Pattern Classification using Neural Networks, 1989, and Jain et al, Artificial Neural Networks, 1996, each of which is incorporated herein by reference.
- feature sets larger than 100 are considered extremely large. Document classification may require more than 50,000.
- More recent work includes statistical modeling of documents, unsupervised clustering (where documents are not labeled with topics and the goal is to discover coherent clusters, as described in Anick and Vaithyanathan, Exploiting Clustering and Phrases for Content - based Information Retrieval, 1997, which is incorporated herein by reference), supervised classification (as in Apte et al, Automated Learning of Decision Rules for Text Categorization, 1994, and Cohen and Singer, Context Sensitive Learning Methods for Text Categorization, 1996.
- Koller and Sahami Hierarchically Classifying Documents Using Very Few Words, International Conference on Machine Learning, July 1997 and Yang and Pedersen, A comparative study on feature selection in text categorization, International Conference on Machine Learning, July 1997 discuss classification.
- Koller et al propose a sophisticated feature selection algorithm that uses a Bayesian net to learn interterm dependencies.
- the complexity in the number of features is supralinear (e.g., quadratic in the number of starting terms and exponential in the degree of dependence between terms). Consequently, the reported experiments have been restricted to a few thousand features and documents.
- TAPER standing for Taxonomy And Path Enhanced Retrieval, as described herein.
- TAPER separates feature and noise terms by computing the best discriminant terms for that node. This is accomplished, by computing a measure of discrimination capability or power for each term in each document of a training set.
- a cut-off point is computed, for which terms having discrimination powers above the cut-off are considered feature (or discriminant) terms and terms having discrimination powers below the cut-off are considered stop terms or noise terms.
- a discrimination measure derived from mutual information (for example, as described by T. M.
- Statistical models are constructed for each topic in the taxonomy, using the feature terms determined locally for that topic. In preferred embodiments, a Bernoulli or binary model of text generation may be assumed.
- a taxonomy has been used for illustration purposes for the discussion above. However, the taxonomy may take on various forms, and, in some embodiments of the present invention, is not necessary.
- FIG. 1 shows a diagram representing a hardware environment for one preferred embodiment of the present invention.
- FIG. 2 shows a tree-shaped topic hierarchy which may be employed with one embodiment of the present invention.
- FIG. 3 shows a block diagram representing a classifier training and feature selection system according to a preferred embodiment of the present invention.
- FIG. 4 is a flow diagram illustrating the steps performed by the present invention for reorganizing, training, and testing.
- FIG. 5 shows a graphical representation of an example of a function of error rate versus number of terms for a single document.
- FIG. 6 shows a graphical representation of an example of a function of accuracy versus prefix size.
- FIG. 7 shows a block diagram of the main statistics table maintained during training in one preferred embodiment.
- FIG. 8 shows an illustration of the data structures involved in the computation of discrimination power of each term during feature selection, in one preferred embodiment.
- FIG. 9 shows an illustration of the organization of the table of terms and their discrimination powers in one preferred embodiment.
- FIG. 10 shows a diagram showing how indexed statistics are computed after feature selection in one preferred embodiment.
- a search query would not elicit a list of documents, but instead would elicit a list of topic paths.
- a topic path is a path in a topic taxonomy.
- the search query “jaguar speed” may elicit a list of topics, as follows:
- a concept is more general than a keyword.
- the concept “animal” together with the keyword “jaguar” is a better query than “jaguar” alone.
- a keyword is syntactically embedded or implied by a document, whereas a concept is a semantic attribute.
- a concept could even be functional (e.g., irrespective of syntactic content, a site can be categorized as “commercial” or “academic”).
- the same keyword may induce different concepts (e.g., “jaguar” in the context of “animal” vs. “car”).
- the user may restrict the search to only one topic path or a few selected topic paths, by selecting one or more topics in the list. Depending upon the depth of the taxonomy, the selection of one or more topic paths may result in a list of further topic paths, representing further levels of the taxonomy.
- designing the query to enforce or forbid additional keywords may not always be as effective as restricting the search to particular topic paths.
- the ability to restrict searches to particular topic paths may be very useful for multicast channels as well.
- user profiles will be topic paths rather than keywords.
- Another paradigm of information retrieval is filtering, in which a continual stream of documents are generated on-line, as in newsgroups and newsfeed.
- the system collects interest profiles from users and uses them to implement either content-based or collaborative filtering. i.e., it notifies the user only of those documents they are likely to be interested in.
- a profile may be a set of terms and phrases specified explicitly by the user. This has the same problem as querying without topic context as discussed above.
- a better notion of a profile is the set of documents the user has seen and/or liked, perhaps with scores. This may work well with small systems, but for thousands of users and millions of documents, a system storing this level of detail will not scale.
- a promising alternative is to characterize profiles, not at the level of individual documents, but at the level of narrow but canonical topics. This can be realized as one embodiment of the present invention.
- An exhaustive keyword index employed by such systems as Alta VistaTM is perhaps more of a problem than a solution.
- the IR literature has advanced further, and now prototypes exist that extract signature terms, which are then used for indexing. These signatures can also be used as summaries or thumbnails.
- the descriptive power of signatures can often compare favorably with that of arbitrary sentences as extracted by popular search engines.
- the signatures are also effective for describing a document cluster. Many approaches have been used for signature extraction, and in one common approach, the most frequent terms that are not stopwords are selected.
- a document abstract or signature as a function of the document alone is of limited utility.
- a useful signature is a function of both the document and the reference node.
- the signature includes terms that are “surprising” given the path from the root to the reference node.
- car and auto may be good signature terms at the top level, or even at the Recreation level, but not when the user has drilled down to Adventure:Automotive.
- the first level classification is Health.
- the top signature terms are computed with respect to Health as follows:
- Finding term associations is another application of context-sensitive signatures.
- the use of phrases for search and classification can potentially boost accuracy.
- the usual way to find phrases is to test a set of terms for occurrence rate far above that predicted by assuming independence between terms.
- associations that are strong for a section of the corpus may not be strong globally and may go unnoticed.
- the term “precision” may be visibly associated with the term “recall” in a set of documents on IR. but not in a collection also including documents on machine tools.
- Computing signatures at each node can expose all such associations.
- stopwords It is tricky to hand-craft the stopwords out of domain knowledge of the language. For example, the term “can” is frequently included in stopword lists. However, that term should not be a stop term for a corpus on waste management. The contents of a stopword list should be highly dependent on the corpus. This issue looms large in searching using categories and clusters. In hierarchical categories, the importance of a search term depends on the position in the hierarchy.
- Feature selection is also useful in any setting in which salient distinctions are sought between two or more sets of documents.
- a set of documents e.g. a keyword query result
- the clusters can be regarded as classes, and feature selection can be used to find these keywords.
- the present invention relates, generally, to a process, system and article of manufacture for organizing, classifying, and indexing information items by topic, such as text and hypertext documents.
- information items comprise documents accessible on the internet.
- further embodiments of the invention are applicable to information items accessible in local network environments, dedicated database environments, or the like.
- FIG. 1 An example hardware environment for an internet embodiment is shown in FIG. 1, which includes a user computer 10 , a user display 11 , and a user interface 12 .
- the display 11 is preferably a visual display device, such as a cathode ray tube monitor, a liquid crystal monitor or other suitable display device.
- the user interface 12 preferably comprises one or more of a key board, mouse, touch-screen device or other suitable input device.
- the computer 10 operates in accordance with a software program stored on a computer readable medium. such as a floppy disk 13 , hard disk (not shown) or other suitable storage medium.
- the computer 10 is linked, through an internet connection, and operates in accordance with a suitable software program to access information items stored in at least one information database.
- the information items comprise text documents stored or accessible through one or more server locations.
- server locations may store or provide access to additional documents.
- preferred embodiments of the present invention include a system comprising a computer, display and user interface, which operate in accordance with a process stored as a program on a computer readable medium, to organize and classify information items. While such information items may be text documents accessible through an internet connection, as shown in FIG. 2, it will be understood that, in further embodiments, the system may operate with information items other than those available on the internet, including, but not limited to information items accessible through local networks, dedicated databases, or the like. However, for purposes of simplifying the present disclosure, embodiments of the present invention are described herein primarily with reference to a search and classification system which operates with information items in the form of text documents that are accessible through an internet connection with the internet.
- embodiments of the invention relate to an automatic process for learning topics from examples and later identifying topics of new documents (also called “test documents”).
- the process employs a multilevel taxonomy having a plurality of nodes, including a root node, at least one intermediate node associated with and under the root node and a plurality of terminal nodes associated with and under each intermediate node.
- a different set of feature terms are associated with each intermediate node, which are used to classify test documents.
- the feature term sets are determined, according to preferred embodiments, during a training procedure.
- the training procedure employs a plurality of training documents that have been pre-assigned manually to various terminal and intermediate nodes in the taxonomy.
- the training documents are tokenized, and information related to the frequency of terms or tokens is recorded in a database.
- a discrimination value is determined for each term in the training documents, and a minimum discrimination value is determined.
- a set of feature terms is selected, where the feature terms are those that are in the training documents associated with the intermediate node or any of its descendants and that have discrimination values equal to or above the minimum discrimination value for the intermediate node.
- test documents are analyzed.
- a text document is first tokenized. Of all the tokens in the document, only those that are also in the feature set of the root topic in the taxonomy are considered useful.
- the statistics related to these useful terms are retrieved from the database, and the statistics are used to compute a score for each of the children of the root node (nodes comprising the next level connected to the root node). A few children with high scores are then picked for further exploration. If any child is an intermediate node, it has associated with it another feature set. The set of all tokens in the test document is now intersected with this new feature set, and the procedure continues from the child in the same manner.
- the system also computes, for each topic node with suitably high score, the terms in the test document that are significantly more frequent than in the training set for that topic. These are then used for building a term index.
- the above system can be used to process a search query.
- the search query is received from the user, for example, through a user input device in the form of keywords.
- the user also restricts the topical context using a suitable selection on the taxonomy.
- a plurality of relevant documents which also adhere to the topic restrictions is retrieved.
- each document in the database has been pre-classified using the above system.
- the user is presented with a suitable display of those portions of the taxonomy where relevant documents were found.
- the user may then enter a command through the user input device to cause the system to select at least one of the displayed sub-topics. This process is repeated as necessary to refine the query topic until the user's information need is satisfied.
- Organization and classification of information items involves a topic hierarchy, or “taxonomy,” preferably having a plurality of levels of nodes. While embodiments of the invention may employ any directed acyclic graph hierarchy structure, embodiments are described herein with reference to a tree-like topic hierarchy.
- FIG. 2 An example of a tree-like topic hierarchy, or taxonomy, for organizing a database of topical documents is shown in FIG. 2.
- the tree 20 includes a first level comprising a single node 22 titled “All Topics.”
- a second level of the tree may divide the first level “All Topics” node into several further nodes directed to general topic categories, such as Business and Economy 24 , Recreation 26 , Science 28 , and so forth.
- Each of the second level nodes may be divided, at the third level, into several further nodes directed to more specific topics within each second level topic.
- the Business and economy topic 24 may be divided into Companies, Stock Markets, and so forth.
- each of the other second level topics may be divided at the third level to further topics.
- taxonomy may be included in the topic hierarchy, or taxonomy.
- the final level of each path in the taxonomy terminates at a terminal or leaf node, labeled c in the diagram.
- the taxonomy in the diagram is provided as an example for purposes of simplifying the present disclosure and is not intended to limit the invention to the specific illustration of the taxonomy.
- an appropriate topic hierarchy, or taxonomy is provided by the user, based on the material (for example text documents) that are intended to be classified and searched.
- the material for example text documents
- a taxonomy might appear as shown in FIG. 2.
- the material to be classified and searched includes all U.S. patents, a taxonomy which more closely follows the U.S. Patent and Trademark Office classification system might be employed.
- system training is performed by providing an initial collection of documents for which classifications are known in advance. With reference to the block diagram of a training system 30 of FIG. 3, this may be accomplished, for example, by collecting a number of documents 32 .
- the document collection may be performed with a suitable web crawler.
- a sample document collection may be provided with the system software 13 (FIG. 1) or manually collected from any suitable source.
- the sample document collection is divided into two sets.
- One set of documents is set aside as a testing set 34 .
- the other set is manually classified or otherwise pre-designated as corresponding to a particular class or terminal node (or, in some cases, intermediate node) within the given topic hierarchy and becomes the training set 38 .
- the training set of documents 38 is split, preferably randomly, into a statistics collection set of documents 42 and a model validation set of documents 44 .
- statistics are collected from the statistics collection set 42 , based on terms appearing in those documents and the known classes for those documents. These statistics are used in the determination of the discriminating power of terms in the documents from the collection set 42 .
- the statistics are calculated for each node in the taxonomy, such that, for any one node, the discriminating power is calculated for the terms in all of the documents that are classified in terminal (and intermediate) nodes below that node. That is, the power that each term has to discriminate between classes in the next level below each node is calculated.
- the statistics calculated for each intermediate node in the hierarchy includes quantities that enable computing the “discriminating power” of each term found in some training document under the node. Based on these statistics, terms are ordered by decreasing discriminating power and the top discriminating terms (those terms with the highest discriminating power) are selected as feature terms for use in classification, while the remaining terms are characterized as stop terms that have little value in distinguishing between topics in the immediate context. The determination of which terms in the order are feature terms and which terms are stop terms is provided by selecting a cut-off point within the ordering.
- the “statistics collection” subset of the training documents are used to collect term frequency information. Then, in block 48 , feature terms and stop terms are determined for each internal topic node based on the model validation set 44 . Finally, class models are constructed over the chosen features in block 49 , preferably as described below in the section titled “Document Models.”
- the class models and statistical information calculated in block 46 are provided to the classifier 50 , for classifying the test documents 34 in a testing mode, as well as new documents when the system is deployed. Classification of test (or new) documents is carried out in the taxonomy, such that each test (or new) document is ultimately classified to correspond to one or more classes, designated by terminal or leaf nodes (or, in some cases, intermediate nodes in the hierarchy).
- FIG. 4 is a flow diagram illustrating the steps performed by the present invention for reorganizing, training, and testing. After starting at block 60 , in block 62 , an action type is determined.
- Reorganization refers to a change in the structure of the topic tree. For example, one type of change involves collapsing two topics into one (e.g., a “science” topic may be collapsed with a “mathematics” topic). Another type of change involves expanding a topic into multiple topics (e.g., a “mathematics” topic may be expanded into a “calculus” topic, a “linear algebra” topic, and other topics). Once statistics are stored in the raw database in block 70 , processing continues at block 72 .
- compaction refers to merging an unsorted table into the main sorted table. If compaction is needed, then tables are compacted in block 74 . If compaction is not needed or after compaction, the processing continues to block 76 . In block 76 , features are selected. Feature selection will be discussed in more detail below under the “Feature Selection” heading. In block 78 , indexed statistics are written to tables for testing. The statistics are stored in the indexed database in block 90 .
- the action type is “new training document”, then the document is tokenized in block 66 .
- statistics are appended.
- the appended statistics are stored in the raw statistics database. Also, the appended statistics are stored in the indexed database in block 90 . Then, processing continues at block 72 , as discussed above.
- documents are tokenized in block 92 .
- the root topic is selected as a starting point.
- a top topic is picked form the pool (i.e., a topic with a high goodness score).
- the children of the picked topic are evaluated and the best ones (i.e., those with high goodness scores) are added to the pool.
- a Bernoulli or binary model of document generation is assumed.
- a document d is generated by first picking a class c.
- Each class has an associated multi-faced coin, with each face representing a term t and having some success probability ⁇ (c,t).
- ⁇ (c,t) is arbitrarily fixed, and each term is generated by flipping the coin.
- a document is a set of terms with counts zero or one, and ⁇ (c,t) is an estimate of the fraction of documents in class c that contain term t at least once.
- the focus is on whether a term occurs, and so a term is either associated with zero (i.e. occurs) or one (i.e., does not occur).
- the focus is on how many times a term occurs, and so the model keeps track of “buckets” for the number of times a term occurs (e.g., once, twice, three times, . . . , n times).
- Buckets for the number of times a term occurs (e.g., once, twice, three times, . . . , n times).
- it may be relevant that a term occurred once, twice, three times or four or more times while it is not relevant that the term occurred specifically four times, five times, . . . , n times.
- the “buckets” could be specified to hold terms that occur “once”, “two to three times”, “four to seven times”, etc.
- the classifier database will be organized as a three-dimensional table.
- the first axis is for terms
- the second axis is for documents
- the third axis is for classes or topics.
- the measure maintained along these dimensions, (t,d,c), is called n(t,d,c), which is the number of times t occurs in d ⁇ c. This number is non-zero only when t ⁇ d ⁇ c.
- t E d means that term t occurs in document d
- d ⁇ c means that d is a training sample for class c.
- a super-class of c i.e., an ancestor in the topic tree, inherits all d ⁇ c.
- n ⁇ ( d ) ⁇ i ⁇ ⁇ n ⁇ ( t , d , c ) .
- [0118] is the multinomial coefficient. A corresponding expression can be easily derived for the binary model as well.
- the Bernoulli model makes the assumption that term occurrences are uncorrelated, which is not accurate. First, given that a term has occurred once in a document it is more likely to occur again when compared to a term about which information is not available. Second, the term frequency distributions are dependent.
- naive Bayes classifier In essence, builds density functions, which are marginally independent, for each class, and then classifies a data point based on which density function has the maximum value at that point.
- density function has the maximum value at that point.
- these simple classifiers perform surprisingly well compared to more sophisticated ones that attempt to approximate the dependence between attributes.
- L(c) is the size of the lexicon (the number of distinct terms found in the training documents) of class c.
- a classifier inputs a document and outputs a class. If the class is not the one from which the document was generated, the classifier is said to have misclassified that document. In the case of a topic hierarchy, one may wish to give the classifier “partial credit” for correctly finding the first few levels of the “true” topic. This is ignored in the current discussion and will be commented on later. For now the discussion focuses on how to find the best leaf topics.
- a distinct classifier is associated with each internal node in the taxonomy, including the root.
- a set of feature terms is generated for each of such nodes.
- the goal of the classification process is to find a leaf node c such that the probability that the document d was generated from class c (called the posterior probability Pr[c
- Hierarchical classification has the benefit of greatly increased speed of classification. As described next, classification of a test document starts at the taxonomy root by assigning a score to each child of the root. In many cases, it will be possible to eliminate most of the topic sub-trees as unlikely candidates. Thus, large sub-trees in the topic tree can be eliminated forthwith if the score of the root of those sub-trees are very poor. Text database population is not the only application of fast multi-level classification. With increasing connectivity, it will be inevitable that some searches will go out to remote text servers and retrieve results that must then be classified in real time.
- d] log Pr[c i ⁇ 1
- the edge (c i ⁇ 1 ,c i ) is marked with the edge cost ⁇ log Pr[c i
- a property of preferred embodiments of the present invention that distinguishes it from prior art is the use of different feature sets computed separately for each internal node. This prevents the classifier from losing accuracy even though it inspects very few of the classes in the taxonomy to pick the best leaf topics.
- the challenge is to select suitable feature terms from a lexicon that can be as large as a hundred thousand terms.
- the selection process is constrained both ways: the highly discriminating terms should not be missed, and every term should not be included, because the frequencies of some terms are noisy and not indicative of content. This is called the feature-selection problem in the statistical pattern recognition literature.
- the following approach is carried out: first a merit measure is assigned to each term, and then a prefix of terms with highest merit are selected.
- the merit measure comprises an index based on mutual information or on Fisher's linear discriminant.
- Mutual information is a well-known statistical measure of dependence between random variables (see Cover and Thomas). It is straight-forward to apply mutual information to the binary document model, but it is more complicated to apply it to the Bernoulli model, and more expensive to evaluate.
- the Fisher discriminant measure that was used in the present invention is described, and this measure was found to be more effective than mutual information.
- ⁇ y 1 ⁇ y ⁇ ⁇ ⁇ y ⁇ Y ⁇ ⁇ y .
- ⁇ y ⁇ ⁇ 1 y ⁇ ⁇ y ⁇ Y ⁇ ⁇ ( y - ⁇ y ) ⁇ ( y - ⁇ y ) T .
- Fisher's discriminant approach seeks to find a vector ⁇ such that the ratio of the projected difference in means,
- Computing ⁇ involves a generalized eigenvalue problem involving the covariance matrices.
- the matrix size k is typically a few hundred at most.
- the matrix size k is typically 50,000 to 100,000, and the covariance matrices may not be suitably sparse for efficient computation.
- a preferred approach, therefore, will be to take the directions ⁇ as given, namely, a coordinate axes for each term.
- the remaining exercise having sorted terms in decreasing order of Fisher index, is to pick a suitable number of feature terms starting with those having the highest index.
- F be the list of terms in our lexicon sorted by decreasing Fisher index.
- a preferred heuristic is to pick from F a prefix F k of the k most discriminating terms.
- F k must include most useful features and exclude most noise terms.
- a short F k enables holding a larger taxonomy in memory and hence fast classification. Too large an F k will degrade not only performance, but also accuracy because of the phenomenon of over-fitting: the classifier will fit the training data very well, but will result in degraded accuracy for test data.
- the current invention prefers the technique of minimization of the classification error rate on a separate validation set.
- the pre-classified samples are partitioned, preferably randomly, into the training set T (shown as block 42 in FIG. 3) and the validation set V (shown as block 44 in FIG. 3).
- the Fisher index of each term based on documents in set T is computed, and then documents in set V are classified using various prefixes F k .
- N k be the number of documents incorrectly classified when a prefix of k features is used, then (the smallest) k for which N k is minimized is sought.
- N k ⁇ d ⁇ ⁇ N k ⁇ ( d ) ⁇
- N k ⁇ ( d ) ⁇ 1 , ⁇ c ⁇ c * ⁇ ( d ) : Pr [ c ⁇ ⁇ d , F k ] > Pr [ c * ⁇ ( d ) ⁇ ⁇ d , F k ] 0 , otherwise .
- the overall plot is constructed of the faction of documents incorrectly classified against the number of features used by averaging the per-document function written above.
- the class for that document is first estimated based only on the highest Fisher value term in the order. If the class estimate is erroneous, the error is 1 for that document. If the class estimate is correct, then an error rate of 0 (zero) is plotted for the term number 1, for that document.
- the aggregate or average error will typically decrease steeply as terms are initially added to the feature set, reach a minimum, and then show an upward trend.
- k* is the smallest number of feature terms for which N k achieves (close to) its global minimum. Then these k* terms are picked as features for the intermediate node of the taxonomy under discussion. In this manner, a generally distinct set of feature terms are derived for classification at each intermediate node in the taxonomy.
- other means for determining a cut-off point in the order of discriminating powers may be employed, including but not limited to defining a preset number of terms as the cut-off point.
- one approach is to concatenate the training documents vassociated with c into a super document d c and then rank the terms t ⁇ d c in decreasing order of the number of standard deviations that n(d,t) is away from ⁇ (c,t).
- the previously described simplistic document model may not suffice.
- a term that has occurred once in a document is more likely to occur again. Since the Bernoulli model does not take this into account, frequent terms often remain surprising all along the taxonomy path.
- Matters are improved by moving to another simple model, the binary model.
- Preferred embodiments of the current invention have the following distinctive features as an “industrial strength” topic analyzer:
- Training has near real-time response, as needed by crawling and indexing applications. Training makes a single pass over the corpus.
- the prototype permits efficient incremental updates to a fixed taxonomy with new documents, deletion of documents, or moving documents from one class to another. With some more work, it is also possible to reorganize entire topic sub-trees.
- System modules include:
- TID's numeric term ID's
- a module for applying the classifier on a test/new document is a module for applying the classifier on a test/new document.
- the goal of this module is to collect term statistics from a training document and dispense with it as fast as possible.
- the document is converted to a sequence of 32-bit TID's (term ID's).
- TID corresponds to a term that occurs in some document belonging to a class corresponding to KCID (child or kid class ID).
- the Parent Class ID (PCID) represents the parent of KCID (zero if KCID is the root).
- the class identifications CID's (KCID's and PCID's) are numbered from one onwards. The explicit presence of PCID is only to simplify the current disclosure. In an actual system, the class tree data structure is preferably always available to map from KCID's to PCID's, and, therefore, PCID does not need to be explicitly stored in the table.
- the main table is kept sorted on the keys TID and KCID. There is another unsorted table with the same row format. There are four other numeric fields per row. All of these four numbers are additive over documents, so for each document d and term t, a row is appended to the unsorted table, with SMC set to one, SNC set to the number of times t occurred in d, called n(d,t), SF1 set to n(d, t)/ ⁇ r n(d,t), and SF2 set to SF1 2 . SMC is used in the binary model, while SNC is needed in the Bernoulli model.
- An indexed access approach could be used instead of the frequency table.
- the system would look up on the (TID, KCID) key and update SMC.
- the frequency table is a temporary file and no direct indexed access to it is actually required later.
- Another benefit is compactness: this is the most space-intensive phase of training, and the storage overheads of indexed access are avoided, while explicit control of compaction is obtained.
- the space overhead of storing TID redundantly is moderate, as the rest of each row is already 18 ⁇ bytes long.
- the frequency table is aggregated one last time, if needed to eliminate all duplicates.
- the frequency table is rewound and prepared for scanning.
- all rows with the same TID are collected in a contiguous run going through all CID's where that TID occurred (see FIG. 7).
- preparation takes place to output another file, called the fisher table.
- FIG. 8 a format shown in FIG. 8 is assumed.
- the format includes rows that are keyed by PCID and a floating point number FI (Fisher index), where for each fixed PCID the rows are sorted in decreasing order of FI.
- the last column is the TID (term ID) whose corresponding PCID and FI are the first and second columns.
- TID is the primary key in the frequency table, as it is scanned, a sequence of runs are obtained, each run having a fixed TID. Associated with each topic node in memory, a few words of statistics are kept (derived from SMC, SNC, etc.). When a run is started for a given TID, these statistics are cleared. As the various KCID's are scanned for the given TID in the frequency table, the node corresponding to the KCID in the taxonomy is located, and these statistics are updated. In a large taxonomy, very few of the nodes will be updated during a run. If a node is updated, its parent will be updated as well. These statistics efficiently can, therefore, be reset after each run.
- the Fisher index of that term is computed for every internal node (identified by its PCID) in the taxonomy as described in the section titled “Feature and Noise Terms.” For each of these PCID's. a row is appended to the Fisher table. Next, the Fisher table is sorted on the key (PCID.FI). This collects all PCID's into contiguous segments, and for each PCID, orders terms by decreasing values of FI.
- both the frequency and fisher table are sorted once again, this time with (PCID,TID) as the key. After these sorts, a merge is performed. Rows of the Fisher table are considered one by one. For each row, once the beginning of a key-matched row of the frequency table is found, the row is read as long as the key remains unchanged, constructing a vector in memory where each element has the form (KCID,SMC,SNC). This buffer is then written into a hash table on disk.
- FIG. 9 shows an illustration of the organization of the table of terms and their discrimination powers in one preferred embodiment. The table of terms is used in the feature selection process.
- N k the number of incorrectly classified documents as a function of k, the number of features used
- N k the number of incorrectly classified documents as a function of k, the number of features used
- the technique used holds N k (the number of incorrectly classified documents as a function of k, the number of features used) in memory as against scanning the validation documents for different values of k.
- N k is stored for every value of k, and the lexicon is of size 500,000 (a sample of 266,000 documents from YahooTM required 600,000), only 2MB is needed. As each document d is scanned, N k (d) is aggregated into the N k array.
- indexed statistics could be computed, given the optimal number of k* of terms at each node.
- the discussion is directed to how the k* is determined.
- the documents in the validation set are considered one by one.
- Each document d has a pre-assigned “correct” class label c*.
- the parent c of this class node is located in the topic tree. Recall that c has associated with it a ranked list of terms. These are intersected with the terms in d, and the common terms are sorted by the rank.
- feature selection is not an interactive operation. For example, on a database with 2000 classes, with an average of 150 documents per class, and an average of 100 terms per document, it may take a couple of hours. So feature selection is invoked only when there is reason to believe that the refreshed statistics will improve classification. In further preferred embodiments, such times to perform feature selection is automatically determined, for example, based on the occurrence of one or more predetermined events.
- Another issue is deletion of documents and moving of documents from one class to another (perhaps because classification was poor or erroneous for those documents). Since feature selection is always preceded by a frequency table aggregation, negative “correction” entries may be placed in it. That is, a frequency table row is produced, corresponding to each term in the deleted document, and SNIC, SNC, SF1 and SF2 are negated for the class(es) from which the document is being deleted. Here, it cannot be ensured that the document was originally included in the aggregate, but that can be done by preserving ID's for training documents.
- Optimized index lookup Initialize all p c, to 0 For each term t ⁇ d ⁇ F(c 0 ) Skip if key (c 0 , t) is not in index Otherwise, retrieve record for (c 0 , t) For each c, that appears in the record Update p c, Normalize etc.
- the experimental computers were between 133 and 266 MHz, with 128-256 MB of memory. Once a document is in memory, typical training time is 140 ⁇ s and typical testing time was 30 ⁇ s. Training and testing on Reuters takes 20 minutes overall. The YahooTM sample, with 2118 classes and 266,000 documents, takes 19 hours to train. Bernoulli was found to be superior to binary for all our experiments.
- the Patent data set had three intermediate nodes below the root and twelve leaf nodes. A few hundred terms out of the lexicon of about 30,000 terms were sufficient to minimize error; the overall accuracy to the leaves was 66% (i.e., 66% patents were correctly classified) and the accuracy at the root node was 75%. For Reuters, the accuracy was 87%, as against the earlier best known technique's accuracy of 81% by Apte et al.
- the feature selection technique is applied to find salient differences between various interesting sets of documents.
- One application is to find descriptions for clusters in unsupervised document clustering.
- the query mouse provides hundreds of responses from the US patent database.
- Clustering the responses and applying TAPER with the clusters treated as classes yields the automatically generated cluster descriptions shown in Table 2 below.
- TABLE 2 Tissue, thymus, transplanted, hematopoietic, treatment, organ, trypsin, . . . Computer, keyboard, hand, edge, top, location, keys, support, cleaning, . . . Point, select, environment, object, display, correspondence. image, . . .
- a taxonomy will be initially designed by hand, and training documents obtained. Once training is completed, the accuracy of the system can be estimated by comparing the class output of the classifier 50 with the known classifications of the testing documents 34 (FIG. 3). If the accuracy is inadequate, a further training procedure, using a different collection of documents, or a reorganization of the taxonomy, may be carried out to retrain the system. Once satisfactory accuracy is achieved, the system may be used for a number of purposes such as search, filtering, and indexing as described above.
- the current invention can be distinguished from prior work in its use of a context dependent statistics, and its emphasis on scalability and speed in dealing with corpora ranging into tens to hundreds of gigabytes, the use of efficient disk data structures, and efficient update mechanisms.
- the present invention has focused on techniques that have good statistical foundation while remaining within almost linear time and one pass over the corpus, even when doing feature selection simultaneously for many nodes in a large topic taxonomy.
Abstract
A system, process, and article of manufacture for organizing a large text database into a hierarchy of topics and for maintaining this organization as documents are added and deleted and as the topic hierarchy changes. Given sample documents belonging to various nodes in the topic hierarchy, the tokens (terms. phrases, dates, or other usable feature in the document) that are most useful at each internal decision node for the purpose of routing new documents to the children of that node are automatically detected. Using feature terms, statistical models are constructed for each topic node. The models are used in an estimation technique to assign topic paths to new unlabeled documents. The hierarchical technique, in which feature terms can be very different at different nodes, leads to an efficient context-sensitive classification technique. The hierarchical technique can handle millions of documents and tens of thousands of topics. A resulting taxonomy and path enhanced retrieval system (TAPER) is used to generate context-dependent document indexing terms. The topic paths are used, in addition to keywords, for better focused searching and browsing of the text database.
Description
- The present application claims the benefit of U.S. Provisional Application Ser. No. 60/050,611, entitled “USING TAXONOMY, DISCRIMINANTS, AND SIGNATURES FOR NAVIGATING IN TEXT DATABASES”, filed Jun. 24, 1997, by Rakesh Agrawal, et al., attorney's reference number AM9-97-060, which is incorporated herein by reference, in its entirety.
- 1. Field of the Invention
- The present invention relates, generally, to a process, system and article of manufacture for organizing and indexing information items such as documents by topic, and in preferred embodiments, to such a process, system and article which employ a topic hierarchy and involve a determination of discriminating terms and stop terms at each internal node in the topic hierarchy.
- 2. Description of Related Art
- With modem advances in computer technology, modem speeds and network and internet technologies, vast amounts of information have become readily available in homes, businesses and educational and government institutions throughout the world. Indeed, many businesses, individuals and institutions rely on computer-accessible information on a daily basis. This global popularity has further increased the demand for even greater amounts of computer-accessible information. However, as the total amount of accessible information increases, the ability to locate specific items of information within the totality becomes increasingly more difficult.
- The format with which the accessible information is arranged also affects the level of difficulty in locating specific items of information within the totality. For example, searching through vast amounts of information arranged in a free-form format can be substantially more difficult and time consuming than searching through information arranged in a pre-defined order, such as by topic, date, category, or the like. However, due to the nature of certain on-line systems, such as the internet, much of the accessible information is placed on-line in the form of free-format text. Moreover, the amount of on-line data in the form of free-format text continues to grow very rapidly.
- Search schemes employed to locate specific items of information among the on-line information content, typically depend upon the presence or absence of key words (words included in the user-entered query) in the searchable text. Such search schemes identify those textual information items that include (or omit) the key words. However, in systems, such as the web, or large intranets, where the total information content is relatively large and free-form, key word searching can be problematic, for example, resulting in the identification of numerous text items that contain (or omit) the selected key words, but which are not relevant to the actual subject matter to which the user intended to direct the search.
- As text repositories grow in number and size and global connectivity improves, there is a pressing need to support efficient and effective information retrieval (IR), searching and filtering. A manifestation of this need is the recent proliferation of over one hundred commercial text search engines that crawl and index the web, and several subscription-based information multicast mechanisms. Nevertheless, there is little structure on the overwhelming information content of the internet.
- Common practices for managing such information complexity on the internet or in database structures typically involve tree-structured hierarchical indices. Many internet directories. such as Yahoo!™ (http://www.yahoo.com) and Infoseek (http://www.infoseek.com) are largely manually organized in preset hierarchies. International Business Machine Corporation has implemented a patent database (http://www.ibm.com/patents) which is organized by the US Patent Office's class codes, which form a preset hierarchy. Digital libraries that mimic hardcopy libraries support some form of subject indexing such as the Library of Congress Catalogue, which is also hierarchical. Such topic hierarchies are referred to herein as “taxonomies.” Taxonomies can provide a means for designing vastly enhanced searching, browsing and filtering systems. Querying with respect to a topic can be more reliable than depending only on the presence or absence of specific words in documents. By the same token, multicast systems such as PointCast (http://www.pointcast.com) are likely to achieve higher quality by registering a user profile in terms of classes in a taxonomy rather than key words.
- The danger in querying or filtering by keywords alone is that there may be many aspects to, and often different interpretations of the key words, and many of these aspects and interpretations are irrelevant to the subject matter that the searcher intended to find.
- Consider, for example, a situation in which a wildlife researcher is attempting to find information about the running speed of the jaguar, using the conventional Alta Vista™ internet search engine (http://www.altavista.digital.com), with the query “jaguar speed”. In a test search conducted with the above-noted search engine and query, a variety of responses were generated, spanning the car, the Atari™ video game, the football team, and a LAN server, in no particular order. The first page about the animal was ranked 183, and was directed to a fable.
- To eliminate the responses on cars, the test query was then changed to “jaguar speed−car−auto”. The top response in the generated results read as follows:
- “If you own a classic Jaguar, you are no doubt aware how difficult it can be to find certain replacement parts. This is particularly true of gearbox parts.”
- The words car and auto do not occur on this page. There was no cat in the first 50 pages of the generated response. Some search engines such as Alta Vista™ propose additional keywords to refine the query, but, at the time of writing, all of the keyword were related to cars or football.
- Even the query “jaguar speed+cat” gave unsatisfactory results. The responses included the word “cat”, but were often about automobiles. The 25th page was the first with information about jaguars, but did not contain the desired information.
- In contrast, if a topic taxonomy such as Yahoo™ is used, there is no problem in insisting that the user seeks documents containing “jaguar” in the topical context of animals, not cars. Unfortunately, it is labor-intensive to maintain Yahoo™ manually as the web changes and grows faster than ever. In our test case, even though the search was easily restricted to within animals, no answer could be found within the relatively small collection returned.
- Search engines are still an immature technology. Other areas have been researched intensively long before web search engines were devised, and the following discussion surveys the following overlapping areas of related research: Information Retrieval (IR) systems and text databases, data mining, statistical pattern recognition, and machine learning.
- For data mining, machine learning, and pattern recognition, the supervised classification problem has been addressed in statistical decision theory (both classical, as in Wald,Statistical Decision Functions, 1950, and Bayesian, as in Berger, Statistical Decision Theory and Bayesian Analysis, 1985, each of which is incorporated herein by reference), in statistical pattern recognition (as in Duda and Hart, Pattern Classification and Scene Analysis, 1973 and Fukunaga, An Introduction to Statistical Pattern Recognition, 1990, each of which is incorporated herein by reference), in machine learning (as in Weiss and Kulikowski, Computer Systems that Learn, 1990, Natarajan, Machine Learning: A Theoretical Approach, 1991, and Langley, Elements of Machine Learning, 1996, each of which is incorporated herein by reference).
- Classifiers can be parametric or non-parametric. Two well-known classes of non-parametric classifiers are decision trees, such as CART (as in Breiman et al,Classification and Regression Trees, 1984, which is incorporated herein by reference) and C4.5 (as in Quinlan, C4.5: Programs for Machine Learning, 1993, which is incorporated herein by reference), and neural networks (as in Hush and Horne, Progress in Supervised Neural Networks, 1993, Lippmann, Pattern Classification using Neural Networks, 1989, and Jain et al, Artificial Neural Networks, 1996, each of which is incorporated herein by reference. For such classifiers, feature sets larger than 100 are considered extremely large. Document classification may require more than 50,000.
- The most mature ideas in IR systems and text databases, which are also successfully integrated into commercial text search systems such as Verity, ConText, and Alta Vista, involve processing at a relatively syntactic level (e.g., stopword filtering, tokenizing, stemming, building inverted indices, computing heuristic term weights, and computing similarity measures between documents and queries in the vector-space model. as described by Rijsbergen,Information Retrieval, 1979, Salton and McGill, Introduction to Modern Information Retrieval, 1983, or Frakes and Baeza-Yates, Information Retrieval: Data Structures and Algorithms, 1992, each of which is incorporated herein by reference). More recent work includes statistical modeling of documents, unsupervised clustering (where documents are not labeled with topics and the goal is to discover coherent clusters, as described in Anick and Vaithyanathan, Exploiting Clustering and Phrases for Content-based Information Retrieval, 1997, which is incorporated herein by reference), supervised classification (as in Apte et al, Automated Learning of Decision Rules for Text Categorization, 1994, and Cohen and Singer, Context Sensitive Learning Methods for Text Categorization, 1996. each of which is incorporated herein by reference), query expansion (as in Schutze et al, A Comparison of Classifiers and Document Representations for the Routing Problem, 1995, and Voorhees, Using WordNet to Disambiguate Word Senses for text Retrieval, 1993. each of which is incorporated herein by reference).
- Singular value decomposition on the term-document matrix has been found to cluster semantically related documents together even if they do not share keywords (as discussed in Deerwester et al,Indexing by Latent Semantic Analysis, 1990, and Papadimitriou et al, Latent Semantic Indexing: A Probabilistic Analysis, 1996, each of which is incorporated herein by reference). None of these works address the supervised topic analysis problem in a hierarchy or how to use context-dependent words for indexing, how to automatically and efficiently compute good feature sets, and how to maintain disk data structures as training documents and the topic structure changes with time.
- Koller and Sahami,Hierarchically Classifying Documents Using Very Few Words, International Conference on Machine Learning, July 1997 and Yang and Pedersen, A comparative study on feature selection in text categorization, International Conference on Machine Learning, July 1997 discuss classification. Koller et al propose a sophisticated feature selection algorithm that uses a Bayesian net to learn interterm dependencies. The complexity in the number of features is supralinear (e.g., quadratic in the number of starting terms and exponential in the degree of dependence between terms). Consequently, the reported experiments have been restricted to a few thousand features and documents. Yang and Pedersen's experiments appear to indicate that much simpler methods suffice, in particular, that the approach of Apte et al of picking a fixed fraction of most frequent terms per class performs reasonably. This fraction may be very sensitive to corpus and methodology (e.g. whether stemming and stopwording is performed). T his is indicated by the poor performance of methods observed in recent work by Mladenic, Feature Subset Selection In Text Learning, 10th European Conference on Machine Learning, 1998.
- Accordingly, it is an object of the present invention to enable scalable, efficient. reliable, and semi-automatic organization and reorganization of a database of information entities, such as text and hypertext documents, into a topic hierarchy with the express, but not exclusive, purpose of facilitating searching and filtering of documents as per the user's information need.
- It is an object of further preferred embodiments to provide a system which uses the topic taxonomy to present the user with a series of progressively refined views of document collections in response to queries, and to discover and highlight salient differences between 1:0 two or more collections of documents.
- It is an object of yet further preferred embodiments to provide such a system which is sufficiently fast, for example, for use in conjunction with a crawler or newswire service.
- It is yet another object of further preferred embodiments to provide such a system which efficiently updates its knowledge when it makes mistakes and a human intervenes, or when there is a need to modify the topic taxonomy to accommodate new contents.
- These and other objects are accomplished, according to preferred embodiments of the present invention, with a system called TAPER, standing for Taxonomy And Path Enhanced Retrieval, as described herein. For every internal node in the taxonomy, TAPER separates feature and noise terms by computing the best discriminant terms for that node. This is accomplished, by computing a measure of discrimination capability or power for each term in each document of a training set. In addition, a cut-off point is computed, for which terms having discrimination powers above the cut-off are considered feature (or discriminant) terms and terms having discrimination powers below the cut-off are considered stop terms or noise terms. In a preferred embodiment, a discrimination measure derived from mutual information (for example, as described by T. M. Cover and J. A. Thomas,Elements of Information Theory, John Wiley and Sons, Inc., 1991. incorporated herein by reference) or Fisher's discriminant (for example, as described by R. Duda and P. Hart, Pattern Classification and Scene Analysis, Wiley. New York, 1973, incorporated herein by reference) may be used.
- Statistical models are constructed for each topic in the taxonomy, using the feature terms determined locally for that topic. In preferred embodiments, a Bernoulli or binary model of text generation may be assumed.
- When classifying new documents, only the feature terms therein are used. Such feature terms are relatively few in number, so the class models are small and classification may be accomplished relatively quickly, in contrast to existing classifiers that employ a flat set of classes, the feature set changes by context as the classification process proceeds down the taxonomy. As a result, jargon common to lower nodes (or levels) of the taxonomy are filtered out, as stop words, at each node (or level), and the classification accuracy remains high in spite of the reduction in the number of terms and candidate classes inspected.
- Addition and deletion of documents to given topics, as well as reorganization of the topic hierarchy itself, are easily handled. The text models built at each node also yield a means to summarize a number of documents using a few descriptive keywords, referred to herein as their signature.
- A taxonomy has been used for illustration purposes for the discussion above. However, the taxonomy may take on various forms, and, in some embodiments of the present invention, is not necessary.
- FIG. 1 shows a diagram representing a hardware environment for one preferred embodiment of the present invention.
- FIG. 2 shows a tree-shaped topic hierarchy which may be employed with one embodiment of the present invention.
- FIG. 3 shows a block diagram representing a classifier training and feature selection system according to a preferred embodiment of the present invention.
- FIG. 4 is a flow diagram illustrating the steps performed by the present invention for reorganizing, training, and testing.
- FIG. 5 shows a graphical representation of an example of a function of error rate versus number of terms for a single document.
- FIG. 6 shows a graphical representation of an example of a function of accuracy versus prefix size.
- FIG. 7 shows a block diagram of the main statistics table maintained during training in one preferred embodiment.
- FIG. 8 shows an illustration of the data structures involved in the computation of discrimination power of each term during feature selection, in one preferred embodiment.
- FIG. 9 shows an illustration of the organization of the table of terms and their discrimination powers in one preferred embodiment.
- FIG. 10 shows a diagram showing how indexed statistics are computed after feature selection in one preferred embodiment.
- Querying in a Taxonomy
- According to preferred embodiments of the present invention, a search query would not elicit a list of documents, but instead would elicit a list of topic paths. A topic path is a path in a topic taxonomy. For example, the search query “jaguar speed” may elicit a list of topics, as follows:
- Business_and_Economy:Companies:Automotive
- Recreation:
- Automotive
- Games:Video_Games
- Sports:Football
- Science:Biology:Animal_Behavior
- Thereafter, the user may restrict queries by concept, not by keyword. A concept is more general than a keyword. For example, the concept “animal” together with the keyword “jaguar” is a better query than “jaguar” alone. A keyword is syntactically embedded or implied by a document, whereas a concept is a semantic attribute. A concept could even be functional (e.g., irrespective of syntactic content, a site can be categorized as “commercial” or “academic”). Moreover, the same keyword may induce different concepts (e.g., “jaguar” in the context of “animal” vs. “car”).
- Upon obtaining a list of topic paths, the user may restrict the search to only one topic path or a few selected topic paths, by selecting one or more topics in the list. Depending upon the depth of the taxonomy, the selection of one or more topic paths may result in a list of further topic paths, representing further levels of the taxonomy.
- As shown above, designing the query to enforce or forbid additional keywords may not always be as effective as restricting the search to particular topic paths. The ability to restrict searches to particular topic paths may be very useful for multicast channels as well. Additionally, user profiles will be topic paths rather than keywords.
- Scalable Filtering Using a Topic Taxonomy
- Another paradigm of information retrieval is filtering, in which a continual stream of documents are generated on-line, as in newsgroups and newsfeed. The system collects interest profiles from users and uses them to implement either content-based or collaborative filtering. i.e., it notifies the user only of those documents they are likely to be interested in.
- In its simplest form, a profile may be a set of terms and phrases specified explicitly by the user. This has the same problem as querying without topic context as discussed above. A better notion of a profile is the set of documents the user has seen and/or liked, perhaps with scores. This may work well with small systems, but for thousands of users and millions of documents, a system storing this level of detail will not scale. A promising alternative is to characterize profiles, not at the level of individual documents, but at the level of narrow but canonical topics. This can be realized as one embodiment of the present invention.
- Context Dependent Signatures
- An exhaustive keyword index employed by such systems as Alta Vista™ is perhaps more of a problem than a solution. The IR literature has advanced further, and now prototypes exist that extract signature terms, which are then used for indexing. These signatures can also be used as summaries or thumbnails. The descriptive power of signatures can often compare favorably with that of arbitrary sentences as extracted by popular search engines. The signatures are also effective for describing a document cluster. Many approaches have been used for signature extraction, and in one common approach, the most frequent terms that are not stopwords are selected.
- A document abstract or signature as a function of the document alone is of limited utility. In the case of a taxonomy, a useful signature is a function of both the document and the reference node. The signature includes terms that are “surprising” given the path from the root to the reference node. In the above example, car and auto may be good signature terms at the top level, or even at the Recreation level, but not when the user has drilled down to Recreation:Automotive.
- The following is an illustration of text from a document in Health Nursing (http://www2.best.com/goodnews/practice/faq.htm):
- “Beware of the too-good-to-be-true baby that is sleeping and sleeping and doesn't want to nurse. Especially monitor the number of wet diapers, as seriously jaundiced babies are lethargic.”
- The first level classification is Health. The top signature terms are computed with respect to Health as follows:
- Jaundice, dampen, dehydration, lethargic, hydrate, forcibly, caregiver, laxative, disposable.
- This indicates that the document is about treating jaundice. The second level classification is Health:Nursing. Shifting the reference class, the new signature is computed to be:
- Baby, water, breast-feed, monitor, new-born, hormone.
- This now indicates that the document is about nursing babies. This information comes from both the path and the signatures. Techniques for computing context-sensitive signatures are described herein. Thus, significant improvement in search quality may be possible by maintaining functionally separate indices at each taxonomy node, using only a few signature terms from each document.
- Context Dependent Term Associations
- Finding term associations is another application of context-sensitive signatures. The use of phrases for search and classification can potentially boost accuracy. The usual way to find phrases is to test a set of terms for occurrence rate far above that predicted by assuming independence between terms. Unfortunately, associations that are strong for a section of the corpus may not be strong globally and may go unnoticed. For example, the term “precision” may be visibly associated with the term “recall” in a set of documents on IR. but not in a collection also including documents on machine tools. Computing signatures at each node can expose all such associations.
- Context Dependent Feature Selection
- Separating feature terms from noise terms is central to preferred embodiments described herein. In the above examples, car and auto should be “stopwords” within Recreation:Automotive and, therefore, should be pruned from the signatures. Feature and noise terms must be determined at each node in the taxonomy.
- It is tricky to hand-craft the stopwords out of domain knowledge of the language. For example, the term “can” is frequently included in stopword lists. However, that term should not be a stop term for a corpus on waste management. The contents of a stopword list should be highly dependent on the corpus. This issue looms large in searching using categories and clusters. In hierarchical categories, the importance of a search term depends on the position in the hierarchy.
- Other Applications of Feature Selection
- Feature selection is also useful in any setting in which salient distinctions are sought between two or more sets of documents. Consider the scenario in which a set of documents (e.g. a keyword query result) has been clustered into subsets, and a user wishes to annotate the clusters using salient keywords. The clusters can be regarded as classes, and feature selection can be used to find these keywords.
- Another application of feature selection is in the discovery of succinct differences between two sets of documents, such as patents filed by two companies, or by the same company over two different time intervals, to expose interesting trends over time.
- Hardware Environment
- As noted above, the present invention relates, generally, to a process, system and article of manufacture for organizing, classifying, and indexing information items by topic, such as text and hypertext documents. In one example embodiment, such information items comprise documents accessible on the internet. However, it will be understood that further embodiments of the invention are applicable to information items accessible in local network environments, dedicated database environments, or the like.
- An example hardware environment for an internet embodiment is shown in FIG. 1, which includes a
user computer 10, auser display 11, and auser interface 12. Thedisplay 11 is preferably a visual display device, such as a cathode ray tube monitor, a liquid crystal monitor or other suitable display device. Theuser interface 12 preferably comprises one or more of a key board, mouse, touch-screen device or other suitable input device. Thecomputer 10 operates in accordance with a software program stored on a computer readable medium. such as afloppy disk 13, hard disk (not shown) or other suitable storage medium. - The
computer 10 is linked, through an internet connection, and operates in accordance with a suitable software program to access information items stored in at least one information database. In the illustrated embodiment, the information items comprise text documents stored or accessible through one or more server locations. For example, with reference to FIG. 1, a set of text documents on a variety of topics are stored or accessible through theServer 1 location, and a further set of text documents on a variety of topics are stored or accessible through theServer 2 location. Further, server locations (not shown) may store or provide access to additional documents. - As described in farther detail herein, preferred embodiments of the present invention include a system comprising a computer, display and user interface, which operate in accordance with a process stored as a program on a computer readable medium, to organize and classify information items. While such information items may be text documents accessible through an internet connection, as shown in FIG. 2, it will be understood that, in further embodiments, the system may operate with information items other than those available on the internet, including, but not limited to information items accessible through local networks, dedicated databases, or the like. However, for purposes of simplifying the present disclosure, embodiments of the present invention are described herein primarily with reference to a search and classification system which operates with information items in the form of text documents that are accessible through an internet connection with the internet.
- As described in further detail below, embodiments of the invention relate to an automatic process for learning topics from examples and later identifying topics of new documents (also called “test documents”). The process employs a multilevel taxonomy having a plurality of nodes, including a root node, at least one intermediate node associated with and under the root node and a plurality of terminal nodes associated with and under each intermediate node. A different set of feature terms are associated with each intermediate node, which are used to classify test documents. The feature term sets are determined, according to preferred embodiments, during a training procedure.
- The training procedure, according to preferred embodiments, employs a plurality of training documents that have been pre-assigned manually to various terminal and intermediate nodes in the taxonomy. The training documents are tokenized, and information related to the frequency of terms or tokens is recorded in a database. A discrimination value is determined for each term in the training documents, and a minimum discrimination value is determined. Then, for each intermediate node, a set of feature terms is selected, where the feature terms are those that are in the training documents associated with the intermediate node or any of its descendants and that have discrimination values equal to or above the minimum discrimination value for the intermediate node.
- In one preferred embodiment, such as related to finding salient differences between two or more sets of documents, the process ends here, after having output the best features.
- In another embodiment, once the system is trained, test documents are analyzed. During this phase, a text document is first tokenized. Of all the tokens in the document, only those that are also in the feature set of the root topic in the taxonomy are considered useful. The statistics related to these useful terms are retrieved from the database, and the statistics are used to compute a score for each of the children of the root node (nodes comprising the next level connected to the root node). A few children with high scores are then picked for further exploration. If any child is an intermediate node, it has associated with it another feature set. The set of all tokens in the test document is now intersected with this new feature set, and the procedure continues from the child in the same manner.
- In a related embodiment, the system also computes, for each topic node with suitably high score, the terms in the test document that are significantly more frequent than in the training set for that topic. These are then used for building a term index.
- In yet another preferred embodiment, the above system can be used to process a search query. The search query is received from the user, for example, through a user input device in the form of keywords. Optionally, the user also restricts the topical context using a suitable selection on the taxonomy. Then a plurality of relevant documents which also adhere to the topic restrictions is retrieved. In a preferred embodiment, each document in the database has been pre-classified using the above system. The user is presented with a suitable display of those portions of the taxonomy where relevant documents were found. The user may then enter a command through the user input device to cause the system to select at least one of the displayed sub-topics. This process is repeated as necessary to refine the query topic until the user's information need is satisfied.
- Topic Hierarchy
- Organization and classification of information items, according to embodiments of the present invention, involves a topic hierarchy, or “taxonomy,” preferably having a plurality of levels of nodes. While embodiments of the invention may employ any directed acyclic graph hierarchy structure, embodiments are described herein with reference to a tree-like topic hierarchy.
- An example of a tree-like topic hierarchy, or taxonomy, for organizing a database of topical documents is shown in FIG. 2. The
tree 20 includes a first level comprising asingle node 22 titled “All Topics.” A second level of the tree may divide the first level “All Topics” node into several further nodes directed to general topic categories, such as Business andEconomy 24,Recreation 26,Science 28, and so forth. Each of the second level nodes may be divided, at the third level, into several further nodes directed to more specific topics within each second level topic. For example, at the third level, the Business andEconomy topic 24 may be divided into Companies, Stock Markets, and so forth. Similarly, each of the other second level topics may be divided at the third level to further topics. Also, in a similar fashion, further levels under the third level may be included in the topic hierarchy, or taxonomy. The final level of each path in the taxonomy terminates at a terminal or leaf node, labeled c in the diagram. The taxonomy in the diagram is provided as an example for purposes of simplifying the present disclosure and is not intended to limit the invention to the specific illustration of the taxonomy. - According to preferred embodiments of the present invention, an appropriate topic hierarchy, or taxonomy, is provided by the user, based on the material (for example text documents) that are intended to be classified and searched. For example, if the material to be classified and searched includes major topics on the internet, a taxonomy might appear as shown in FIG. 2. As another example, if the material to be classified and searched includes all U.S. patents, a taxonomy which more closely follows the U.S. Patent and Trademark Office classification system might be employed.
- System Training
- Before the system can analyze the topics of new documents, the system is provided with examples of documents belonging to given topic(s). Given a topic hierarchy (taxonomy), system training is performed by providing an initial collection of documents for which classifications are known in advance. With reference to the block diagram of a
training system 30 of FIG. 3, this may be accomplished, for example, by collecting a number of documents 32. For example, for classification and searching of documents available on the internet, the document collection may be performed with a suitable web crawler. Alternatively, a sample document collection may be provided with the system software 13 (FIG. 1) or manually collected from any suitable source. - The sample document collection is divided into two sets. One set of documents is set aside as a
testing set 34. The other set is manually classified or otherwise pre-designated as corresponding to a particular class or terminal node (or, in some cases, intermediate node) within the given topic hierarchy and becomes the training set 38. Atblock 40, the training set ofdocuments 38 is split, preferably randomly, into a statistics collection set ofdocuments 42 and a model validation set ofdocuments 44. - In
block 46, statistics are collected from the statistics collection set 42, based on terms appearing in those documents and the known classes for those documents. These statistics are used in the determination of the discriminating power of terms in the documents from the collection set 42. The statistics are calculated for each node in the taxonomy, such that, for any one node, the discriminating power is calculated for the terms in all of the documents that are classified in terminal (and intermediate) nodes below that node. That is, the power that each term has to discriminate between classes in the next level below each node is calculated. - Thus, with reference to the hierarchy represented in the FIG. 2, statistics are calculated for the “science”
node 28, based on the terms in all of the documents d from the collection set 42 that are classified in classes represented by nodes (terminal and intermediate) below the “science”node 28. including the nodes labeled “biology,” “chemistry,” “electronics,” and all children nodes of those nodes. Similarly, statistics are calculated for the terms in all of the documents (from the collection set 42) under each of those intermediate nodes and each of the other intermediate nodes in the hierarchy. - The statistics calculated for each intermediate node in the hierarchy includes quantities that enable computing the “discriminating power” of each term found in some training document under the node. Based on these statistics, terms are ordered by decreasing discriminating power and the top discriminating terms (those terms with the highest discriminating power) are selected as feature terms for use in classification, while the remaining terms are characterized as stop terms that have little value in distinguishing between topics in the immediate context. The determination of which terms in the order are feature terms and which terms are stop terms is provided by selecting a cut-off point within the ordering.
- At
block 42 in FIG. 3, the “statistics collection” subset of the training documents are used to collect term frequency information. Then, inblock 48, feature terms and stop terms are determined for each internal topic node based on the model validation set 44. Finally, class models are constructed over the chosen features inblock 49, preferably as described below in the section titled “Document Models.” - The class models and statistical information calculated in
block 46 are provided to theclassifier 50, for classifying thetest documents 34 in a testing mode, as well as new documents when the system is deployed. Classification of test (or new) documents is carried out in the taxonomy, such that each test (or new) document is ultimately classified to correspond to one or more classes, designated by terminal or leaf nodes (or, in some cases, intermediate nodes in the hierarchy). - FIG. 4 is a flow diagram illustrating the steps performed by the present invention for reorganizing, training, and testing. After starting at
block 60, inblock 62, an action type is determined. - If the action type is “reorganize”, then, statistics are updated in
block 64 and stored in the raw statistics database inblock 70. Reorganization refers to a change in the structure of the topic tree. For example, one type of change involves collapsing two topics into one (e.g., a “science” topic may be collapsed with a “mathematics” topic). Another type of change involves expanding a topic into multiple topics (e.g., a “mathematics” topic may be expanded into a “calculus” topic, a “linear algebra” topic, and other topics). Once statistics are stored in the raw database inblock 70, processing continues atblock 72. - In
block 72, whether compaction is needed is determined. As will be discussed under the “Statistics Collection” heading below, compaction refers to merging an unsorted table into the main sorted table. If compaction is needed, then tables are compacted inblock 74. If compaction is not needed or after compaction, the processing continues to block 76. Inblock 76, features are selected. Feature selection will be discussed in more detail below under the “Feature Selection” heading. Inblock 78, indexed statistics are written to tables for testing. The statistics are stored in the indexed database inblock 90. - If the action type is “new training document”, then the document is tokenized in
block 66. Next, inblock 68, statistics are appended. Inblock 70, the appended statistics are stored in the raw statistics database. Also, the appended statistics are stored in the indexed database inblock 90. Then, processing continues atblock 72, as discussed above. - If the action type is “test”, documents are tokenized in
block 92. Inblock 94, the root topic is selected as a starting point. Inblock 96, a top topic is picked form the pool (i.e., a topic with a high goodness score). Inblock 98, using indexed statistics from block 90 (as indicated by the arrow fromblock 90 to block 98), the children of the picked topic are evaluated and the best ones (i.e., those with high goodness scores) are added to the pool. Inblock 100, it is determined whether there are enough leaf topics. If there are not enough leaf topics, atblock 96, another topic is picked. Otherwise, the context-dependent signatures are computed inblock 102. The computation of context-dependent signatures is discussed in more detail under the “Extracting Document Signatures” heading below. These context-dependent signatures are displayed or indexed inblock 104. - Document Models
- There have been many proposals for statistical models of text generation. One of the earliest indicators of the power of simple statistical tests on term frequencies is Zipfs law. The models most frequently used in the IR community are Poisson and Poisson mixtures. If X is distributed Poisson with rate μ, denoted X˜P(x), then Pr[X=x]=e−μμx/x!, and if Y is distributed Bernoulli with n trials and mean np, denoted Y˜B(n,p), then
- converge to each other.
- According to preferred embodiments described herein, a Bernoulli or binary model of document generation is assumed. In the Bernoulli model, a document d is generated by first picking a class c. Each class has an associated multi-faced coin, with each face representing a term t and having some success probability θ(c,t). Then a document length n(d) is arbitrarily fixed, and each term is generated by flipping the coin. In the binary model, a document is a set of terms with counts zero or one, and θ(c,t) is an estimate of the fraction of documents in class c that contain term t at least once.
- Generally, in a binary model, the focus is on whether a term occurs, and so a term is either associated with zero (i.e. occurs) or one (i.e., does not occur). In a Bernoulli model, the focus is on how many times a term occurs, and so the model keeps track of “buckets” for the number of times a term occurs (e.g., once, twice, three times, . . . , n times). However, a variety of other models could be used. For example, in one model, it may be relevant that a term occurred once, twice, three times or four or more times, while it is not relevant that the term occurred specifically four times, five times, . . . , n times. In another model, the “buckets” could be specified to hold terms that occur “once”, “two to three times”, “four to seven times”, etc.
- Conceptually, as the training text is being scanned, the classifier database will be organized as a three-dimensional table. The first axis is for terms, the second axis is for documents, and the third axis is for classes or topics. The measure maintained along these dimensions, (t,d,c), is called n(t,d,c), which is the number of times t occurs in d∈c. This number is non-zero only when t∈d∈c. t E d means that term t occurs in document d, and d∈c means that d is a training sample for class c. A super-class of c, i.e., an ancestor in the topic tree, inherits all d∈c.
- Aggregation along some dimensions gives some important statistics about the corpus that is used by the classifier. The following is a list of these statistics:
-
- The length of all documents can be found using a GROUP BY on (d,c).
- The total length of the training documents in class c, denoted n(c).
- The total number of times that term t occurred over all training documents of class c, denoted n(t,c).
-
- that term t occurs in document d. The sum of f and f2 over all documents in a class is needed.
- The number m(t,c) of training documents in class c that have at least one occurrence of term t. This will be needed for the binary model.
- The number of training documents in class c, denoted |c|.
-
- is the multinomial coefficient. A corresponding expression can be easily derived for the binary model as well.
- The Bernoulli model makes the assumption that term occurrences are uncorrelated, which is not accurate. First, given that a term has occurred once in a document it is more likely to occur again when compared to a term about which information is not available. Second, the term frequency distributions are dependent.
- Our independence assumption leads to what is called a naive Bayes classifier. (A naive Bayes classifier, in essence, builds density functions, which are marginally independent, for each class, and then classifies a data point based on which density function has the maximum value at that point.) In practice, these simple classifiers perform surprisingly well compared to more sophisticated ones that attempt to approximate the dependence between attributes.
- Recently this phenomenon has been investigated in depth by Friedman in On Bias, Variance, 0/1 Loss, and the Curse-of-dimensionality, Data Mining and Knowledge Discovery, 1(1), pp. 55-77, 1997, incorporated herein by reference). A classifier that uses an estimate of class densities is subject to bias (decision boundaries that are shifted from the “best” position, because the model is inaccurate) and variance (decision boundaries fit to noisy data). Friedman analyzes how the low variance of naive density estimates can mitigate the high bias to give simple classifiers that can often beat more sophisticated ones. It will also be clear from a description of the system that this simplicity enables designing of a system that can handle enormous problem sets.
- Rare Events and Laws of Succession
- The average English speaker uses about 20,000 of the 1,000,000 or more terms in an English dictionary. In that sense, many terms that occur in documents are “rare events.” This means that with reasonably small sample sets, there will be zero occurrences of many terms in many classes, and the maximum likelihood estimate θ(c,t)=n(c,t)/n(c) wilt be problematic: a class with θ(c,t)=0 will reject any document containing t even a single time.
-
-
-
- where L(c) is the size of the lexicon (the number of distinct terms found in the training documents) of class c.
- Hierarchical Classification
- A classifier inputs a document and outputs a class. If the class is not the one from which the document was generated, the classifier is said to have misclassified that document. In the case of a topic hierarchy, one may wish to give the classifier “partial credit” for correctly finding the first few levels of the “true” topic. This is ignored in the current discussion and will be commented on later. For now the discussion focuses on how to find the best leaf topics.
- According to preferred embodiments of the present invention, a distinct classifier is associated with each internal node in the taxonomy, including the root. During classifier training, a set of feature terms is generated for each of such nodes. Given a new document d, the goal of the classification process is to find a leaf node c such that the probability that the document d was generated from class c (called the posterior probability Pr[c|d] ) is maximized among all the leaves.
- Hierarchical classification has the benefit of greatly increased speed of classification. As described next, classification of a test document starts at the taxonomy root by assigning a score to each child of the root. In many cases, it will be possible to eliminate most of the topic sub-trees as unlikely candidates. Thus, large sub-trees in the topic tree can be eliminated forthwith if the score of the root of those sub-trees are very poor. Text database population is not the only application of fast multi-level classification. With increasing connectivity, it will be inevitable that some searches will go out to remote text servers and retrieve results that must then be classified in real time.
- This benefit of increased speed may be useless if an error is made in choosing a topic early in the process at a shallow level of the tree. Thus, a greedy search for the best leaf may be risky. Let the path to a leaf c from the root c1 be c1,c2, . . . , ck=c. Since the root subsumes all classes, the probability that any document d is within the root class c1 is Pr[c1|d]−1. Thereafter, Pr[ci|d]=Pr[ci=1|d]Pr[ci|ci−1,d], for i=2, . . . , k. Taking logs, log Pr[ci|d]=log Pr[ci−1|d]+log Pr[ci|ci−1,d]. Suppose that, in the taxonomy, the edge (ci−1,ci) is marked with the edge cost−log Pr[ci|ci−1,d]. Then the least-cost path from the root to some leaf is being sought.
-
-
- using Bayes rule, and this can be evaluated from the model parameters. Care is needed here with finite-precision numbers and underflow, because the probabilities are very small and the scaling needed to condition the probability prevents maintaining the numbers always in log-form.
- Feature and Noise Terms
- The above application of Bayes rule depended on a document model. This was embedded in the θ(c,t) parameters. These parameters are estimated during the training phase. using sample documents from the statistics collection set42 (FIG. 3). When building a model for each class from a training set, a determination is made as to whether a term appears only incidentally, or sufficiently consistently to suspect a causal connection; the term is accordingly a noise term (also called a stopword) or a feature term. Only feature terms should be used for classifying new documents.
- A property of preferred embodiments of the present invention that distinguishes it from prior art (as in Apte, Damerau, and Weiss, 1994) is the use of different feature sets computed separately for each internal node. This prevents the classifier from losing accuracy even though it inspects very few of the classes in the taxonomy to pick the best leaf topics.
- The challenge is to select suitable feature terms from a lexicon that can be as large as a hundred thousand terms. The selection process is constrained both ways: the highly discriminating terms should not be missed, and every term should not be included, because the frequencies of some terms are noisy and not indicative of content. This is called the feature-selection problem in the statistical pattern recognition literature. In general, there is a need to find a set of terms that minimizes the probability that a document is misclassified, with the understanding that only terms in the intersection of the document and the feature set are used by the classifier.
- It is not possible to search for the best feature set, because it is not known what the best possible classifier does, and because there are too many terms in the lexicon. Therefore, in practice this is done for a fixed classifier. A heuristic is desired that is essentially linear in the original number of terms and preferably makes only one pass over the training corpus.
- Therefore, the following approach is carried out: first a merit measure is assigned to each term, and then a prefix of terms with highest merit are selected. In preferred embodiments, the merit measure comprises an index based on mutual information or on Fisher's linear discriminant. Mutual information is a well-known statistical measure of dependence between random variables (see Cover and Thomas). It is straight-forward to apply mutual information to the binary document model, but it is more complicated to apply it to the Bernoulli model, and more expensive to evaluate. Next, the Fisher discriminant measure that was used in the present invention is described, and this measure was found to be more effective than mutual information.
- Fisher's Discriminant
- Suppose two sets of points are given in k-dimensional Euclidean space, interpreted as two classes. Fisher's approach finds a direction on which to project all the points so as to maximize (in the resulting one-dimensional space) the relative class separation as measured by the ratio of inter-class to intra-class variance. More specifically, let X and Y be the point sets, and μX, μY be their respective centroids, i.e.,
-
-
-
- Fisher's discriminant approach seeks to find a vector α such that the ratio of the projected difference in means, |αT(μX−μY)|, to the average variance, ½αT(ΣX+ΣY)α=αTΣα, is maximized. It can be shown that α=Σ−1(μX−μY) achieves the extremum, provided Σ−1 exists. Furthermore, when X and Y are drawn from multivariate Gaussian distributions with ΣX=ΣY, this is the optimal discriminator, in that computing αTq for a test point q and comparing the result to a suitable threshold is the minimum error classifier.
- Computing α involves a generalized eigenvalue problem involving the covariance matrices. In applications such as signal processing where Fisher's discriminant is used, the matrix size k is typically a few hundred at most. In the text domain, the matrix size k is typically 50,000 to 100,000, and the covariance matrices may not be suitably sparse for efficient computation. Moreover, it is difficult to interpret a discriminant that is a linear sum of term frequencies, possibly with negative coefficients. A preferred approach, therefore, will be to take the directions α as given, namely, a coordinate axes for each term. A figure of merit is assigned to each term, which is called its Fisher index, based on the variance figures above, which is
- in the two-class case. For each term t, α=ε1, is a unit vector in the direction of t.
-
- Selecting a Cut-Off
- The remaining exercise, having sorted terms in decreasing order of Fisher index, is to pick a suitable number of feature terms starting with those having the highest index. Let F be the list of terms in our lexicon sorted by decreasing Fisher index. A preferred heuristic is to pick from F a prefix Fk of the k most discriminating terms. Fk must include most useful features and exclude most noise terms. A short Fk enables holding a larger taxonomy in memory and hence fast classification. Too large an Fk will degrade not only performance, but also accuracy because of the phenomenon of over-fitting: the classifier will fit the training data very well, but will result in degraded accuracy for test data. There are various techniques for pruning feature sets. The current invention prefers the technique of minimization of the classification error rate on a separate validation set. There exist other approaches in the prior art using the minimum description length principle, resampling or cross validation, but these make too many passes over the text corpus.
- The pre-classified samples are partitioned, preferably randomly, into the training set T (shown as
block 42 in FIG. 3) and the validation set V (shown asblock 44 in FIG. 3). The Fisher index of each term based on documents in set T is computed, and then documents in set V are classified using various prefixes Fk. Let Nk be the number of documents incorrectly classified when a prefix of k features is used, then (the smallest) k for which Nk is minimized is sought. -
-
- In effect, the overall plot is constructed of the faction of documents incorrectly classified against the number of features used by averaging the per-document function written above. For a fixed document in the model validation set, the class for that document is first estimated based only on the highest Fisher value term in the order. If the class estimate is erroneous, the error is 1 for that document. If the class estimate is correct, then an error rate of 0 (zero) is plotted for the
term number 1, for that document. - Further terms in the Fisher value order are added, one at a time, to the estimation process, to add points to the plot of Nk(d). Thus, a plot of the error rate versus term number function for any one particular document might appear as shown in FIG. 5. Typically, the document will be erroneously classified until some number m of terms are employed in the class estimation. Accordingly, an error rate of 1 is plotted, up to m terms, upon which the document is correctly classified and the error plot drops to 0. As further terms (in Fisher value order) are added to the class estimation, eventually an added term may be a noise term and may cause misclassification of the document, upon which the error jumps back to 1. Thereafter, further terms may cause fluctuations in the Nk(d) function, between 0 and 1.
- The process is repeated for each document in the model validation set44, such that a function of the error rate versus number of terms is generated for each document. The functions for all of the documents in the model validation set are added together to provide an overall error rate versus term number function that appears, for example, as shown in FIG. 6., which is discussed in more detail below under the section titled “Feature Selection”. In preferred embodiments, this aggregate error for all required number of feature terms is computed while scanning each validation document only once through the whole process.
- The aggregate or average error will typically decrease steeply as terms are initially added to the feature set, reach a minimum, and then show an upward trend. Suppose k* is the smallest number of feature terms for which Nk achieves (close to) its global minimum. Then these k* terms are picked as features for the intermediate node of the taxonomy under discussion. In this manner, a generally distinct set of feature terms are derived for classification at each intermediate node in the taxonomy. However, it will be understood that other means for determining a cut-off point in the order of discriminating powers may be employed, including but not limited to defining a preset number of terms as the cut-off point.
- Extracting Document Signatures
- Up to a point, the user can sift a query response based only on the topic paths. However, even the leaf classes are necessarily coarser than individual documents. Support is therefore needed to browse quickly through many documents without looking into the documents in detail. Most search engines attach a few lines from each document. Often these are the title and first few lines, or they are sentences with the most search terms. For many documents, better keyword extraction is needed. Moreover, it would be more advantageous for these signatures to be extracted relative to a node in the taxonomy.
- Given this reference node c, one approach is to concatenate the training documents vassociated with c into a super document dc and then rank the terms t∈dc in decreasing order of the number of standard deviations that n(d,t) is away from θ(c,t). Here, the previously described simplistic document model may not suffice. As mentioned above, a term that has occurred once in a document is more likely to occur again. Since the Bernoulli model does not take this into account, frequent terms often remain surprising all along the taxonomy path.
- Matters are improved by moving to another simple model, the binary model. First, consider a single test document d, and consider t∈d. If the observed fraction of training documents in class c containing term t at least once is θ(c,t), all t∈d are sorted in increasing order of θ(c,t) and report the top few. Second, if there are l>1 test documents, then the fraction φ(t) is found that contains t at least once, and the terms are sorted in increasing order of
- before presenting the results. Both, in fact, correspond to p-values computed using the normal approximation to the binomial distribution.
- Data Structures and Pseudocode
- Preferred embodiments of the current invention have the following distinctive features as an “industrial strength” topic analyzer:
- Thousands of classes and millions of documents can be handled. The current limits in a prototype implementation are 216 classes in the taxonomy, 232 unique tokens and 232 documents. A reasonable assumption is made that a simple pointer representation of the taxonomy together with a few words of data per class can be held in the computer memory at all times.
- Training has near real-time response, as needed by crawling and indexing applications. Training makes a single pass over the corpus.
- The prototype permits efficient incremental updates to a fixed taxonomy with new documents, deletion of documents, or moving documents from one class to another. With some more work, it is also possible to reorganize entire topic sub-trees.
- System modules, according to preferred embodiments of the invention, include:
- A mapping from textual terms to numeric term ID's (called TID's), which may be derived using a global counter, or a suitable hash function. A reverse mapping is also maintained.
- Similar mappings between classes and numeric class ID's (called CID's).
- A pointer-based tree data structure in memory for storing class-specific information, such as the number of training documents, etc.
- A module for statistics.
- A module for feature selection.
- A module for applying the classifier on a test/new document.
- The last three modules are described in detail next.
- Statistics Collection
- The goal of this module is to collect term statistics from a training document and dispense with it as fast as possible. After simple stopword filtering and stemming while scanning, the document is converted to a sequence of 32-bit TID's (term ID's). The main table, for example, maintained on a computer readable disk, is a frequency table, such as that shown in FIG. 7. TID corresponds to a term that occurs in some document belonging to a class corresponding to KCID (child or kid class ID). The Parent Class ID (PCID) represents the parent of KCID (zero if KCID is the root). The class identifications CID's (KCID's and PCID's) are numbered from one onwards. The explicit presence of PCID is only to simplify the current disclosure. In an actual system, the class tree data structure is preferably always available to map from KCID's to PCID's, and, therefore, PCID does not need to be explicitly stored in the table.
- In the statistics collection phase the main table is kept sorted on the keys TID and KCID. There is another unsorted table with the same row format. There are four other numeric fields per row. All of these four numbers are additive over documents, so for each document d and term t, a row is appended to the unsorted table, with SMC set to one, SNC set to the number of times t occurred in d, called n(d,t), SF1 set to n(d, t)/Σrn(d,t), and SF2 set to SF12. SMC is used in the binary model, while SNC is needed in the Bernoulli model.
- This approach trades off space for time, and the unsorted table grows rather quickly, but with a lot of duplicate keys. Depending on how much disk space exists, once in a while the system sorts and aggregates the unsorted table and then merges the result into the main sorted table. Various simple heuristics may be used to decide when to initiate a sort-merge. In one implementation, processing documents stops while the sort-merge is in progress. To meet tough real-time requirements, one can open a new frequency table and fork a thread, perhaps on another processor, to aggregate the last run while more documents continue to be accepted.
- An indexed access approach could be used instead of the frequency table. As each document is scanned, the system would look up on the (TID, KCID) key and update SMC. SNC, SF1 and SF2. That would result in index lookups and random IO, potentially for every term in the training set. For large corpora, it is far more efficient to append statistics in a logged fashion as in the preferred embodiment. The frequency table is a temporary file and no direct indexed access to it is actually required later. Another benefit is compactness: this is the most space-intensive phase of training, and the storage overheads of indexed access are avoided, while explicit control of compaction is obtained. The space overhead of storing TID redundantly is moderate, as the rest of each row is already 18˜bytes long.
- Computing Fisher Indices
- Before beginning feature selection, the frequency table is aggregated one last time, if needed to eliminate all duplicates. The frequency table is rewound and prepared for scanning. At this stage, all rows with the same TID are collected in a contiguous run going through all CID's where that TID occurred (see FIG. 7). Also, preparation takes place to output another file, called the fisher table. For the following description, a format shown in FIG. 8 is assumed. The format includes rows that are keyed by PCID and a floating point number FI (Fisher index), where for each fixed PCID the rows are sorted in decreasing order of FI. The last column is the TID (term ID) whose corresponding PCID and FI are the first and second columns.
- Because TID is the primary key in the frequency table, as it is scanned, a sequence of runs are obtained, each run having a fixed TID. Associated with each topic node in memory, a few words of statistics are kept (derived from SMC, SNC, etc.). When a run is started for a given TID, these statistics are cleared. As the various KCID's are scanned for the given TID in the frequency table, the node corresponding to the KCID in the taxonomy is located, and these statistics are updated. In a large taxonomy, very few of the nodes will be updated during a run. If a node is updated, its parent will be updated as well. These statistics efficiently can, therefore, be reset after each run.
- When the run for a given TID completes, exploring only the updated nodes, the Fisher index of that term is computed for every internal node (identified by its PCID) in the taxonomy as described in the section titled “Feature and Noise Terms.” For each of these PCID's. a row is appended to the Fisher table. Next, the Fisher table is sorted on the key (PCID.FI). This collects all PCID's into contiguous segments, and for each PCID, orders terms by decreasing values of FI.
- Consider now the case in which, for each internal topic c, the number k*(c) of features to pick is specified to TAPER directly. (The next section discusses how k*(c) is determined in one pass over the portion of the training documents set apart earlier for model pruning.) Given k*(c), the sorted Fisher table is scanned while copying the first k*(c) rows for the run corresponding to class c to an output table and discarding the remaining terms. This involves completely sequential IO.
- As shown in FIG. 10, once feature selection is performed on the Fisher table, both the frequency and fisher table are sorted once again, this time with (PCID,TID) as the key. After these sorts, a merge is performed. Rows of the Fisher table are considered one by one. For each row, once the beginning of a key-matched row of the frequency table is found, the row is read as long as the key remains unchanged, constructing a vector in memory where each element has the form (KCID,SMC,SNC). This buffer is then written into a hash table on disk.
- Feature Selection
- Given terms in decreasing Fisher index order for a fixed class c, it is desirable to find a good value for k*, the number of chosen features, as described in the section titled “Selecting a cut-off.” In a preferred embodiment, this is done in only one pass over the validation documents in set V. FIG. 9 shows an illustration of the organization of the table of terms and their discrimination powers in one preferred embodiment. The table of terms is used in the feature selection process.
- The technique used holds Nk (the number of incorrectly classified documents as a function of k, the number of features used) in memory as against scanning the validation documents for different values of k. The following is pseudocode:
Naïve pseudocode: For all or suitably many values of k Prepare models with only k features For each validation document d ε V, Determine Vk (d) IO-efficient pseudocode: Compute Fisher ordering and initialize Nk = 0 for k = 0,1,2, . . . For each validation document d ε V Compute Pr[c′|c, d, F0] for every child c′ using priors only If a wrong class has highest Pr[c′| . . .] increment N0 For k = 1,2, . . . Check if the k -th feature occurs in d If so, find Pr[c′|c, d, Fk] using Pr[c′|c, d, Fk-1] If a wrong class has highest Pr[c′| . . .] increment Nk - Even if Nk is stored for every value of k, and the lexicon is of size 500,000 (a sample of 266,000 documents from Yahoo™ required 600,000), only 2MB is needed. As each document d is scanned, Nk (d) is aggregated into the Nk array.
- Earlier, it was discussed that indexed statistics could be computed, given the optimal number of k* of terms at each node. Here, the discussion is directed to how the k* is determined. The documents in the validation set are considered one by one. Each document d has a pre-assigned “correct” class label c*. The parent c of this class node is located in the topic tree. Recall that c has associated with it a ranked list of terms. These are intersected with the terms in d, and the common terms are sorted by the rank.
- Now a sequence of progressively more detailed classifiers are constructed at c. The first one has only one feature, the top-ranked term in d. The second classifier has the top two features, etc. For each such classifier, it is checked whether d is correctly routed from c to c*.
- This gives a sequence of zero-one readings (i.e. zero if correct, and one if not) on a scale indexed by the rank of the last included term. The following is an example sequence for a particular document:
- 0-29 1 (classifiers indexed 0-29 did not place d in c*)
- 29-500 0 (classifiers that used terms ranked 0 through k, for k=29, 30, . . . 500 did place d in c*)
- 500-550 1 (classifiers that included even more terms went wrong again, because they included noise)
- Attached with each internal node, such as c above, is a file in which such sequences are appended for each validation document d. Once processing of (c, c*) is complete, processing continues up the tree, etc., until the processing of d is completely finished. These sequence files are then sorted on the index and a running sum is computed to generate the error rate curve as illustrated in FIG. 6.
- Updates to the Database
- For a batch job, the large frequency and fisher tables can now be deleted, leaving the relatively smaller indexed topic statistics and the much smaller term-to-TID maps. If the system is used to determine where new documents will be added to classes, it is necessary to preserve the frequency table. It continues to be used in the same way as before: rows are appended and occasionally it is compacted to aggregate duplicate keys. Running feature selection integrates the new data into the indexed statistics.
- Like running statistics generation for a relational server, feature selection is not an interactive operation. For example, on a database with 2000 classes, with an average of 150 documents per class, and an average of 100 terms per document, it may take a couple of hours. So feature selection is invoked only when there is reason to believe that the refreshed statistics will improve classification. In further preferred embodiments, such times to perform feature selection is automatically determined, for example, based on the occurrence of one or more predetermined events.
- Another issue is deletion of documents and moving of documents from one class to another (perhaps because classification was poor or erroneous for those documents). Since feature selection is always preceded by a frequency table aggregation, negative “correction” entries may be placed in it. That is, a frequency table row is produced, corresponding to each term in the deleted document, and SNIC, SNC, SF1 and SF2 are negated for the class(es) from which the document is being deleted. Here, it cannot be ensured that the document was originally included in the aggregate, but that can be done by preserving ID's for training documents.
- A more difficult issue is the reorganization of the taxonomy itself. Notice that in TAPER, a parent class inherits, in an additive fashion, the statistics of its children, since each training document generates rows for each topic node from the assigned topic up to the root. The preferred means of reorganization therefore involves reassigning some CID's and writing out a new frequency table with some negative “correction” entries.
- For example, consider detaching a subtree under node c1 and attaching it elsewhere under node c2. Statistics at or above the least common ancestor c, of c1 and c2 remain unchanged. Negative (respectively, positive) rows are appended to the frequency table corresponding to all classes on the path between cl inclusive and c1 (respectively, c2) exclusive. Finally, the parent and child links have to be modified in the taxonomy tree.
- Classification
- The rationale for the data organization described above is now explained by a description of the procedure of classifying a new or test document. When analyzing a new or test document, the taxonomy and associated pre-computed statistics are first loaded, and then the document is submitted to the system in the form of a map from terms to frequencies. In the present model, the probability that the document is generated by the root topic is 1, by definition, and decreases down any path in the taxonomy. Accordingly, the user also specifies a probability cut-off for topics reported back as close matches.
- Consider the document d at some internal node c0 with children c1,c2, . . . . The system first intersects d with the feature set at c0, then, for each retained term t∈d∩F(c0), and then the system looks up the model parameters for subtopics c1, c2, etc. It is thus best for both space and IO efficiency to index the statistics by (c0, t) and include in the record a vector of statistics for each child ci, for i=1,2, . . . , of node c0. The obvious pseudocode has to be slightly modified to reflect this optimization (i.e., pc denotes log probabilities).
Naïve index lookup: For each child c, of c0, i = 1,2 . . . Initialize pc, to 0 For each term t ε d ∩ F(c0) Lookup statistics for term t for class c, Update pc, Normalize Σ,ep,, to 1 and add pc0 to each pc,. Optimized index lookup: Initialize all pc, to 0 For each term t ε d ∩ F(c0) Skip if key (c0, t) is not in index Otherwise, retrieve record for (c0, t) For each c, that appears in the record Update pc, Normalize etc. - Caching and Batch Classification
- Two additional optimizations may be needed in the above technique to rapidly compute class estimates for test/new documents. First, observe that in practice, most terms in a document will not be found in F(c0). It can be wasteful to frequently look up the index on disk only to discard terms. Since few terms are found useful, it is relatively acceptable to look up these statistics on disk. Thus, a fast cache of feature TID's, even without any attached statistics, just to check if a TID is a feature or not, greatly increases speed.
- Second, given not one, but several documents to classify, it is desirable to amortize lookup of common terms by using a B-Tree access approach on the statistics (rather than a hash table), pre-sorting the TID's and making indexed scans on the statistics table. It is to be understood that any such locality-based performance optimizations and reordering of accesses to disk data are obvious to one trained in the art.
- Incomplete Classification
- With some documents, it may not be possible to narrow down the topic to a leaf node of the taxonomy. For example, there may be strong elements of both “Art:Paintina” and “Arts:Photography” in a document, so that the best classification should be just “Arts.” Various schemes may be proposed for dealing with such situations. In a preferred embodiment the system can trade off between the benefit of narrowing down the topic against the cost of doing so with low confidence.
-
- If this quantity is smaller than 1, the system stops at c0 in a preferred embodiment. It is to be understood that any other such test of confidence, with the purpose of stopping the classifier prematurely, is within the spirit of the present invention.
- Performance (Speed and Accuracy) Example
- We used three data sets for evaluation: the Reuters newswire widely used in the IR community, a portion from the US Patent database, and a sample from Yahoo™.
- The experimental computers were between 133 and 266 MHz, with 128-256 MB of memory. Once a document is in memory, typical training time is 140ìs and typical testing time was 30ìs. Training and testing on Reuters takes 20 minutes overall. The Yahoo™ sample, with 2118 classes and 266,000 documents, takes 19 hours to train. Bernoulli was found to be superior to binary for all our experiments.
- The Patent data set had three intermediate nodes below the root and twelve leaf nodes. A few hundred terms out of the lexicon of about 30,000 terms were sufficient to minimize error; the overall accuracy to the leaves was 66% (i.e., 66% patents were correctly classified) and the accuracy at the root node was 75%. For Reuters, the accuracy was 87%, as against the earlier best known technique's accuracy of 81% by Apte et al.
- The best features in the patent taxonomy for the root and its three children are listed in Table 1 below. Observe how distinct the sets are.
TABLE 1 Patent: signal, modulate, motor, receive, antenna, telephone, transmit, frequency, demodulate, current, voltage, data . . . Communication: Electricity: motor, Electronics: amplifier. antenna. telephone, heat, voltage, oscillator, input, modulator, transistor, output, output, frequency, demodulator, circuit, . . . transistor, . . . signal, . . . - The feature selection technique is applied to find salient differences between various interesting sets of documents. One application is to find descriptions for clusters in unsupervised document clustering. For example, the query mouse provides hundreds of responses from the US patent database. Clustering the responses and applying TAPER with the clusters treated as classes yields the automatically generated cluster descriptions shown in Table 2 below.
TABLE 2 Tissue, thymus, transplanted, hematopoietic, treatment, organ, trypsin, . . . Computer, keyboard, hand, edge, top, location, keys, support, cleaning, . . . Point, select, environment, object, display, correspondence. image, . . . - These “cluster digests” clearly identify different contexts of the occurrence of “mouse” and help the user easily refine the query.
- As yet another example application of the TAPER system, the patent portfolios of two assignees, or one assignee over two different intervals, were compared. For example, a comparison between “Sun Microsystems” and “Silicon Graphics” gave the result shown in Table 3 below.
TABLE 3 Sun Microsystems Silicon Graphics General purpose Information processing programmable digital system organization computer systems Data representation, Electrical computers and computer graphics data processing system Sufrace detail, texture Integrated circuit, Adjusting resolution or processor level of detail - Note how the commonality between the assignees (“UNIX workstations”) is filtered out and the difference brought out.
- An even more interesting example is comparing Intel patents in 1993-94 with those in 1994-95. TAPER detects a new line of research and patenting in the latter year, as shown in Table 4 below.
TABLE 4 Intel, 1993-94 Intel, 1994-95 General purpose Interactive television programmable digital bandwidth reduction computer systems system Chip, fabrication, counter, Involving difference input transmission Field or frame difference - Note that it is possible to get coherent phrases in this case because the patent meta-data stores these as effectively single terms.
- In a typical application, a taxonomy will be initially designed by hand, and training documents obtained. Once training is completed, the accuracy of the system can be estimated by comparing the class output of the
classifier 50 with the known classifications of the testing documents 34 (FIG. 3). If the accuracy is inadequate, a further training procedure, using a different collection of documents, or a reorganization of the taxonomy, may be carried out to retrain the system. Once satisfactory accuracy is achieved, the system may be used for a number of purposes such as search, filtering, and indexing as described above. - Broadly, the current invention can be distinguished from prior work in its use of a context dependent statistics, and its emphasis on scalability and speed in dealing with corpora ranging into tens to hundreds of gigabytes, the use of efficient disk data structures, and efficient update mechanisms. The present invention has focused on techniques that have good statistical foundation while remaining within almost linear time and one pass over the corpus, even when doing feature selection simultaneously for many nodes in a large topic taxonomy.
- Having thus described exemplary embodiments of the present invention, it should be understood by those skilled in the art that the above disclosures are exemplary only and that various other alternatives, adaptations and modifications may be made within the scope of the present invention. The presently disclosed embodiments are to be considered in all respects as illustrative and not restrictive. The scope of the invention being indicated by the appended claims, rather than the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are, therefore, intended to be embraced therein.
Claims (48)
1. A process for classifying new documents containing features under nodes defining a multilevel taxonomy, based on features derived from a training set of documents that have been classified under respective nodes of the taxonomy, the process comprising:
associating a respective set of features with each one of said plurality of nodes, each given set of features comprising a plurality of features that are in at least one training document classified under the associated node; and
classifying each new document under at least one node, based on the set of features associated with said at least one node.
2. A process as recited in , wherein said step of associating comprises the steps of:
claim 1
determining a discrimination value for each term in at least one training document which is classified under each one of a plurality of the nodes of the taxonomy; and
determining a minimum discrimination value for each of said plurality of nodes;
wherein the features in each given set of features have discrimination values equal to or above the minimum discrimination value determined for the node associated with the given set of features.
3. A process as recited in , wherein said step of classifying comprises:
claim 1
scanning each new document to determine the features in the document; and
defining, for each of said plurality of said plurality of nodes and for each new document, the probability that the new document is classified under the node, based on the set of features associated with the node and the features in the document.
4. A process as recited in , wherein said step of defining the probability comprises the step of applying a statistical model to define said probability that features in each given new document would occur at the frequency at which they do occur in the given new document.
claim 3
5. A process as recited in , wherein said statistical model comprises a Bernoulli model.
claim 4
6. A process as recited in , wherein said statistical model comprises a Poisson model.
claim 4
7. A process as recited in , wherein said step of classifying further comprises the step of assigning each given new document to at least one respective node in at least one level of the taxonomy, wherein the at least one node to which each given new document is assigned is the node for which the defined probability is above a predefined threshold among all of the nodes at the same level in the taxonomy.
claim 4
8. A process as recited in , wherein the at least one node to which each given new document is assigned is the node for which the defined probability is maximum among all of the nodes at the same level in the taxonomy.
claim 7
9. A process as recited in , wherein said step of assigning each given new document to at least one respective node in at least one level of the taxonomy comprises the step of assigning each given new document to at least one respective node in each of a plurality of levels of the taxonomy.
claim 8
10. A process as recited in , wherein said step of selecting a set of features comprises selecting features that are in a plurality of training documents classified under the associated node and that have discrimination values equal to or above the minimum discrimination value.
claim 2
11. A process as recited in , wherein said step of selecting a set of features comprises selecting features that are in all of training documents classified under the associated node and that have discrimination values equal to or above the minimum discrimination value.
claim 2
12. A process as recited in , wherein said step of determining a discrimination value comprises determining a discrimination value for each feature in a plurality of training documents which are classified under each one of a plurality of the nodes of the taxonomy.
claim 2
13. A process as recited in , wherein said step of determining a discrimination value comprises determining a discrimination value for each feature in all of the training documents which are classified under each one of a plurality of the nodes of the taxonomy.
claim 2
14. A process as recited in , wherein said step of determining a discrimination value for each feature comprises determining a Fisher value for each feature, based on the equation:
claim 2
where: t represents a term, d represents a document,
μ(c,t)=1/|c|Σd∈c X(d,t), and
x(d,t) =the occurrence rate of t in d.
15. A process as recited in , wherein said step of associating a respective set of features with each node comprises the step of determining the number of features to associate with each respective node.
claim 1
16. A process as recited in , wherein said step of associating a respective set of features with each given node comprises the steps of:
claim 15
ranking, by discrimination power, each of a plurality of features that are in at least one training document classified under the each given node;
providing an optimal number N of features for each given node; and
defining the set of features associated with a given node as the features ranked highest to the Nth highest in said step of ranking.
17. A process as recited in , wherein said step of providing an optimal number N comprises the step of determining the number N for each given node based on a test set of documents.
claim 16
18. A process as recited in , further comprising the step of displaying, for given node of a plurality of nodes of the taxonomy, a signature comprising at least one feature associated with the documents classified under the given node.
claim 1
19. A process as recited in , wherein said signature for each given node comprises a plurality of features associated with the documents classified under the given node.
claim 18
20. A process as recited in , wherein said signature for each given node comprises a plurality of features that occur in the documents classified under the given node, but which are determined to have a relatively low frequency of occurrence among documents under the given node.
claim 18
21. A process for searching for documents relevant to a search query from a group of accessible documents containing terms, comprising the steps of:
defining a multilevel taxonomy having a plurality of nodes, including a root node, at least one intermediate node associated with and under the root node and a plurality of terminal nodes associated with and under each intermediate node;
classifying each one of a plurality of training documents with at least one of the terminal and intermediate nodes;
determining a discrimination value for each term in at least one training document which is classified with each one of a plurality of the terminal and intermediate nodes of the taxonomy;
determining a minimum discrimination value for each of said plurality of terminal and intermediate nodes;
selecting a set of feature terms associated with each one of said plurality of terminal and intermediate nodes, said feature terms comprising terms that are in at least one training document classified with the associated node or any node under the associated node and that have discrimination values equal to or above the minimum discrimination value;
receiving a search query:
determining a plurality of search documents, each search document comprising one of the accessible document that is relevant to the search query;
classifying each search document with at least one of the terminal and intermediate nodes of the taxonomy, based on the sets of feature terms associated with the terminal and intermediate nodes of the taxonomy;
displaying a list of nodes with or under which said search documents are classified;
selecting at least one of the displayed nodes; and
displaying at least one search document classified under each selected node.
22. A process as recited in :
claim 21
wherein said search query comprises at least one search term; and
wherein each search document comprises one of the accessible document that contain said at least one search term as one of the terms in the document.
23. A process as recited in :
claim 21
wherein, following said step of selecting at least one of the displayed nodes and prior to said step of displaying at least one search document, said process further includes the steps of displaying a second list of further nodes with or under which said search documents are classified; and selecting at least one of the displayed further nodes; and
wherein said step of displaying at least one search document comprises the step of displaying at least one search document classified under each selected further node.
24. A process as recited in , wherein said step of displaying a list of nodes with or under which said search documents are classified further comprises the step of displaying signature terms associated with said search documents classified with or under each of said nodes in the displayed list.
claim 21
25. A process as recited in , wherein said signature terms comprise a plurality of the most frequently occurring terms in the search documents that are also feature terms.
claim 24
26. A classifier system for classifying new documents containing terms under nodes defining a multilevel taxonomy, based on feature terms derived from a training set of documents which are classified under respective nodes of the taxonomy, the system comprising:
means for determining a discrimination value for each term in at least one training document which is classified under each one of a plurality of the nodes of the taxonomy;
means for determining a minimum discrimination value for each of said plurality of nodes;
means for selecting a set of feature terms associated with each one of said plurality of nodes, said feature terms comprising terms that are in at least one training document classified under the associated node and that have discrimination values equal to or above the minimum discrimination value; and
means for classifying each new document under at least one node, based on the feature terms associated with said at least one node.
27. A system as recited in , wherein said means for classifying comprises:
claim 26
means for scanning each new document to determine the terms in the document; and
means for defining, for each of said plurality of said plurality of nodes and for each new document, the probability that the new document is classified under the node, based on the feature terms associated with the node and the terms in the document.
28. A system as recited in , wherein said means for defining the probability comprises means for applying a Bernoulli model to define said probability for each of said plurality of nodes.
claim 27
29. A system as recited in , wherein said means for selecting a set of feature terms comprises means for selecting terms that are in a plurality of training documents classified under the associated node and that have discrimination values equal to or above the minimum discrimination value.
claim 26
30. A system as recited in , wherein said means for selecting a set of feature terms comprises means for selecting terms that are in all of training documents classified under the associated node and that have discrimination values equal to or above the minimum discrimination value.
claim 26
31. A system as recited in , wherein said means for determining a discrimination value comprises means for determining a discrimination value for each term in a plurality of training documents which are classified under each one of a plurality of the nodes of the taxonomy.
claim 26
32. A system as recited in , wherein said means for determining a discrimination value comprises means for determining a discrimination value for each term in all of the training documents which are classified under each one of a plurality of the nodes of the taxonomy.
claim 26
33. A system as recited in , wherein said means for determining a discrimination value for each term comprises means for determining a Fisher value for each term, based on the equation:
claim 26
where: t represents a term, d represents a document,
μ(c,t)=1/|c|Σd∈cx(d,t), and
x(d,t)=the occurrence rate of t in d.
34. A system for searching for documents relevant to a search query from a group of accessible documents containing terms, comprising:
means for defining a multilevel taxonomy having a plurality of nodes, including a root node, at least one intermediate node associated with and under the root node and a plurality of terminal nodes associated with and under each intermediate node;
means for classifying each one of a plurality of training documents with at least one of the terminal and intermediate nodes;
means for determining a discrimination value for each term in at least one training document which is classified with each one of a plurality of the terminal and intermediate nodes of the taxonomy;
means for determining a minimum discrimination value for each of said plurality of terminal and intermediate nodes;
means for selecting a set of feature terms associated with each one of said plurality of terminal and intermediate nodes, said feature terms comprising terms that are in at least one training document classified with the associated node or any node under the associated node and that have discrimination values equal to or above the minimum discrimination value;
means for receiving a search query;
means for determining a plurality of search documents, each search document comprising one of the accessible document that is relevant to the search query;
means for classifying each search document with at least one of the terminal and intermediate nodes of the taxonomy, based on the sets of feature terms associated with the terminal and intermediate nodes of the taxonomy;
means for displaying a list of nodes with or under which said search documents are classified;
means for selecting at least one of the displayed nodes; and
means for displaying at least one search document classified under each selected node.
35. A system as recited in :
claim 34
wherein said search query comprises at least one search term; and
wherein each search document comprises one of the accessible document that contain said at least one search term as one of the terms in the document.
36. A system as recited in , wherein said means for displaying a list of nodes with or under which said search documents are classified further comprises means for displaying signature terms associated with said search documents classified with or under each of said nodes in the displayed list.
claim 34
37. A system as recited in , wherein said signature terms comprise a plurality of the most frequently occurring terms in the search documents that are also feature terms.
claim 34
38. An article of manufacture comprising a computer program carrier readable by a computer and embodying one or more instructions executable by the computer to perform a process for classifying new documents containing terms under nodes defining a multilevel taxonomy, based on feature terms derived from a training set of documents which are classified under respective nodes of the taxonomy, the process comprising:
determining a discrimination value for each term in at least one training document which is classified under each one of a plurality of the nodes of the taxonomy;
determining a minimum discrimination value for each of said plurality of nodes;
selecting a set of feature terms associated with each one of said plurality of nodes, said feature terms comprising terms that are in at least one training document classified under the associated node and that have discrimination values equal to or above the minimum discrimination value; and
classifying each new document under at least one node, based on the feature terms associated with said at least one node.
39. An article as recited in , wherein said step of classifying comprises:
claim 38
scanning each new document to determine the terms in the document; and
defining, for each of said plurality of said plurality of nodes and for each new document, the probability that the new document is classified under the node, based on the feature terms associated with the node and the terms in the document.
40. An article as recited in , wherein said step of defining the probability comprises the step of applying a Bernoulli model to define said probability for each of said plurality of nodes.
claim 39
41. An article as recited in , wherein said step of selecting a set of feature terms comprises selecting terms that are in a plurality of training documents classified under the associated node and that have discrimination values equal to or above the minimum discrimination value.
claim 38
42. An article as recited in , wherein said step of selecting a set of feature terms comprises selecting terms that are in all of training documents classified under the associated node and that have discrimination values equal to or above the minimum discrimination value.
claim 38
43. An article as recited in , wherein said step of determining a discrimination value comprises determining a discrimination value for each term in a plurality of training documents which are classified under each one of a plurality of the nodes of the taxonomy.
claim 38
44. An article as recited in , wherein said step of determining a discrimination value comprises determining a discrimination value for each term in all of the training documents which are classified under each one of a plurality of the nodes of the taxonomy.
claim 38
45. An article as recited in , wherein said step of determining a discrimination value for each term comprises determining a Fisher value for each term, based on the equation:
claim 38
where: t represents a term, d represents a document,
μ(c,t)=1/|c|Σd∈cx(d,t), and
x(d,t)=the occurrence rate of t in d.
46. An article of manufacture comprising a computer program carrier readable by a computer and embodying one or more instructions executable by the computer for searching for documents relevant to a search query from a group of accessible documents containing terms, comprising the steps of:
defining a multilevel taxonomy having a plurality of nodes, including a root node, at least one intermediate node associated with and under the root node and a plurality of terminal nodes associated with and under each intermediate node;
classifying each one of a plurality of training documents with at least one of the terminal and intermediate nodes;
determining a discrimination value for each term in at least one training document which is classified with each one of a plurality of the terminal and intermediate nodes of the taxonomy;
determining a minimum discrimination value for each of said plurality of terminal and intermediate nodes;
selecting a set of feature terms associated with each one of said plurality of terminal and intermediate nodes, said feature terms comprising terms that are in at least one training document classified with the associated node or any node under the associated node and that have discrimination values equal to or above the minimum discrimination value;
receiving a search query;
determining a plurality of search documents, each search document comprising one of the accessible document that is relevant to the search query;
classifying each search document with at least one of the terminal and intermediate nodes of the taxonomy, based on the sets of feature terms associated with the terminal and intermediate nodes of the taxonomy;
displaying a list of nodes with or under which said search documents are classified;
selecting at least one of the displayed nodes; and
displaying at least one search document classified under each selected node.
47. An article as recited in , wherein said step of displaying a list of nodes with or under which said search documents are classified further comprises the step of displaying signature terms associated with said search documents classified with or under each of said nodes in the displayed list.
claim 46
48. An article as recited in , wherein said signature terms comprise a plurality of the most frequently occurring terms in the search documents that are also feature terms.
claim 47
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/777,278 US20010037324A1 (en) | 1997-06-24 | 2001-02-05 | Multilevel taxonomy based on features derived from training documents classification using fisher values as discrimination values |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US5061197P | 1997-06-24 | 1997-06-24 | |
US09/102,861 US6233575B1 (en) | 1997-06-24 | 1998-06-23 | Multilevel taxonomy based on features derived from training documents classification using fisher values as discrimination values |
US09/777,278 US20010037324A1 (en) | 1997-06-24 | 2001-02-05 | Multilevel taxonomy based on features derived from training documents classification using fisher values as discrimination values |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/102,861 Division US6233575B1 (en) | 1997-06-24 | 1998-06-23 | Multilevel taxonomy based on features derived from training documents classification using fisher values as discrimination values |
Publications (1)
Publication Number | Publication Date |
---|---|
US20010037324A1 true US20010037324A1 (en) | 2001-11-01 |
Family
ID=26728460
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/102,861 Expired - Fee Related US6233575B1 (en) | 1997-06-24 | 1998-06-23 | Multilevel taxonomy based on features derived from training documents classification using fisher values as discrimination values |
US09/777,278 Abandoned US20010037324A1 (en) | 1997-06-24 | 2001-02-05 | Multilevel taxonomy based on features derived from training documents classification using fisher values as discrimination values |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/102,861 Expired - Fee Related US6233575B1 (en) | 1997-06-24 | 1998-06-23 | Multilevel taxonomy based on features derived from training documents classification using fisher values as discrimination values |
Country Status (1)
Country | Link |
---|---|
US (2) | US6233575B1 (en) |
Cited By (127)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030101191A1 (en) * | 2001-11-28 | 2003-05-29 | Yu-Chwin Lin | Publication of commercial entity information and method for classifying commercial entity information |
US20030163454A1 (en) * | 2002-02-26 | 2003-08-28 | Brian Jacobsen | Subject specific search engine |
US20030172357A1 (en) * | 2002-03-11 | 2003-09-11 | Kao Anne S.W. | Knowledge management using text classification |
US20030194689A1 (en) * | 2002-04-12 | 2003-10-16 | Mitsubishi Denki Kabushiki Kaisha | Structured document type determination system and structured document type determination method |
US6640224B1 (en) * | 1997-12-15 | 2003-10-28 | International Business Machines Corporation | System and method for dynamic index-probe optimizations for high-dimensional similarity search |
US20030217055A1 (en) * | 2002-05-20 | 2003-11-20 | Chang-Huang Lee | Efficient incremental method for data mining of a database |
US20040049514A1 (en) * | 2002-09-11 | 2004-03-11 | Sergei Burkov | System and method of searching data utilizing automatic categorization |
US20040111419A1 (en) * | 2002-12-05 | 2004-06-10 | Cook Daniel B. | Method and apparatus for adapting a search classifier based on user queries |
US20040117388A1 (en) * | 2002-09-02 | 2004-06-17 | Yasuhiko Inaba | Method, apparatus and programs for delivering information |
US20040153307A1 (en) * | 2001-03-30 | 2004-08-05 | Naftali Tishby | Discriminative feature selection for data sequences |
US20040220901A1 (en) * | 2003-04-30 | 2004-11-04 | Benq Corporation | System and method for association itemset mining |
US20040225645A1 (en) * | 2003-05-06 | 2004-11-11 | Rowney Kevin T. | Personal computing device -based mechanism to detect preselected data |
US20050027723A1 (en) * | 2002-09-18 | 2005-02-03 | Chris Jones | Method and apparatus to report policy violations in messages |
US20050086252A1 (en) * | 2002-09-18 | 2005-04-21 | Chris Jones | Method and apparatus for creating an information security policy based on a pre-configured template |
US20050132046A1 (en) * | 2003-12-10 | 2005-06-16 | De La Iglesia Erik | Method and apparatus for data capture and analysis system |
US20050187892A1 (en) * | 2004-02-09 | 2005-08-25 | Xerox Corporation | Method for multi-class, multi-label categorization using probabilistic hierarchical modeling |
US20060074632A1 (en) * | 2004-09-30 | 2006-04-06 | Nanavati Amit A | Ontology-based term disambiguation |
US20060074907A1 (en) * | 2004-09-27 | 2006-04-06 | Singhal Amitabh K | Presentation of search results based on document structure |
US20060155662A1 (en) * | 2003-07-01 | 2006-07-13 | Eiji Murakami | Sentence classification device and method |
US20060184549A1 (en) * | 2005-02-14 | 2006-08-17 | Rowney Kevin T | Method and apparatus for modifying messages based on the presence of pre-selected data |
US20060195415A1 (en) * | 2005-02-14 | 2006-08-31 | France Telecom | Method and device for the generation of a classification tree to unify the supervised and unsupervised approaches, corresponding computer package and storage means |
US7107254B1 (en) * | 2001-05-07 | 2006-09-12 | Microsoft Corporation | Probablistic models and methods for combining multiple content classifiers |
US20060224589A1 (en) * | 2005-02-14 | 2006-10-05 | Rowney Kevin T | Method and apparatus for handling messages containing pre-selected data |
US20060242140A1 (en) * | 2005-04-26 | 2006-10-26 | Content Analyst Company, Llc | Latent semantic clustering |
US20060242190A1 (en) * | 2005-04-26 | 2006-10-26 | Content Analyst Comapny, Llc | Latent semantic taxonomy generation |
US20060242098A1 (en) * | 2005-04-26 | 2006-10-26 | Content Analyst Company, Llc | Generating representative exemplars for indexing, clustering, categorization and taxonomy |
US20060282442A1 (en) * | 2005-04-27 | 2006-12-14 | Canon Kabushiki Kaisha | Method of learning associations between documents and data sets |
US20070005535A1 (en) * | 2005-04-27 | 2007-01-04 | Abdolreza Salahshour | System and methods for IT resource event situation classification and semantics |
US20070011154A1 (en) * | 2005-04-11 | 2007-01-11 | Textdigger, Inc. | System and method for searching for a query |
US7203669B2 (en) * | 2003-03-17 | 2007-04-10 | Intel Corporation | Detector tree of boosted classifiers for real-time object detection and tracking |
US20070156665A1 (en) * | 2001-12-05 | 2007-07-05 | Janusz Wnek | Taxonomy discovery |
US20070192442A1 (en) * | 2001-07-24 | 2007-08-16 | Brightplanet Corporation | System and method for efficient control and capture of dynamic database content |
US20070203903A1 (en) * | 2006-02-28 | 2007-08-30 | Ilial, Inc. | Methods and apparatus for visualizing, managing, monetizing, and personalizing knowledge search results on a user interface |
US20070234232A1 (en) * | 2006-03-29 | 2007-10-04 | Gheorghe Adrian Citu | Dynamic image display |
US20080059451A1 (en) * | 2006-04-04 | 2008-03-06 | Textdigger, Inc. | Search system and method with text function tagging |
US7363593B1 (en) * | 2001-11-30 | 2008-04-22 | Versata Development Group, Inc. | System and method for presenting information organized by hierarchical levels |
US20080114573A1 (en) * | 2006-11-10 | 2008-05-15 | Institute For Information Industry | Tag organization methods and systems |
US7376635B1 (en) * | 2000-07-21 | 2008-05-20 | Ford Global Technologies, Llc | Theme-based system and method for classifying documents |
US20080195567A1 (en) * | 2007-02-13 | 2008-08-14 | International Business Machines Corporation | Information mining using domain specific conceptual structures |
WO2008097891A2 (en) * | 2007-02-02 | 2008-08-14 | Musgrove Technology Enterprises Llc | Method and apparatus for aligning multiple taxonomies |
US7472114B1 (en) * | 2002-09-18 | 2008-12-30 | Symantec Corporation | Method and apparatus to define the scope of a search for information from a tabular data source |
US20090228499A1 (en) * | 2008-03-05 | 2009-09-10 | Schmidtler Mauritius A R | Systems and methods for organizing data sets |
US20090254540A1 (en) * | 2007-11-01 | 2009-10-08 | Textdigger, Inc. | Method and apparatus for automated tag generation for digital content |
US7650340B2 (en) * | 1998-12-21 | 2010-01-19 | Adobe Systems Incorporated | Describing documents and expressing document structure |
US7657104B2 (en) | 2005-11-21 | 2010-02-02 | Mcafee, Inc. | Identifying image type in a capture system |
US20100030752A1 (en) * | 2008-07-30 | 2010-02-04 | Lev Goldentouch | System, methods and applications for structured document indexing |
US7673344B1 (en) | 2002-09-18 | 2010-03-02 | Symantec Corporation | Mechanism to search information content for preselected data |
US20100076984A1 (en) * | 2008-03-27 | 2010-03-25 | Alkis Papadopoullos | System and method for query expansion using tooltips |
US7689614B2 (en) | 2006-05-22 | 2010-03-30 | Mcafee, Inc. | Query generation for a capture system |
US7730011B1 (en) | 2005-10-19 | 2010-06-01 | Mcafee, Inc. | Attributes of captured objects in a capture system |
US20100162347A1 (en) * | 2008-12-22 | 2010-06-24 | Ian Barile | Adaptive data loss prevention policies |
US20100185577A1 (en) * | 2009-01-16 | 2010-07-22 | Microsoft Corporation | Object classification using taxonomies |
US7774604B2 (en) | 2003-12-10 | 2010-08-10 | Mcafee, Inc. | Verifying captured objects before presentation |
US7814327B2 (en) | 2003-12-10 | 2010-10-12 | Mcafee, Inc. | Document registration |
US7818326B2 (en) * | 2005-08-31 | 2010-10-19 | Mcafee, Inc. | System and method for word indexing in a capture system and querying thereof |
US20100332481A1 (en) * | 2002-09-18 | 2010-12-30 | Rowney Kevin T | Secure and scalable detection of preselected data embedded in electronically transmitted messages |
US7899828B2 (en) | 2003-12-10 | 2011-03-01 | Mcafee, Inc. | Tag data structure for maintaining relational data over captured objects |
US7907608B2 (en) | 2005-08-12 | 2011-03-15 | Mcafee, Inc. | High speed packet capture |
US7908260B1 (en) | 2006-12-29 | 2011-03-15 | BrightPlanet Corporation II, Inc. | Source editing, internationalization, advanced configuration wizard, and summary page selection for information automation systems |
US7930540B2 (en) | 2004-01-22 | 2011-04-19 | Mcafee, Inc. | Cryptographic policy enforcement |
US7949849B2 (en) | 2004-08-24 | 2011-05-24 | Mcafee, Inc. | File system for a capture system |
US7958227B2 (en) | 2006-05-22 | 2011-06-07 | Mcafee, Inc. | Attributes of captured objects in a capture system |
US7962591B2 (en) | 2004-06-23 | 2011-06-14 | Mcafee, Inc. | Object classification in a capture system |
US7962490B1 (en) * | 2008-01-07 | 2011-06-14 | Amdocs Software Systems Limited | System, method, and computer program product for analyzing and decomposing a plurality of rules into a plurality of contexts |
US7996374B1 (en) | 2008-03-28 | 2011-08-09 | Symantec Corporation | Method and apparatus for automatically correlating related incidents of policy violations |
US7996373B1 (en) | 2008-03-28 | 2011-08-09 | Symantec Corporation | Method and apparatus for detecting policy violations in a data repository having an arbitrary data schema |
US8010689B2 (en) | 2006-05-22 | 2011-08-30 | Mcafee, Inc. | Locational tagging in a capture system |
US20110214080A1 (en) * | 2010-02-26 | 2011-09-01 | Microsoft Corporation | Taxonomy Editor |
US20110231411A1 (en) * | 2008-08-08 | 2011-09-22 | Holland Bloorview Kids Rehabilitation Hospital | Topic Word Generation Method and System |
US20110264699A1 (en) * | 2008-12-30 | 2011-10-27 | Telecom Italia S.P.A. | Method and system for content classification |
US8065739B1 (en) | 2008-03-28 | 2011-11-22 | Symantec Corporation | Detecting policy violations in information content containing data in a character-based language |
US20110314024A1 (en) * | 2010-06-18 | 2011-12-22 | Microsoft Corporation | Semantic content searching |
US8122005B1 (en) * | 2009-10-22 | 2012-02-21 | Google Inc. | Training set construction for taxonomic classification |
US20120078969A1 (en) * | 2010-09-24 | 2012-03-29 | International Business Machines Corporation | System and method to extract models from semi-structured documents |
US20120109642A1 (en) * | 1999-02-05 | 2012-05-03 | Stobbs Gregory A | Computer-implemented patent portfolio analysis method and apparatus |
US8205242B2 (en) | 2008-07-10 | 2012-06-19 | Mcafee, Inc. | System and method for data mining and security policy management |
WO2012174640A1 (en) * | 2011-06-22 | 2012-12-27 | Rogers Communications Inc. | Systems and methods for creating an interest profile for a user |
US20130046797A1 (en) * | 2005-05-10 | 2013-02-21 | Netseer, Inc. | Methods and apparatus for distributed community finding |
US8447722B1 (en) | 2009-03-25 | 2013-05-21 | Mcafee, Inc. | System and method for data mining and security policy management |
US8473442B1 (en) | 2009-02-25 | 2013-06-25 | Mcafee, Inc. | System and method for intelligent state management |
US20130166563A1 (en) * | 2011-12-21 | 2013-06-27 | Sap Ag | Integration of Text Analysis and Search Functionality |
US8489538B1 (en) | 2010-05-25 | 2013-07-16 | Recommind, Inc. | Systems and methods for predictive coding |
US8504537B2 (en) | 2006-03-24 | 2013-08-06 | Mcafee, Inc. | Signature distribution in a document registration system |
US8515972B1 (en) | 2010-02-10 | 2013-08-20 | Python 4 Fun, Inc. | Finding relevant documents |
US20130218904A1 (en) * | 2012-02-22 | 2013-08-22 | Salesforce.Com, Inc. | System and method for inferring reporting relationships from a contact database |
US8548170B2 (en) | 2003-12-10 | 2013-10-01 | Mcafee, Inc. | Document de-registration |
US8560534B2 (en) | 2004-08-23 | 2013-10-15 | Mcafee, Inc. | Database for a capture system |
US8645298B2 (en) | 2010-10-26 | 2014-02-04 | Microsoft Corporation | Topic models |
US8656039B2 (en) | 2003-12-10 | 2014-02-18 | Mcafee, Inc. | Rule parser |
US8667121B2 (en) | 2009-03-25 | 2014-03-04 | Mcafee, Inc. | System and method for managing data and policies |
US20140086497A1 (en) * | 2012-06-14 | 2014-03-27 | The Board of Trustees for the Leland Stanford, Junior, University | Method and System for Optimizing Accuracy-Specificity Trade-offs in Large Scale Visual Recognition |
US8700561B2 (en) | 2011-12-27 | 2014-04-15 | Mcafee, Inc. | System and method for providing data protection workflows in a network environment |
US8706709B2 (en) | 2009-01-15 | 2014-04-22 | Mcafee, Inc. | System and method for intelligent term grouping |
US20140172652A1 (en) * | 2012-12-19 | 2014-06-19 | Yahoo! Inc. | Automated categorization of products in a merchant catalog |
US8806615B2 (en) | 2010-11-04 | 2014-08-12 | Mcafee, Inc. | System and method for protecting specified data combinations |
US8826443B1 (en) | 2008-09-18 | 2014-09-02 | Symantec Corporation | Selective removal of protected content from web requests sent to an interactive website |
US8849716B1 (en) * | 2001-04-20 | 2014-09-30 | Jpmorgan Chase Bank, N.A. | System and method for preventing identity theft or misuse by restricting access |
US8850591B2 (en) | 2009-01-13 | 2014-09-30 | Mcafee, Inc. | System and method for concept building |
US8862619B1 (en) | 2008-01-07 | 2014-10-14 | Amdocs Software Systems Limited | System, method, and computer program product for filtering a data stream utilizing a plurality of contexts |
US8862580B1 (en) * | 2004-03-01 | 2014-10-14 | Radix Holdings, Llc | Category-based search |
US8935752B1 (en) | 2009-03-23 | 2015-01-13 | Symantec Corporation | System and method for identity consolidation |
US20150095017A1 (en) * | 2013-09-27 | 2015-04-02 | Google Inc. | System and method for learning word embeddings using neural language models |
US9110985B2 (en) | 2005-05-10 | 2015-08-18 | Neetseer, Inc. | Generating a conceptual association graph from large-scale loosely-grouped content |
US9245029B2 (en) | 2006-01-03 | 2016-01-26 | Textdigger, Inc. | Search system with query refinement and search method |
US9253154B2 (en) | 2008-08-12 | 2016-02-02 | Mcafee, Inc. | Configuration management for a capture/registration system |
US20160140207A1 (en) * | 2014-11-14 | 2016-05-19 | Symantec Corporation | Systems and methods for aggregating information-asset classifications |
US20160182669A1 (en) * | 2014-12-22 | 2016-06-23 | Here Global B.V. | Optimal Coding Method for Efficient Matching Of Hierarchical Categories In Publish-Subscribe Systems |
US9436726B2 (en) | 2011-06-23 | 2016-09-06 | BCM International Regulatory Analytics LLC | System, method and computer program product for a behavioral database providing quantitative analysis of cross border policy process and related search capabilities |
US9443018B2 (en) | 2006-01-19 | 2016-09-13 | Netseer, Inc. | Systems and methods for creating, navigating, and searching informational web neighborhoods |
US20170091590A1 (en) * | 2013-03-15 | 2017-03-30 | Sri International | Computer vision as a service |
US9785634B2 (en) | 2011-06-04 | 2017-10-10 | Recommind, Inc. | Integration and combination of random sampling and document batching |
US9817902B2 (en) | 2006-10-27 | 2017-11-14 | Netseer Acquisition, Inc. | Methods and apparatus for matching relevant content to user intention |
US10025804B2 (en) | 2014-05-04 | 2018-07-17 | Veritas Technologies Llc | Systems and methods for aggregating information-asset metadata from multiple disparate data-management systems |
US20190051294A1 (en) * | 2002-10-31 | 2019-02-14 | Promptu Systems Corporation | Efficient empirical determination, computation, and use of acoustic confusability measures |
US10311085B2 (en) | 2012-08-31 | 2019-06-04 | Netseer, Inc. | Concept-level user intent profile extraction and applications |
US10387892B2 (en) | 2008-05-06 | 2019-08-20 | Netseer, Inc. | Discovering relevant concept and context for content node |
US20190392078A1 (en) * | 2018-06-22 | 2019-12-26 | Microsoft Technology Licensing, Llc | Topic set refinement |
US20190392073A1 (en) * | 2018-06-22 | 2019-12-26 | Microsoft Technology Licensing, Llc | Taxonomic tree generation |
US10586169B2 (en) * | 2015-10-16 | 2020-03-10 | Microsoft Technology Licensing, Llc | Common feature protocol for collaborative machine learning |
US10635645B1 (en) | 2014-05-04 | 2020-04-28 | Veritas Technologies Llc | Systems and methods for maintaining aggregate tables in databases |
US10902066B2 (en) | 2018-07-23 | 2021-01-26 | Open Text Holdings, Inc. | Electronic discovery using predictive filtering |
US11281995B2 (en) | 2018-05-21 | 2022-03-22 | International Business Machines Corporation | Finding optimal surface for hierarchical classification task on an ontology |
US20220245378A1 (en) * | 2021-02-03 | 2022-08-04 | Aon Risk Services, Inc. Of Maryland | Document analysis using model intersections |
US20230045330A1 (en) * | 2013-09-26 | 2023-02-09 | Groupon, Inc. | Multi-term query subsumption for document classification |
US11676043B2 (en) | 2019-03-04 | 2023-06-13 | International Business Machines Corporation | Optimizing hierarchical classification with adaptive node collapses |
US11704370B2 (en) | 2018-04-20 | 2023-07-18 | Microsoft Technology Licensing, Llc | Framework for managing features across environments |
US11928879B2 (en) * | 2021-02-03 | 2024-03-12 | Aon Risk Services, Inc. Of Maryland | Document analysis using model intersections |
Families Citing this family (393)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3597697B2 (en) * | 1998-03-20 | 2004-12-08 | 富士通株式会社 | Document summarizing apparatus and method |
US6510429B1 (en) * | 1998-04-29 | 2003-01-21 | International Business Machines Corporation | Message broker apparatus, method and computer program product |
US7711672B2 (en) * | 1998-05-28 | 2010-05-04 | Lawrence Au | Semantic network methods to disambiguate natural language meaning |
US8396824B2 (en) * | 1998-05-28 | 2013-03-12 | Qps Tech. Limited Liability Company | Automatic data categorization with optimally spaced semantic seed terms |
US20070294229A1 (en) * | 1998-05-28 | 2007-12-20 | Q-Phrase Llc | Chat conversation methods traversing a provisional scaffold of meanings |
JP3665480B2 (en) * | 1998-06-24 | 2005-06-29 | 富士通株式会社 | Document organizing apparatus and method |
US6446061B1 (en) * | 1998-07-31 | 2002-09-03 | International Business Machines Corporation | Taxonomy generation for document collections |
US6665837B1 (en) * | 1998-08-10 | 2003-12-16 | Overture Services, Inc. | Method for identifying related pages in a hyperlinked database |
US20040230409A1 (en) * | 1998-08-31 | 2004-11-18 | Nutech Solutions, Inc. | Method for performing social computation |
GB9821787D0 (en) * | 1998-10-06 | 1998-12-02 | Data Limited | Apparatus for classifying or processing data |
US6751606B1 (en) * | 1998-12-23 | 2004-06-15 | Microsoft Corporation | System for enhancing a query interface |
US6460029B1 (en) * | 1998-12-23 | 2002-10-01 | Microsoft Corporation | System for improving search text |
US6601059B1 (en) | 1998-12-23 | 2003-07-29 | Microsoft Corporation | Computerized searching tool with spell checking |
US6721759B1 (en) * | 1998-12-24 | 2004-04-13 | Sony Corporation | Techniques for spatial representation of data and browsing based on similarity |
US6385619B1 (en) * | 1999-01-08 | 2002-05-07 | International Business Machines Corporation | Automatic user interest profile generation from structured document access information |
US7024416B1 (en) * | 1999-03-31 | 2006-04-04 | Verizon Laboratories Inc. | Semi-automatic index term augmentation in document retrieval |
AU4328000A (en) * | 1999-03-31 | 2000-10-16 | Verizon Laboratories Inc. | Techniques for performing a data query in a computer system |
US8275661B1 (en) | 1999-03-31 | 2012-09-25 | Verizon Corporate Services Group Inc. | Targeted banner advertisements |
US8572069B2 (en) | 1999-03-31 | 2013-10-29 | Apple Inc. | Semi-automatic index term augmentation in document retrieval |
US6665681B1 (en) * | 1999-04-09 | 2003-12-16 | Entrieva, Inc. | System and method for generating a taxonomy from a plurality of documents |
US6564197B2 (en) * | 1999-05-03 | 2003-05-13 | E.Piphany, Inc. | Method and apparatus for scalable probabilistic clustering using decision trees |
US6442545B1 (en) * | 1999-06-01 | 2002-08-27 | Clearforest Ltd. | Term-level text with mining with taxonomies |
JP2000348041A (en) * | 1999-06-03 | 2000-12-15 | Nec Corp | Document retrieval method, device therefor and mechanically readable recording medium |
US6711585B1 (en) * | 1999-06-15 | 2004-03-23 | Kanisa Inc. | System and method for implementing a knowledge management system |
US6418434B1 (en) * | 1999-06-25 | 2002-07-09 | International Business Machines Corporation | Two stage automated electronic messaging system |
US7181438B1 (en) * | 1999-07-21 | 2007-02-20 | Alberti Anemometer, Llc | Database access system |
US6718363B1 (en) * | 1999-07-30 | 2004-04-06 | Verizon Laboratories, Inc. | Page aggregation for web sites |
US6578065B1 (en) * | 1999-09-23 | 2003-06-10 | Hewlett-Packard Development Company L.P. | Multi-threaded processing system and method for scheduling the execution of threads based on data received from a cache memory |
US6665656B1 (en) * | 1999-10-05 | 2003-12-16 | Motorola, Inc. | Method and apparatus for evaluating documents with correlating information |
US6651059B1 (en) * | 1999-11-15 | 2003-11-18 | International Business Machines Corporation | System and method for the automatic recognition of relevant terms by mining link annotations |
US6742023B1 (en) | 2000-04-28 | 2004-05-25 | Roxio, Inc. | Use-sensitive distribution of data files between users |
US8271316B2 (en) * | 1999-12-17 | 2012-09-18 | Buzzmetrics Ltd | Consumer to business data capturing system |
US7181454B1 (en) * | 1999-12-29 | 2007-02-20 | International Business Machines Corporation | Asset locator |
US6397211B1 (en) * | 2000-01-03 | 2002-05-28 | International Business Machines Corporation | System and method for identifying useless documents |
US6931403B1 (en) * | 2000-01-19 | 2005-08-16 | International Business Machines Corporation | System and architecture for privacy-preserving data mining |
US6701314B1 (en) * | 2000-01-21 | 2004-03-02 | Science Applications International Corporation | System and method for cataloguing digital information for searching and retrieval |
US6704727B1 (en) * | 2000-01-31 | 2004-03-09 | Overture Services, Inc. | Method and system for generating a set of search terms |
US6868525B1 (en) * | 2000-02-01 | 2005-03-15 | Alberti Anemometer Llc | Computer graphic display visualization system and method |
US6502091B1 (en) * | 2000-02-23 | 2002-12-31 | Hewlett-Packard Company | Apparatus and method for discovering context groups and document categories by mining usage logs |
WO2001067225A2 (en) * | 2000-03-06 | 2001-09-13 | Kanisa Inc. | A system and method for providing an intelligent multi-step dialog with a user |
US6728701B1 (en) * | 2000-04-18 | 2004-04-27 | Claritech Corporation | Method and apparatus for database retrieval utilizing vector optimization |
US7356604B1 (en) * | 2000-04-18 | 2008-04-08 | Claritech Corporation | Method and apparatus for comparing scores in a vector space retrieval process |
US6704728B1 (en) | 2000-05-02 | 2004-03-09 | Iphase.Com, Inc. | Accessing information from a collection of data |
US6711561B1 (en) * | 2000-05-02 | 2004-03-23 | Iphrase.Com, Inc. | Prose feedback in information access system |
US7310624B1 (en) * | 2000-05-02 | 2007-12-18 | International Business Machines Corporation | Methods and apparatus for generating decision trees with discriminants and employing same in data classification |
US6745181B1 (en) * | 2000-05-02 | 2004-06-01 | Iphrase.Com, Inc. | Information access method |
US8478732B1 (en) | 2000-05-02 | 2013-07-02 | International Business Machines Corporation | Database aliasing in information access system |
US7127450B1 (en) * | 2000-05-02 | 2006-10-24 | International Business Machines Corporation | Intelligent discard in information access system |
US6912525B1 (en) | 2000-05-08 | 2005-06-28 | Verizon Laboratories, Inc. | Techniques for web site integration |
US6446083B1 (en) * | 2000-05-12 | 2002-09-03 | Vastvideo, Inc. | System and method for classifying media items |
US6865600B1 (en) * | 2000-05-19 | 2005-03-08 | Napster, Inc. | System and method for selecting internet media channels |
US6697800B1 (en) * | 2000-05-19 | 2004-02-24 | Roxio, Inc. | System and method for determining affinity using objective and subjective data |
US6876997B1 (en) | 2000-05-22 | 2005-04-05 | Overture Services, Inc. | Method and apparatus for indentifying related searches in a database search system |
US7062561B1 (en) | 2000-05-23 | 2006-06-13 | Richard Reisman | Method and apparatus for utilizing the social usage learned from multi-user feedback to improve resource identity signifier mapping |
US7096220B1 (en) | 2000-05-24 | 2006-08-22 | Reachforce, Inc. | Web-based customer prospects harvester system |
US7120629B1 (en) | 2000-05-24 | 2006-10-10 | Reachforce, Inc. | Prospects harvester system for providing contact data about customers of product or service offered by business enterprise extracting text documents selected from newsgroups, discussion forums, mailing lists, querying such data to provide customers who confirm to business profile data |
US7082427B1 (en) | 2000-05-24 | 2006-07-25 | Reachforce, Inc. | Text indexing system to index, query the archive database document by keyword data representing the content of the documents and by contact data associated with the participant who generated the document |
US7003517B1 (en) * | 2000-05-24 | 2006-02-21 | Inetprofit, Inc. | Web-based system and method for archiving and searching participant-based internet text sources for customer lead data |
US7028250B2 (en) * | 2000-05-25 | 2006-04-11 | Kanisa, Inc. | System and method for automatically classifying text |
US8290768B1 (en) | 2000-06-21 | 2012-10-16 | International Business Machines Corporation | System and method for determining a set of attributes based on content of communications |
US9699129B1 (en) | 2000-06-21 | 2017-07-04 | International Business Machines Corporation | System and method for increasing email productivity |
US6408277B1 (en) | 2000-06-21 | 2002-06-18 | Banter Limited | System and method for automatic task prioritization |
US7117215B1 (en) | 2001-06-07 | 2006-10-03 | Informatica Corporation | Method and apparatus for transporting data for data warehousing applications that incorporates analytic data interface |
US7389208B1 (en) * | 2000-06-30 | 2008-06-17 | Accord Solutions, Inc. | System and method for dynamic knowledge construction |
US6738759B1 (en) * | 2000-07-07 | 2004-05-18 | Infoglide Corporation, Inc. | System and method for performing similarity searching using pointer optimization |
WO2002006993A1 (en) * | 2000-07-17 | 2002-01-24 | Asymmetry, Inc. | System and methods for web resource discovery |
US6714941B1 (en) * | 2000-07-19 | 2004-03-30 | University Of Southern California | Learning data prototypes for information extraction |
JP2002041544A (en) * | 2000-07-25 | 2002-02-08 | Toshiba Corp | Text information analyzing device |
US6687696B2 (en) * | 2000-07-26 | 2004-02-03 | Recommind Inc. | System and method for personalized search, information filtering, and for generating recommendations utilizing statistical latent class models |
US6675159B1 (en) * | 2000-07-27 | 2004-01-06 | Science Applic Int Corp | Concept-based search and retrieval system |
US7062488B1 (en) * | 2000-08-30 | 2006-06-13 | Richard Reisman | Task/domain segmentation in applying feedback to command control |
US6647383B1 (en) * | 2000-09-01 | 2003-11-11 | Lucent Technologies Inc. | System and method for providing interactive dialogue and iterative search functions to find information |
AUPR033800A0 (en) * | 2000-09-25 | 2000-10-19 | Telstra R & D Management Pty Ltd | A document categorisation system |
US20020147578A1 (en) * | 2000-09-29 | 2002-10-10 | Lingomotors, Inc. | Method and system for query reformulation for searching of information |
US20050144114A1 (en) * | 2000-09-30 | 2005-06-30 | Ruggieri Thomas P. | System and method for providing global information on risks and related hedging strategies |
US7043531B1 (en) | 2000-10-04 | 2006-05-09 | Inetprofit, Inc. | Web-based customer lead generator system with pre-emptive profiling |
US7330850B1 (en) | 2000-10-04 | 2008-02-12 | Reachforce, Inc. | Text mining system for web-based business intelligence applied to web site server logs |
US7197470B1 (en) * | 2000-10-11 | 2007-03-27 | Buzzmetrics, Ltd. | System and method for collection analysis of electronic discussion methods |
US7185065B1 (en) | 2000-10-11 | 2007-02-27 | Buzzmetrics Ltd | System and method for scoring electronic messages |
US6694337B1 (en) * | 2000-10-26 | 2004-02-17 | Intel Corporation | Synchronizing databases |
US7027974B1 (en) | 2000-10-27 | 2006-04-11 | Science Applications International Corporation | Ontology-based parser for natural language processing |
US7200606B2 (en) * | 2000-11-07 | 2007-04-03 | The Regents Of The University Of California | Method and system for selecting documents by measuring document quality |
US6678694B1 (en) * | 2000-11-08 | 2004-01-13 | Frank Meik | Indexed, extensible, interactive document retrieval system |
US7295991B1 (en) * | 2000-11-10 | 2007-11-13 | Erc Dataplus, Inc. | Employment sourcing system |
US6640228B1 (en) * | 2000-11-10 | 2003-10-28 | Verizon Laboratories Inc. | Method for detecting incorrectly categorized data |
US7363308B2 (en) * | 2000-12-28 | 2008-04-22 | Fair Isaac Corporation | System and method for obtaining keyword descriptions of records from a large database |
US7644057B2 (en) * | 2001-01-03 | 2010-01-05 | International Business Machines Corporation | System and method for electronic communication management |
US6766316B2 (en) | 2001-01-18 | 2004-07-20 | Science Applications International Corporation | Method and system of ranking and clustering for document indexing and retrieval |
US6892189B2 (en) * | 2001-01-26 | 2005-05-10 | Inxight Software, Inc. | Method for learning and combining global and local regularities for information extraction and classification |
US7013289B2 (en) * | 2001-02-21 | 2006-03-14 | Michel Horn | Global electronic commerce system |
US20020133392A1 (en) * | 2001-02-22 | 2002-09-19 | Angel Mark A. | Distributed customer relationship management systems and methods |
US7213069B2 (en) * | 2001-02-28 | 2007-05-01 | Microsoft Corporation | Category name service able to override the category name based on requestor privilege information |
US6584470B2 (en) * | 2001-03-01 | 2003-06-24 | Intelliseek, Inc. | Multi-layered semiotic mechanism for answering natural language questions using document retrieval combined with information extraction |
US6985950B1 (en) | 2001-03-06 | 2006-01-10 | Microsoft Corporation | System for creating a space-efficient document categorizer for training and testing of automatic categorization engines |
US20020169735A1 (en) * | 2001-03-07 | 2002-11-14 | David Kil | Automatic mapping from data to preprocessing algorithms |
US20020138492A1 (en) * | 2001-03-07 | 2002-09-26 | David Kil | Data mining application with improved data mining algorithm selection |
US20020129342A1 (en) * | 2001-03-07 | 2002-09-12 | David Kil | Data mining apparatus and method with user interface based ground-truth tool and user algorithms |
US7080328B1 (en) * | 2001-03-28 | 2006-07-18 | Ebay, Inc. | Graphical user interface for filtering a population of items |
US7136846B2 (en) | 2001-04-06 | 2006-11-14 | 2005 Keel Company, Inc. | Wireless information retrieval |
US7155668B2 (en) * | 2001-04-19 | 2006-12-26 | International Business Machines Corporation | Method and system for identifying relationships between text documents and structured variables pertaining to the text documents |
US6947936B1 (en) * | 2001-04-30 | 2005-09-20 | Hewlett-Packard Development Company, L.P. | Method for a topic hierarchy classification system |
US7194483B1 (en) | 2001-05-07 | 2007-03-20 | Intelligenxia, Inc. | Method, system, and computer program product for concept-based multi-dimensional analysis of unstructured information |
US6938025B1 (en) * | 2001-05-07 | 2005-08-30 | Microsoft Corporation | Method and apparatus for automatically determining salient features for object classification |
USRE46973E1 (en) | 2001-05-07 | 2018-07-31 | Ureveal, Inc. | Method, system, and computer program product for concept-based multi-dimensional analysis of unstructured information |
US7627588B1 (en) | 2001-05-07 | 2009-12-01 | Ixreveal, Inc. | System and method for concept based analysis of unstructured data |
US7536413B1 (en) | 2001-05-07 | 2009-05-19 | Ixreveal, Inc. | Concept-based categorization of unstructured objects |
US6978266B2 (en) * | 2001-05-07 | 2005-12-20 | Microsoft Corporation | Determining a rating for a collection of documents |
US6826576B2 (en) * | 2001-05-07 | 2004-11-30 | Microsoft Corporation | Very-large-scale automatic categorizer for web content |
US6970881B1 (en) | 2001-05-07 | 2005-11-29 | Intelligenxia, Inc. | Concept-based method and system for dynamically analyzing unstructured information |
US7024400B2 (en) * | 2001-05-08 | 2006-04-04 | Sunflare Co., Ltd. | Differential LSI space-based probabilistic document classifier |
US20020169872A1 (en) * | 2001-05-14 | 2002-11-14 | Hiroshi Nomiyama | Method for arranging information, information processing apparatus, storage media and program tranmission apparatus |
US6980984B1 (en) | 2001-05-16 | 2005-12-27 | Kanisa, Inc. | Content provider systems and methods using structured data |
US7162643B1 (en) | 2001-06-15 | 2007-01-09 | Informatica Corporation | Method and system for providing transfer of analytic application data over a network |
US6820073B1 (en) * | 2001-06-20 | 2004-11-16 | Microstrategy Inc. | System and method for multiple pass cooperative processing |
US7003512B1 (en) * | 2001-06-20 | 2006-02-21 | Microstrategy, Inc. | System and method for multiple pass cooperative processing |
US7043506B1 (en) | 2001-06-28 | 2006-05-09 | Microsoft Corporation | Utility-based archiving |
US7720842B2 (en) | 2001-07-16 | 2010-05-18 | Informatica Corporation | Value-chained queries in analytic applications |
US20030033263A1 (en) * | 2001-07-31 | 2003-02-13 | Reel Two Limited | Automated learning system |
WO2003014975A1 (en) * | 2001-08-08 | 2003-02-20 | Quiver, Inc. | Document categorization engine |
US6609124B2 (en) * | 2001-08-13 | 2003-08-19 | International Business Machines Corporation | Hub for strategic intelligence |
US7257568B2 (en) * | 2001-10-16 | 2007-08-14 | Sizatola, Llc | Process and system for matching products and markets |
US7644102B2 (en) * | 2001-10-19 | 2010-01-05 | Xerox Corporation | Methods, systems, and articles of manufacture for soft hierarchical clustering of co-occurring objects |
US20030084066A1 (en) * | 2001-10-31 | 2003-05-01 | Waterman Scott A. | Device and method for assisting knowledge engineer in associating intelligence with content |
US20030115191A1 (en) * | 2001-12-17 | 2003-06-19 | Max Copperman | Efficient and cost-effective content provider for customer relationship management (CRM) or other applications |
US7206778B2 (en) | 2001-12-17 | 2007-04-17 | Knova Software Inc. | Text search ordered along one or more dimensions |
AUPR958901A0 (en) * | 2001-12-18 | 2002-01-24 | Telstra New Wave Pty Ltd | Information resource taxonomy |
US7197493B2 (en) * | 2001-12-21 | 2007-03-27 | Lifestory Productions, Inc. | Collection management database of arbitrary schema |
US7162480B2 (en) | 2001-12-26 | 2007-01-09 | Sbc Technology Resources, Inc. | Usage-based adaptable taxonomy |
US6978264B2 (en) * | 2002-01-03 | 2005-12-20 | Microsoft Corporation | System and method for performing a search and a browse on a query |
US7225183B2 (en) * | 2002-01-28 | 2007-05-29 | Ipxl, Inc. | Ontology-based information management system and method |
JP3860046B2 (en) * | 2002-02-15 | 2006-12-20 | インターナショナル・ビジネス・マシーンズ・コーポレーション | Program, system and recording medium for information processing using random sample hierarchical structure |
US6820077B2 (en) | 2002-02-22 | 2004-11-16 | Informatica Corporation | Method and system for navigating a large amount of data |
US7343372B2 (en) * | 2002-02-22 | 2008-03-11 | International Business Machines Corporation | Direct navigation for information retrieval |
US7716207B2 (en) * | 2002-02-26 | 2010-05-11 | Odom Paul S | Search engine methods and systems for displaying relevant topics |
US20060004732A1 (en) * | 2002-02-26 | 2006-01-05 | Odom Paul S | Search engine methods and systems for generating relevant search results and advertisements |
US20040098385A1 (en) * | 2002-02-26 | 2004-05-20 | Mayfield James C. | Method for indentifying term importance to sample text using reference text |
US7340466B2 (en) * | 2002-02-26 | 2008-03-04 | Kang Jo Mgmt. Limited Liability Company | Topic identification and use thereof in information retrieval systems |
US8589413B1 (en) | 2002-03-01 | 2013-11-19 | Ixreveal, Inc. | Concept-based method and system for dynamically analyzing results from search engines |
US7188107B2 (en) * | 2002-03-06 | 2007-03-06 | Infoglide Software Corporation | System and method for classification of documents |
US7031909B2 (en) * | 2002-03-12 | 2006-04-18 | Verity, Inc. | Method and system for naming a cluster of words and phrases |
KR100457375B1 (en) * | 2002-03-19 | 2004-11-16 | (주) 위즈도메인 | Method for fast searching and displaying of patent genealogical status from a patent database |
US7051009B2 (en) * | 2002-03-29 | 2006-05-23 | Hewlett-Packard Development Company, L.P. | Automatic hierarchical classification of temporal ordered case log documents for detection of changes |
JP4255239B2 (en) * | 2002-03-29 | 2009-04-15 | 富士通株式会社 | Document search method |
US20030220917A1 (en) * | 2002-04-03 | 2003-11-27 | Max Copperman | Contextual search |
US20030225763A1 (en) * | 2002-04-15 | 2003-12-04 | Microsoft Corporation | Self-improving system and method for classifying pages on the world wide web |
US7085771B2 (en) * | 2002-05-17 | 2006-08-01 | Verity, Inc | System and method for automatically discovering a hierarchy of concepts from a corpus of documents |
US7167871B2 (en) * | 2002-05-17 | 2007-01-23 | Xerox Corporation | Systems and methods for authoritativeness grading, estimation and sorting of documents in large heterogeneous document collections |
US7296020B2 (en) * | 2002-06-05 | 2007-11-13 | International Business Machines Corp | Automatic evaluation of categorization system quality |
US7165068B2 (en) * | 2002-06-12 | 2007-01-16 | Zycus Infotech Pvt Ltd. | System and method for electronic catalog classification using a hybrid of rule based and statistical method |
US7016884B2 (en) * | 2002-06-27 | 2006-03-21 | Microsoft Corporation | Probability estimate for K-nearest neighbor |
US6990485B2 (en) * | 2002-08-02 | 2006-01-24 | Hewlett-Packard Development Company, L.P. | System and method for inducing a top-down hierarchical categorizer |
US7158983B2 (en) | 2002-09-23 | 2007-01-02 | Battelle Memorial Institute | Text analysis technique |
MXPA05003249A (en) * | 2002-09-25 | 2005-07-05 | Microsoft Corp | Method and apparatus for automatically determining salient features for object classification. |
US7395499B2 (en) * | 2002-11-27 | 2008-07-01 | Accenture Global Services Gmbh | Enforcing template completion when publishing to a content management system |
US7200614B2 (en) * | 2002-11-27 | 2007-04-03 | Accenture Global Services Gmbh | Dual information system for contact center users |
US8275811B2 (en) * | 2002-11-27 | 2012-09-25 | Accenture Global Services Limited | Communicating solution information in a knowledge management system |
US20040100493A1 (en) * | 2002-11-27 | 2004-05-27 | Reid Gregory S. | Dynamically ordering solutions |
US8572058B2 (en) * | 2002-11-27 | 2013-10-29 | Accenture Global Services Limited | Presenting linked information in a CRM system |
US7769622B2 (en) * | 2002-11-27 | 2010-08-03 | Bt Group Plc | System and method for capturing and publishing insight of contact center users whose performance is above a reference key performance indicator |
US20050014116A1 (en) * | 2002-11-27 | 2005-01-20 | Reid Gregory S. | Testing information comprehension of contact center users |
US7418403B2 (en) | 2002-11-27 | 2008-08-26 | Bt Group Plc | Content feedback in a multiple-owner content management system |
US7062505B2 (en) * | 2002-11-27 | 2006-06-13 | Accenture Global Services Gmbh | Content management system for the telecommunications industry |
US9396473B2 (en) * | 2002-11-27 | 2016-07-19 | Accenture Global Services Limited | Searching within a contact center portal |
US7320000B2 (en) * | 2002-12-04 | 2008-01-15 | International Business Machines Corporation | Method and apparatus for populating a predefined concept hierarchy or other hierarchical set of classified data items by minimizing system entrophy |
US7035838B2 (en) * | 2002-12-06 | 2006-04-25 | General Electric Company | Methods and systems for organizing information stored within a computer network-based system |
US8037496B1 (en) | 2002-12-27 | 2011-10-11 | At&T Intellectual Property Ii, L.P. | System and method for automatically authoring interactive television content |
US20040133560A1 (en) * | 2003-01-07 | 2004-07-08 | Simske Steven J. | Methods and systems for organizing electronic documents |
US6961733B2 (en) * | 2003-03-10 | 2005-11-01 | Unisys Corporation | System and method for storing and accessing data in an interlocking trees datastore |
US7107264B2 (en) * | 2003-04-04 | 2006-09-12 | Yahoo, Inc. | Content bridge for associating host content and guest content wherein guest content is determined by search |
US7917483B2 (en) * | 2003-04-24 | 2011-03-29 | Affini, Inc. | Search engine and method with improved relevancy, scope, and timeliness |
US8495002B2 (en) | 2003-05-06 | 2013-07-23 | International Business Machines Corporation | Software tool for training and testing a knowledge base |
US20050187913A1 (en) | 2003-05-06 | 2005-08-25 | Yoram Nelken | Web-based customer service interface |
US6985920B2 (en) * | 2003-06-23 | 2006-01-10 | Protego Networks Inc. | Method and system for determining intra-session event correlation across network address translation devices |
US7644365B2 (en) * | 2003-09-12 | 2010-01-05 | Cisco Technology, Inc. | Method and system for displaying network security incidents |
US20060101018A1 (en) * | 2004-11-08 | 2006-05-11 | Mazzagatti Jane C | Method for processing new sequences being recorded into an interlocking trees datastore |
US8516004B2 (en) * | 2003-09-19 | 2013-08-20 | Unisys Corporation | Method for processing K node count fields using an intensity variable |
US7313574B2 (en) | 2003-10-02 | 2007-12-25 | Nokia Corporation | Method for clustering and querying media items |
US7421458B1 (en) | 2003-10-16 | 2008-09-02 | Informatica Corporation | Querying, versioning, and dynamic deployment of database objects |
US7319998B2 (en) * | 2003-11-14 | 2008-01-15 | Universidade De Coimbra | Method and system for supporting symbolic serendipity |
US7444403B1 (en) | 2003-11-25 | 2008-10-28 | Microsoft Corporation | Detecting sexually predatory content in an electronic communication |
US20050114313A1 (en) * | 2003-11-26 | 2005-05-26 | Campbell Christopher S. | System and method for retrieving documents or sub-documents based on examples |
DE10356127A1 (en) * | 2003-12-02 | 2005-07-07 | Robert Bosch Gmbh | Device for controlling a memory |
US7254590B2 (en) * | 2003-12-03 | 2007-08-07 | Informatica Corporation | Set-oriented real-time data processing based on transaction boundaries |
US20050125280A1 (en) * | 2003-12-05 | 2005-06-09 | Hewlett-Packard Development Company, L.P. | Real-time aggregation and scoring in an information handling system |
US7340471B2 (en) | 2004-01-16 | 2008-03-04 | Unisys Corporation | Saving and restoring an interlocking trees datastore |
US20050177599A1 (en) * | 2004-02-09 | 2005-08-11 | Microsoft Corporation | System and method for complying with anti-spam rules, laws, and regulations |
CA2556023A1 (en) * | 2004-02-20 | 2005-09-09 | Dow Jones Reuters Business Interactive, Llc | Intelligent search and retrieval system and method |
US7725414B2 (en) | 2004-03-16 | 2010-05-25 | Buzzmetrics, Ltd An Israel Corporation | Method for developing a classifier for classifying communications |
US7565369B2 (en) * | 2004-05-28 | 2009-07-21 | International Business Machines Corporation | System and method for mining time-changing data streams |
US7660463B2 (en) * | 2004-06-03 | 2010-02-09 | Microsoft Corporation | Foreground extraction using iterated graph cuts |
JP4254623B2 (en) * | 2004-06-09 | 2009-04-15 | 日本電気株式会社 | Topic analysis method, apparatus thereof, and program |
US7313575B2 (en) * | 2004-06-14 | 2007-12-25 | Hewlett-Packard Development Company, L.P. | Data services handler |
US7593923B1 (en) | 2004-06-29 | 2009-09-22 | Unisys Corporation | Functional operations for accessing and/or building interlocking trees datastores to enable their use with applications software |
US7428530B2 (en) * | 2004-07-01 | 2008-09-23 | Microsoft Corporation | Dispersing search engine results by using page category information |
US7698333B2 (en) | 2004-07-22 | 2010-04-13 | Factiva, Inc. | Intelligent query system and method using phrase-code frequency-inverse phrase-code document frequency module |
US7383260B2 (en) * | 2004-08-03 | 2008-06-03 | International Business Machines Corporation | Method and apparatus for ontology-based classification of media content |
US20060029275A1 (en) * | 2004-08-06 | 2006-02-09 | Microsoft Corporation | Systems and methods for image data separation |
US7917480B2 (en) | 2004-08-13 | 2011-03-29 | Google Inc. | Document compression system and method for use with tokenspace repository |
US8407239B2 (en) | 2004-08-13 | 2013-03-26 | Google Inc. | Multi-stage query processing system and method for use with tokenspace repository |
US7680648B2 (en) * | 2004-09-30 | 2010-03-16 | Google Inc. | Methods and systems for improving text segmentation |
US8051096B1 (en) | 2004-09-30 | 2011-11-01 | Google Inc. | Methods and systems for augmenting a token lexicon |
US7996208B2 (en) * | 2004-09-30 | 2011-08-09 | Google Inc. | Methods and systems for selecting a language for text segmentation |
US7523085B2 (en) * | 2004-09-30 | 2009-04-21 | Buzzmetrics, Ltd An Israel Corporation | Topical sentiments in electronically stored communications |
US7213041B2 (en) * | 2004-10-05 | 2007-05-01 | Unisys Corporation | Saving and restoring an interlocking trees datastore |
US7765178B1 (en) * | 2004-10-06 | 2010-07-27 | Shopzilla, Inc. | Search ranking estimation |
US7716241B1 (en) | 2004-10-27 | 2010-05-11 | Unisys Corporation | Storing the repository origin of data inputs within a knowledge store |
US7908240B1 (en) | 2004-10-28 | 2011-03-15 | Unisys Corporation | Facilitated use of column and field data for field record universe in a knowledge store |
US7499932B2 (en) * | 2004-11-08 | 2009-03-03 | Unisys Corporation | Accessing data in an interlocking trees data structure using an application programming interface |
US20060101048A1 (en) * | 2004-11-08 | 2006-05-11 | Mazzagatti Jane C | KStore data analyzer |
US20060100845A1 (en) * | 2004-11-08 | 2006-05-11 | Mazzagatti Jane C | Multiple stream real time data simulation adapted for a KStore data structure |
US7348980B2 (en) * | 2004-11-08 | 2008-03-25 | Unisys Corporation | Method and apparatus for interface for graphic display of data from a Kstore |
US20060155530A1 (en) * | 2004-12-14 | 2006-07-13 | International Business Machines Corporation | Method and apparatus for generation of text documents |
US10002325B2 (en) | 2005-03-30 | 2018-06-19 | Primal Fusion Inc. | Knowledge representation systems and methods incorporating inference rules |
US7849090B2 (en) * | 2005-03-30 | 2010-12-07 | Primal Fusion Inc. | System, method and computer program for faceted classification synthesis |
US9104779B2 (en) | 2005-03-30 | 2015-08-11 | Primal Fusion Inc. | Systems and methods for analyzing and synthesizing complex knowledge representations |
US7596574B2 (en) * | 2005-03-30 | 2009-09-29 | Primal Fusion, Inc. | Complex-adaptive system for providing a facted classification |
US7844565B2 (en) | 2005-03-30 | 2010-11-30 | Primal Fusion Inc. | System, method and computer program for using a multi-tiered knowledge representation model |
US9177248B2 (en) | 2005-03-30 | 2015-11-03 | Primal Fusion Inc. | Knowledge representation systems and methods incorporating customization |
US7606781B2 (en) * | 2005-03-30 | 2009-10-20 | Primal Fusion Inc. | System, method and computer program for facet analysis |
US8849860B2 (en) | 2005-03-30 | 2014-09-30 | Primal Fusion Inc. | Systems and methods for applying statistical inference techniques to knowledge representations |
US9378203B2 (en) | 2008-05-01 | 2016-06-28 | Primal Fusion Inc. | Methods and apparatus for providing information of interest to one or more users |
US7546294B2 (en) * | 2005-03-31 | 2009-06-09 | Microsoft Corporation | Automated relevance tuning |
US7409380B1 (en) | 2005-04-07 | 2008-08-05 | Unisys Corporation | Facilitated reuse of K locations in a knowledge store |
US7809736B2 (en) * | 2005-05-02 | 2010-10-05 | Brown University | Importance ranking for a hierarchical collection of objects |
US20060253423A1 (en) * | 2005-05-07 | 2006-11-09 | Mclane Mark | Information retrieval system and method |
US7389301B1 (en) | 2005-06-10 | 2008-06-17 | Unisys Corporation | Data aggregation user interface and analytic adapted for a KStore |
US9158855B2 (en) | 2005-06-16 | 2015-10-13 | Buzzmetrics, Ltd | Extracting structured data from weblogs |
US20060287980A1 (en) * | 2005-06-21 | 2006-12-21 | Microsoft Corporation | Intelligent search results blending |
US7698335B1 (en) * | 2005-06-27 | 2010-04-13 | Microsoft Corporation | Cluster organization of electronically-stored items |
US7403932B2 (en) * | 2005-07-01 | 2008-07-22 | The Boeing Company | Text differentiation methods, systems, and computer program products for content analysis |
CN100585591C (en) * | 2005-07-15 | 2010-01-27 | 国际商业机器公司 | Interacting viewing system and method |
US20070100779A1 (en) * | 2005-08-05 | 2007-05-03 | Ori Levy | Method and system for extracting web data |
US7882262B2 (en) * | 2005-08-18 | 2011-02-01 | Cisco Technology, Inc. | Method and system for inline top N query computation |
US7966395B1 (en) | 2005-08-23 | 2011-06-21 | Amazon Technologies, Inc. | System and method for indicating interest of online content |
US7774335B1 (en) * | 2005-08-23 | 2010-08-10 | Amazon Technologies, Inc. | Method and system for determining interest levels of online content navigation paths |
US20070073745A1 (en) * | 2005-09-23 | 2007-03-29 | Applied Linguistics, Llc | Similarity metric for semantic profiling |
US7620607B1 (en) * | 2005-09-26 | 2009-11-17 | Quintura Inc. | System and method for using a bidirectional neural network to identify sentences for use as document annotations |
WO2007038713A2 (en) * | 2005-09-28 | 2007-04-05 | Epacris Inc. | Search engine determining results based on probabilistic scoring of relevance |
CA2625493C (en) * | 2005-10-11 | 2014-12-16 | Intelligenxia Inc. | System, method & computer program product for concept based searching & analysis |
US7917519B2 (en) * | 2005-10-26 | 2011-03-29 | Sizatola, Llc | Categorized document bases |
WO2007064639A2 (en) * | 2005-11-29 | 2007-06-07 | Scientigo, Inc. | Methods and systems for providing personalized contextual search results |
US7502765B2 (en) | 2005-12-21 | 2009-03-10 | International Business Machines Corporation | Method for organizing semi-structured data into a taxonomy, based on tag-separated clustering |
US7657522B1 (en) * | 2006-01-12 | 2010-02-02 | Recommind, Inc. | System and method for providing information navigation and filtration |
US7747631B1 (en) | 2006-01-12 | 2010-06-29 | Recommind, Inc. | System and method for establishing relevance of objects in an enterprise system |
WO2007084616A2 (en) * | 2006-01-18 | 2007-07-26 | Ilial, Inc. | System and method for context-based knowledge search, tagging, collaboration, management and advertisement |
US7676485B2 (en) * | 2006-01-20 | 2010-03-09 | Ixreveal, Inc. | Method and computer program product for converting ontologies into concept semantic networks |
US20070203865A1 (en) * | 2006-02-09 | 2007-08-30 | Hirsch Martin C | Apparatus and methods for an item retrieval system |
US20070195776A1 (en) * | 2006-02-23 | 2007-08-23 | Zheng Danyang R | System and method for channeling network traffic |
US8019763B2 (en) * | 2006-02-27 | 2011-09-13 | Microsoft Corporation | Propagating relevance from labeled documents to unlabeled documents |
US20070214153A1 (en) * | 2006-03-10 | 2007-09-13 | Mazzagatti Jane C | Method for processing an input particle stream for creating upper levels of KStore |
US7461289B2 (en) * | 2006-03-16 | 2008-12-02 | Honeywell International Inc. | System and method for computer service security |
US7734571B2 (en) * | 2006-03-20 | 2010-06-08 | Unisys Corporation | Method for processing sensor data within a particle stream by a KStore |
US20070220069A1 (en) * | 2006-03-20 | 2007-09-20 | Mazzagatti Jane C | Method for processing an input particle stream for creating lower levels of a KStore |
US20080275842A1 (en) * | 2006-03-20 | 2008-11-06 | Jane Campbell Mazzagatti | Method for processing counts when an end node is encountered |
US7689571B1 (en) | 2006-03-24 | 2010-03-30 | Unisys Corporation | Optimizing the size of an interlocking tree datastore structure for KStore |
US7720848B2 (en) * | 2006-03-29 | 2010-05-18 | Xerox Corporation | Hierarchical clustering with real-time updating |
US8238351B2 (en) * | 2006-04-04 | 2012-08-07 | Unisys Corporation | Method for determining a most probable K location |
US7478075B2 (en) * | 2006-04-11 | 2009-01-13 | Sun Microsystems, Inc. | Reducing the size of a training set for classification |
US20070255732A1 (en) * | 2006-04-27 | 2007-11-01 | Moss Barrie J | Method and Apparatus for Implementing a Semantic Environment Including Multi-Search Term Storage and Retrieval of Data and Content |
US7676330B1 (en) | 2006-05-16 | 2010-03-09 | Unisys Corporation | Method for processing a particle using a sensor structure |
US8233388B2 (en) * | 2006-05-30 | 2012-07-31 | Cisco Technology, Inc. | System and method for controlling and tracking network content flow |
US8370127B2 (en) * | 2006-06-16 | 2013-02-05 | Nuance Communications, Inc. | Systems and methods for building asset based natural language call routing application with limited resources |
US20070294223A1 (en) * | 2006-06-16 | 2007-12-20 | Technion Research And Development Foundation Ltd. | Text Categorization Using External Knowledge |
US7809723B2 (en) * | 2006-06-26 | 2010-10-05 | Microsoft Corporation | Distributed hierarchical text classification framework |
EP1876540A1 (en) * | 2006-07-06 | 2008-01-09 | British Telecommunications Public Limited Company | Organising and storing documents |
US7519619B2 (en) * | 2006-08-21 | 2009-04-14 | Microsoft Corporation | Facilitating document classification using branch associations |
US7660783B2 (en) * | 2006-09-27 | 2010-02-09 | Buzzmetrics, Inc. | System and method of ad-hoc analysis of data |
JP2010506308A (en) * | 2006-10-03 | 2010-02-25 | キューピーエス テック. リミテッド ライアビリティ カンパニー | Mechanism for automatic matching of host content and guest content by categorization |
US20080091423A1 (en) * | 2006-10-13 | 2008-04-17 | Shourya Roy | Generation of domain models from noisy transcriptions |
US20080120294A1 (en) * | 2006-11-17 | 2008-05-22 | X.Com.Inc | Computer-implemented systems and methods for media asset searching and access |
US8676802B2 (en) | 2006-11-30 | 2014-03-18 | Oracle Otc Subsidiary Llc | Method and system for information retrieval with clustering |
US20080172380A1 (en) * | 2007-01-17 | 2008-07-17 | Wojciech Czyz | Information retrieval based on information location in the information space. |
US7873583B2 (en) * | 2007-01-19 | 2011-01-18 | Microsoft Corporation | Combining resilient classifiers |
US7603348B2 (en) * | 2007-01-26 | 2009-10-13 | Yahoo! Inc. | System for classifying a search query |
US7853595B2 (en) * | 2007-01-30 | 2010-12-14 | The Boeing Company | Method and apparatus for creating a tool for generating an index for a document |
US7610185B1 (en) * | 2007-02-26 | 2009-10-27 | Quintura, Inc. | GUI for subject matter navigation using maps and search terms |
US7529743B1 (en) * | 2007-02-26 | 2009-05-05 | Quintura, Inc. | GUI for subject matter navigation using maps and search terms |
EP1973045A1 (en) * | 2007-03-20 | 2008-09-24 | British Telecommunications Public Limited Company | Organising and storing documents |
US20080270451A1 (en) * | 2007-04-24 | 2008-10-30 | Interse A/S | System and Method of Generating a Metadata Model for Use in Classifying and Searching for Information Objects Maintained in Heterogeneous Data Stores |
US20080301033A1 (en) * | 2007-06-01 | 2008-12-04 | Netseer, Inc. | Method and apparatus for optimizing long term revenues in online auctions |
US8935249B2 (en) | 2007-06-26 | 2015-01-13 | Oracle Otc Subsidiary Llc | Visualization of concepts within a collection of information |
US8005643B2 (en) * | 2007-06-26 | 2011-08-23 | Endeca Technologies, Inc. | System and method for measuring the quality of document sets |
US20090028164A1 (en) * | 2007-07-23 | 2009-01-29 | Semgine, Gmbh | Method and apparatus for semantic serializing |
CA2595541A1 (en) * | 2007-07-26 | 2009-01-26 | Hamid Htami-Hanza | Assisted knowledge discovery and publication system and method |
US20090089381A1 (en) * | 2007-09-28 | 2009-04-02 | Microsoft Corporation | Pending and exclusive electronic mail inbox |
US8671104B2 (en) * | 2007-10-12 | 2014-03-11 | Palo Alto Research Center Incorporated | System and method for providing orientation into digital information |
US8165985B2 (en) * | 2007-10-12 | 2012-04-24 | Palo Alto Research Center Incorporated | System and method for performing discovery of digital information in a subject area |
US8073682B2 (en) * | 2007-10-12 | 2011-12-06 | Palo Alto Research Center Incorporated | System and method for prospecting digital information |
US7415460B1 (en) | 2007-12-10 | 2008-08-19 | International Business Machines Corporation | System and method to customize search engine results by picking documents |
US8347326B2 (en) | 2007-12-18 | 2013-01-01 | The Nielsen Company (US) | Identifying key media events and modeling causal relationships between key events and reported feelings |
US20090164418A1 (en) * | 2007-12-19 | 2009-06-25 | Valentina Pulnikova | Retrieval system and method of searching information in the Internet |
US8150869B2 (en) * | 2008-03-17 | 2012-04-03 | Microsoft Corporation | Combined web browsing and searching |
JP5530425B2 (en) | 2008-05-01 | 2014-06-25 | プライマル フュージョン インコーポレイテッド | Method, system, and computer program for dynamic generation of user-driven semantic networks and media integration |
US9361365B2 (en) | 2008-05-01 | 2016-06-07 | Primal Fusion Inc. | Methods and apparatus for searching of content using semantic synthesis |
US8676732B2 (en) | 2008-05-01 | 2014-03-18 | Primal Fusion Inc. | Methods and apparatus for providing information of interest to one or more users |
US8082248B2 (en) * | 2008-05-29 | 2011-12-20 | Rania Abouyounes | Method and system for document classification based on document structure and written style |
US20090300009A1 (en) * | 2008-05-30 | 2009-12-03 | Netseer, Inc. | Behavioral Targeting For Tracking, Aggregating, And Predicting Online Behavior |
US8010544B2 (en) * | 2008-06-06 | 2011-08-30 | Yahoo! Inc. | Inverted indices in information extraction to improve records extracted per annotation |
US8145654B2 (en) * | 2008-06-20 | 2012-03-27 | Lexisnexis Group | Systems and methods for document searching |
US9268843B2 (en) * | 2008-06-27 | 2016-02-23 | Cbs Interactive Inc. | Personalization engine for building a user profile |
US8214346B2 (en) | 2008-06-27 | 2012-07-03 | Cbs Interactive Inc. | Personalization engine for classifying unstructured documents |
US20090327210A1 (en) * | 2008-06-27 | 2009-12-31 | Microsoft Corporation | Advanced book page classification engine and index page extraction |
US20100023319A1 (en) * | 2008-07-28 | 2010-01-28 | International Business Machines Corporation | Model-driven feedback for annotation |
US8209616B2 (en) * | 2008-08-28 | 2012-06-26 | Palo Alto Research Center Incorporated | System and method for interfacing a web browser widget with social indexing |
US8010545B2 (en) * | 2008-08-28 | 2011-08-30 | Palo Alto Research Center Incorporated | System and method for providing a topic-directed search |
US20100057536A1 (en) * | 2008-08-28 | 2010-03-04 | Palo Alto Research Center Incorporated | System And Method For Providing Community-Based Advertising Term Disambiguation |
US20100057577A1 (en) * | 2008-08-28 | 2010-03-04 | Palo Alto Research Center Incorporated | System And Method For Providing Topic-Guided Broadening Of Advertising Targets In Social Indexing |
CA2734756C (en) * | 2008-08-29 | 2018-08-21 | Primal Fusion Inc. | Systems and methods for semantic concept definition and semantic concept relationship synthesis utilizing existing domain definitions |
US8364663B2 (en) * | 2008-09-05 | 2013-01-29 | Microsoft Corporation | Tokenized javascript indexing system |
US7730061B2 (en) * | 2008-09-12 | 2010-06-01 | International Business Machines Corporation | Fast-approximate TFIDF |
US8417695B2 (en) * | 2008-10-30 | 2013-04-09 | Netseer, Inc. | Identifying related concepts of URLs and domain names |
US8549016B2 (en) * | 2008-11-14 | 2013-10-01 | Palo Alto Research Center Incorporated | System and method for providing robust topic identification in social indexes |
US20100185672A1 (en) * | 2009-01-21 | 2010-07-22 | Rising Iii Hawley K | Techniques for spatial representation of data and browsing based on similarity |
US8239397B2 (en) * | 2009-01-27 | 2012-08-07 | Palo Alto Research Center Incorporated | System and method for managing user attention by detecting hot and cold topics in social indexes |
US8452781B2 (en) * | 2009-01-27 | 2013-05-28 | Palo Alto Research Center Incorporated | System and method for using banded topic relevance and time for article prioritization |
US8356044B2 (en) * | 2009-01-27 | 2013-01-15 | Palo Alto Research Center Incorporated | System and method for providing default hierarchical training for social indexing |
US8527658B2 (en) * | 2009-04-07 | 2013-09-03 | Verisign, Inc | Domain traffic ranking |
US9292612B2 (en) | 2009-04-22 | 2016-03-22 | Verisign, Inc. | Internet profile service |
US9245243B2 (en) * | 2009-04-14 | 2016-01-26 | Ureveal, Inc. | Concept-based analysis of structured and unstructured data using concept inheritance |
US8370504B2 (en) * | 2009-07-15 | 2013-02-05 | Verisign, Inc. | Method and system for predicting domain name registration renewal probability |
US9002857B2 (en) * | 2009-08-13 | 2015-04-07 | Charite-Universitatsmedizin Berlin | Methods for searching with semantic similarity scores in one or more ontologies |
CN102012900B (en) * | 2009-09-04 | 2013-01-30 | 阿里巴巴集团控股有限公司 | An information retrieval method and system |
US20110060645A1 (en) * | 2009-09-08 | 2011-03-10 | Peter Sweeney | Synthesizing messaging using context provided by consumers |
US9292855B2 (en) * | 2009-09-08 | 2016-03-22 | Primal Fusion Inc. | Synthesizing messaging using context provided by consumers |
US20110060644A1 (en) * | 2009-09-08 | 2011-03-10 | Peter Sweeney | Synthesizing messaging using context provided by consumers |
US8965893B2 (en) * | 2009-10-15 | 2015-02-24 | Rogers Communications Inc. | System and method for grouping multiple streams of data |
US9262520B2 (en) | 2009-11-10 | 2016-02-16 | Primal Fusion Inc. | System, method and computer program for creating and manipulating data structures using an interactive graphical interface |
US8392175B2 (en) | 2010-02-01 | 2013-03-05 | Stratify, Inc. | Phrase-based document clustering with automatic phrase extraction |
US8296290B2 (en) * | 2010-02-05 | 2012-10-23 | Fti Consulting, Inc. | System and method for propagating classification decisions |
US8392432B2 (en) * | 2010-04-12 | 2013-03-05 | Microsoft Corporation | Make and model classifier |
US9031944B2 (en) | 2010-04-30 | 2015-05-12 | Palo Alto Research Center Incorporated | System and method for providing multi-core and multi-level topical organization in social indexes |
US20110289025A1 (en) * | 2010-05-19 | 2011-11-24 | Microsoft Corporation | Learning user intent from rule-based training data |
US8874727B2 (en) | 2010-05-31 | 2014-10-28 | The Nielsen Company (Us), Llc | Methods, apparatus, and articles of manufacture to rank users in an online social network |
US10474647B2 (en) | 2010-06-22 | 2019-11-12 | Primal Fusion Inc. | Methods and devices for customizing knowledge representation systems |
US9235806B2 (en) | 2010-06-22 | 2016-01-12 | Primal Fusion Inc. | Methods and devices for customizing knowledge representation systems |
WO2012049883A1 (en) * | 2010-10-15 | 2012-04-19 | 日本電気株式会社 | Data structure, index creation device, data search device, index creation method, data search method, and computer-readable recording medium |
US11294977B2 (en) | 2011-06-20 | 2022-04-05 | Primal Fusion Inc. | Techniques for presenting content to a user based on the user's preferences |
US9092516B2 (en) | 2011-06-20 | 2015-07-28 | Primal Fusion Inc. | Identifying information of interest based on user preferences |
US8527450B2 (en) * | 2011-03-24 | 2013-09-03 | Yahoo! Inc. | Apparatus and methods for analyzing and using short messages from commercial accounts |
US9208438B2 (en) * | 2011-08-02 | 2015-12-08 | Cavium, Inc. | Duplication in decision trees |
US9183244B2 (en) | 2011-08-02 | 2015-11-10 | Cavium, Inc. | Rule modification in decision trees |
US10229139B2 (en) | 2011-08-02 | 2019-03-12 | Cavium, Llc | Incremental update heuristics |
KR101476114B1 (en) | 2011-08-02 | 2014-12-23 | 캐비엄, 인코포레이티드 | Lookup front end input processor |
US8954436B2 (en) * | 2012-01-26 | 2015-02-10 | International Business Machines Corporation | Monitoring content repositories, identifying misclassified content objects, and suggesting reclassification |
US20130238608A1 (en) * | 2012-03-07 | 2013-09-12 | Microsoft Corporation | Search results by mapping associated with disparate taxonomies |
US9069798B2 (en) | 2012-05-24 | 2015-06-30 | Mitsubishi Electric Research Laboratories, Inc. | Method of text classification using discriminative topic transformation |
US10261938B1 (en) | 2012-08-31 | 2019-04-16 | Amazon Technologies, Inc. | Content preloading using predictive models |
US20140214835A1 (en) * | 2013-01-29 | 2014-07-31 | Richard Thomas Oehrle | System and method for automatically classifying documents |
US10083200B2 (en) | 2013-03-14 | 2018-09-25 | Cavium, Inc. | Batch incremental update |
US9195939B1 (en) | 2013-03-15 | 2015-11-24 | Cavium, Inc. | Scope in decision trees |
US9298814B2 (en) | 2013-03-15 | 2016-03-29 | Maritz Holdings Inc. | Systems and methods for classifying electronic documents |
US9595003B1 (en) | 2013-03-15 | 2017-03-14 | Cavium, Inc. | Compiler with mask nodes |
US10229144B2 (en) | 2013-03-15 | 2019-03-12 | Cavium, Llc | NSP manager |
US9336332B2 (en) | 2013-08-28 | 2016-05-10 | Clipcard Inc. | Programmatic data discovery platforms for computing applications |
US9659045B2 (en) * | 2013-11-08 | 2017-05-23 | Oracle International Corporation | Generic indexing for efficiently supporting ad-hoc query over hierarchically marked-up data |
US10157239B2 (en) | 2013-12-23 | 2018-12-18 | Oracle International Corporation | Finding common neighbors between two nodes in a graph |
US9544402B2 (en) | 2013-12-31 | 2017-01-10 | Cavium, Inc. | Multi-rule approach to encoding a group of rules |
US9275336B2 (en) | 2013-12-31 | 2016-03-01 | Cavium, Inc. | Method and system for skipping over group(s) of rules based on skip group rule |
US9667446B2 (en) | 2014-01-08 | 2017-05-30 | Cavium, Inc. | Condition code approach for comparing rule and packet data that are provided in portions |
US10255355B2 (en) * | 2014-05-28 | 2019-04-09 | Battelle Memorial Institute | Method and system for information retrieval and aggregation from inferred user reasoning |
US9881072B2 (en) * | 2014-08-14 | 2018-01-30 | McAFEE, LLC. | Dynamic feature set management |
US9646512B2 (en) * | 2014-10-24 | 2017-05-09 | Lingualeo, Inc. | System and method for automated teaching of languages based on frequency of syntactic models |
CN106202124B (en) * | 2015-05-08 | 2019-12-31 | 广州市动景计算机科技有限公司 | Webpage classification method and device |
CN104866606B (en) * | 2015-06-02 | 2019-02-01 | 浙江师范大学 | A kind of MapReduce parallelization big data file classification method |
US10726060B1 (en) * | 2015-06-24 | 2020-07-28 | Amazon Technologies, Inc. | Classification accuracy estimation |
CN105956610B (en) * | 2016-04-22 | 2019-02-22 | 中国人民解放军军事医学科学院卫生装备研究所 | A kind of remote sensing images classification of landform method based on multi-layer coding structure |
US10313348B2 (en) * | 2016-09-19 | 2019-06-04 | Fortinet, Inc. | Document classification by a hybrid classifier |
US11158012B1 (en) | 2017-02-14 | 2021-10-26 | Casepoint LLC | Customizing a data discovery user interface based on artificial intelligence |
US11275794B1 (en) * | 2017-02-14 | 2022-03-15 | Casepoint LLC | CaseAssist story designer |
US10740557B1 (en) | 2017-02-14 | 2020-08-11 | Casepoint LLC | Technology platform for data discovery |
CN108959295B (en) | 2017-05-19 | 2021-04-16 | 腾讯科技(深圳)有限公司 | Method and device for identifying native object |
CN109145108A (en) * | 2017-06-16 | 2019-01-04 | 贵州小爱机器人科技有限公司 | Classifier training method, classification method, device and computer equipment is laminated in text |
US20190096525A1 (en) * | 2017-09-26 | 2019-03-28 | Owned Outcomes Inc. | Patient data management system |
US10509782B2 (en) * | 2017-12-11 | 2019-12-17 | Sap Se | Machine learning based enrichment of database objects |
US10866989B1 (en) * | 2018-04-06 | 2020-12-15 | Architecture Technology Corporation | Real time recommender modeling system, methods of construction, and methods of use |
US11715042B1 (en) | 2018-04-20 | 2023-08-01 | Meta Platforms Technologies, Llc | Interpretability of deep reinforcement learning models in assistant systems |
US11307880B2 (en) | 2018-04-20 | 2022-04-19 | Meta Platforms, Inc. | Assisting users with personalized and contextual communication content |
US11676220B2 (en) | 2018-04-20 | 2023-06-13 | Meta Platforms, Inc. | Processing multimodal user input for assistant systems |
US20190327330A1 (en) | 2018-04-20 | 2019-10-24 | Facebook, Inc. | Building Customized User Profiles Based on Conversational Data |
US11886473B2 (en) | 2018-04-20 | 2024-01-30 | Meta Platforms, Inc. | Intent identification for agent matching by assistant systems |
JP2019204246A (en) * | 2018-05-23 | 2019-11-28 | 株式会社日立製作所 | Learning data creation method and learning data creation device |
US11226955B2 (en) | 2018-06-28 | 2022-01-18 | Oracle International Corporation | Techniques for enabling and integrating in-memory semi-structured data and text document searches with in-memory columnar query processing |
US20200012930A1 (en) * | 2018-07-06 | 2020-01-09 | Global Elmeast Inc. | Techniques for knowledge neuron enhancements |
US10395169B1 (en) | 2018-07-06 | 2019-08-27 | Global Elmeast Inc. | Self learning neural knowledge artifactory for autonomous decision making |
US10311058B1 (en) | 2018-07-06 | 2019-06-04 | Global Elmeast Inc. | Techniques for processing neural queries |
US11610107B2 (en) | 2018-07-06 | 2023-03-21 | Global Elmeast Inc. | Methodology to automatically incorporate feedback to enable self learning in neural learning artifactories |
WO2020059123A1 (en) * | 2018-09-21 | 2020-03-26 | 富士通株式会社 | Determination method and determination program |
US11157478B2 (en) | 2018-12-28 | 2021-10-26 | Oracle International Corporation | Technique of comprehensively support autonomous JSON document object (AJD) cloud service |
US11610277B2 (en) | 2019-01-25 | 2023-03-21 | Open Text Holdings, Inc. | Seamless electronic discovery system with an enterprise data portal |
US11562592B2 (en) | 2019-01-28 | 2023-01-24 | International Business Machines Corporation | Document retrieval through assertion analysis on entities and document fragments |
CN111967515A (en) * | 2020-08-14 | 2020-11-20 | Oppo广东移动通信有限公司 | Image information extraction method, training method and device, medium and electronic equipment |
US11640380B2 (en) | 2021-03-10 | 2023-05-02 | Oracle International Corporation | Technique of comprehensively supporting multi-value, multi-field, multilevel, multi-position functional index over stored aggregately stored data in RDBMS |
CN113378907B (en) * | 2021-06-04 | 2024-01-09 | 南京大学 | Automated software traceability recovery method for enhancing data preprocessing process |
Family Cites Families (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH021057A (en) | 1988-01-20 | 1990-01-05 | Ricoh Co Ltd | Document retrieving device |
US4975975A (en) | 1988-05-26 | 1990-12-04 | Gtx Corporation | Hierarchical parametric apparatus and method for recognizing drawn characters |
DE69032712T2 (en) | 1989-06-14 | 1999-07-01 | Hitachi Ltd | HIERARCHICAL PRE-SEARCH TYPE DOCUMENT SEARCH METHOD, DEVICE THEREFOR, AND A MAGNETIC DISK ARRANGEMENT FOR THIS DEVICE |
US5469354A (en) | 1989-06-14 | 1995-11-21 | Hitachi, Ltd. | Document data processing method and apparatus for document retrieval |
JPH03129472A (en) * | 1989-07-31 | 1991-06-03 | Ricoh Co Ltd | Processing method for document retrieving device |
US5325298A (en) | 1990-11-07 | 1994-06-28 | Hnc, Inc. | Methods for generating or revising context vectors for a plurality of word stems |
US5317507A (en) | 1990-11-07 | 1994-05-31 | Gallant Stephen I | Method for document retrieval and for word sense disambiguation using neural networks |
US5557794A (en) | 1991-09-02 | 1996-09-17 | Fuji Xerox Co., Ltd. | Data management system for a personal data base |
JP3303926B2 (en) | 1991-09-27 | 2002-07-22 | 富士ゼロックス株式会社 | Structured document classification apparatus and method |
US5428778A (en) | 1992-02-13 | 1995-06-27 | Office Express Pty. Ltd. | Selective dissemination of information |
US5659724A (en) | 1992-11-06 | 1997-08-19 | Ncr | Interactive data analysis apparatus employing a knowledge base |
EP0602932B1 (en) * | 1992-12-18 | 2001-03-14 | Raytheon Company | Improved pattern recognition system for sonar and other applications |
JP3175399B2 (en) | 1993-05-18 | 2001-06-11 | セイコーエプソン株式会社 | Card data management device |
US5506984A (en) | 1993-06-30 | 1996-04-09 | Digital Equipment Corporation | Method and system for data retrieval in a distributed system using linked location references on a plurality of nodes |
JP3053153B2 (en) | 1993-09-20 | 2000-06-19 | 株式会社日立製作所 | How to start application of document management system |
US5576954A (en) | 1993-11-05 | 1996-11-19 | University Of Central Florida | Process for determination of text relevancy |
US5625767A (en) * | 1995-03-13 | 1997-04-29 | Bartell; Brian | Method and system for two-dimensional visualization of an information taxonomy and of text documents based on topical content of the documents |
DE69624755T2 (en) | 1995-05-22 | 2003-04-24 | Matsushita Electric Ind Co Ltd | Information search device for searching text to retrieve strings that match a keyword |
US5675710A (en) * | 1995-06-07 | 1997-10-07 | Lucent Technologies, Inc. | Method and apparatus for training a text classifier |
US5918240A (en) * | 1995-06-28 | 1999-06-29 | Xerox Corporation | Automatic method of extracting summarization using feature probabilities |
US5826260A (en) * | 1995-12-11 | 1998-10-20 | International Business Machines Corporation | Information retrieval system and method for displaying and ordering information based on query element contribution |
-
1998
- 1998-06-23 US US09/102,861 patent/US6233575B1/en not_active Expired - Fee Related
-
2001
- 2001-02-05 US US09/777,278 patent/US20010037324A1/en not_active Abandoned
Cited By (243)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6640224B1 (en) * | 1997-12-15 | 2003-10-28 | International Business Machines Corporation | System and method for dynamic index-probe optimizations for high-dimensional similarity search |
US7650340B2 (en) * | 1998-12-21 | 2010-01-19 | Adobe Systems Incorporated | Describing documents and expressing document structure |
US20120109642A1 (en) * | 1999-02-05 | 2012-05-03 | Stobbs Gregory A | Computer-implemented patent portfolio analysis method and apparatus |
US9710457B2 (en) * | 1999-02-05 | 2017-07-18 | Gregory A. Stobbs | Computer-implemented patent portfolio analysis method and apparatus |
US7376635B1 (en) * | 2000-07-21 | 2008-05-20 | Ford Global Technologies, Llc | Theme-based system and method for classifying documents |
US20040153307A1 (en) * | 2001-03-30 | 2004-08-05 | Naftali Tishby | Discriminative feature selection for data sequences |
US10380374B2 (en) | 2001-04-20 | 2019-08-13 | Jpmorgan Chase Bank, N.A. | System and method for preventing identity theft or misuse by restricting access |
US8849716B1 (en) * | 2001-04-20 | 2014-09-30 | Jpmorgan Chase Bank, N.A. | System and method for preventing identity theft or misuse by restricting access |
US7107254B1 (en) * | 2001-05-07 | 2006-09-12 | Microsoft Corporation | Probablistic models and methods for combining multiple content classifiers |
US7676555B2 (en) | 2001-07-24 | 2010-03-09 | Brightplanet Corporation | System and method for efficient control and capture of dynamic database content |
US8380735B2 (en) | 2001-07-24 | 2013-02-19 | Brightplanet Corporation II, Inc | System and method for efficient control and capture of dynamic database content |
US20070192442A1 (en) * | 2001-07-24 | 2007-08-16 | Brightplanet Corporation | System and method for efficient control and capture of dynamic database content |
US20030101191A1 (en) * | 2001-11-28 | 2003-05-29 | Yu-Chwin Lin | Publication of commercial entity information and method for classifying commercial entity information |
US7363593B1 (en) * | 2001-11-30 | 2008-04-22 | Versata Development Group, Inc. | System and method for presenting information organized by hierarchical levels |
US20070156665A1 (en) * | 2001-12-05 | 2007-07-05 | Janusz Wnek | Taxonomy discovery |
US20030163454A1 (en) * | 2002-02-26 | 2003-08-28 | Brian Jacobsen | Subject specific search engine |
US7949648B2 (en) | 2002-02-26 | 2011-05-24 | Soren Alain Mortensen | Compiling and accessing subject-specific information from a computer network |
US20030172357A1 (en) * | 2002-03-11 | 2003-09-11 | Kao Anne S.W. | Knowledge management using text classification |
US7673234B2 (en) | 2002-03-11 | 2010-03-02 | The Boeing Company | Knowledge management using text classification |
WO2003079234A3 (en) * | 2002-03-11 | 2004-06-10 | Boeing Co | Knowledge management using text classification |
US20030194689A1 (en) * | 2002-04-12 | 2003-10-16 | Mitsubishi Denki Kabushiki Kaisha | Structured document type determination system and structured document type determination method |
US20030217055A1 (en) * | 2002-05-20 | 2003-11-20 | Chang-Huang Lee | Efficient incremental method for data mining of a database |
US20040117388A1 (en) * | 2002-09-02 | 2004-06-17 | Yasuhiko Inaba | Method, apparatus and programs for delivering information |
US20040049514A1 (en) * | 2002-09-11 | 2004-03-11 | Sergei Burkov | System and method of searching data utilizing automatic categorization |
US20100083377A1 (en) * | 2002-09-18 | 2010-04-01 | Rowney Kevin T | Method and apparatus to define the scope of a search for information from a tabular data source |
US8595849B2 (en) | 2002-09-18 | 2013-11-26 | Symantec Corporation | Method and apparatus to report policy violations in messages |
US7996385B2 (en) * | 2002-09-18 | 2011-08-09 | Symantec Corporation | Method and apparatus to define the scope of a search for information from a tabular data source |
US7886359B2 (en) | 2002-09-18 | 2011-02-08 | Symantec Corporation | Method and apparatus to report policy violations in messages |
US20100332481A1 (en) * | 2002-09-18 | 2010-12-30 | Rowney Kevin T | Secure and scalable detection of preselected data embedded in electronically transmitted messages |
US8225371B2 (en) | 2002-09-18 | 2012-07-17 | Symantec Corporation | Method and apparatus for creating an information security policy based on a pre-configured template |
US7472114B1 (en) * | 2002-09-18 | 2008-12-30 | Symantec Corporation | Method and apparatus to define the scope of a search for information from a tabular data source |
US20090106205A1 (en) * | 2002-09-18 | 2009-04-23 | Rowney Kevin T | Method and apparatus to define the scope of a search for information from a tabular data source |
US20050027723A1 (en) * | 2002-09-18 | 2005-02-03 | Chris Jones | Method and apparatus to report policy violations in messages |
US8312553B2 (en) | 2002-09-18 | 2012-11-13 | Symantec Corporation | Mechanism to search information content for preselected data |
US7673344B1 (en) | 2002-09-18 | 2010-03-02 | Symantec Corporation | Mechanism to search information content for preselected data |
US8566305B2 (en) * | 2002-09-18 | 2013-10-22 | Symantec Corporation | Method and apparatus to define the scope of a search for information from a tabular data source |
US20050086252A1 (en) * | 2002-09-18 | 2005-04-21 | Chris Jones | Method and apparatus for creating an information security policy based on a pre-configured template |
US8813176B2 (en) | 2002-09-18 | 2014-08-19 | Symantec Corporation | Method and apparatus for creating an information security policy based on a pre-configured template |
US8661498B2 (en) | 2002-09-18 | 2014-02-25 | Symantec Corporation | Secure and scalable detection of preselected data embedded in electronically transmitted messages |
US9515998B2 (en) | 2002-09-18 | 2016-12-06 | Symantec Corporation | Secure and scalable detection of preselected data embedded in electronically transmitted messages |
US10748527B2 (en) * | 2002-10-31 | 2020-08-18 | Promptu Systems Corporation | Efficient empirical determination, computation, and use of acoustic confusability measures |
US20190051294A1 (en) * | 2002-10-31 | 2019-02-14 | Promptu Systems Corporation | Efficient empirical determination, computation, and use of acoustic confusability measures |
US11587558B2 (en) | 2002-10-31 | 2023-02-21 | Promptu Systems Corporation | Efficient empirical determination, computation, and use of acoustic confusability measures |
US7266559B2 (en) * | 2002-12-05 | 2007-09-04 | Microsoft Corporation | Method and apparatus for adapting a search classifier based on user queries |
US20040111419A1 (en) * | 2002-12-05 | 2004-06-10 | Cook Daniel B. | Method and apparatus for adapting a search classifier based on user queries |
US20070276818A1 (en) * | 2002-12-05 | 2007-11-29 | Microsoft Corporation | Adapting a search classifier based on user queries |
US7203669B2 (en) * | 2003-03-17 | 2007-04-10 | Intel Corporation | Detector tree of boosted classifiers for real-time object detection and tracking |
US20040220901A1 (en) * | 2003-04-30 | 2004-11-04 | Benq Corporation | System and method for association itemset mining |
US8041719B2 (en) | 2003-05-06 | 2011-10-18 | Symantec Corporation | Personal computing device-based mechanism to detect preselected data |
US20040225645A1 (en) * | 2003-05-06 | 2004-11-11 | Rowney Kevin T. | Personal computing device -based mechanism to detect preselected data |
US8751506B2 (en) | 2003-05-06 | 2014-06-10 | Symantec Corporation | Personal computing device-based mechanism to detect preselected data |
US20060155662A1 (en) * | 2003-07-01 | 2006-07-13 | Eiji Murakami | Sentence classification device and method |
US7567954B2 (en) * | 2003-07-01 | 2009-07-28 | Yamatake Corporation | Sentence classification device and method |
US7814327B2 (en) | 2003-12-10 | 2010-10-12 | Mcafee, Inc. | Document registration |
US9092471B2 (en) | 2003-12-10 | 2015-07-28 | Mcafee, Inc. | Rule parser |
US8271794B2 (en) | 2003-12-10 | 2012-09-18 | Mcafee, Inc. | Verifying captured objects before presentation |
US8301635B2 (en) | 2003-12-10 | 2012-10-30 | Mcafee, Inc. | Tag data structure for maintaining relational data over captured objects |
US8762386B2 (en) | 2003-12-10 | 2014-06-24 | Mcafee, Inc. | Method and apparatus for data capture and analysis system |
US9374225B2 (en) | 2003-12-10 | 2016-06-21 | Mcafee, Inc. | Document de-registration |
US7984175B2 (en) | 2003-12-10 | 2011-07-19 | Mcafee, Inc. | Method and apparatus for data capture and analysis system |
US8166307B2 (en) | 2003-12-10 | 2012-04-24 | McAffee, Inc. | Document registration |
US20050132046A1 (en) * | 2003-12-10 | 2005-06-16 | De La Iglesia Erik | Method and apparatus for data capture and analysis system |
US8656039B2 (en) | 2003-12-10 | 2014-02-18 | Mcafee, Inc. | Rule parser |
US8548170B2 (en) | 2003-12-10 | 2013-10-01 | Mcafee, Inc. | Document de-registration |
US7774604B2 (en) | 2003-12-10 | 2010-08-10 | Mcafee, Inc. | Verifying captured objects before presentation |
US7899828B2 (en) | 2003-12-10 | 2011-03-01 | Mcafee, Inc. | Tag data structure for maintaining relational data over captured objects |
US7930540B2 (en) | 2004-01-22 | 2011-04-19 | Mcafee, Inc. | Cryptographic policy enforcement |
US8307206B2 (en) | 2004-01-22 | 2012-11-06 | Mcafee, Inc. | Cryptographic policy enforcement |
US20050187892A1 (en) * | 2004-02-09 | 2005-08-25 | Xerox Corporation | Method for multi-class, multi-label categorization using probabilistic hierarchical modeling |
US7139754B2 (en) * | 2004-02-09 | 2006-11-21 | Xerox Corporation | Method for multi-class, multi-label categorization using probabilistic hierarchical modeling |
US11163802B1 (en) | 2004-03-01 | 2021-11-02 | Huawei Technologies Co., Ltd. | Local search using restriction specification |
US11860921B2 (en) | 2004-03-01 | 2024-01-02 | Huawei Technologies Co., Ltd. | Category-based search |
US8862580B1 (en) * | 2004-03-01 | 2014-10-14 | Radix Holdings, Llc | Category-based search |
US7962591B2 (en) | 2004-06-23 | 2011-06-14 | Mcafee, Inc. | Object classification in a capture system |
US8560534B2 (en) | 2004-08-23 | 2013-10-15 | Mcafee, Inc. | Database for a capture system |
US8707008B2 (en) | 2004-08-24 | 2014-04-22 | Mcafee, Inc. | File system for a capture system |
US7949849B2 (en) | 2004-08-24 | 2011-05-24 | Mcafee, Inc. | File system for a capture system |
US20060074907A1 (en) * | 2004-09-27 | 2006-04-06 | Singhal Amitabh K | Presentation of search results based on document structure |
US9031898B2 (en) * | 2004-09-27 | 2015-05-12 | Google Inc. | Presentation of search results based on document structure |
US20060074632A1 (en) * | 2004-09-30 | 2006-04-06 | Nanavati Amit A | Ontology-based term disambiguation |
US8011003B2 (en) | 2005-02-14 | 2011-08-30 | Symantec Corporation | Method and apparatus for handling messages containing pre-selected data |
US7584168B2 (en) * | 2005-02-14 | 2009-09-01 | France Telecom | Method and device for the generation of a classification tree to unify the supervised and unsupervised approaches, corresponding computer package and storage means |
US20060184549A1 (en) * | 2005-02-14 | 2006-08-17 | Rowney Kevin T | Method and apparatus for modifying messages based on the presence of pre-selected data |
US20060195415A1 (en) * | 2005-02-14 | 2006-08-31 | France Telecom | Method and device for the generation of a classification tree to unify the supervised and unsupervised approaches, corresponding computer package and storage means |
US20060224589A1 (en) * | 2005-02-14 | 2006-10-05 | Rowney Kevin T | Method and apparatus for handling messages containing pre-selected data |
US9400838B2 (en) | 2005-04-11 | 2016-07-26 | Textdigger, Inc. | System and method for searching for a query |
US20070011154A1 (en) * | 2005-04-11 | 2007-01-11 | Textdigger, Inc. | System and method for searching for a query |
US20060242098A1 (en) * | 2005-04-26 | 2006-10-26 | Content Analyst Company, Llc | Generating representative exemplars for indexing, clustering, categorization and taxonomy |
US20060242190A1 (en) * | 2005-04-26 | 2006-10-26 | Content Analyst Comapny, Llc | Latent semantic taxonomy generation |
US20060242140A1 (en) * | 2005-04-26 | 2006-10-26 | Content Analyst Company, Llc | Latent semantic clustering |
US7844566B2 (en) | 2005-04-26 | 2010-11-30 | Content Analyst Company, Llc | Latent semantic clustering |
US20060282442A1 (en) * | 2005-04-27 | 2006-12-14 | Canon Kabushiki Kaisha | Method of learning associations between documents and data sets |
US7461044B2 (en) | 2005-04-27 | 2008-12-02 | International Business Machines Corporation | It resource event situation classification and semantics |
US20090006298A1 (en) * | 2005-04-27 | 2009-01-01 | International Business Machines Corporation | It resource event situation classification and semantics |
US7895137B2 (en) | 2005-04-27 | 2011-02-22 | International Business Machines Corporation | Rules generation for IT resource event situation classification |
US7730007B2 (en) | 2005-04-27 | 2010-06-01 | International Business Machines Corporation | IT event data classifier configured to label messages if message identifiers map directly to classification categories or parse for feature extraction if message identifiers do not map directly to classification categories |
US20070005535A1 (en) * | 2005-04-27 | 2007-01-04 | Abdolreza Salahshour | System and methods for IT resource event situation classification and semantics |
US20090276383A1 (en) * | 2005-04-27 | 2009-11-05 | International Business Machines Corporation | Rules generation for it resource event situation classification |
US9110985B2 (en) | 2005-05-10 | 2015-08-18 | Neetseer, Inc. | Generating a conceptual association graph from large-scale loosely-grouped content |
US8838605B2 (en) * | 2005-05-10 | 2014-09-16 | Netseer, Inc. | Methods and apparatus for distributed community finding |
US20130046797A1 (en) * | 2005-05-10 | 2013-02-21 | Netseer, Inc. | Methods and apparatus for distributed community finding |
US8825654B2 (en) | 2005-05-10 | 2014-09-02 | Netseer, Inc. | Methods and apparatus for distributed community finding |
US8730955B2 (en) | 2005-08-12 | 2014-05-20 | Mcafee, Inc. | High speed packet capture |
US7907608B2 (en) | 2005-08-12 | 2011-03-15 | Mcafee, Inc. | High speed packet capture |
US7818326B2 (en) * | 2005-08-31 | 2010-10-19 | Mcafee, Inc. | System and method for word indexing in a capture system and querying thereof |
US8554774B2 (en) | 2005-08-31 | 2013-10-08 | Mcafee, Inc. | System and method for word indexing in a capture system and querying thereof |
US8176049B2 (en) | 2005-10-19 | 2012-05-08 | Mcafee Inc. | Attributes of captured objects in a capture system |
US7730011B1 (en) | 2005-10-19 | 2010-06-01 | Mcafee, Inc. | Attributes of captured objects in a capture system |
US8463800B2 (en) | 2005-10-19 | 2013-06-11 | Mcafee, Inc. | Attributes of captured objects in a capture system |
US8200026B2 (en) | 2005-11-21 | 2012-06-12 | Mcafee, Inc. | Identifying image type in a capture system |
US7657104B2 (en) | 2005-11-21 | 2010-02-02 | Mcafee, Inc. | Identifying image type in a capture system |
US9245029B2 (en) | 2006-01-03 | 2016-01-26 | Textdigger, Inc. | Search system with query refinement and search method |
US9928299B2 (en) | 2006-01-03 | 2018-03-27 | Textdigger, Inc. | Search system with query refinement and search method |
US9443018B2 (en) | 2006-01-19 | 2016-09-13 | Netseer, Inc. | Systems and methods for creating, navigating, and searching informational web neighborhoods |
US8843434B2 (en) | 2006-02-28 | 2014-09-23 | Netseer, Inc. | Methods and apparatus for visualizing, managing, monetizing, and personalizing knowledge search results on a user interface |
US20070203903A1 (en) * | 2006-02-28 | 2007-08-30 | Ilial, Inc. | Methods and apparatus for visualizing, managing, monetizing, and personalizing knowledge search results on a user interface |
US8504537B2 (en) | 2006-03-24 | 2013-08-06 | Mcafee, Inc. | Signature distribution in a document registration system |
US20070234232A1 (en) * | 2006-03-29 | 2007-10-04 | Gheorghe Adrian Citu | Dynamic image display |
US10540406B2 (en) | 2006-04-04 | 2020-01-21 | Exis Inc. | Search system and method with text function tagging |
US8862573B2 (en) | 2006-04-04 | 2014-10-14 | Textdigger, Inc. | Search system and method with text function tagging |
US20080059451A1 (en) * | 2006-04-04 | 2008-03-06 | Textdigger, Inc. | Search system and method with text function tagging |
US8005863B2 (en) | 2006-05-22 | 2011-08-23 | Mcafee, Inc. | Query generation for a capture system |
US7689614B2 (en) | 2006-05-22 | 2010-03-30 | Mcafee, Inc. | Query generation for a capture system |
US9094338B2 (en) | 2006-05-22 | 2015-07-28 | Mcafee, Inc. | Attributes of captured objects in a capture system |
US7958227B2 (en) | 2006-05-22 | 2011-06-07 | Mcafee, Inc. | Attributes of captured objects in a capture system |
US8010689B2 (en) | 2006-05-22 | 2011-08-30 | Mcafee, Inc. | Locational tagging in a capture system |
US8683035B2 (en) | 2006-05-22 | 2014-03-25 | Mcafee, Inc. | Attributes of captured objects in a capture system |
US8307007B2 (en) | 2006-05-22 | 2012-11-06 | Mcafee, Inc. | Query generation for a capture system |
US9817902B2 (en) | 2006-10-27 | 2017-11-14 | Netseer Acquisition, Inc. | Methods and apparatus for matching relevant content to user intention |
US20080114573A1 (en) * | 2006-11-10 | 2008-05-15 | Institute For Information Industry | Tag organization methods and systems |
US7908260B1 (en) | 2006-12-29 | 2011-03-15 | BrightPlanet Corporation II, Inc. | Source editing, internationalization, advanced configuration wizard, and summary page selection for information automation systems |
WO2008097891A2 (en) * | 2007-02-02 | 2008-08-14 | Musgrove Technology Enterprises Llc | Method and apparatus for aligning multiple taxonomies |
US20090037457A1 (en) * | 2007-02-02 | 2009-02-05 | Musgrove Technology Enterprises, Llc (Mte) | Method and apparatus for aligning multiple taxonomies |
US8732197B2 (en) | 2007-02-02 | 2014-05-20 | Musgrove Technology Enterprises Llc (Mte) | Method and apparatus for aligning multiple taxonomies |
WO2008097891A3 (en) * | 2007-02-02 | 2008-10-09 | Musgrove Technology Entpr Llc | Method and apparatus for aligning multiple taxonomies |
US8805843B2 (en) | 2007-02-13 | 2014-08-12 | International Business Machines Corporation | Information mining using domain specific conceptual structures |
US20080243889A1 (en) * | 2007-02-13 | 2008-10-02 | International Business Machines Corporation | Information mining using domain specific conceptual structures |
US20080195567A1 (en) * | 2007-02-13 | 2008-08-14 | International Business Machines Corporation | Information mining using domain specific conceptual structures |
US20090254540A1 (en) * | 2007-11-01 | 2009-10-08 | Textdigger, Inc. | Method and apparatus for automated tag generation for digital content |
US8862619B1 (en) | 2008-01-07 | 2014-10-14 | Amdocs Software Systems Limited | System, method, and computer program product for filtering a data stream utilizing a plurality of contexts |
US8868563B2 (en) * | 2008-01-07 | 2014-10-21 | Amdocs Software Systems Limited | System, method, and computer program product for analyzing and decomposing a plurality of rules into a plurality of contexts |
US7962490B1 (en) * | 2008-01-07 | 2011-06-14 | Amdocs Software Systems Limited | System, method, and computer program product for analyzing and decomposing a plurality of rules into a plurality of contexts |
US20110231412A1 (en) * | 2008-01-07 | 2011-09-22 | Amdocs Software Systems Limited | System, method, and computer program product for analyzing and decomposing a plurality of rules into a plurality of contexts |
US20090228499A1 (en) * | 2008-03-05 | 2009-09-10 | Schmidtler Mauritius A R | Systems and methods for organizing data sets |
US20100262571A1 (en) * | 2008-03-05 | 2010-10-14 | Schmidtler Mauritius A R | Systems and methods for organizing data sets |
US9082080B2 (en) | 2008-03-05 | 2015-07-14 | Kofax, Inc. | Systems and methods for organizing data sets |
US8321477B2 (en) | 2008-03-05 | 2012-11-27 | Kofax, Inc. | Systems and methods for organizing data sets |
US20100076984A1 (en) * | 2008-03-27 | 2010-03-25 | Alkis Papadopoullos | System and method for query expansion using tooltips |
US9235629B1 (en) | 2008-03-28 | 2016-01-12 | Symantec Corporation | Method and apparatus for automatically correlating related incidents of policy violations |
US7996374B1 (en) | 2008-03-28 | 2011-08-09 | Symantec Corporation | Method and apparatus for automatically correlating related incidents of policy violations |
US7996373B1 (en) | 2008-03-28 | 2011-08-09 | Symantec Corporation | Method and apparatus for detecting policy violations in a data repository having an arbitrary data schema |
US8255370B1 (en) | 2008-03-28 | 2012-08-28 | Symantec Corporation | Method and apparatus for detecting policy violations in a data repository having an arbitrary data schema |
US8065739B1 (en) | 2008-03-28 | 2011-11-22 | Symantec Corporation | Detecting policy violations in information content containing data in a character-based language |
US10387892B2 (en) | 2008-05-06 | 2019-08-20 | Netseer, Inc. | Discovering relevant concept and context for content node |
US11475465B2 (en) | 2008-05-06 | 2022-10-18 | Netseer, Inc. | Discovering relevant concept and context for content node |
US8601537B2 (en) | 2008-07-10 | 2013-12-03 | Mcafee, Inc. | System and method for data mining and security policy management |
US8635706B2 (en) | 2008-07-10 | 2014-01-21 | Mcafee, Inc. | System and method for data mining and security policy management |
US8205242B2 (en) | 2008-07-10 | 2012-06-19 | Mcafee, Inc. | System and method for data mining and security policy management |
US8554800B2 (en) * | 2008-07-30 | 2013-10-08 | Portool Ltd. | System, methods and applications for structured document indexing |
US20100030752A1 (en) * | 2008-07-30 | 2010-02-04 | Lev Goldentouch | System, methods and applications for structured document indexing |
US8335787B2 (en) * | 2008-08-08 | 2012-12-18 | Quillsoft Ltd. | Topic word generation method and system |
US20110231411A1 (en) * | 2008-08-08 | 2011-09-22 | Holland Bloorview Kids Rehabilitation Hospital | Topic Word Generation Method and System |
US9253154B2 (en) | 2008-08-12 | 2016-02-02 | Mcafee, Inc. | Configuration management for a capture/registration system |
US10367786B2 (en) | 2008-08-12 | 2019-07-30 | Mcafee, Llc | Configuration management for a capture/registration system |
US9118720B1 (en) | 2008-09-18 | 2015-08-25 | Symantec Corporation | Selective removal of protected content from web requests sent to an interactive website |
US8826443B1 (en) | 2008-09-18 | 2014-09-02 | Symantec Corporation | Selective removal of protected content from web requests sent to an interactive website |
US20100162347A1 (en) * | 2008-12-22 | 2010-06-24 | Ian Barile | Adaptive data loss prevention policies |
US8613040B2 (en) | 2008-12-22 | 2013-12-17 | Symantec Corporation | Adaptive data loss prevention policies |
US20110264699A1 (en) * | 2008-12-30 | 2011-10-27 | Telecom Italia S.P.A. | Method and system for content classification |
US9916381B2 (en) * | 2008-12-30 | 2018-03-13 | Telecom Italia S.P.A. | Method and system for content classification |
US8850591B2 (en) | 2009-01-13 | 2014-09-30 | Mcafee, Inc. | System and method for concept building |
US8706709B2 (en) | 2009-01-15 | 2014-04-22 | Mcafee, Inc. | System and method for intelligent term grouping |
US8275726B2 (en) * | 2009-01-16 | 2012-09-25 | Microsoft Corporation | Object classification using taxonomies |
US20100185577A1 (en) * | 2009-01-16 | 2010-07-22 | Microsoft Corporation | Object classification using taxonomies |
US9602548B2 (en) | 2009-02-25 | 2017-03-21 | Mcafee, Inc. | System and method for intelligent state management |
US8473442B1 (en) | 2009-02-25 | 2013-06-25 | Mcafee, Inc. | System and method for intelligent state management |
US9195937B2 (en) | 2009-02-25 | 2015-11-24 | Mcafee, Inc. | System and method for intelligent state management |
US8935752B1 (en) | 2009-03-23 | 2015-01-13 | Symantec Corporation | System and method for identity consolidation |
US8918359B2 (en) | 2009-03-25 | 2014-12-23 | Mcafee, Inc. | System and method for data mining and security policy management |
US8667121B2 (en) | 2009-03-25 | 2014-03-04 | Mcafee, Inc. | System and method for managing data and policies |
US9313232B2 (en) | 2009-03-25 | 2016-04-12 | Mcafee, Inc. | System and method for data mining and security policy management |
US8447722B1 (en) | 2009-03-25 | 2013-05-21 | Mcafee, Inc. | System and method for data mining and security policy management |
US8484194B1 (en) | 2009-10-22 | 2013-07-09 | Google Inc. | Training set construction for taxonomic classification |
US8122005B1 (en) * | 2009-10-22 | 2012-02-21 | Google Inc. | Training set construction for taxonomic classification |
US8515972B1 (en) | 2010-02-10 | 2013-08-20 | Python 4 Fun, Inc. | Finding relevant documents |
US8527893B2 (en) * | 2010-02-26 | 2013-09-03 | Microsoft Corporation | Taxonomy editor |
US20110214080A1 (en) * | 2010-02-26 | 2011-09-01 | Microsoft Corporation | Taxonomy Editor |
US9595005B1 (en) | 2010-05-25 | 2017-03-14 | Recommind, Inc. | Systems and methods for predictive coding |
US8489538B1 (en) | 2010-05-25 | 2013-07-16 | Recommind, Inc. | Systems and methods for predictive coding |
US8554716B1 (en) | 2010-05-25 | 2013-10-08 | Recommind, Inc. | Systems and methods for predictive coding |
US11282000B2 (en) | 2010-05-25 | 2022-03-22 | Open Text Holdings, Inc. | Systems and methods for predictive coding |
US11023828B2 (en) | 2010-05-25 | 2021-06-01 | Open Text Holdings, Inc. | Systems and methods for predictive coding |
US20110314024A1 (en) * | 2010-06-18 | 2011-12-22 | Microsoft Corporation | Semantic content searching |
US8380719B2 (en) * | 2010-06-18 | 2013-02-19 | Microsoft Corporation | Semantic content searching |
US10089390B2 (en) * | 2010-09-24 | 2018-10-02 | International Business Machines Corporation | System and method to extract models from semi-structured documents |
US20120078969A1 (en) * | 2010-09-24 | 2012-03-29 | International Business Machines Corporation | System and method to extract models from semi-structured documents |
US8645298B2 (en) | 2010-10-26 | 2014-02-04 | Microsoft Corporation | Topic models |
US11316848B2 (en) | 2010-11-04 | 2022-04-26 | Mcafee, Llc | System and method for protecting specified data combinations |
US8806615B2 (en) | 2010-11-04 | 2014-08-12 | Mcafee, Inc. | System and method for protecting specified data combinations |
US9794254B2 (en) | 2010-11-04 | 2017-10-17 | Mcafee, Inc. | System and method for protecting specified data combinations |
US10666646B2 (en) | 2010-11-04 | 2020-05-26 | Mcafee, Llc | System and method for protecting specified data combinations |
US10313337B2 (en) | 2010-11-04 | 2019-06-04 | Mcafee, Llc | System and method for protecting specified data combinations |
US9785634B2 (en) | 2011-06-04 | 2017-10-10 | Recommind, Inc. | Integration and combination of random sampling and document batching |
US9116979B2 (en) | 2011-06-22 | 2015-08-25 | Rogers Communications Inc. | Systems and methods for creating an interest profile for a user |
WO2012174640A1 (en) * | 2011-06-22 | 2012-12-27 | Rogers Communications Inc. | Systems and methods for creating an interest profile for a user |
US9436726B2 (en) | 2011-06-23 | 2016-09-06 | BCM International Regulatory Analytics LLC | System, method and computer program product for a behavioral database providing quantitative analysis of cross border policy process and related search capabilities |
US20130166563A1 (en) * | 2011-12-21 | 2013-06-27 | Sap Ag | Integration of Text Analysis and Search Functionality |
US8700561B2 (en) | 2011-12-27 | 2014-04-15 | Mcafee, Inc. | System and method for providing data protection workflows in a network environment |
US9430564B2 (en) | 2011-12-27 | 2016-08-30 | Mcafee, Inc. | System and method for providing data protection workflows in a network environment |
US20130218904A1 (en) * | 2012-02-22 | 2013-08-22 | Salesforce.Com, Inc. | System and method for inferring reporting relationships from a contact database |
US9477698B2 (en) * | 2012-02-22 | 2016-10-25 | Salesforce.Com, Inc. | System and method for inferring reporting relationships from a contact database |
US9779291B2 (en) * | 2012-06-14 | 2017-10-03 | The Board Of Trustees Of The Leland Stanford Junior University | Method and system for optimizing accuracy-specificity trade-offs in large scale visual recognition |
US20160162731A1 (en) * | 2012-06-14 | 2016-06-09 | The Board Of Trustees Of The Leland Stanford Junior University | Method and System for Optimizing Accuracy-Specificity Trade-offs in Large Scale Visual Recognition |
US20140086497A1 (en) * | 2012-06-14 | 2014-03-27 | The Board of Trustees for the Leland Stanford, Junior, University | Method and System for Optimizing Accuracy-Specificity Trade-offs in Large Scale Visual Recognition |
US9158965B2 (en) * | 2012-06-14 | 2015-10-13 | The Board Of Trustees Of The Leland Stanford Junior University | Method and system for optimizing accuracy-specificity trade-offs in large scale visual recognition |
US10311085B2 (en) | 2012-08-31 | 2019-06-04 | Netseer, Inc. | Concept-level user intent profile extraction and applications |
US10860619B2 (en) | 2012-08-31 | 2020-12-08 | Netseer, Inc. | Concept-level user intent profile extraction and applications |
US20140172652A1 (en) * | 2012-12-19 | 2014-06-19 | Yahoo! Inc. | Automated categorization of products in a merchant catalog |
US10528907B2 (en) * | 2012-12-19 | 2020-01-07 | Oath Inc. | Automated categorization of products in a merchant catalog |
US20170091590A1 (en) * | 2013-03-15 | 2017-03-30 | Sri International | Computer vision as a service |
US20230045330A1 (en) * | 2013-09-26 | 2023-02-09 | Groupon, Inc. | Multi-term query subsumption for document classification |
US20150095017A1 (en) * | 2013-09-27 | 2015-04-02 | Google Inc. | System and method for learning word embeddings using neural language models |
US10078668B1 (en) | 2014-05-04 | 2018-09-18 | Veritas Technologies Llc | Systems and methods for utilizing information-asset metadata aggregated from multiple disparate data-management systems |
US10025804B2 (en) | 2014-05-04 | 2018-07-17 | Veritas Technologies Llc | Systems and methods for aggregating information-asset metadata from multiple disparate data-management systems |
US10817510B1 (en) | 2014-05-04 | 2020-10-27 | Veritas Technologies Llc | Systems and methods for navigating through a hierarchy of nodes stored in a database |
US10635645B1 (en) | 2014-05-04 | 2020-04-28 | Veritas Technologies Llc | Systems and methods for maintaining aggregate tables in databases |
US10073864B1 (en) | 2014-05-04 | 2018-09-11 | Veritas Technologies Llc | Systems and methods for automated aggregation of information-source metadata |
US20160140207A1 (en) * | 2014-11-14 | 2016-05-19 | Symantec Corporation | Systems and methods for aggregating information-asset classifications |
US10095768B2 (en) * | 2014-11-14 | 2018-10-09 | Veritas Technologies Llc | Systems and methods for aggregating information-asset classifications |
US20160182669A1 (en) * | 2014-12-22 | 2016-06-23 | Here Global B.V. | Optimal Coding Method for Efficient Matching Of Hierarchical Categories In Publish-Subscribe Systems |
US10158738B2 (en) * | 2014-12-22 | 2018-12-18 | Here Global B.V. | Optimal coding method for efficient matching of hierarchical categories in publish-subscribe systems |
US10586169B2 (en) * | 2015-10-16 | 2020-03-10 | Microsoft Technology Licensing, Llc | Common feature protocol for collaborative machine learning |
US11704370B2 (en) | 2018-04-20 | 2023-07-18 | Microsoft Technology Licensing, Llc | Framework for managing features across environments |
US11334816B2 (en) | 2018-05-21 | 2022-05-17 | International Business Machines Corporation | Finding optimal surface for hierarchical classification task on an ontology |
US11281995B2 (en) | 2018-05-21 | 2022-03-22 | International Business Machines Corporation | Finding optimal surface for hierarchical classification task on an ontology |
US11157539B2 (en) * | 2018-06-22 | 2021-10-26 | Microsoft Technology Licensing, Llc | Topic set refinement |
US10929439B2 (en) * | 2018-06-22 | 2021-02-23 | Microsoft Technology Licensing, Llc | Taxonomic tree generation |
US20190392073A1 (en) * | 2018-06-22 | 2019-12-26 | Microsoft Technology Licensing, Llc | Taxonomic tree generation |
US20190392078A1 (en) * | 2018-06-22 | 2019-12-26 | Microsoft Technology Licensing, Llc | Topic set refinement |
US10902066B2 (en) | 2018-07-23 | 2021-01-26 | Open Text Holdings, Inc. | Electronic discovery using predictive filtering |
US11676043B2 (en) | 2019-03-04 | 2023-06-13 | International Business Machines Corporation | Optimizing hierarchical classification with adaptive node collapses |
US20220245378A1 (en) * | 2021-02-03 | 2022-08-04 | Aon Risk Services, Inc. Of Maryland | Document analysis using model intersections |
US11928879B2 (en) * | 2021-02-03 | 2024-03-12 | Aon Risk Services, Inc. Of Maryland | Document analysis using model intersections |
Also Published As
Publication number | Publication date |
---|---|
US6233575B1 (en) | 2001-05-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6233575B1 (en) | Multilevel taxonomy based on features derived from training documents classification using fisher values as discrimination values | |
Chakrabarti et al. | Scalable feature selection, classification and signature generation for organizing large text databases into hierarchical topic taxonomies | |
Chakrabarti et al. | Using taxonomy, discriminants, and signatures for navigating in text databases | |
Chakrabarti et al. | Enhanced hypertext categorization using hyperlinks | |
US6389436B1 (en) | Enhanced hypertext categorization using hyperlinks | |
Sahami | Using machine learning to improve information access | |
Aggarwal et al. | On the merits of building categorization systems by supervised clustering | |
US6360227B1 (en) | System and method for generating taxonomies with applications to content-based recommendations | |
Liu et al. | Personalized web search for improving retrieval effectiveness | |
Wang et al. | Building hierarchical classifiers using class proximity | |
Zamir | Clustering web documents: a phrase-based method for grouping search engine results | |
US6922699B2 (en) | System and method for quantitatively representing data objects in vector space | |
US7085771B2 (en) | System and method for automatically discovering a hierarchy of concepts from a corpus of documents | |
Raghavan | Information retrieval algorithms: A survey | |
Brücher et al. | Document classification methods for organizing explicit knowledge | |
Aggarwal et al. | On using partial supervision for text categorization | |
US20030110181A1 (en) | System and method for clustering data objects in a collection | |
Lin et al. | ACIRD: intelligent Internet document organization and retrieval | |
Lang | A tolerance rough set approach to clustering web search results | |
Marques et al. | MUSE: A content-based image search and retrieval system using relevance feedback | |
Chen et al. | Multimodal browsing of images in web documents | |
Su et al. | Automatic hierarchical classification of structured deep web databases | |
Dobrynin et al. | SOPHIA: an interactive cluster-based retrieval system for the OHSUMED collection | |
Freeman et al. | Web content management by self-organization | |
WO2001039008A1 (en) | Method and system for collecting topically related resources |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |