Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
  • G06F40/40—Processing or translation of natural language
  • G06F40/42—Data-driven translation
  • G06F40/49—Data-driven translation using very large corpora, e.g. the web
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
  • G06F16/35—Clustering; Classification
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/10—File systems; File servers
  • G06F16/14—Details of searching files based on file metadata
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/24—Querying
  • G06F16/245—Query processing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/24—Querying
  • G06F16/245—Query processing
  • G06F16/2452—Query translation
  • G06F16/24522—Translation of natural language queries to structured queries
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/24—Querying
  • G06F16/245—Query processing
  • G06F16/2455—Query execution
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
  • G06F16/33—Querying
  • G06F16/332—Query formulation
  • G06F16/3329—Natural language query formulation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
  • G06F16/33—Querying
  • G06F16/3331—Query processing
  • G06F16/3332—Query translation
  • G06F16/3334—Selection or weighting of terms from queries, including natural language queries
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
  • G06F16/35—Clustering; Classification
  • G06F16/353—Clustering; Classification into predefined classes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
  • G06F16/48—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/93—Document management systems
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
  • G06F40/20—Natural language analysis
  • G06F40/205—Parsing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
  • G06F40/20—Natural language analysis
  • G06F40/205—Parsing
  • G06F40/216—Parsing using statistical methods
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
  • G06F40/20—Natural language analysis
  • G06F40/279—Recognition of textual entities
  • G06F40/284—Lexical analysis, e.g. tokenisation or collocates

Definitions

• Keyword extraction typically serves as the core component of contextual advertising systems, where advertisements that match webpage content are chosen based on keywords automatically selected from the page text.
• keywords In order to display ads relevant to the webpage, and thus potentially more interesting to the user, numerous features present in the text need to be assessed to make a decision as to which keywords accurately reflect the content of the page.
• a keyword extraction system takes a page url as input and returns 10 keyword phrases ranked by the system as top keyword candidates.
• the system first processes webpage text and uses its structure to extract phrases which serve as a keyword candidate pool.
• Each phrase can then be described by a set of features such as its frequency on the webpage, location in the text, capitalization and its linguistic structure (for example, whether it constitutes a noun phrase).
• the system learns how these features contribute to the decision whether a candidate phrase is likely to be a "good" keyword. Once it has been trained in this manner, the system can be used to identify keywords on previously unseen webpages (i.e., that were not in the training set).
• An exemplary system embodiment improves this approach by using natural langauge processing techniques in order to achieve improved performance.
• One or more exemplary embodiments employ a novel keyword candidate extraction method that is sensitive to phrase structure, and may include additional linguistic features that lead to better machine learning results.
• One exemplary aspect comprises a computer system comprising: (a) a
• the tf-idf weight (term frequency-inverse document frequency) is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.
• preprocessing unit that extracts text from a webpage to produce at least a first set of candidate keywords, applies language processing to produce at least a second set of candidate keywords, and combines said first and second sets of candidate keywords into a first candidate pool; (b) a candidate extraction unit that receives data from said
• preprocessing unit describing at least said first candidate pool and produces a second candidate pool
• a feature extraction unit that receives data describing at least said second candidate pool and analyzes said second candidate pool for general features and linguistic features
• a classification unit that receives said data describing at least said second candidate pool and related data from said feature extraction unit, and determines a likelihood of each candidate in said second candidate pool being a primary or secondary keyword.
• At least part of said language processing is performed by a tokenizer and a parser; (2) at least part of said language processing is performed by a tokenizer, a parser, a part of speech tagger, and a named entity tagger; (3) at least part of said language processing is performed by a tokenizer; (4) at least part of said language processing is performed by a parser; (5) at least part of said language processing is performed by a part of speech tagger; (6) at least part of said language processing is performed by a named entity tagger; (7) said first set of candidate keywords comprises metadata text; (8) said second candidate pool comprises noun phrases and noun sequences; (9) said second candidate pool comprises noun phrases, noun sequences, and n-grams; (10) said general features comprise one or more of frequency, position in the document, and capitalization; (11) said linguistic features relate to one or more of part of speech, phrase structure, and named entity information; (12) said general features comprise frequency features, and said frequency features, and said frequency features, and said frequency features, and said
• Another aspect comprises A method comprising steps implemented by a computer processing system, said steps comprising: (a) extracting text from a webpage to produce at least a first set of candidate keywords, applying language processing to produce at least a second set of candidate keywords, and combining said first and second sets of candidate keywords into a first candidate pool; (b) receiving data describing at least said first candidate pool and producing a second candidate pool; (c) receiving data describing at least said second candidate pool and analyzing said second candidate pool for general features and linguistic features; and (d) receiving said data describing at least said second candidate pool and related data from said feature extraction unit, and determining a likelihood of each candidate in said second candidate pool being a primary or secondary keyword.
• Another aspect comprises a tangible computer readable medium storing software operable to perform steps comprising: (a) extracting text from a webpage to produce at least a first set of candidate keywords, applying language processing to produce at least a second set of candidate keywords, and combining said first and second sets of candidate keywords into a first candidate pool; (b) receiving data describing at least said first candidate pool and producing a second candidate pool; (c) receiving data describing at least said second candidate pool and analyzing said second candidate pool for general features and linguistic features; and (d) receiving said data describing at least said second candidate pool and related data from said feature extraction unit, and determining a likelihood of each candidate in said second candidate pool being a primary or secondary keyword.
• FIG. 1 depicts an overview of processing of an exemplary embodiment.
• FIG. 2 depicts a computer system over which an exemplary embodiment may be implemented.
• FIG. 1. Each component is described in further detail in the remaining sections of this description.
• plain text of the page may be extracted from the HTML format.
• this text may be processed further to obtain information about its structure that can be useful to the keyword extraction system.
• the preprocessing unit of the system preferably performs extraction as well as tagging and formatting webpage text, to provide suitable input for the stages of candidate phrase selection and feature extraction that follow.
• content text may be first extracted from the webpage using BoilerPipe (see, e.g., [9]), which removes boilerplate content and preserves only the main text body of the page. Aside from the body text, header information such as title, meta-description, and meta-keywords may be extracted and combined with
• BoilerPipe output to form plain text input for further processing.
• the page text may then be tokenized and the tokenizer output passed to a part-of- speech tagger (see, e.g., [18]) and a parser (see, e.g., [13]). Since there is a tendency for keywords to constitute noun phrases, parser output may be used to find noun phrases in the text.
• the use of a parser rather than a chunker may be motivated by the desire to obtain finer-grained information on hierarchical phrase structure, as opposed to basic noun phrase chunks, in order to improve keyword candidate extraction.
• NE Named Entities
• Two different NE systems see, e.g., [18], [4]) preferably are used in order to provide coverage of a larger set of entity types.
• Candidate extraction may be used to select phrases that are potential keywords and can be used as input for the classifier which estimates the likelihood that a given phrase is a keyword.
• better accuracy of candidate extraction helps to filter word combinations that are not likely keywords and thus reduces the amount of negative training samples, thereby improving the ratio of positive to negative training data (the keyword extraction task has an imbalance between positive and negative samples, with very few positive label data).
• a keyword extraction method performs as follows. First, a base candidate set is formed by recursively extracting all noun phrases from parsed text. Then all candidate subsequences (extracted left to right) that consist of nouns only are added to the candidate set (for example, if best Nixon camera accessories is the candidate, Nixon camera accessories, camera accessories and accessories would be added to the candidate set). Finally, the candidate set is augmented with all unigrams, bigrams, and trigrams extracted from the candidate phrases.
• the candidate set may also be filtered against a stoplist of most frequent English words. Unigrams or bigrams containing a stopword preferably are removed from the candidate set. However, longer phrases containing a word from the stoplist in the middle of the phrase may be retained.
• an exemplary embodiment employs a classifier that uses the input (features of the candidate phrase) to estimate the probability that the phrase is a keyword, and assigns an output label
• the classifier function that maps the feature input to a keyword label may be obtained using supervised machine learning. That is, the mapping may be learned by the classifier system based on a dataset where "correct" output labels have been provided by human annotators.
• a maximum entropy (ME) model may be used (this is sometimes called a the logistic regression model; for an introduction, see [11]).
• An ME model derives constraints from the training data and assumes a distribution of maximum entropy in cases not covered by the training set.
• the ME classifier input consists of vectors of values for each keyword candidate, which are used by the model to learn the weights associated with each feature. Given new input data, the trained classifer can then compute the probability that a phrase is a keyword given the input values for that candidate phrase.
• f is a joint- feature (a function of the input vector and the label) and a is a weight assigned to that feature.
• CG Natural Language Toolkit
• rbf kernel support- vector machines
• CG refers to the Conjugate Gradient method, a standard iterative method to solve sparse linear equation systems that is provided as one of the training methods in the classifier library.
• CG requires the scipy package (http://www.scipy.org/) to be installed with Python and NLTK. candidates with the highest probabilities in a given webpage.
• a set of feature values may be computed for each keyword candidate and used as classifier input.
• the choice of features plays an important role in classifier performance.
• the features may be divided into two types: (a) general, non-linguistic features, and (b) linguistic features.
• General features may be similar to the features employed by the system described in [17] and include information such as frequency, position in the document, and capitalization. Linguistic features make use of part of speech, phrase structure, and named entity information. The two types of features are described in more detail below.
• HasCap 1 if at least binary YES one word in
• IsNoun 1 if all words binary YES(but in keyword defined candidate are differently nouns, 0 with a otherwise. distinction between proper an generic nouns) hasNoun 1 if at least binary YES one word in
• HasNE_oak 1 if keyword binary NO
• Frequency features provide information similar to TFxIDF . Frequency features
• TFxIDF refers to term frequency-inverse document frequency and is a standard score used in information retrieval to evaluate the relative importance of a term. It is based on frequency of the term in a given may include relative term frequency within the document, log of term frequency, as well as DF (frequency in document collection) and log DF values. DF values may be approximated using frequencies from Google Ngram corpus. Preferably only unigram and bigram frequency information are used to calculate DF. For candidate phrases longer than 2 words, the average of DFs for all bigrams in the phrase may be used as the DF value. Averages may be used in order to obtain a similar range of values for phrases of different length. Also, DF values computed for the entire blog collection may be used, instead of the frequencies from the Google Ngram corpus.
• Capitalized words include proper names or words marked as important terms in a given document. Exemplary capitalization features are: whether all words in keyword candidate are capitalized, and whether at least one word in a candidate phrase is capitalized.
• wikipedia traffic statistics may be used to reflect the popularity of keyword candidates as frequent search/query items. This set of features may include: whether the candidate phrase is a wikipedia title (including redirects), and the traffic figure for the candidate phrase (0 if the candidate is not a wikipedia title). Traffic statistics may be based on hourly wikipedia logs aggregated over a certain period (e.g., a 20 day period in June 2010).
• the candidate is a Noun Phrase or contains a Noun Phrase.
• the candidate phrase contains at least one noun, and whether the candidate phrase consists of nouns only.
• a keyword candidate is a Named Entity, whether it contains a Named Entity and the Named Entity tag assigned to the candidate ("O" if the candidate phrase is not an NE).
• Pointwise mutual information reflects whether a phrase is likely to be a collocation.
• a PMI score of a candidate phrase may be calculated as follows:
• PMI may be set to the average of PMI scores for all bigrams in the phrase.
• the training data may comprise, say, 500 web pages (selected randomly from a blog page corpus; see [3]). Annotators may be presented with plain-text extracted from the blog page and instructed to select keywords that best express the content of the page. Meta information from the header preferably is not included in the annotated text.
• Keywords there is no limit on the number of keywords that may be chosen for a single page. Additional pages may also be annotated and set aside as a test set not used for training.
• the keywords preferably are selected by two annotators.
• the inter- annotator agreement on this task is might not be high (for example, in one implementation, the kappa score 4 of annotators was 0.49.
• Low kappa scores may be due to the following: First, annotators may tag similar phrases that are only partial matches. Second, when a maximum number of keywords that can be selected is not specified, one annotator may choose to select a greater number of keywords than another for a given text.
• GS Golden Standard
• annotators may be instructed to also select whether the keyword is a "primary keyword” or a "secondary keyword.”
• Primary keywords may be defined as keywords
• Cohen's kappa coefficient is a statistical measure commonly employed to measure agreement between
• Kappa is calculated as where P(A ) is the observed l - P(E)
• kappa scores between each annotator and the standard were 0.75 for annotator 1 and and 0.74 for annotator 2.
• Detailed agreeement statistics for primary and secondary keywords are shown in Table 2 below.
• an exemplary embodiment uses noun phrases as a base candidate set, but augments the candidate pool with noun sequences and unigrams, bigrams, and trigrams extracted from the noun phrases.
• One prior art method of obtaining all possible candidate phrases from a text is to include all n-grams up to length n (typically 3-5) in the candidate set.
• n typically 3-5
• a serious disadvantage of this n-gram method is that it introduces substantial noise, in the form of word sequences that are not meaningful phrases and/or are not likely to be potential keywords. The n-gram method thus suffers from low precision.
• An alternative prior art method is to use language structure cues to extract candidates. Since keywords tend to be noun phrases, all noun phrases from the text can be used to form the candidate pool. However, this method has a markedly lower recall than the n-gram extraction method, which means that many potential keywords are not included in the candidate set.
• the n-gram approach has a recall above 80%, but it also has the lowest precision of the three methods (i.e., the candidate set includes a substantial amount of noise). Extracting noun phrases as candidates has the advantage of increasing precision, but this method has a very low recall (only 26%), so there is a high chance of missing potential keywords.
• an exemplary embodiment of the inventive method results in an improvement in recall compared to extracting noun phrases.
• the recall of this approach is comparable to the n-gram method, but the precision is higher. Evaluation results of how the different methods combine with classifier performance are described below.
• the results achieved by the inventive system were compared to a baseline, based on [17].
• the candidate extraction method is the n-gram method, and features consist of general non-linguistic features (plus a simple set of NP/Noun features). How system
• Top- 10 score (like R-Precision but with a cut-off at top- 10 results, i.e. all ⁇ > 10 are set to 10).
• the top- 10 measure was used for evaluation since it provides an estimate of how the classifier performs as an extraction system when the candidates with top- 10 scores are selected as the keyword output.
• System performance was tested on a held-out test set of 100 webpages which were never used in classifier training (see Table 4) and
• Table 4 Top- 10 score results for the held-out set.
• Table 5 Top-10 score results for cross-validation tests.
• keyword extraction preferably comprises: (a) preprocessing, which includes text extraction from the webpage as well as linguistic processing such as part of speech tagging and parsing; (b) extraction of keyword candidate phrases; and (c) candidate classification using supervised machine learning.
• the inventive systems and methods may achieve improved performance due to use of linguistic information, both at the candidate selection and at the feature extraction stage.
• An exemplary embodiment comprises candidate selection that uses hierarchical phrase structure, resulting in a less noisy candidate pool.
• Features that may be used for classification also include linguistic features such as part of speech and named entity information, resulting in improved classifier performance.
• Embodiments comprise computer components and computer-implemented steps that will be apparent to those skilled in the art. For example, calculations and communications can be performed electronically, and results can be displayed using a graphical user interface.
• Computers 100 communicate via network 110 with a server 130.
• a plurality of sources of data 120-121 also communicate via network 110 with a server 130, processor 150, and/or other components operable to calculate and/or transmit information.
• Server(s) 130 may be coupled to one or more storage devices 140, one or more processors 150, and software 160.
• Server 130 may facilitate communication of data from a storage device 140 to and from processor(s) 150, and communications to computers 100.
• Processor 150 may optionally include or communicate with local or networked storage (not shown) which may be used to store temporary or other information.
• Software 160 can be installed locally at a computer 100, processor 150 and/or centrally supported for facilitating calculations and applications.
• processing and decision-making may be performed by functionally equivalent circuits such as a digital signal processor circuit or an application specific integrated circuit.
• Boilerplate detection using shallow text features WSDM '10: Proceedings of the third ACM international conference on Web search and data mining, pages 441—450, New York, NY, USA, 2010. ACM. [10] Matsuo, Y. and Ishizuka, M. Keyword Extraction from a Document using Word Co-occurrence Statistical Information. Transactions of the Japanese Society for Artificial Intelligence, 17:217-223, 2002.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• Databases & Information Systems (AREA)
• Data Mining & Analysis (AREA)
• Computational Linguistics (AREA)
• Artificial Intelligence (AREA)
• Audiology, Speech & Language Pathology (AREA)
• General Health & Medical Sciences (AREA)
• Health & Medical Sciences (AREA)
• Mathematical Physics (AREA)
• Human Computer Interaction (AREA)
• Library & Information Science (AREA)
• Probability & Statistics with Applications (AREA)
• Multimedia (AREA)
• Business, Economics & Management (AREA)
• General Business, Economics & Management (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Machine Translation (AREA)
• Document Processing Apparatus (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=46020615

Family Applications (1)

Country Status (6)

Cited By (2)

Families Citing this family (72)

Citations (4)

Family Cites Families (22)

• 2011
  • 2011-11-02 US US13/287,294 patent/US8874568B2/en active Active
  • 2011-11-02 JP JP2013537776A patent/JP5990178B2/ja active Active
  • 2011-11-02 WO PCT/US2011/058899 patent/WO2012061462A1/en not_active Ceased
  • 2011-11-02 KR KR1020137011659A patent/KR101672579B1/ko active Active
  • 2011-11-02 CN CN2011800531753A patent/CN103201718A/zh active Pending
  • 2011-11-02 EP EP11838723.2A patent/EP2635965A4/en not_active Withdrawn

Patent Citations (4)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events