US20230385704A1 - Systems and method for performing contextual classification using supervised and unsupervised training - Google Patents
Systems and method for performing contextual classification using supervised and unsupervised training Download PDFInfo
- Publication number
- US20230385704A1 US20230385704A1 US18/446,980 US202318446980A US2023385704A1 US 20230385704 A1 US20230385704 A1 US 20230385704A1 US 202318446980 A US202318446980 A US 202318446980A US 2023385704 A1 US2023385704 A1 US 2023385704A1
- Authority
- US
- United States
- Prior art keywords
- comments
- objects
- user
- classification
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012549 training Methods 0.000 title claims abstract description 105
- 238000000034 method Methods 0.000 title claims abstract description 93
- 238000004458 analytical method Methods 0.000 claims abstract description 16
- 238000012545 processing Methods 0.000 claims description 48
- 238000010801 machine learning Methods 0.000 claims description 37
- 238000004422 calculation algorithm Methods 0.000 claims description 32
- 230000008569 process Effects 0.000 claims description 19
- 230000015654 memory Effects 0.000 claims description 14
- 238000012706 support-vector machine Methods 0.000 claims description 10
- 238000001914 filtration Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 abstract description 27
- 238000012552 review Methods 0.000 abstract description 4
- 230000008901 benefit Effects 0.000 description 6
- 239000013598 vector Substances 0.000 description 5
- 238000013145 classification model Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 241000288113 Gallirallus australis Species 0.000 description 3
- 238000013499 data model Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 238000003066 decision tree Methods 0.000 description 3
- 239000012634 fragment Substances 0.000 description 3
- 239000003795 chemical substances by application Substances 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000010200 validation analysis Methods 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 230000000881 depressing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000036651 mood Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
- G06N5/022—Knowledge engineering; Knowledge acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/285—Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
Definitions
- the present disclosure relates to content assessment and computerized systems and methods for performing contextual classification of objects. More particularly, and without limitation, the present disclosure relates to systems and methods for selecting among a plurality of algorithms or models, in parallel, on a pluggable platform, and performing contextual classification using both supervised and unsupervised training.
- Contextual classification of objects enables content providers to determine that objects share similar properties and group those objects with one another to meet various needs or objectives. For example, content providers may seek to group like objects together, so that content consumers can easily access similar content. As another example, objects may be contextually classified to facilitate filtering of content, such as by classifying and removing inappropriate content.
- Contextual classification may be performed on a wide variety of objects, such as web pages or finite objects like text, images, videos, and other content.
- contextual classification may be performed to group web pages by topic, thus improving the organization of a web site.
- content providers make it easier for users to find content that suits their interests, thus increasing the likelihood that users will view multiple web pages within a web site upon finding a single web page on that web site that suits their interests or needs.
- Prior methods for contextual classification have relied primarily on manual input by editors or users. For example, some prior techniques have relied on manual tagging by persons designated as content reviewers. Such content reviewers may be employees of a particular company, such as the editors of a particular web site, or they may also be users of a web site. Regardless of the criteria for selection of persons to review and classify content, there are several disadvantages associated with the manual classification of objects. For example, manual tagging of objects is time-consuming. For content providers with numerous objects to classify, manual classification may take so long as to decrease the utility of the classification.
- an online news provider that relies on manual classification of news stories will need stories classified quickly in order to guide users to the stories in which they are interested before other news sources capture the users' attention (or before the news becomes stale). If a content provider releases a large volume of new content per day, it may be unfeasible to have each object tagged within an acceptable time frame.
- Contextual classification may involve classification of an object based on a single parameter or on numerous parameters. For example, a web page may be classified based on genre (e.g., politics, religion, sports), audience (e.g., children, adults), mood (e.g., funny, depressing, inciting), and numerous other parameters. Further, whereas some parameters may be assigned a binary value (e.g., true/false, yes/no), other parameters may be assigned a wide variety of values and require more analysis to determine the appropriate value.
- the present disclosure is directed to addressing one or more of the above-referenced challenges or drawbacks with conventional methods and techniques for contextual classification.
- the present disclosure provides improved systems and methods for performing contextual classification using supervised and unsupervised training.
- certain embodiments of the present disclosure may utilize parallel machine learning to determine the best model/parameter combination for contextual classification of objects based on supervised and unsupervised training.
- Exemplary implementations of the disclosed embodiments include article classification and comment moderation.
- systems and methods are provided for performing contextual classification of objects using supervised and unsupervised training.
- Embodiments consistent with the present disclosure include, for example, systems and methods for collecting supervised training data from content reviewers, preprocessing the supervised training data to create vectorized objects, and training models using the vectorized objects.
- the disclosed systems and methods may be used to automatically classify objects based on the models.
- a computer-implemented method for performing contextual classification of objects.
- supervised training data is received from at least one content reviewer and stored in a database.
- the supervised training data may be preprocessed to form at least one vectorized object.
- the at least one vectorized object may be used to train a plurality of models by applying a plurality of machine learning algorithms thereto.
- An optimal model may be identified from the plurality of models and used to classify a candidate object.
- a computer-implemented system for performing contextual classification of objects.
- the system includes a database and a processor.
- the database may be used to store supervised training data collected from at least one content reviewer.
- the processor may be configured to preprocess the supervised training data to form at least one vectorized object.
- the processor may be configured to train a plurality of models by applying a plurality of machine learning algorithms to the at least one vectorized object. From the plurality of models, an optimal model may be identified and used to classify a candidate object.
- a computer-readable medium including instructions for performing contextual classification of objects.
- the instructions When executed, the instructions perform steps including collecting supervised training data from at least one content reviewer. The steps further include preprocessing the supervised training data to form at least one vectorized object.
- the at least one vectorized object may be used to train a plurality of models by applying a plurality of machine learning algorithms thereto. An optimal model may be identified from the plurality of models and used to classify a candidate object.
- FIG. 1 illustrates exemplary phases from an exemplary method for performing contextual classification of objects, in accordance with an embodiment of the present disclosure.
- FIG. 2 illustrates a flow diagram of an exemplary method for performing contextual classification of objects, in accordance with an embodiment of the present disclosure.
- FIG. 3 illustrates an exemplary diagram for employing parallel machine learning to classify objects, in accordance with an embodiment of the present disclosure.
- FIG. 4 illustrates exemplary steps for preprocessing training data to create vectorized objects, in accordance an embodiment of the present disclosure.
- FIG. 5 illustrates exemplary steps for training one or more models using vectorized objects, in accordance with an embodiment of the present disclosure.
- FIG. 6 illustrates an exemplary system environment for implementing embodiments of the present disclosure.
- FIG. 7 illustrates a flow diagram of an exemplary method for performing contextual classification of objects, in accordance with an embodiment of the present disclosure.
- prior methods of contextual classification have relied primarily on manual classification of objects. Specifically, prior methods have required content providers to task personnel with analyzing and classifying content according to specified criteria. This sort of manual classification of content can also be referred to as “supervised” classification. Although supervised classification of objects can suffer from certain inefficiencies (e.g., time-consuming, costly), supervised classification can also provide certain benefits. Perhaps the biggest advantage of supervised classification of objects is accuracy or reliability of classification. Content providers that employ editors dedicated to the sole task of classifying content are likely to have more reliably classified content than those providers that may classify content solely through automated methods.
- unsupervised classification may be performed automatically using machine learning techniques, thus overcoming the above-discussed disadvantages of supervised classification methods.
- supervised classification methods can be faulted for being time-consuming and costly, the automated nature of unsupervised classification provides for contextual classification of objects to be performed quickly and cheaply.
- a drawback of unsupervised classification is that unsupervised classification, if not properly tuned, may be unreliable. Accordingly, to perform contextual classification of objects quickly, cheaply, and reliably, embodiments consistent with the present disclosure may incorporate both supervised and unsupervised classification methods.
- supervised classification may be performed by persons employed by a content provider for that purpose. This may be particularly suitable for content providers that produce a high volume of content and have the resources to employ persons dedicated to contextual classification of their content.
- supervised classification of a particular content item e.g., an article
- content providers may allow users (i.e., content consumers) to perform supervised classification of content. This responsibility may be provided to all users or only to users who meet certain criteria. For example, the authority to perform supervised classification of content may be provided to certain users who have been given “moderator” or “expert” status.
- a user may achieve such a status upon a determination that a threshold number of classifications made by the user were proper. Such a determination may be made, for example, when a threshold number of provisional classifications provided by a user were ultimately adopted, either through validation by employees of the content provider or through automated validation methods (e.g., the unsupervised classification methods disclosed herein).
- Supervised classification of objects may be performed by, among other methods, tagging objects with pre-defined tags.
- a content reviewer may be presented with a pre-defined set of tags that may either be selected or left unselected.
- tags may be associated with a binary value (e.g., true/false, yes/no, 0/1) to indicate whether the tag is properly associated with the object.
- tags may be assigned one of several different values. For example, whereas a tag for “Scary?” may be assigned a binary value (e.g., yes/no, true/false, scary/not scary), a tag for “genre” may be assigned one of a broader variety of values, as genre is not susceptible to binary classification.
- a genre tag may be assigned one of a pre-defined set of values: politics, religion, sports, opinion, or business.
- the possible values for a given tag are pre-defined by an editor associated with the content provider who is familiar with the range of topics that are likely to be published by the content provider.
- content providers may ensure that similar content is tagged similarly. In other words, two different content reviewers will not be able to tag one article as relating to the “election” genre and another article as relating to the “politics” genre when the content provider would prefer all political articles to be clustered together as belonging to the “politics” genre.
- tags may be assigned open-ended values, such as by providing an input box associated with a tag.
- a content reviewer may review an object and assign the object the value that he or she sees fit, without being restricted only to a pre-defined set of values.
- this alternative embodiment may allow for more finely tuned or better descriptive values, similar articles are likely to be tagged differently because of their dependence on the subjective judgments of persons having different vocabularies.
- embodiments disclosed herein may utilize unsupervised classification methods.
- unsupervised classification may be performed automatically without direct human supervision or interaction, thus allowing unsupervised classification to be performed more cheaply and quickly than supervised classification.
- objects may be analyzed to identify key terms that are descriptive of the object as a whole.
- analyses may involve preprocessing of an object to create a list of terms (e.g., all terms associated with an object or only key terms), such as by identifying terms based on tokenization, stemming, n-gram analysis, phrasal analysis, or any other method for segmentation and representation of objects by a plurality of terms. These terms may be used automatically to assign tags to objects.
- the terms may be compared to one or more lists of terms known to be associated with candidate tags.
- the object may be tagged.
- the tags/classifications and the known terms assigned thereto may have been designated by an editor or other person associated with a content provider, thus incorporating an element of supervised classification.
- Unsupervised classification methods may also use previously classified objects to classify other objects. For example, an object may be classified using either supervised classification (e.g., manually tagging by editors) or unsupervised classification (e.g., automated tagging based on term analysis) and then compared to unclassified objects for similarities. If a classified object is sufficiently similar to an unclassified object, the unclassified object may be tagged with one or more of the same classifications as the classified object. The similarity of objects may be analyzed based on key terms derived from the object or a comparison of all terms from the objects.
- supervised classification e.g., manually tagging by editors
- unsupervised classification e.g., automated tagging based on term analysis
- classification of objects is performed using machine learning techniques. Such techniques may be used to design algorithms that recognize patterns within objects and make intelligent decisions based on input data.
- contextual classification of objects incorporates machine learning to identify key terms or phrases within objects and used that information to classify objects into various categories based on training data. Further, unlike prior methods of machine learning, certain embodiments take advantage of parallel processing, so that the training and classification processes may be performed much more quickly than in prior machine learning techniques.
- FIG. 1 depicts two phases of an exemplary machine learning process, in accordance with one embodiment.
- machine learning may comprise both a learning phase 110 and a classifying phase 120 .
- objects may be tagged with various values that describe the objects.
- individual terms or phrases within an object may be tagged as being associated with certain values.
- the learning phase 110 may utilize the supervised classification methods discussed herein. For example, in the learning phase 110 , information regarding the tags assigned to an object by editors or others performing supervised classification may be collected to learn what sort of information is tied to each tag/classification. This information may then be used to train one or more models for use in the classifying phase 120 .
- one or more models trained in the learning phase 110 are used to classify previously unclassified objects.
- This phase may be considered an unsupervised element of the machine learning process, as such classifications are performed automatically, albeit based on data input during a supervised portion of the machine learning process (i.e., the learning phase 110 ).
- object data may be parsed in accordance with a model developed during the learning phase 110 . As discussed in more detail below, the nature of the parsing applied to an object may vary, as the optimal model for one class of objects may differ from that corresponding to another class of objects.
- the model is then utilized to classify the object into one or more classifications. In one embodiment, this classification may be verified using supervised classification techniques (i.e., an editor may validate the classification). This classification may also be used as additional training data that is input into the learning phase 110 and used to train a new model and/or refine a previous model.
- objects are analyzed to determine whether they may be classified as “business” objects.
- objects are preprocessed to identify various terms and/or phrases for classification.
- the terms are identified by editors or other persons as part of supervised classification.
- these terms and/or phrases may include sentences or sentence fragments from an object.
- terms identified from objects for training in the learning phase 110 may be single words, n-grams, tokens, stems, phrases, or other combinations of letters and/or words, as discussed in more detail herein.
- the identified terms have been classified as either “business” or “non-business.”
- these classifications are supervised classifications. In other words, the classifications are made by editors, users, or other persons who have analyzed the objects.
- the terms “Investments are taxed as . . . ” and “the top tax bracket for . . . ” have been classified as business terms. Accordingly, the objects associated with these terms may be classified as business objects. In one embodiment, the classification of an object containing these terms may be based on a determination of whether a threshold number of terms classified as business terms are present in the object. Moreover, the determination of whether to classify an object containing these business terms as a business object may take into consideration the number of terms classified as non-business terms, as a high number of non-business terms relative to a low number of business terms may indicate that the object as a whole is less likely to be appropriately classified as a business object. As shown in FIG.
- an automated (i.e., unsupervised) process may have determined that this term should be classified as non-business due to the proximity of the phrase “hedge fund” to the word “nephew.”
- Data pertaining to these classified terms and/or objects are the used to train one or more models, which may then be used to classify additional terms and/or objects.
- the term “capital gains to be taxed” is input into one or more models that were trained during the learning phase 110 .
- the one or more models compare the term with other terms to determine whether the term should be classified as a business or a non-business term.
- the term “capital gains to be taxed . . . ” is classified as a business term. This determination may be based on a comparison of the term to terms or objects classified during the learning phase 110 and input into the one or more models employed as part of the classifying phase 120 .
- one or more terms, or sub-terms, from within the term “capital gains to be taxed . . . ” may be compared with the business and non-business terms used to train the one or more models employed in the classifying phase 120 .
- the terms “capital gains” and “taxed” may be compared to terms, or sub-terms, analyzed during the learning phase 110 to determine that these sub-terms are business terms.
- the term “capital gains to be taxed . . . ,” as a whole, may be classified as a business term.
- FIG. 2 depicts a general processing flow of contextual classification of objects using supervised and unsupervised training, in accordance with an embodiment.
- objects and/or their terms may be classified into various classifications to create training objects (or “training docs”) 210 .
- training docs are then preprocessed 220 to create one or more vectorized objects (or “vectorized docs”) 230 .
- vectorized docs 230 are used in training 240 to develop one or more models 250 .
- an optimal model for contextual classification may be determined and used for future classification of objects.
- Training docs 210 may be developed in a manner similar to that described with respect to learning phase 110 of FIG. 1 . As described with respect to FIG. 1 , this may include supervised classification performed by editors or users who have reviewed one or more objects and assigned one or more classifications to those objects and/or one or more terms within those objects. Further, as described above, these classifications may be performed by classifying an object, or a term within an object, by selecting one or more classifications from a pre-defined set of classifications, such as by clicking on a button associated with a particular classification while reviewing the object or term. Alternatively, the classifications may be performed by typing a descriptive label in an input box associated with the object or term. In one embodiment, all objects and terms that have been classified are used in the creation of training docs.
- a subset of objects and/or terms that have been classified using the supervised classification techniques disclosed herein may be used as training docs.
- persons involved in the supervised classification of objects, or a subset of such persons comprised of managers or supervisors may determine which objects or terms are most indicative of a particular classification and, thus, best candidates for training docs.
- content providers may ensure that only the objects or terms that are most closely tied to a particular classification are used in subsequent steps of the general processing flow shown in FIG. 2 to train models for classification of yet unclassified objects or terms.
- preprocessing 220 may employ one or more of tokenizers, stopword filters, stemmers, feature filters, or vectorizers. Each of these preprocessors may parse information from training docs and put the information in a form better suited for training one or more models used for contextual classification. For example, preprocessing 220 may remove stopwords from training docs that are unlikely to be strongly connected to any particular classification.
- Exemplary stopwords may include articles, such as “a,” “an,” and “the,” as well as prepositions, such as “of,” “at,” and “on.” Alternatively, stopwords may be left in training docs to support other processing functions, such as phrasal analysis, which would be adversely affected by the removal of certain commonly used terms (e.g., prepositions common used in prepositional phrases).
- Preprocessing 220 may also include tokenization of one or more terms or phrases of the trainings docs. Tokenization may separate the words or phrases associated with a training doc into smaller words, phrases, or fragments (i.e., smaller “tokens”). Numerous methods may be employed for tokenizing training docs as part of preprocessing 220 . For example, phrases associated with a training doc may be separated into individual words, as determined by spacing or any other means for detecting separation of words. Alternatively, words or phrases associated with a training doc may be separated into n-grams. Accordingly, preprocessing 220 may divide words or phrases in a training doc into fragments consisting of a pre-defined number of characters (i.e., n-grams).
- one n-gram size may be pre-defined for all training docs used by a content provider employing the contextual classification techniques disclosed herein.
- n-gram size may vary based on the subject matter of a particular object or training document or based on the average word size for the words associated with the object or training doc.
- n-gram size may be set either manually by an editor or other content reviewer or be set automatically.
- Preprocessing 220 may include stemming of words associated with an object or training doc.
- Stemming is a process of reducing words to their stem, base, or root form. Stemming allows for different words with similar stems to be treated similarly in the training processes disclosed herein. For example, if a training doc designates the words “election,” “electoral,” and “electable” as associated with politics, these words may each be stemmed to the term “elect,” and the word “elect” may be classified as being associated with politics.
- later processing steps e.g., training
- preprocessing 220 converts the one or more training docs 210 into one or more vectorized docs 230 .
- Vectorized docs 230 may be clustered such that similar docs or objects are grouped together for training.
- vectorized docs 230 that relate to business may be clustered together, such that these documents may be input together into the processes of training 240 .
- Training 240 inputs data from vectorized docs 230 into various algorithms to generate one or more models 250 .
- training 240 may support and/or implement a variety of algorithms, such as, but not limited to, support vector machine (SVM), Perceptron/Winnow, Bayesian, decision tree, and AdaBoost.
- SVM support vector machine
- Perceptron/Winnow Perceptron/Winnow
- Bayesian Bayesian
- AdaBoost AdaBoost
- different objects or object types will be better suited to some algorithms than others. Accordingly, training 240 may involve running each vectorized doc (or group of vectorized docs) through multiple algorithms, so that the best algorithm for each vectorized doc (or group of vectorized docs) may be determined.
- vectorized docs 230 may be input into an SVM algorithm to develop one or more models appropriate for contextual classification of a specified object type.
- Vectorized docs 230 may comprise data linking terms or phrases to one of two possible categories, such as the “business” or “non-business” classifications discussed above with respect to FIG. 1 .
- the SVM may take the vectorized docs 230 and build a model that assigns new data into one category or the other.
- the SVM algorithm may train a model based on the vectorized docs, such that the model will properly classify future data (e.g., vectorized docs or objects) as falling into one of two potential categories.
- Prior machine learning techniques run data through algorithms sequentially, thus potentially requiring a large amount of time to develop models corresponding to each algorithm and each set of data.
- training 240 is performed on multiple vectorized docs (or groups of vectorized docs) 230 in parallel.
- the parallel machine learning techniques of the embodiments disclosed herein provide significant advantages over prior machine learning techniques. For example, by dividing machine learning processes over fifty machines, the training processes associated with contextually classifying objects may be performed up to fifty times faster than if the processes were performed sequentially on one machine. This increased processing speed can allow for content providers to generate and compare a much larger number of classification models, thus increasing the likelihood that the optimal model will be chosen. Moreover, content providers that produce a large volume of new and constantly changing content may need to determine optimal models for classification of new content types quickly, so that the content is properly classified before it becomes stale.
- Certain embodiments may leverage parallelism in a Map/Reduce paradigm in order to generate a large number of machine learning models and find the best algorithm and parameters. Moreover, certain embodiments may also provide a pluggable placeholder for heterogeneous tools and algorithms.
- the parallel machine learning processes disclosed herein may be implemented using Hadoop, as discussed below in connection with FIG. 5 .
- Several parallel machine learning tools are available, including Mahout, Mallet, libsvm, and Weka. Mahout does not yet support the necessary classification algorithms, and other currently available tools do not run natively on Hadoop. Accordingly, the embodiments disclosed herein are designed for a flexible machine learning platform running Hadoop. This platform supports a wide range of algorithms and leverages publicly available implementations.
- this platform may be used efficiently to generate and test tens or hundreds of thousands of models and choose the best models from among this large pool to use for future classification of objects.
- prior sequential parallel machine learning techniques employed in other fields took up to a week, depending on the data input into the models and the number of total models trained, the disclosed parallel machine learning techniques may generate an optimal model for contextual classification of an object pool within an hour.
- FIG. 3 depicts an exemplary parallel processing flow for contextually classifying objects, in accordance with certain embodiments.
- training docs 310 may be preprocessed to generate one or more vectorized docs (or vectorized doc groups) 320 , 330 , and 340 .
- Each of these vectorized doc groups may then be used to train one or more models, such as models 350 - 390 , using one or more machine learning algorithms.
- both the preprocessing of training docs and training of models using vectorized docs may be performed in parallel over one or more machines, such as by using Hadoop.
- FIG. 4 depicts an exemplary preprocessing technique, in accordance with certain embodiments.
- training data may be comprised of term-classification pairs, which may be stored in training docs, such as training docs 210 or 310 .
- the exemplary training data 410 shown in FIG. 4 , indicates that the term “Investments are taxed as capital gains” is paired with the “business” classification and “I am afraid we may be headed for . . . ” is paired with the “none” classification (or is not assigned to any particular classification).
- preprocessing may be performed on training data 410 , in accordance with preprocessing request 420 . As seen in FIG.
- preprocessing request 420 may specify parameters, such as n-grams, stems, or stopwords, as well as one or more values for each of those parameters.
- Preprocessing 430 may be performed as specified by preprocessing request 420 to generate one or more vectors, such as vectors 440 - 465 .
- preprocessing 430 may be performed sequentially on one machine.
- preprocessing 430 may be performed over multiple machines in parallel (e.g., a Hadoop cluster). These vectors may then be used to train models for use in future classification of objects.
- FIG. 5 illustrates an exemplary training flow, in accordance with certain embodiments.
- Vectors 510 containing preprocessed training data are used in conjunction with one or more train requests, such as train request 520 , to train one or more models, such as models 540 - 565 .
- Train request 520 may include information specifying one or more algorithms, along with associated parameters, for application to vectors 510 .
- train request 520 depicts six parameter sets, each of which relates to the balanced winnow training algorithm.
- train requests are run sequentially on one machine, as part of training 530 .
- train requests may be run in training 530 in parallel over multiple machines (e.g., a Hadoop cluster).
- Training 530 may be performed using one or more machine learning tools, such as Mahout, Weka, Mallet or Mallet+SVMLite to generate models 540 - 565 .
- the disclosed methods and systems for contextual classification of objects may be used in a variety of applications and implementations.
- the disclosed methods and systems may be used for article classification.
- articles may be classified with one or more tags using the supervised classification methods described herein.
- an article may be tagged by the author of the article, an editor of the article, another employee of the content provider publishing the article, or a reader of the article.
- Such a person may select one or more pre-defined tags and/or input one or more additional tags, such as by using a text input box.
- the article may then be suitable for use in training one or more models to classify other articles.
- the article is first preprocessed to form one or more vectorized objects, such as by performing tokenization, stemming, and/or n-gram analysis on the article.
- the one or more resultant vectorized objects are then used to train one or more models for classification of future articles.
- the tags/classifications applied to an article by its author, editor, reader, or other reviewer may be used to develop one or more models for automatically classifying similar articles in the future.
- many models may be trained and compared, such that the most efficient and accurate model may be chosen for future use as a classifying model.
- the disclosed systems and methods may be used for comment moderation.
- Content providers often provide the ability for their consumers to comment on their content.
- user comments are displayed along with (usually below) the subject content.
- comments are generally written by third-parties.
- user comments should also be reviewed to ensure that their content will not be objectionable to readers.
- manual review of comments is unfeasible. Accordingly, the disclosed embodiments may be employed to filter automatically through user comments and reject or publish the comments, as appropriate.
- comment moderation may begin with supervised classification of comments.
- supervised classification may be performed, for example, by the author or editor of the article that is the subject of the comment, some other employee or agent of the content provider, or a third-party user or subscriber of the content provider.
- a third-party user or subscriber may be given authority to moderate comments based on a determination that the user is a “trusted” user. For example, the user may be given “trusted” user status if a threshold of comments flagged by the user as inappropriate were also flagged by other users as inappropriate or classified as inappropriate by the unsupervised (i.e., automated) classification techniques disclosed herein.
- supervised classification of comments may be comprised simply of tagging or flagging the comment as abusive or not abusive.
- other classifications may be applied to comments, such as helpful, funny, recommended, or off-topic.
- the comments may then be preprocessed to obtain vectorized data for use in training models for moderation of future comments. For example, key terms may be identified (e.g., by tokenization, stemming, or n-gram analysis) and associated with a certain classification (e.g., abusive, non-abusive). Accordingly, profane or obscene terms may be classified as abusive in these vectorized objects.
- the models Once the models have been trained using the vectorized data, they may be compared to determine the best model to use for comment moderation. In one embodiment, different models may be developed for moderation of comments directed to different subject matter.
- comments deemed aggressive or abusive when directed to an article focused on education may not be deemed aggressive or abusive when directed to an article focused on sports.
- the model may be applied to moderate future comments directed to the same article or subject matter.
- Embodiments herein include computer-implemented methods, tangible non-transitory computer-readable mediums, and computer-implemented systems.
- the computer-implemented methods may be executed, for example, by one or more processors that receive instructions from one or more non-transitory computer-readable storage mediums.
- the systems disclosed herein may include at least one processor and memory, and the memory may be a non-transitory computer-readable storage medium.
- a non-transitory computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage medium. Singular terms, such as “memory” and “computer-readable storage medium,” may additionally refer to multiple structures, such a plurality of memories and/or computer-readable storage mediums.
- a “memory” may comprise any type of computer-readable storage medium unless otherwise specified.
- a computer-readable storage medium may store instructions for execution by a processor, including instructions for causing the processor to perform steps or stages consistent with an embodiment herein. Additionally, one or more computer-readable storage mediums may be utilized in implementing a computer-implemented method.
- the term “computer-readable storage medium” should be understood to include tangible items and exclude carrier waves and transient signals.
- FIG. 6 illustrates an exemplary system environment for implementing embodiments of the present disclosure.
- the learning phase 110 of FIG. 1 may use data 610 , 620 , and 630 stored in object database 640 to train one or more models using processing engine 650 .
- classifying phase 120 of FIG. 1 may involve using processing engine 650 to classify models 670 , 680 , and 690 and store them in model database 660 .
- the training docs/data 210 , 310 , and 410 shown in FIGS. 2 - 4 may correspond to objects 610 , 620 , and 630 , which may be stored in object database 640 .
- Models 250 , 350 - 390 , and 540 - 565 , shown in FIGS. 2 , 3 , and 5 may correspond to models 670 , 680 , and 690 , which may be stored in model database 660 .
- the preprocessing and training elements shown in certain figures, such as preprocessing 220 and 430 and training 240 and 530 may be performed using processing engine 650 of FIG. 6 .
- the system depicted in FIG. 6 is exemplary, and any other suitable means disclosed herein for implementing the disclosed embodiments may be used, such as a system employing one or more servers, as well as additional databases (or other storage devices) and processors.
- system 600 may be configured to perform methods for contextual classification of objects, such as those described herein.
- system 600 may be implemented to store and process information pertaining to objects, such as web pages, articles, and comments, and process such information to train models for use in contextual classification of other objects.
- System 600 may include one or more server systems, databases, and/or computing systems configured to receive information from entities in a network, such as network 695 , process the information, and/or communicate the information with other entities in the network, according to the exemplary embodiments described herein. More specifically, system 600 may be configured to receive and store data associated with objects transferred over a network such as the Internet, process/analyze the data to create vectorized objects, utilize the vectorized objects to train models according to machine learning techniques, identify one or more optimal models for contextual classification of objects, and classify one or more objects using the one or more identified optimal models.
- a network such as the Internet
- various components of system 600 may include an assembly of hardware, software, and/or firmware, including a memory, a central processing unit (“CPU”), and/or a user interface.
- Memory may include any type of RAM or ROM embodied in a physical storage medium, such as magnetic storage including floppy disk, hard disk, or magnetic tape; semiconductor storage such as solid state disk (SSD) or flash memory; optical disc storage; or magneto-optical disc storage.
- a CPU may include one or more processors for processing data according to a set of programmable instructions or software stored in the memory. The functions of each processor may be provided by a single dedicated processor or by a plurality of processors.
- processors may include, without limitation, digital signal processor (DSP) hardware, or any other hardware capable of executing software.
- DSP digital signal processor
- An optional user interface may include any type or combination of input/output devices, such as a display monitor, keyboard, and/or mouse.
- system 600 may store one or more objects, such as articles 610 , comments 620 , and other web data 630 , in an object database 640 .
- objects such as articles 610 , comments 620 , and other web data 630
- object database 640 may be any suitable type of large scale data storage device, which may optionally include any type or combination of slave databases, load balancers, dummy servers, firewalls, back-up databases, and/or any other desired database components.
- system 600 may also include a processing engine 650 .
- Processing engine 650 may comprise a Hadoop cluster including a Hadoop distributed file system (“HDFS”) that is configured to stage input data, perform data processing, and store large-volume data output.
- HDFS Hadoop distributed file system
- the HDFS may include any desired number or arrangement of clustered servers or machines, as needed to provide suitable efficiency, storage space, and/or processing power.
- processing engine 650 may be configured to preprocess training objects, such as objects 610 - 630 , to form vectorized objects, as illustrated, for example, by FIG. 4 .
- processing engine 650 may utilize tokenizers, stopword filters, stemmers, feature filters, and/or vectorizers to preprocess training objects, as discussed in further detail herein.
- Processing engine 650 may also be configured to train one or more models using the vectorized objects, as illustrated, for example, by FIG. 5 .
- processing engine 650 may utilize one or more machine learning tools, such as Mahout, Mallet, libsvm, and Weka, and implement one or more algorithms, such as SVM, Perceptron/Winnow, Bayesian, decision tree, and AdaBoost, as described in additional detail herein. Moreover, processing engine 650 may compare the one or more models to determine the optimal model for performing contextual classification on a specific set of objects.
- machine learning tools such as Mahout, Mallet, libsvm, and Weka
- algorithms such as SVM, Perceptron/Winnow, Bayesian, decision tree, and AdaBoost, as described in additional detail herein.
- processing engine 650 may compare the one or more models to determine the optimal model for performing contextual classification on a specific set of objects.
- system 600 may store the optimal classification models identified by processing engine 650 in model database 660 .
- model database 660 may be any suitable type of large scale data storage device, which may optionally include any type or combination of slave databases, load balancers, dummy servers, firewalls, back-up databases, and/or any other desired database components.
- model database 660 may store one or more optimal models for each object type analyzed by processing engine 650 , such as article model 670 , comment model 680 , and other web data model 690 .
- model database 660 may store one or more models directed to specific subclasses within the article, comment, and other web data classes, such as a sports model, politics model, and religion model (not shown). After processing engine 650 has created one or more optimal classification models and stored the models in model database 660 , these models may be used by processing engine 650 to classify additional objects.
- object database 640 , processing engine 650 , and model database 660 may transmit information over a network 695 to implement the contextual classification methods disclosed herein.
- network 695 may be a public network, such as the Internet.
- network 695 may be a private network, such as a corporate local area network.
- Object database 640 may communicate with network 695 to collect and store objects, such as articles 610 , comments 620 , and other web data 630 , as well as information pertaining to content reviewers' tags/classifications of those objects.
- Model database 660 may communicate information regarding classification models, such as article model 670 , comment model 680 , and other web data model 690 , over network 695 .
- processing engine 650 may collect training data directly from network 695 , preprocess the training data and train models based on the collected data, and use those models to classify objects retrieved directly from network 695 .
- any suitable configuration of software, processors, and data storage devices may be selected to carry out the disclosed features and aspects of system 600 .
- the software and hardware associated with system 600 may be selected to enable quick response to various business needs, relatively fast prototyping, and delivery of high-quality solutions and results. An emphasis may be placed on achieving high performance through scaling on a distributed architecture.
- the selected software and hardware may be flexible, to allow for quick reconfiguration, repurposing, and prototyping for research purposes.
- the data flows and processes described herein are merely exemplary, and may be reconfigured, merged, compartmentalized, and combined as desired.
- the exemplary modular architecture described herein may be desirable for performing data intensive analysis.
- a modular architecture may also be desired to enable efficient integration with external platforms, such as content analysis systems, various plug-ins and services, etc.
- the exemplary hardware and modular architecture may be provided with various system monitoring, reporting, and troubleshooting tools.
- FIG. 7 depicts a flow diagram for an exemplary method for performing contextual classification of objects, in accordance with an embodiment of the present disclosure.
- the exemplary method of FIG. 7 may be implemented with a computerized system or platform, such as that illustrated in FIG. 6 .
- supervised training data may be received from at least one content reviewer.
- supervised training data may be transmitted over a network, such as network 695 , from an editor, an author, or a third-party user.
- this training data may be transmitted over a network to a server, database, or processing engine, such as object database 640 or processing engine 650 .
- the supervised training data is stored in a database, such as object database 640 , in step 720 .
- the supervised training data may be preprocessed to form at least one vectorized object.
- the preprocessing of step 730 may comprise one or more of stemming, tokenization, and n-gram analysis.
- the at least one vectorized object may be used to train a plurality of models by applying a plurality of machine learning algorithms thereto in step 740 .
- step 740 may include applying one or more of the SVM, Perceptron/Winnow, Bayesian, decision tree, and AdaBoost algorithms to the at least one vectorized object to obtain a plurality of models.
- Steps 730 and 740 may be performed on one or more processors, such as the one or more processors comprising processing engine 650 .
- the preprocessing of step 730 of multiple objects may be distributed over multiple processors of processing engine 650 to improve efficiency.
- the plurality of models trained in step 740 may be trained using multiple processors of processing engine 650 as well.
- at least one processor may be used for steps 730 and 740 , if the volume of data to be preprocessed and trained is sufficiently small.
- steps 730 and 740 as well as other steps of the exemplary method depicted in FIG. 7 , may be performed using one or more machines forming a Hadoop cluster.
- An optimal model may be identified in step 750 from the models trained in step 740 .
- the models trained in step 740 may be compared in step 750 using processing engine 650 , or any other processor or processors, to determine the most efficient model for classifying a set of objects.
- Optimal models such as article model 670 , comment model 680 , and other web data model 690 , may be stored in model database 660 for future use.
- an optimal model may be used to classify a candidate object. For example, an optimal model may be used to classify an article or to filter an abusive comment.
- step 760 may be performed using processing engine 650 on candidate objects obtained from network 695 .
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Medical Informatics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
- This application claims the benefit of U.S. Provisional Application No. 61/542,537, filed Oct. 3, 2011, the disclosure of which is expressly incorporated herein by reference in its entirety.
- The present disclosure relates to content assessment and computerized systems and methods for performing contextual classification of objects. More particularly, and without limitation, the present disclosure relates to systems and methods for selecting among a plurality of algorithms or models, in parallel, on a pluggable platform, and performing contextual classification using both supervised and unsupervised training.
- Contextual classification of objects enables content providers to determine that objects share similar properties and group those objects with one another to meet various needs or objectives. For example, content providers may seek to group like objects together, so that content consumers can easily access similar content. As another example, objects may be contextually classified to facilitate filtering of content, such as by classifying and removing inappropriate content.
- Contextual classification may be performed on a wide variety of objects, such as web pages or finite objects like text, images, videos, and other content. In the context of web pages, contextual classification may be performed to group web pages by topic, thus improving the organization of a web site. By placing similar content together in one location, content providers make it easier for users to find content that suits their interests, thus increasing the likelihood that users will view multiple web pages within a web site upon finding a single web page on that web site that suits their interests or needs.
- Prior methods for contextual classification have relied primarily on manual input by editors or users. For example, some prior techniques have relied on manual tagging by persons designated as content reviewers. Such content reviewers may be employees of a particular company, such as the editors of a particular web site, or they may also be users of a web site. Regardless of the criteria for selection of persons to review and classify content, there are several disadvantages associated with the manual classification of objects. For example, manual tagging of objects is time-consuming. For content providers with numerous objects to classify, manual classification may take so long as to decrease the utility of the classification. By way of example, an online news provider that relies on manual classification of news stories will need stories classified quickly in order to guide users to the stories in which they are interested before other news sources capture the users' attention (or before the news becomes stale). If a content provider releases a large volume of new content per day, it may be unfeasible to have each object tagged within an acceptable time frame.
- Contextual classification may involve classification of an object based on a single parameter or on numerous parameters. For example, a web page may be classified based on genre (e.g., politics, religion, sports), audience (e.g., children, adults), mood (e.g., funny, depressing, inciting), and numerous other parameters. Further, whereas some parameters may be assigned a binary value (e.g., true/false, yes/no), other parameters may be assigned a wide variety of values and require more analysis to determine the appropriate value.
- In order manually to classify a large volume of content, a large number of people must be involved, which leads to the second drawback of manual tagging—it can be very expensive. A content provider that produces numerous web pages or articles must dedicate a large number of editors to classifying content if the content is to be classified accurately and in a timely fashion. Further, depending on the nature of the content produced, content providers may require that such editors have certain educational qualifications or prior work experience. As the number and skill level of editors increases, the cost of manual contextual classification increases as well.
- The present disclosure is directed to addressing one or more of the above-referenced challenges or drawbacks with conventional methods and techniques for contextual classification. The present disclosure provides improved systems and methods for performing contextual classification using supervised and unsupervised training. Among other features and advantages, certain embodiments of the present disclosure may utilize parallel machine learning to determine the best model/parameter combination for contextual classification of objects based on supervised and unsupervised training. Exemplary implementations of the disclosed embodiments include article classification and comment moderation.
- Consistent with the present disclosure, systems and methods are provided for performing contextual classification of objects using supervised and unsupervised training. Embodiments consistent with the present disclosure include, for example, systems and methods for collecting supervised training data from content reviewers, preprocessing the supervised training data to create vectorized objects, and training models using the vectorized objects. The disclosed systems and methods may be used to automatically classify objects based on the models.
- In accordance with one disclosed exemplary embodiment, a computer-implemented method is provided for performing contextual classification of objects. According to the method, supervised training data is received from at least one content reviewer and stored in a database. The supervised training data may be preprocessed to form at least one vectorized object. The at least one vectorized object may be used to train a plurality of models by applying a plurality of machine learning algorithms thereto. An optimal model may be identified from the plurality of models and used to classify a candidate object.
- In accordance with another exemplary embodiment, a computer-implemented system is provided for performing contextual classification of objects. The system includes a database and a processor. The database may be used to store supervised training data collected from at least one content reviewer. The processor may be configured to preprocess the supervised training data to form at least one vectorized object. Further, the processor may be configured to train a plurality of models by applying a plurality of machine learning algorithms to the at least one vectorized object. From the plurality of models, an optimal model may be identified and used to classify a candidate object.
- According to still another exemplary embodiment, a computer-readable medium is provided including instructions for performing contextual classification of objects. When executed, the instructions perform steps including collecting supervised training data from at least one content reviewer. The steps further include preprocessing the supervised training data to form at least one vectorized object. Moreover, the at least one vectorized object may be used to train a plurality of models by applying a plurality of machine learning algorithms thereto. An optimal model may be identified from the plurality of models and used to classify a candidate object.
- It is to be understood that the disclosure is not limited in its application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. The present disclosure encompasses embodiments in addition to those described herein, as well as those practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein, as well as in the abstract, are for the purpose of description and should not be regarded as limiting.
- As such, those skilled in the art will appreciate that the conception upon which this disclosure is based may readily be utilized as a basis for designing other structures, methods, and systems for carrying out the several purposes of the present disclosure. It is important, therefore, to recognize that the claims should be regarded as including such equivalent constructions insofar as they do not depart from the spirit and scope of the present disclosure.
- The accompanying drawings, which are incorporated in and constitute part of this specification, and together with the description, illustrate and serve to explain the principles of various exemplary embodiments.
-
FIG. 1 illustrates exemplary phases from an exemplary method for performing contextual classification of objects, in accordance with an embodiment of the present disclosure. -
FIG. 2 illustrates a flow diagram of an exemplary method for performing contextual classification of objects, in accordance with an embodiment of the present disclosure. -
FIG. 3 illustrates an exemplary diagram for employing parallel machine learning to classify objects, in accordance with an embodiment of the present disclosure. -
FIG. 4 illustrates exemplary steps for preprocessing training data to create vectorized objects, in accordance an embodiment of the present disclosure. -
FIG. 5 illustrates exemplary steps for training one or more models using vectorized objects, in accordance with an embodiment of the present disclosure. -
FIG. 6 illustrates an exemplary system environment for implementing embodiments of the present disclosure. -
FIG. 7 illustrates a flow diagram of an exemplary method for performing contextual classification of objects, in accordance with an embodiment of the present disclosure. - Reference will now be made in detail to the exemplary embodiments implemented according to the disclosure, the examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.
- As discussed above, prior methods of contextual classification have relied primarily on manual classification of objects. Specifically, prior methods have required content providers to task personnel with analyzing and classifying content according to specified criteria. This sort of manual classification of content can also be referred to as “supervised” classification. Although supervised classification of objects can suffer from certain inefficiencies (e.g., time-consuming, costly), supervised classification can also provide certain benefits. Perhaps the biggest advantage of supervised classification of objects is accuracy or reliability of classification. Content providers that employ editors dedicated to the sole task of classifying content are likely to have more reliably classified content than those providers that may classify content solely through automated methods.
- Unfortunately, for high-volume content providers, supervised classification of all content is simply unfeasible. To accommodate the needs of such providers, the disclosed embodiments incorporate “unsupervised” classification methods for contextual classification. According to the disclosed embodiments, unsupervised classification may be performed automatically using machine learning techniques, thus overcoming the above-discussed disadvantages of supervised classification methods. Whereas supervised classification methods can be faulted for being time-consuming and costly, the automated nature of unsupervised classification provides for contextual classification of objects to be performed quickly and cheaply. A drawback of unsupervised classification, however, is that unsupervised classification, if not properly tuned, may be unreliable. Accordingly, to perform contextual classification of objects quickly, cheaply, and reliably, embodiments consistent with the present disclosure may incorporate both supervised and unsupervised classification methods.
- In accordance with certain embodiments, supervised classification may be performed by persons employed by a content provider for that purpose. This may be particularly suitable for content providers that produce a high volume of content and have the resources to employ persons dedicated to contextual classification of their content. In an alternative embodiment, supervised classification of a particular content item (e.g., an article) may be performed by the creator of that content item (e.g., the author of the article). In yet another embodiment, content providers may allow users (i.e., content consumers) to perform supervised classification of content. This responsibility may be provided to all users or only to users who meet certain criteria. For example, the authority to perform supervised classification of content may be provided to certain users who have been given “moderator” or “expert” status. A user may achieve such a status upon a determination that a threshold number of classifications made by the user were proper. Such a determination may be made, for example, when a threshold number of provisional classifications provided by a user were ultimately adopted, either through validation by employees of the content provider or through automated validation methods (e.g., the unsupervised classification methods disclosed herein).
- Supervised classification of objects may be performed by, among other methods, tagging objects with pre-defined tags. For example, a content reviewer may be presented with a pre-defined set of tags that may either be selected or left unselected. In such a situation, such tags may be associated with a binary value (e.g., true/false, yes/no, 0/1) to indicate whether the tag is properly associated with the object. Alternatively, tags may be assigned one of several different values. For example, whereas a tag for “Scary?” may be assigned a binary value (e.g., yes/no, true/false, scary/not scary), a tag for “genre” may be assigned one of a broader variety of values, as genre is not susceptible to binary classification. As an example, a genre tag may be assigned one of a pre-defined set of values: politics, religion, sports, opinion, or business. In one embodiment, the possible values for a given tag are pre-defined by an editor associated with the content provider who is familiar with the range of topics that are likely to be published by the content provider. By providing a pre-defined set of values, content providers may ensure that similar content is tagged similarly. In other words, two different content reviewers will not be able to tag one article as relating to the “election” genre and another article as relating to the “politics” genre when the content provider would prefer all political articles to be clustered together as belonging to the “politics” genre.
- In an alternative embodiment, tags may be assigned open-ended values, such as by providing an input box associated with a tag. For example, a content reviewer may review an object and assign the object the value that he or she sees fit, without being restricted only to a pre-defined set of values. Whereas this alternative embodiment may allow for more finely tuned or better descriptive values, similar articles are likely to be tagged differently because of their dependence on the subjective judgments of persons having different vocabularies.
- In addition to, or as an alternative to, the disclosed methods for supervised classification of objects, embodiments disclosed herein may utilize unsupervised classification methods. As the name implies, unsupervised classification may be performed automatically without direct human supervision or interaction, thus allowing unsupervised classification to be performed more cheaply and quickly than supervised classification. In one embodiment, objects may be analyzed to identify key terms that are descriptive of the object as a whole. As discussed herein, such analyses may involve preprocessing of an object to create a list of terms (e.g., all terms associated with an object or only key terms), such as by identifying terms based on tokenization, stemming, n-gram analysis, phrasal analysis, or any other method for segmentation and representation of objects by a plurality of terms. These terms may be used automatically to assign tags to objects. For example, the terms may be compared to one or more lists of terms known to be associated with candidate tags. Upon identifying a threshold number of matches between the terms derived from an object and a list of terms known to be associated with a tag/classification, the object may be tagged. In one embodiment, the tags/classifications and the known terms assigned thereto may have been designated by an editor or other person associated with a content provider, thus incorporating an element of supervised classification.
- Unsupervised classification methods may also use previously classified objects to classify other objects. For example, an object may be classified using either supervised classification (e.g., manually tagging by editors) or unsupervised classification (e.g., automated tagging based on term analysis) and then compared to unclassified objects for similarities. If a classified object is sufficiently similar to an unclassified object, the unclassified object may be tagged with one or more of the same classifications as the classified object. The similarity of objects may be analyzed based on key terms derived from the object or a comparison of all terms from the objects.
- In accordance with certain embodiments, classification of objects is performed using machine learning techniques. Such techniques may be used to design algorithms that recognize patterns within objects and make intelligent decisions based on input data. In accordance with certain embodiments, contextual classification of objects incorporates machine learning to identify key terms or phrases within objects and used that information to classify objects into various categories based on training data. Further, unlike prior methods of machine learning, certain embodiments take advantage of parallel processing, so that the training and classification processes may be performed much more quickly than in prior machine learning techniques.
-
FIG. 1 depicts two phases of an exemplary machine learning process, in accordance with one embodiment. As shown inFIG. 1 , machine learning may comprise both alearning phase 110 and aclassifying phase 120. In thelearning phase 110, objects may be tagged with various values that describe the objects. Moreover, individual terms or phrases within an object may be tagged as being associated with certain values. In one embodiment, thelearning phase 110 may utilize the supervised classification methods discussed herein. For example, in thelearning phase 110, information regarding the tags assigned to an object by editors or others performing supervised classification may be collected to learn what sort of information is tied to each tag/classification. This information may then be used to train one or more models for use in theclassifying phase 120. - In the
classifying phase 120, one or more models trained in thelearning phase 110 are used to classify previously unclassified objects. This phase may be considered an unsupervised element of the machine learning process, as such classifications are performed automatically, albeit based on data input during a supervised portion of the machine learning process (i.e., the learning phase 110). In theclassifying phase 120, object data may be parsed in accordance with a model developed during thelearning phase 110. As discussed in more detail below, the nature of the parsing applied to an object may vary, as the optimal model for one class of objects may differ from that corresponding to another class of objects. Once the object has been parsed, the model is then utilized to classify the object into one or more classifications. In one embodiment, this classification may be verified using supervised classification techniques (i.e., an editor may validate the classification). This classification may also be used as additional training data that is input into thelearning phase 110 and used to train a new model and/or refine a previous model. - In the exemplary illustration of machine learning techniques for contextual classification depicted in
FIG. 1 , objects are analyzed to determine whether they may be classified as “business” objects. In thelearning phrase 110, objects are preprocessed to identify various terms and/or phrases for classification. In one embodiment, the terms are identified by editors or other persons as part of supervised classification. As shown inFIG. 1 , these terms and/or phrases may include sentences or sentence fragments from an object. Alternatively, terms identified from objects for training in thelearning phase 110 may be single words, n-grams, tokens, stems, phrases, or other combinations of letters and/or words, as discussed in more detail herein. InFIG. 1 , the identified terms have been classified as either “business” or “non-business.” In accordance with one embodiment, these classifications are supervised classifications. In other words, the classifications are made by editors, users, or other persons who have analyzed the objects. - In the example depicted in
FIG. 1 , the terms “Investments are taxed as . . . ” and “the top tax bracket for . . . ” have been classified as business terms. Accordingly, the objects associated with these terms may be classified as business objects. In one embodiment, the classification of an object containing these terms may be based on a determination of whether a threshold number of terms classified as business terms are present in the object. Moreover, the determination of whether to classify an object containing these business terms as a business object may take into consideration the number of terms classified as non-business terms, as a high number of non-business terms relative to a low number of business terms may indicate that the object as a whole is less likely to be appropriately classified as a business object. As shown inFIG. 1 , the terms “Are you dense or just clueless?” and “My nephew is a hedge fund manager . . . ” have been designated as non-business terms. With respect to the term “My nephew is a hedge fund manager . . . ,” an editor or other reviewer of the object containing this term may have determined that, despite its inclusion of the sub-term “hedge fund,” the term as a whole is insufficiently tied to business to qualify as a business term. Alternatively, an automated (i.e., unsupervised) process may have determined that this term should be classified as non-business due to the proximity of the phrase “hedge fund” to the word “nephew.” Data pertaining to these classified terms and/or objects are the used to train one or more models, which may then be used to classify additional terms and/or objects. - As shown in
FIG. 1 , once terms and/or objects have been classified and used to train one or more models, other terms and/or objects may then be classified using the one or more models. For example, the term “capital gains to be taxed” is input into one or more models that were trained during thelearning phase 110. The one or more models compare the term with other terms to determine whether the term should be classified as a business or a non-business term. In this example, the term “capital gains to be taxed . . . ” is classified as a business term. This determination may be based on a comparison of the term to terms or objects classified during thelearning phase 110 and input into the one or more models employed as part of theclassifying phase 120. In one embodiment, one or more terms, or sub-terms, from within the term “capital gains to be taxed . . . ” may be compared with the business and non-business terms used to train the one or more models employed in theclassifying phase 120. For example, the terms “capital gains” and “taxed” may be compared to terms, or sub-terms, analyzed during thelearning phase 110 to determine that these sub-terms are business terms. Thus, the term “capital gains to be taxed . . . ,” as a whole, may be classified as a business term. -
FIG. 2 depicts a general processing flow of contextual classification of objects using supervised and unsupervised training, in accordance with an embodiment. First, objects and/or their terms may be classified into various classifications to create training objects (or “training docs”) 210. These training docs are then preprocessed 220 to create one or more vectorized objects (or “vectorized docs”) 230. Thesevectorized docs 230 are used intraining 240 to develop one ormore models 250. After one or more models have been trained and analyzed, an optimal model for contextual classification may be determined and used for future classification of objects. -
Training docs 210 may be developed in a manner similar to that described with respect to learningphase 110 ofFIG. 1 . As described with respect toFIG. 1 , this may include supervised classification performed by editors or users who have reviewed one or more objects and assigned one or more classifications to those objects and/or one or more terms within those objects. Further, as described above, these classifications may be performed by classifying an object, or a term within an object, by selecting one or more classifications from a pre-defined set of classifications, such as by clicking on a button associated with a particular classification while reviewing the object or term. Alternatively, the classifications may be performed by typing a descriptive label in an input box associated with the object or term. In one embodiment, all objects and terms that have been classified are used in the creation of training docs. In another embodiment, a subset of objects and/or terms that have been classified using the supervised classification techniques disclosed herein may be used as training docs. For example, persons involved in the supervised classification of objects, or a subset of such persons comprised of managers or supervisors, may determine which objects or terms are most indicative of a particular classification and, thus, best candidates for training docs. By focusing on only the best training docs, content providers may ensure that only the objects or terms that are most closely tied to a particular classification are used in subsequent steps of the general processing flow shown inFIG. 2 to train models for classification of yet unclassified objects or terms. - Once training docs have been created, they are preprocessed. Several preprocessors may be utilized as part of
preprocessing 220. For example, preprocessing 220 may employ one or more of tokenizers, stopword filters, stemmers, feature filters, or vectorizers. Each of these preprocessors may parse information from training docs and put the information in a form better suited for training one or more models used for contextual classification. For example, preprocessing 220 may remove stopwords from training docs that are unlikely to be strongly connected to any particular classification. Exemplary stopwords may include articles, such as “a,” “an,” and “the,” as well as prepositions, such as “of,” “at,” and “on.” Alternatively, stopwords may be left in training docs to support other processing functions, such as phrasal analysis, which would be adversely affected by the removal of certain commonly used terms (e.g., prepositions common used in prepositional phrases). -
Preprocessing 220 may also include tokenization of one or more terms or phrases of the trainings docs. Tokenization may separate the words or phrases associated with a training doc into smaller words, phrases, or fragments (i.e., smaller “tokens”). Numerous methods may be employed for tokenizing training docs as part ofpreprocessing 220. For example, phrases associated with a training doc may be separated into individual words, as determined by spacing or any other means for detecting separation of words. Alternatively, words or phrases associated with a training doc may be separated into n-grams. Accordingly, preprocessing 220 may divide words or phrases in a training doc into fragments consisting of a pre-defined number of characters (i.e., n-grams). In one embodiment, one n-gram size may be pre-defined for all training docs used by a content provider employing the contextual classification techniques disclosed herein. In another embodiment, n-gram size may vary based on the subject matter of a particular object or training document or based on the average word size for the words associated with the object or training doc. Moreover, n-gram size may be set either manually by an editor or other content reviewer or be set automatically. -
Preprocessing 220 may include stemming of words associated with an object or training doc. Stemming is a process of reducing words to their stem, base, or root form. Stemming allows for different words with similar stems to be treated similarly in the training processes disclosed herein. For example, if a training doc designates the words “election,” “electoral,” and “electable” as associated with politics, these words may each be stemmed to the term “elect,” and the word “elect” may be classified as being associated with politics. By stemming similar words to the same root word, or stem, later processing steps (e.g., training) may run more efficiently, as duplicitous terms are reduced. - In accordance with one embodiment, preprocessing 220 converts the one or
more training docs 210 into one or morevectorized docs 230.Vectorized docs 230 may be clustered such that similar docs or objects are grouped together for training. For example,vectorized docs 230 that relate to business may be clustered together, such that these documents may be input together into the processes oftraining 240. -
Training 240 inputs data fromvectorized docs 230 into various algorithms to generate one ormore models 250. In order to determine the best model for contextual classification,training 240 may support and/or implement a variety of algorithms, such as, but not limited to, support vector machine (SVM), Perceptron/Winnow, Bayesian, decision tree, and AdaBoost. Moreover, different objects or object types will be better suited to some algorithms than others. Accordingly,training 240 may involve running each vectorized doc (or group of vectorized docs) through multiple algorithms, so that the best algorithm for each vectorized doc (or group of vectorized docs) may be determined. - For example,
vectorized docs 230 may be input into an SVM algorithm to develop one or more models appropriate for contextual classification of a specified object type.Vectorized docs 230 may comprise data linking terms or phrases to one of two possible categories, such as the “business” or “non-business” classifications discussed above with respect toFIG. 1 . The SVM may take thevectorized docs 230 and build a model that assigns new data into one category or the other. In other words, the SVM algorithm may train a model based on the vectorized docs, such that the model will properly classify future data (e.g., vectorized docs or objects) as falling into one of two potential categories. - Prior machine learning techniques run data through algorithms sequentially, thus potentially requiring a large amount of time to develop models corresponding to each algorithm and each set of data. In accordance with certain embodiments,
training 240 is performed on multiple vectorized docs (or groups of vectorized docs) 230 in parallel. The parallel machine learning techniques of the embodiments disclosed herein provide significant advantages over prior machine learning techniques. For example, by dividing machine learning processes over fifty machines, the training processes associated with contextually classifying objects may be performed up to fifty times faster than if the processes were performed sequentially on one machine. This increased processing speed can allow for content providers to generate and compare a much larger number of classification models, thus increasing the likelihood that the optimal model will be chosen. Moreover, content providers that produce a large volume of new and constantly changing content may need to determine optimal models for classification of new content types quickly, so that the content is properly classified before it becomes stale. - Certain embodiments may leverage parallelism in a Map/Reduce paradigm in order to generate a large number of machine learning models and find the best algorithm and parameters. Moreover, certain embodiments may also provide a pluggable placeholder for heterogeneous tools and algorithms. In one embodiment, the parallel machine learning processes disclosed herein may be implemented using Hadoop, as discussed below in connection with
FIG. 5 . Several parallel machine learning tools are available, including Mahout, Mallet, libsvm, and Weka. Mahout does not yet support the necessary classification algorithms, and other currently available tools do not run natively on Hadoop. Accordingly, the embodiments disclosed herein are designed for a flexible machine learning platform running Hadoop. This platform supports a wide range of algorithms and leverages publicly available implementations. Contrary to prior machine learning environments, this platform may be used efficiently to generate and test tens or hundreds of thousands of models and choose the best models from among this large pool to use for future classification of objects. Whereas prior sequential parallel machine learning techniques employed in other fields took up to a week, depending on the data input into the models and the number of total models trained, the disclosed parallel machine learning techniques may generate an optimal model for contextual classification of an object pool within an hour. -
FIG. 3 depicts an exemplary parallel processing flow for contextually classifying objects, in accordance with certain embodiments. In similar fashion to that discussed above with respect toFIG. 2 ,training docs 310 may be preprocessed to generate one or more vectorized docs (or vectorized doc groups) 320, 330, and 340. Each of these vectorized doc groups may then be used to train one or more models, such as models 350-390, using one or more machine learning algorithms. In one embodiment, both the preprocessing of training docs and training of models using vectorized docs may be performed in parallel over one or more machines, such as by using Hadoop. -
FIG. 4 depicts an exemplary preprocessing technique, in accordance with certain embodiments. As shown inFIG. 4 , training data may be comprised of term-classification pairs, which may be stored in training docs, such astraining docs exemplary training data 410, shown inFIG. 4 , indicates that the term “Investments are taxed as capital gains” is paired with the “business” classification and “I am afraid we may be headed for . . . ” is paired with the “none” classification (or is not assigned to any particular classification). Next, preprocessing may be performed ontraining data 410, in accordance withpreprocessing request 420. As seen inFIG. 4 , preprocessingrequest 420 may specify parameters, such as n-grams, stems, or stopwords, as well as one or more values for each of those parameters.Preprocessing 430 may be performed as specified by preprocessingrequest 420 to generate one or more vectors, such as vectors 440-465. In one embodiment, preprocessing 430 may be performed sequentially on one machine. In another embodiment, preprocessing 430 may be performed over multiple machines in parallel (e.g., a Hadoop cluster). These vectors may then be used to train models for use in future classification of objects. -
FIG. 5 illustrates an exemplary training flow, in accordance with certain embodiments.Vectors 510 containing preprocessed training data are used in conjunction with one or more train requests, such astrain request 520, to train one or more models, such as models 540-565.Train request 520 may include information specifying one or more algorithms, along with associated parameters, for application tovectors 510. For example,train request 520, as shown inFIG. 5 , depicts six parameter sets, each of which relates to the balanced winnow training algorithm. In one embodiment, train requests are run sequentially on one machine, as part oftraining 530. In another embodiment, train requests may be run intraining 530 in parallel over multiple machines (e.g., a Hadoop cluster).Training 530 may be performed using one or more machine learning tools, such as Mahout, Weka, Mallet or Mallet+SVMLite to generate models 540-565. - The disclosed methods and systems for contextual classification of objects may be used in a variety of applications and implementations. In one embodiment, the disclosed methods and systems may be used for article classification. In this embodiment, articles may be classified with one or more tags using the supervised classification methods described herein. For example, an article may be tagged by the author of the article, an editor of the article, another employee of the content provider publishing the article, or a reader of the article. Such a person may select one or more pre-defined tags and/or input one or more additional tags, such as by using a text input box.
- After an article has been classified using the supervised classification methods disclosed herein, the article may then be suitable for use in training one or more models to classify other articles. In one embodiment, the article is first preprocessed to form one or more vectorized objects, such as by performing tokenization, stemming, and/or n-gram analysis on the article. The one or more resultant vectorized objects are then used to train one or more models for classification of future articles. Accordingly, the tags/classifications applied to an article by its author, editor, reader, or other reviewer, may be used to develop one or more models for automatically classifying similar articles in the future. As discussed in further detail herein, many models may be trained and compared, such that the most efficient and accurate model may be chosen for future use as a classifying model. By using the disclosed systems and methods of contextual classification for article classification, articles may be classified quickly and reliably, such that their intended readers may locate them easily.
- In another embodiment, the disclosed systems and methods may be used for comment moderation. Content providers often provide the ability for their consumers to comment on their content. Generally, user comments are displayed along with (usually below) the subject content. Unlike articles or other content authored and/or edited by the content provider, or its employees/agents, comments are generally written by third-parties. Like articles published by a content provider, user comments should also be reviewed to ensure that their content will not be objectionable to readers. Unfortunately, due to the magnitude of user comments received by larger content providers, manual review of comments is unfeasible. Accordingly, the disclosed embodiments may be employed to filter automatically through user comments and reject or publish the comments, as appropriate.
- Similar to article classification, comment moderation may begin with supervised classification of comments. Such supervised classification may be performed, for example, by the author or editor of the article that is the subject of the comment, some other employee or agent of the content provider, or a third-party user or subscriber of the content provider. In one embodiment, a third-party user or subscriber may be given authority to moderate comments based on a determination that the user is a “trusted” user. For example, the user may be given “trusted” user status if a threshold of comments flagged by the user as inappropriate were also flagged by other users as inappropriate or classified as inappropriate by the unsupervised (i.e., automated) classification techniques disclosed herein. In certain embodiments, supervised classification of comments may be comprised simply of tagging or flagging the comment as abusive or not abusive. In other embodiments, other classifications may be applied to comments, such as helpful, funny, recommended, or off-topic.
- After comments have been classified (e.g., as abusive or non-abusive) by using the supervised classification methods disclosed herein, the comments may then be preprocessed to obtain vectorized data for use in training models for moderation of future comments. For example, key terms may be identified (e.g., by tokenization, stemming, or n-gram analysis) and associated with a certain classification (e.g., abusive, non-abusive). Accordingly, profane or obscene terms may be classified as abusive in these vectorized objects. Once the models have been trained using the vectorized data, they may be compared to determine the best model to use for comment moderation. In one embodiment, different models may be developed for moderation of comments directed to different subject matter. For example, comments deemed aggressive or abusive when directed to an article focused on education may not be deemed aggressive or abusive when directed to an article focused on sports. Once a model has been developed for moderation of comments directed to a particular article or subject matter, the model may be applied to moderate future comments directed to the same article or subject matter.
- Embodiments herein include computer-implemented methods, tangible non-transitory computer-readable mediums, and computer-implemented systems. The computer-implemented methods may be executed, for example, by one or more processors that receive instructions from one or more non-transitory computer-readable storage mediums. Similarly, the systems disclosed herein may include at least one processor and memory, and the memory may be a non-transitory computer-readable storage medium.
- As used herein, a non-transitory computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage medium. Singular terms, such as “memory” and “computer-readable storage medium,” may additionally refer to multiple structures, such a plurality of memories and/or computer-readable storage mediums.
- As referred to herein, a “memory” may comprise any type of computer-readable storage medium unless otherwise specified. A computer-readable storage medium may store instructions for execution by a processor, including instructions for causing the processor to perform steps or stages consistent with an embodiment herein. Additionally, one or more computer-readable storage mediums may be utilized in implementing a computer-implemented method. The term “computer-readable storage medium” should be understood to include tangible items and exclude carrier waves and transient signals.
-
FIG. 6 illustrates an exemplary system environment for implementing embodiments of the present disclosure. Each of the processes depicted inFIGS. 1-5 may be implemented using the exemplary system shown inFIG. 6 . For example, thelearning phase 110 ofFIG. 1 may usedata object database 640 to train one or more models usingprocessing engine 650. Moreover, classifyingphase 120 ofFIG. 1 may involve usingprocessing engine 650 to classifymodels model database 660. The training docs/data FIGS. 2-4 may correspond toobjects object database 640.Models 250, 350-390, and 540-565, shown inFIGS. 2, 3, and 5 may correspond tomodels model database 660. Moreover, the preprocessing and training elements shown in certain figures, such aspreprocessing training processing engine 650 ofFIG. 6 . Further, it is to be understood that the system depicted inFIG. 6 is exemplary, and any other suitable means disclosed herein for implementing the disclosed embodiments may be used, such as a system employing one or more servers, as well as additional databases (or other storage devices) and processors. - In the exemplary embodiment of
FIG. 6 , asystem 600 is provided. Consistent with the present disclosure,system 600 may be configured to perform methods for contextual classification of objects, such as those described herein. Among other things,system 600 may be implemented to store and process information pertaining to objects, such as web pages, articles, and comments, and process such information to train models for use in contextual classification of other objects. -
System 600 may include one or more server systems, databases, and/or computing systems configured to receive information from entities in a network, such asnetwork 695, process the information, and/or communicate the information with other entities in the network, according to the exemplary embodiments described herein. More specifically,system 600 may be configured to receive and store data associated with objects transferred over a network such as the Internet, process/analyze the data to create vectorized objects, utilize the vectorized objects to train models according to machine learning techniques, identify one or more optimal models for contextual classification of objects, and classify one or more objects using the one or more identified optimal models. - In one embodiment, various components of
system 600 may include an assembly of hardware, software, and/or firmware, including a memory, a central processing unit (“CPU”), and/or a user interface. Memory may include any type of RAM or ROM embodied in a physical storage medium, such as magnetic storage including floppy disk, hard disk, or magnetic tape; semiconductor storage such as solid state disk (SSD) or flash memory; optical disc storage; or magneto-optical disc storage. A CPU may include one or more processors for processing data according to a set of programmable instructions or software stored in the memory. The functions of each processor may be provided by a single dedicated processor or by a plurality of processors. Moreover, processors may include, without limitation, digital signal processor (DSP) hardware, or any other hardware capable of executing software. An optional user interface may include any type or combination of input/output devices, such as a display monitor, keyboard, and/or mouse. - In accordance with certain embodiments,
system 600 may store one or more objects, such asarticles 610, comments 620, andother web data 630, in anobject database 640. Alternatively, only select data pertaining to objects, such as objects 610-630, may be stored inobject database 640, and the objects 610-630 themselves may be stored elsewhere, such as onnetwork 695.Object database 640 may be any suitable type of large scale data storage device, which may optionally include any type or combination of slave databases, load balancers, dummy servers, firewalls, back-up databases, and/or any other desired database components. - In one embodiment,
system 600 may also include aprocessing engine 650.Processing engine 650 may comprise a Hadoop cluster including a Hadoop distributed file system (“HDFS”) that is configured to stage input data, perform data processing, and store large-volume data output. It will be appreciated that the HDFS may include any desired number or arrangement of clustered servers or machines, as needed to provide suitable efficiency, storage space, and/or processing power. Although certain embodiments of the present disclosure are described with reference to a Hadoop cluster, it will be appreciated that any type of processing system may be used in the alternative. - In one embodiment,
processing engine 650 may be configured to preprocess training objects, such as objects 610-630, to form vectorized objects, as illustrated, for example, byFIG. 4 .Processing engine 650 may utilize tokenizers, stopword filters, stemmers, feature filters, and/or vectorizers to preprocess training objects, as discussed in further detail herein.Processing engine 650 may also be configured to train one or more models using the vectorized objects, as illustrated, for example, byFIG. 5 . To train the one or more models,processing engine 650 may utilize one or more machine learning tools, such as Mahout, Mallet, libsvm, and Weka, and implement one or more algorithms, such as SVM, Perceptron/Winnow, Bayesian, decision tree, and AdaBoost, as described in additional detail herein. Moreover,processing engine 650 may compare the one or more models to determine the optimal model for performing contextual classification on a specific set of objects. - In accordance with certain embodiments,
system 600 may store the optimal classification models identified by processingengine 650 inmodel database 660. Similar to objectdatabase 640,model database 660 may be any suitable type of large scale data storage device, which may optionally include any type or combination of slave databases, load balancers, dummy servers, firewalls, back-up databases, and/or any other desired database components. Further,model database 660 may store one or more optimal models for each object type analyzed by processingengine 650, such asarticle model 670,comment model 680, and otherweb data model 690. Moreover,model database 660 may store one or more models directed to specific subclasses within the article, comment, and other web data classes, such as a sports model, politics model, and religion model (not shown). After processingengine 650 has created one or more optimal classification models and stored the models inmodel database 660, these models may be used by processingengine 650 to classify additional objects. - According to certain embodiments,
object database 640,processing engine 650, andmodel database 660 may transmit information over anetwork 695 to implement the contextual classification methods disclosed herein. In one embodiment,network 695 may be a public network, such as the Internet. In an alternative embodiment,network 695 may be a private network, such as a corporate local area network.Object database 640 may communicate withnetwork 695 to collect and store objects, such asarticles 610, comments 620, andother web data 630, as well as information pertaining to content reviewers' tags/classifications of those objects.Model database 660 may communicate information regarding classification models, such asarticle model 670,comment model 680, and otherweb data model 690, overnetwork 695. Moreover, in accordance with certain embodiments,processing engine 650 may collect training data directly fromnetwork 695, preprocess the training data and train models based on the collected data, and use those models to classify objects retrieved directly fromnetwork 695. - It will be appreciated that any suitable configuration of software, processors, and data storage devices may be selected to carry out the disclosed features and aspects of
system 600. The software and hardware associated withsystem 600 may be selected to enable quick response to various business needs, relatively fast prototyping, and delivery of high-quality solutions and results. An emphasis may be placed on achieving high performance through scaling on a distributed architecture. The selected software and hardware may be flexible, to allow for quick reconfiguration, repurposing, and prototyping for research purposes. The data flows and processes described herein are merely exemplary, and may be reconfigured, merged, compartmentalized, and combined as desired. The exemplary modular architecture described herein may be desirable for performing data intensive analysis. A modular architecture may also be desired to enable efficient integration with external platforms, such as content analysis systems, various plug-ins and services, etc. Finally, the exemplary hardware and modular architecture may be provided with various system monitoring, reporting, and troubleshooting tools. -
FIG. 7 depicts a flow diagram for an exemplary method for performing contextual classification of objects, in accordance with an embodiment of the present disclosure. The exemplary method ofFIG. 7 may be implemented with a computerized system or platform, such as that illustrated inFIG. 6 . - In
step 710, supervised training data may be received from at least one content reviewer. For example, supervised training data may be transmitted over a network, such asnetwork 695, from an editor, an author, or a third-party user. In one embodiment, this training data may be transmitted over a network to a server, database, or processing engine, such asobject database 640 orprocessing engine 650. The supervised training data is stored in a database, such asobject database 640, instep 720. - In
step 730, the supervised training data may be preprocessed to form at least one vectorized object. For example, the preprocessing ofstep 730 may comprise one or more of stemming, tokenization, and n-gram analysis. The at least one vectorized object may be used to train a plurality of models by applying a plurality of machine learning algorithms thereto instep 740. For example, step 740 may include applying one or more of the SVM, Perceptron/Winnow, Bayesian, decision tree, and AdaBoost algorithms to the at least one vectorized object to obtain a plurality of models.Steps processing engine 650. In one embodiment, the preprocessing ofstep 730 of multiple objects may be distributed over multiple processors ofprocessing engine 650 to improve efficiency. Similarly, the plurality of models trained instep 740 may be trained using multiple processors ofprocessing engine 650 as well. Alternatively, at least one processor may be used forsteps steps FIG. 7 , may be performed using one or more machines forming a Hadoop cluster. - An optimal model may be identified in
step 750 from the models trained instep 740. In one embodiment, the models trained instep 740 may be compared instep 750 usingprocessing engine 650, or any other processor or processors, to determine the most efficient model for classifying a set of objects. Optimal models, such asarticle model 670,comment model 680, and otherweb data model 690, may be stored inmodel database 660 for future use. Instep 760, an optimal model may be used to classify a candidate object. For example, an optimal model may be used to classify an article or to filter an abusive comment. In one embodiment, step 760 may be performed usingprocessing engine 650 on candidate objects obtained fromnetwork 695. - In this disclosure, various preferred embodiments have been described with reference to the accompanying drawings. It will, however, be evident that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the broader scope of the invention as set forth in the claims that follow. The specification and drawings are accordingly to be regarded in an illustrative rather than restrictive sense.
- For example, advantageous results still could be achieved if steps of the disclosed techniques were performed in a different order and/or if components in the disclosed systems were combined in a different manner and/or replaced or supplemented by other components. Other implementations are within the scope of the following exemplary claims.
- Therefore, it is intended that the disclosed embodiments and examples be considered as exemplary only, with a true scope of the present disclosure being indicated by the following claims and their equivalents.
Claims (21)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/446,980 US20230385704A1 (en) | 2011-10-03 | 2023-08-09 | Systems and method for performing contextual classification using supervised and unsupervised training |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201161542537P | 2011-10-03 | 2011-10-03 | |
US13/573,727 US9104655B2 (en) | 2011-10-03 | 2012-10-03 | Systems and methods for performing contextual classification using supervised and unsupervised training |
US14/792,585 US10565519B2 (en) | 2011-10-03 | 2015-07-06 | Systems and method for performing contextual classification using supervised and unsupervised training |
US16/737,581 US11763193B2 (en) | 2011-10-03 | 2020-01-08 | Systems and method for performing contextual classification using supervised and unsupervised training |
US18/446,980 US20230385704A1 (en) | 2011-10-03 | 2023-08-09 | Systems and method for performing contextual classification using supervised and unsupervised training |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/737,581 Continuation US11763193B2 (en) | 2011-10-03 | 2020-01-08 | Systems and method for performing contextual classification using supervised and unsupervised training |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230385704A1 true US20230385704A1 (en) | 2023-11-30 |
Family
ID=48044123
Family Applications (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/573,727 Active 2033-03-14 US9104655B2 (en) | 2011-10-03 | 2012-10-03 | Systems and methods for performing contextual classification using supervised and unsupervised training |
US14/792,585 Active 2035-10-20 US10565519B2 (en) | 2011-10-03 | 2015-07-06 | Systems and method for performing contextual classification using supervised and unsupervised training |
US16/737,581 Active 2034-11-14 US11763193B2 (en) | 2011-10-03 | 2020-01-08 | Systems and method for performing contextual classification using supervised and unsupervised training |
US18/446,980 Pending US20230385704A1 (en) | 2011-10-03 | 2023-08-09 | Systems and method for performing contextual classification using supervised and unsupervised training |
Family Applications Before (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/573,727 Active 2033-03-14 US9104655B2 (en) | 2011-10-03 | 2012-10-03 | Systems and methods for performing contextual classification using supervised and unsupervised training |
US14/792,585 Active 2035-10-20 US10565519B2 (en) | 2011-10-03 | 2015-07-06 | Systems and method for performing contextual classification using supervised and unsupervised training |
US16/737,581 Active 2034-11-14 US11763193B2 (en) | 2011-10-03 | 2020-01-08 | Systems and method for performing contextual classification using supervised and unsupervised training |
Country Status (3)
Country | Link |
---|---|
US (4) | US9104655B2 (en) |
EP (1) | EP2764472A4 (en) |
WO (1) | WO2013052555A1 (en) |
Families Citing this family (30)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140279595A1 (en) * | 2013-03-13 | 2014-09-18 | Facebook, Inc. | Reviewing advertisement components for compliance with policies of an online system |
US8819335B1 (en) * | 2013-08-30 | 2014-08-26 | NXGN Data, Inc. | System and method for executing map-reduce tasks in a storage device |
WO2015044934A1 (en) * | 2013-09-30 | 2015-04-02 | ABIDIN, Indira Ratna Dewi | A method for adaptively classifying sentiment of document snippets |
US9547471B2 (en) * | 2014-07-03 | 2017-01-17 | Microsoft Technology Licensing, Llc | Generating computer responses to social conversational inputs |
JP6368633B2 (en) * | 2014-12-08 | 2018-08-01 | 日本電信電話株式会社 | Term meaning learning device, term meaning judging device, method, and program |
US9766818B2 (en) | 2014-12-31 | 2017-09-19 | Samsung Electronics Co., Ltd. | Electronic system with learning mechanism and method of operation thereof |
US10460720B2 (en) | 2015-01-03 | 2019-10-29 | Microsoft Technology Licensing, Llc. | Generation of language understanding systems and methods |
US20200143115A1 (en) * | 2015-01-23 | 2020-05-07 | Conversica, Inc. | Systems and methods for improved automated conversations |
WO2016160626A1 (en) | 2015-03-27 | 2016-10-06 | Globallogic, Inc. | Determining actions based on imputing meaning to sensed information in a distributed computing environment |
EP3329412A4 (en) * | 2015-07-31 | 2019-01-23 | Bluvector, Inc. | System and method for in-situ classifier retraining for malware identification and model heterogeneity |
FI20165240A (en) * | 2016-03-22 | 2017-09-23 | Utopia Analytics Oy | PROCEDURES, SYSTEMS AND RESOURCES FOR MODERATING CONTENTS |
CN105912500B (en) * | 2016-03-30 | 2017-11-14 | 百度在线网络技术(北京)有限公司 | Machine learning model generation method and device |
US10176092B2 (en) | 2016-09-21 | 2019-01-08 | Ngd Systems, Inc. | System and method for executing data processing tasks using resilient distributed datasets (RDDs) in a storage device |
US9715494B1 (en) * | 2016-10-27 | 2017-07-25 | International Business Machines Corporation | Contextually and tonally enhanced channel messaging |
CN108228687A (en) * | 2017-06-20 | 2018-06-29 | 上海吉贝克信息技术有限公司 | Big data knowledge excavation and accurate tracking and system |
US10770044B2 (en) | 2017-08-31 | 2020-09-08 | Spotify Ab | Lyrics analyzer |
US11017039B2 (en) * | 2017-12-01 | 2021-05-25 | Facebook, Inc. | Multi-stage ranking optimization for selecting content |
US10509782B2 (en) * | 2017-12-11 | 2019-12-17 | Sap Se | Machine learning based enrichment of database objects |
US20210117417A1 (en) * | 2018-05-18 | 2021-04-22 | Robert Christopher Technologies Ltd. | Real-time content analysis and ranking |
CN109145151B (en) * | 2018-06-20 | 2021-05-18 | 北京达佳互联信息技术有限公司 | Video emotion classification acquisition method and device |
CN109146395B (en) * | 2018-06-29 | 2022-04-05 | 创新先进技术有限公司 | Data processing method, device and equipment |
US11698922B2 (en) | 2018-11-02 | 2023-07-11 | Valve Corporation | Classification and moderation of text |
US11212183B1 (en) * | 2019-06-14 | 2021-12-28 | Zscaler, Inc. | Network device identification |
US11373104B2 (en) * | 2019-07-26 | 2022-06-28 | Bae Systems Information And Electronic Systems Integration Inc. | Connecting OBP objects with knowledge models through context data layer |
CN110825914A (en) * | 2019-10-31 | 2020-02-21 | 广州市百果园信息技术有限公司 | Resource marking management system |
US11670286B2 (en) * | 2019-12-31 | 2023-06-06 | Beijing Didi Infinity Technology And Development Co., Ltd. | Training mechanism of verbal harassment detection systems |
US11620987B2 (en) * | 2019-12-31 | 2023-04-04 | Beijing Didi Infinity Technology And Development Co., Ltd. | Generation of training data for verbal harassment detection |
US11664043B2 (en) | 2019-12-31 | 2023-05-30 | Beijing Didi Infinity Technology And Development Co., Ltd. | Real-time verbal harassment detection system |
US12026465B2 (en) | 2020-12-22 | 2024-07-02 | Direct Cursus Technology L.L.C | Method and system for classifying word as obscene word |
WO2022177806A1 (en) * | 2021-02-19 | 2022-08-25 | President And Fellows Of Harvard College | Hardware-accelerated topic modeling |
Family Cites Families (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5867799A (en) | 1996-04-04 | 1999-02-02 | Lang; Andrew K. | Information system and method for filtering a massive flow of information entities to meet user information classification needs |
US6882990B1 (en) * | 1999-05-01 | 2005-04-19 | Biowulf Technologies, Llc | Methods of identifying biological patterns using multiple data sets |
US6161130A (en) * | 1998-06-23 | 2000-12-12 | Microsoft Corporation | Technique which utilizes a probabilistic classifier to detect "junk" e-mail by automatically updating a training and re-training the classifier based on the updated training set |
US6687696B2 (en) * | 2000-07-26 | 2004-02-03 | Recommind Inc. | System and method for personalized search, information filtering, and for generating recommendations utilizing statistical latent class models |
CA2323883C (en) | 2000-10-19 | 2016-02-16 | Patrick Ryan Morin | Method and device for classifying internet objects and objects stored oncomputer-readable media |
US7062504B2 (en) * | 2002-04-25 | 2006-06-13 | The Regents Of The University Of California | Creating ensembles of oblique decision trees with evolutionary algorithms and sampling |
EP1523717A1 (en) | 2002-07-19 | 2005-04-20 | BRITISH TELECOMMUNICATIONS public limited company | Method and system for classification of semantic content of audio/video data |
US7668376B2 (en) * | 2004-06-30 | 2010-02-23 | National Instruments Corporation | Shape feature extraction and classification |
US7783085B2 (en) * | 2006-05-10 | 2010-08-24 | Aol Inc. | Using relevance feedback in face recognition |
US7756535B1 (en) * | 2006-07-07 | 2010-07-13 | Trend Micro Incorporated | Lightweight content filtering system for mobile phones |
US8287281B2 (en) | 2006-12-06 | 2012-10-16 | Microsoft Corporation | Memory training via visual journal |
US20080249764A1 (en) * | 2007-03-01 | 2008-10-09 | Microsoft Corporation | Smart Sentiment Classifier for Product Reviews |
US20090119242A1 (en) * | 2007-10-31 | 2009-05-07 | Miguel Vargas Martin | System, Apparatus, and Method for Internet Content Detection |
US20100306144A1 (en) * | 2009-06-02 | 2010-12-02 | Scholz Martin B | System and method for classifying information |
US8311956B2 (en) * | 2009-08-11 | 2012-11-13 | At&T Intellectual Property I, L.P. | Scalable traffic classifier and classifier training system |
US8924488B2 (en) * | 2010-07-27 | 2014-12-30 | At&T Intellectual Property I, L.P. | Employing report ratios for intelligent mobile messaging classification and anti-spam defense |
US8954316B2 (en) * | 2011-05-20 | 2015-02-10 | AOL, Inc. | Systems and methods for categorizing and moderating user-generated content in an online environment |
US8676730B2 (en) * | 2011-07-11 | 2014-03-18 | Accenture Global Services Limited | Sentiment classifiers based on feature extraction |
-
2012
- 2012-10-03 WO PCT/US2012/058575 patent/WO2013052555A1/en active Application Filing
- 2012-10-03 US US13/573,727 patent/US9104655B2/en active Active
- 2012-10-03 EP EP12838674.5A patent/EP2764472A4/en not_active Withdrawn
-
2015
- 2015-07-06 US US14/792,585 patent/US10565519B2/en active Active
-
2020
- 2020-01-08 US US16/737,581 patent/US11763193B2/en active Active
-
2023
- 2023-08-09 US US18/446,980 patent/US20230385704A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
WO2013052555A1 (en) | 2013-04-11 |
US20150310352A1 (en) | 2015-10-29 |
US9104655B2 (en) | 2015-08-11 |
US20130151443A1 (en) | 2013-06-13 |
US11763193B2 (en) | 2023-09-19 |
EP2764472A1 (en) | 2014-08-13 |
EP2764472A4 (en) | 2015-10-14 |
US10565519B2 (en) | 2020-02-18 |
US20200143289A1 (en) | 2020-05-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11763193B2 (en) | Systems and method for performing contextual classification using supervised and unsupervised training | |
Saleena | An ensemble classification system for twitter sentiment analysis | |
Sharif et al. | Sentiment analysis of Bengali texts on online restaurant reviews using multinomial Naïve Bayes | |
CN110209764B (en) | Corpus annotation set generation method and device, electronic equipment and storage medium | |
Stamatatos et al. | Clustering by authorship within and across documents | |
US9753916B2 (en) | Automatic generation of a speech by processing raw claims to a set of arguments | |
Inzalkar et al. | A survey on text mining-techniques and application | |
US8676730B2 (en) | Sentiment classifiers based on feature extraction | |
Baltas et al. | An apache spark implementation for sentiment analysis on twitter data | |
Hertling et al. | Webisalod: providing hypernymy relations extracted from the web as linked open data | |
Shirsat et al. | Document level sentiment analysis from news articles | |
Cardie et al. | Text annotation for political science research | |
Verberne et al. | Automatic thematic classification of election manifestos | |
Komalavalli et al. | Challenges in big data analytics techniques: a survey | |
Li et al. | A hybrid model for role-related user classification on twitter | |
Wicaksono et al. | Toward advice mining: Conditional random fields for extracting advice-revealing text units | |
Ijaz et al. | Biasness identification of talk show's host by using twitter data | |
Makrynioti et al. | PaloPro: a platform for knowledge extraction from big social data and the news | |
Rehman et al. | Urdu sentiment analysis | |
Zitnik | Using sentiment analysis to improve business operations | |
Sungsri et al. | The analysis and summarizing system of thai hotel reviews using opinion mining technique | |
Dutta et al. | A Deep Learning Model for Classification of COVID-19 Related Fake News | |
Khabour et al. | Arabic Sentiment Analysis of Mobile Banking Services Reviews | |
Sorvisto et al. | Live Twitter sentiment analysis: Big data mining with Twitter’s public streaming API | |
Stojanovski et al. | Emotion identification in twitter messages for smart city applications |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: YAHOO ASSETS LLC, NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:YAHOO AD TECH LLC (FORMERLY VERIZON MEDIA INC.);REEL/FRAME:064838/0613 Effective date: 20211117 Owner name: YAHOO AD TECH LLC, NEW YORK Free format text: CHANGE OF NAME;ASSIGNOR:VERIZON MEDIA INC.;REEL/FRAME:064838/0586 Effective date: 20211102 Owner name: VERIZON MEDIA INC., NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:OATH INC.;REEL/FRAME:064838/0278 Effective date: 20201005 Owner name: AOL INC., VIRGINIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KYAW, THU;SONG, SANG CHUL;MAHAJAN, VINEET;AND OTHERS;SIGNING DATES FROM 20130114 TO 20130212;REEL/FRAME:064831/0976 Owner name: OATH INC., VIRGINIA Free format text: CHANGE OF NAME;ASSIGNOR:AOL INC.;REEL/FRAME:064839/0518 Effective date: 20170612 |