KR20170139492A - Classifier recall estimation for sparse topics - Google Patents

Abstract

A method for document retrieval includes classifying documents by a first classifier to obtain a first set of final documents and classifying documents by a second classifier to obtain a second set of final documents . The method includes estimating a recall metric of a first classifier based on a first set of final documents and a second set of final documents, revising the first classifier based on the recall metric, And sorting the documents by the revised classifiers to obtain at least one additional document that has been obtained. The method further comprises transmitting the communication based on the classification.

Description

{CLASSIFIER RECALL ESTIMATION FOR SPARSE TOPICS}

This application claims priority to U. S. Provisional Application No. 62 / 069,618, filed October 28, 2014, entitled " METHOD AND SYSTEM FOR RECALL ESTIMATION " § 119 (e), which is incorporated herein by reference in its entirety.

This specification relates to digital information retrieval in the context of social platforms.

In the field of information retrieval, search engines typically implement indexing and retrieval of networked accessible documents in response to search queries. As used herein, documents are units of content, e.g., web pages of a social platform, text files, portable document format files, video files, audio files, image files, and social messages . As can be expected, different types of documents can represent different features, which are generally programmatically detectable properties of a document that can be computationally quantified. For example, a particular text file (often analyzed as important terms) with a particular string of text would represent features that are commonly referred to as term frequency, document length, and inverse term frequency. Certain photographs with a particular captured image will exhibit characteristics associated with color, opacity, and texture. Typically, documents that share common features are indexed with the same corpus, which facilitates programmatically comparing one document to another document (i.e., computing resources compare apples to apples .

One basic aspect of indexing is the classification performed for some search engines to determine which topics are associated with which documents. Moreover, a document can describe more than one topic. For example, a photograph of a celebrity in front of a famous landmark can be associated with both a celebrity and a landmark. Classification is typically implemented with some combination of program resources and human judgment when program resources are leveraging machine learning, and machine learning requires training to be effective. In operation, the program resource (classifier) takes the selected features of the document as input and applies the machine learning to determine whether such documents can be classified as describing the topics that the classifier is assigned to detect. Sampling of classifier judgments is evaluated by human operators, and the evaluations are applied to evaluate the validity of the classifier and to calibrate its machine learning parameters.

One measure of the validity of a classifier is that for implementations that are assigned tasks for assigning topics to documents, the classifier may use a recall (i.e., the ability to properly classify all the documents in the corpus that describes the topic recall).

In order to evaluate recall for classifiers assigned to detect topics populated in the corpus, human manipulations should typically sample only a small portion of the corpus. However, for classifiers assigned to detect sparse (less than 0.1% of the corpus) topics, a much larger sample of corpus is likely to be encountered because small samples are not likely to capture documents describing sparse topics Is required. The time and expense involved in these later efforts can be prohibitively expensive, especially when corpus documents are growing in billions.

In general, the second classifier C ₂ is configured to be sufficiently independent of the classifier C ₁ being evaluated, and both classifiers are configured to detect the same sparse topic of a particular corpus of documents U . Here, independence generally means having different features, i.e., those that are not interwoven. Mathematically speaking, complete independence means orthogonal features in feature space, and full dependency means having features on the same line in feature space. The second classifier (C ₂ ) is then used as a proxy for corpus U in the evaluation of the ability of C ₂ to recall sparse topics and should be sampled from the entire corpus U to evaluate the ability of C _{2 to} recall Instead, one should only sample from a set of documents selected by C ₂ , which is a subset much smaller than U, and thus the required sample size is significantly reduced. Optionally, complete independence is not implemented because incomplete independence may result in acceptable accuracy when estimating a recall of C ₁ . That is, C ₂ may be arbitrarily configured by features that are not independent, but are remotely related to features used by C ₁ .

Generally, in one aspect, one or more embodiments relate to a method for document retrieval. The method includes classifying the documents by a first classifier to obtain a first set of final documents, and classifying the documents by a second classifier to obtain a second set of final documents. The method includes estimating a recall metric of a first classifier based on a first set of final documents and a second set of final documents, revising the first classifier based on the recall metric, And sorting the documents by the revised classifiers to obtain at least one additional document that has been obtained. The method further comprises transmitting the communication based on the classification.

In general, in one aspect, one or more embodiments relates to a computing processor, and a system for recall estimation, including instructions, wherein the instructions, when executed on a computing device processor, By the first classifier, and classify the documents by the second classifier to obtain the second set of final documents. The instructions may include estimating a recall metric of the first classifier based on the first set of final documents and the second set of final documents, revising the first classifier based on the recall metric, And classifying the documents by the revised classifiers to obtain at least one additional document that is being generated. The instructions further comprise transmitting the communication based on the classification.

Generally, in one aspect, one or more embodiments relate to a non-volatile computer-readable medium comprising computer-readable program code, the computer-readable program code including instructions for: Classify the documents by the classifier, and classify the documents by the second classifier to obtain a second set of final documents. The computer readable program code includes instructions for: estimating a recall metric of a first classifier based on a first set of final documents and a second set of final documents; revising a first classifier based on a recall metric; And classifying the documents by the revised classifiers to obtain at least one additional document excluded from the one set. The computer readable program code further comprises transmitting the communication based on the classification.

Some implementations of the present invention require significantly smaller sample sizes than conventional systems in order to effectively evaluate the ability of the classifier to recall sparse topics. Because sampling requires human judgment, a significantly smaller sample size advantageously imposes a significantly reduced time and cost burden that is not prohibited, and therefore allows to perform the evaluations described for corpus with billions of documents.

Other aspects of the invention will be apparent from the following description and the appended claims.

Figures 1A, 1B, and 2 illustrate illustrations of the system in accordance with one or more embodiments of the present invention.
Figures 3 and 4 illustrate flowcharts in accordance with one or more embodiments of the present invention.
Figure 5 illustrates an example in accordance with one or more embodiments of the present invention.
Figure 6 illustrates a computing system in accordance with one or more embodiments of the present invention.

Specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Similar elements in the various figures are represented by like reference numerals for consistency.

In the following detailed description of embodiments of the present invention, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well-known features have not been described in detail in order to avoid unnecessarily complicated descriptions.

Throughout this application, ordinals (e.g., first, second, third, etc.) may be used as an adjective to an element (i.e., any noun in this application). The use of ordinates does not imply any element, either by the use of the terms "before", "after", "single", and such other terminology, unless the context clearly dictates or implies any particular ordering of the elements But is not limited to being a single element. Rather, the use of ordinances is to distinguish elements. By way of example, the first element may be separate from the second element, the first element may encompass more than one element and follow (or precede) the second element in the ordering of the elements.

In general, embodiments of the present invention relate to document retrieval. One or more embodiments have at least two different classifiers that independently classify a plurality of documents to obtain at least two sets of final documents. The classification from at least one classifier is used to generate a recall metric for the other classifier. The recall metric is an estimate of the number of documents that belong to the class but are not classified by the classifier into the class. Other classifiers may be revised based on the estimated amount.

In one or more embodiments, the classifiers are applied to sparse document classification within a large corpus. For example, a large corpus may have millions or billions of documents, and the number of documents that actually belong to a class may be less than 0.1% of the total corpus. One or more embodiments may use the results of one classifier as a proxy for the corpus of documents to generate an estimate of the recall for another classifier. In other words, for the purpose of determining the recall metric, the results of the second classifier serve as a proxy for the entire corpus of documents. In one or more embodiments, the classifiers meet a threshold degree of independence with respect to each other. Independence is discussed below with respect to FIG.

One or more embodiments of the invention may be applied to social networks. Here, the social content is divided into corpuses in which common features or attributes are assigned to the same corpus. In this corpus, a document includes all social content originating from a particular user, and such documents are associated with a user. Thus, a corpus can consist of multiple documents (hundreds of millions or billions) each associated with a particular user. In the case of user agreement, the Ad system may deduce the topics of interest to the user by detecting the topics discussed in the user's document and then provide the targeted ads, I am interested in the topics being discussed. Because of the recall estimation techniques described herein, at least two classifiers for sparse topics can be evaluated and calibrated to effectively recall documents resulting from sparse topics. For example, a first classifier may use "documents" comprised of social media messages from an account, and a second classifier may be using accounts / bio-text / other signals after users. Therefore, the system can find users interested in sparse topics, and although Ad targeting can be very granular (i.e., specific to sparse topics), the corpus of documents of social content is very large . Here, sparse topics can be, for example, specific models of automobiles with specific options, and this particular model provides more accurate Ad targeting than automotive models.

Moreover, users sharing similar topics of interest can be grouped together in the same group, and similar curated content can be provided to users in the group. For example, curated content related to news about foreign policy and sports may be provided to users in groups associated with foreign policy and sports. Curated content related to sports and food news can be provided to other groups of users, one associated with sports and food. Optionally, user grouping may be implemented to facilitate programmed curation. If the curated content is modular so that each module covers one topic that may be of interest to users, then the programmatic cueing system may select the modules of content corresponding to the topics of interest associated with the group It is possible to build curated content for each group of users of the social platform. Since programmatic cues can be implemented with low latency, the content provided can be up-to-date content, even breaking news.

Figure 1A illustrates a schematic diagram of a system for a social network in accordance with one or more embodiments of the present invention. In particular, FIG. 1A illustrates a social network application 100 and a client 120 in accordance with one or more embodiments of the present invention. The social network application 100 may also be referred to as a messaging platform in accordance with one or more embodiments of the present invention.

The social network application 100 connects users to other users (i.e., clients) of the social network application 100, exchanges social networking messages between connected users of the social network application 100, Provides an interface to create and view social network messages. In one or more embodiments of the invention, the social network messages are broadcast social networking messages that are transmitted to at least a set of users. The users in the set (e.g., the sender's followers) can be self-selected or users who meet certain conditions with the sending user (e.g., a group, friend, family, etc.) can be self-selected. The social networking messages may include, for example, comments from users on the document, personal status updates, references to documents, and / or other information or a combination thereof.

In addition, in one or more embodiments of the invention, the social network application 100 may receive an original reference from a user for the document, generate a social network reference from the original reference, send the social network reference to other users And transmitting functions. Thus, a user may share a document via the social network application 100 by sending a message containing a reference to the document to other users or by posting a social network reference to the document. In one or more embodiments of the present invention, the original reference is a reference to the location of the published document, e.g., a uniform resource locator (URL) of a web page. A social network reference is an indirect reference to the location of a published document. Specifically, the social network reference is mapped to the original reference in the discovery repository 148. The social network application may be configured to perform analyzes with connections of social network references and / or shorten the original references. For example, the social network reference and original reference may be a hypertext transfer protocol link or other mechanism that references the location of the document.

1A, a social network application 100 includes a classification engine 150, a machine learning module (not shown), a discovery repository 148, a front end module 125, a routing module 155, Including a fanout module 130, a delivery module 135, a message store 140, a connection graph store 142, a stream store 144, and an account store 146. [ The various components of the social network application 100 may be stored on the same device (e.g., server, mainframe, desktop personal computer (PC), laptop, personal digital assistant (PDA) (E. G., A local area network (LAN), the Internet, etc.) that may be located on a network (e. G., A mobile telephone, a kiosk, a cable box, Lt; / RTI > As well as any combination of these components, more than one component of each individual component executing on the device may be present in a given embodiment of the present invention.

In one or more embodiments of the invention, the social network application 100 is a platform that enables real-time communication between one or more entities. For example, the social network application 100 may store millions of accounts (e.g., pseudonym accounts, new accounts, etc.) of individuals, businesses, and / or other entities. One or more users of each account may use the social network application 100 to send social networking messages to other accounts within and / or to the social network application 100. [ The social network application 100 may be configured to allow users to communicate "in real time ", i. E. To communicate with other users with minimal delay, and to have conversations with one or more other users during concurrent sessions. . In other words, the social network application 100 may allow the user to broadcast social networking messages and may display the social networking messages in one reasonable time frame to one or more other users to facilitate real- can do. The recipients of the social networking message may have a pre-defined graph relationship with the user's account that broadcasts the social networking message. In one or more embodiments of the invention, the user is neither an account holder nor logged in to the account in the social network application 100. In such a case, the social network application 100 may be configured to allow a user to utilize other functionality of the social network application 100 by broadcasting broadcast social networking messages and / or associating the user with a temporary account or identifier .

In one or more embodiments of the invention, the link graph store 142 is configured to store one or more link graphs. Figure IB illustrates an exemplary depiction of a connection graph 299 in accordance with one or more embodiments of the present invention. As shown in FIG. 1B, the connection graph 299 includes the accounts of the social network application 100 (i.e., account A 200, account B 202, account C 204, account D 206, Including accounts E (208), accounts F (210), G (212), and the edges connecting the various nodes.

The connection graph 299 is a data structure that represents relationships (i.e., connections) between one or more accounts. The connection graph 299 represents relationships, such as nodes, representing the accounts and the edges connecting one or more nodes. A relationship may refer to any association between accounts (e.g., follow, add friends, join, track, taste, tagging, etc.). The edges of the connection graph 299 may be oriented and / or non-oriented based on the type of relationship (e.g., bidirectional, unidirectional), according to various embodiments of the present invention.

1A, in one or more embodiments of the invention, the routing module 155 includes functionality to receive one or more social networking messages and store the social networking messages in the message store 140. [ The routing module 155 may include functionality to assign an identifier to a social networking message and notify the sender's graph fanout module 130 of the social networking message.

In one or more embodiments of the invention, the graph fanout module 130 retrieves the graph data from the connection graph store 142 and determines which accounts in the social network application 100 should receive the social networking message Includes functionality to use graph data. The graph data may reflect, for example, whether accounts in the social network application are "following" a particular account and thus are subscribed to receive status social networking messages from a particular account.

In one or more embodiments of the invention, the forwarding module 135 receives a list of accounts from the graph fanout module 130 and a message identifier generated by the routing module 155 and stores the message identifier in each identified Into the stream data associated with the account. The delivery module 135 may then store the message list in the stream store 144. The stream data stored in the stream store 144 may constitute one or more streams associated with one or more accounts of the social network application 100. The stream may be a dynamic list of social networking messages associated with one or more accounts, or may reflect any arbitrary organization of social networking messages favorable to the user of the account.

In one or more embodiments of the invention, front end module 125 is a set of software applications or related software applications that are configured to communicate with external entities (e.g., client 120). The front end module 125 may include an application programming interface (API) and / or any number of other components used to communicate with entities external to the social network application 100. The API may include any number of features that receive requests from the social network application 100 and / or provide data to the application. For example, the functionality provided by the API may provide artist / song recommendations to the requesting client (not shown).

In one or more embodiments of the present invention, the front end module 125 may provide data stores (e. G., Data streams) to define streams that serve social networking messages (E.g., message store 140, connection graph store 142, stream store 144, discovery store 148, and / or account store 146). The user may use any client 120 to receive social networking messages. For example, if a user uses a web-based client to access a social network application 100, the API of the front end module 125 may define one or more streams and / or provide stream data to a user Can be used to serve clients. Similarly, different types of message delivery may be handled by different modules in the front end module 125. [ In one or more embodiments of the invention, the user may specify specific reception preferences, which are implemented by the front end module 125. [

One or more of the data stores (message store 140, connection graph store 142, stream store 144, account store 146, discovery store 148) And / or < / RTI > storage services residing on the above servers. For example, one or more of the data stores may be implemented as a storage service using a service-oriented architecture (SOA), receive requests for data, and send the requested data to the social network application 100. [ To other components of the system. In another example, the message store 140 may include one or more tables in a distributed database management system (DBMS), a clustered database, a stand-alone flat file, and / Lt; RTI ID = 0.0 > software. Examples of storage devices may include, but are not limited to, hard disk drives, solid state drives, and / or other memory devices. Any type of database or storage application may be used in accordance with various embodiments of the present invention.

One or more of the data stores (message store 140, link graph store 142, stream store 144, account store 146, discovery store 148) Is a set of individual applications or applications residing on one or more servers external to (and communicatively coupled to) the network application 100. Alternatively, in one or more embodiments of the invention, one or more of the data stores may be an integrated component of the social network application 100 and / or one or more common hardware devices (e.g., ), ≪ / RTI >

In one or more embodiments of the invention, the discovery repository 148 may store document metadata relating to documents. The document metadata may include a list of keywords, tracking information, date of the document, mapping between the social media references to the documents and the original references, and other information.

In one or more embodiments of the invention, the message store 140 includes functionality for storing social networking messages and social networking message metadata. The social networking message metadata may include an identifier of the originating user of the social networking message, a list of users who have received the social networking message, a number of users who have received the social networking message, statistics (e.g., The ratio of unconnected users to connected users sending a social networking message), the time and date that the social networking message is sent, and other information.

In one or more embodiments of the invention, the link graph store 142 may store information about users. Specifically, the link graph store may associate user identifiers of a user with the preferences and history of a user in the social network application 100. For example, user preferences and history may include language, user connections, topics of interest to the user, and other information.

The classification engine 150 includes functionality to classify documents and determine metrics from the classification according to one or more embodiments of the invention. The classification engine 150 is described in more detail in FIG.

2 illustrates a schematic diagram of a system in accordance with one or more embodiments of the present invention. 2, the system may include an Internet repository 220, a social network repository 208, a classification engine 200, and a communication distributor 218 in accordance with one or more embodiments of the present invention .

The Internet store 220 is any data store available over the Internet. In particular, the Internet repository 220 is any storage unit of the Internet documents 222. For example, the Internet repository 220 may be one or more personal computers, servers, a collection of mobile devices, or any other computing system or a collection thereof connected via the Internet. The Internet store 220 may be controlled by a number of different companies and users.

In one or more embodiments of the present invention, the Internet repository 220 includes functionality to store Internet documents 222. Internet documents 222 are documents available over the Internet. A document is any type of stored information. For example, the document may be a technical document, a paper, an audio file, a web page, a file, a tweet, a posting, a user profile, a message posted or communicated through a social media platform, a user's digital footprint, have. For example, the user ' s digital footprint may include a collection of documents that the user accessed, submitted social media messages, authored, or the user has other direct relationships. A document may have document content and metadata. The document content is the content of the document (e.g., the body of the document) presented to users when the document is displayed or played to the user. Metadata is information about a document. For example, metadata may include the name and size of the document, the author, the editing history (e.g., including the user identifier, time, and tracking information for edits made for each editing in the editing history) Creation time, security features enabled on the document, and other information. For example, the security features may include password protection, public key encryption, or other security features, or a combination of features.

A document can have one or more attributes. Attributes are characteristic of part of document content or metadata. In particular, the attribute may be information that is explicitly present in the document content or metadata, or may be information derived from the document content and / or metadata. For example, the attributes may be a name, a title, a table of content in the document content, a target reader's education level identifier, a number of retransmissions of the document, a document size, a uniform resource identifier, or other information.

The social network repository 208 is any type of storage unit and / or device (e.g., a file system, database, collection of tables, or any other storage mechanism) for storing data. In addition, the social network repository 208 may comprise a number of different storage units and / or devices. The plurality of different storage units and / or devices may or may not be the same type or may be located at the same physical site or not. The social network repository 208 may correspond to any of the repositories shown in FIG. 1A or any combination of repositories.

The social network repository 208 includes profiles 210 for each user, social network documents 212, and classifier results (e.g., classifier X results 214, classifier Y results (216). The profiles 210 correspond to information about the user. For example, the profile may include a user's unique identifier, a name, message preferences for the user, public and private identifiers, location information, pictures, links to messages, connections to other users and groups, Profile information.

The social network documents 212 correspond to any document for the social network. For example, the social network documents 212 may include postings by the user, technical documents stored in the social network repository, user profiles, status updates, or any other document specific to the social network repository.

The classifier results (e.g., classifier X results 214, classifier Y results 216) correspond to the results of the classification. For example, the classifier results may include, for each document classified by the classifier, a unique document identifier of the document and a class of the document. In one or more embodiments of the invention, the classifier results may be stored by a document or separated from the document. For example, the classifier results may be a relationship between the document and the class of the document. As an example, the classifier results may be implemented with tags contained in the indexes of the documents in the corpus.

Continuing with FIG. 2, the system may include a classification engine 200 having a classifier X 202, a classifier Y 204, and an analyzer 206. The classifier is hardware, software, firmware, or any combination thereof including functionality to classify documents (e.g., social network documents 212 and / or Internet documents 222). For example, one or both of the classifiers may be a bidirectional classifier that classifies documents into one of two groups, or a multi-way classifier that classifies documents into one of a plurality of groups. A bidirectional classifier is a judgment rule that, when considering a document, assigns a label {+1, -1} from the set to the document. The +1 label is given to " positive "judgments, indicating that the classification of the problem document meets the search criteria, while the -1 label is given to" negative " Classifiers make judgments using attributes of the input document, such as text in the document, copyright information, or links to other documents. In other words, the classifier performs classifications based on attributes having one or more attributes of the document. In one or more embodiments of the invention, certain attributes used by the classifier may be configurable and automatically updated.

In the equations presented below, various symbols can be used. The following is a table of symbols and corresponding definitions that may appear below.

In one or more embodiments of the invention, the classifiers meet a criticality of independence with respect to each other. In one or more embodiments of the present invention, the amount of classifiers is independent when the classifiers meet:

,

,

Where δ ₁ is the degree of independence between the classifiers (C ₁ and C ₂ ) as defined by the above equations. In one or more embodiments of the invention, the classifiers C ₁ and C ₂ meet the criticality of independence when δ ₁ is greater than the degree of independence threshold. For example, the degree of independence may be .95, .99, .90, or any other configurable value. In one or more embodiments of the invention, the equation may be considered valid and the classifiers use different and non-overlapping attributes of the documents.

Another way of characterizing the degree of independence between classifiers is based on documents that are not in the class (i.e., off-topic documents, T ^C ). The following equation can be used to determine the degree of independence based on the number of documents not in the class.

Analyzer 206 includes functionality to analyze the classifier results (e.g., classifier X results 214, classifier Y results 216) and perform recall estimation on the classifier results. The recall estimate is the propensity of a classifier that did not correctly lose a particular document in the classification. In particular, the accuracy is a measure of the percentage of documents that are added to the class by the classifier and that must be exactly in the class (e.g., the number of documents that are exactly in the class by the classifier). On the other hand, the recall estimate is not in the class as categorized, but is an estimate of the percentage of documents that should be in the class. For example, in a classification where the document is in a class or not in a class, the recall estimate is the percentage of documents added or lost to the class by the classifier or other number of documents. The recall can be defined using the following equation:

,

,

Where r is the recall percentage, the associated documents are all documents that should be classified as classes, and the retrieved documents are documents that are classified as classes.

The recall is distinct from the accuracy, which is the number of documents correctly searched by the classifier. In particular, the accuracy can be defined using the following equation:

Where r is the recall percentage, the associated documents are all documents that should be classified as classes, and the retrieved documents are documents that are classified as classes.

In the case of a large corpus of documents (e.g., social network documents 212, Internet documents 222), it may not be possible to manually determine the percentage. Analyzer 206 includes functionality to execute multiple classifiers (classifier X 202, classifier Y 204) and perform recall estimation based on classification results. Analyzer 206 may further include functionality for revising at least one of the classifiers.

The communication spreader 218 is hardware, software, firmware, or any combination thereof including functionality to distribute communications based on the classifier results. The communication spreader 218 may be part of an application used by a social network application and / or an independent product, e.g., a third party. For example, the communication distributor may be an advertisement server that distributes the correspondences to the advertisements, an advisory engine that includes functionality to send recommendations to the user, or any other communications distributor. By way of example, the recommendations may be recommendations of contacts, including recommendations of wireless, recommendations of profile preferences, or any other recommendation, to the social network.

Although Figure 2 shows two classifiers, additional classifiers may be included without departing from the scope of the present invention. In addition, although Figure 2 illustrates a communication distributor that is connected to a social network repository, any other module that uses classification results may be used without departing from the scope of the present invention.

Although Figures 1A, 1B, and 2 illustrate the construction of components, other configurations may be used without departing from the scope of the present invention. For example, the various components may be combined to produce a single component. As another example, the functionality performed by a single component may be performed by two or more components.

Figures 3 and 4 show a flow diagram according to one or more embodiments of the present invention. Although the various steps in this flowchart are presented and described in order, the ordinary skilled artisan will appreciate that some or all of the steps may be performed in different orders, combined or omitted, and that some or all of the steps may be performed in parallel I will understand. Moreover, the steps may be performed actively or passively. For example, some of the steps may be performed using polling or may be interrupt driven in accordance with one or more embodiments of the present invention. By way of example, the determining steps may not require the processor to process an instruction as long as an interrupt is received to indicate that the condition exists in accordance with one or more embodiments of the present invention. As another example, the determining steps may be performed by performing a test, e.g., by checking the data value to test whether the data value is consistent with the tested condition, in accordance with one or more embodiments of the present invention.

3 illustrates a flow chart for classifying documents associated with a user in accordance with one or more embodiments of the present invention. In Fig. 3, the documents may be a set of digital footprints or documents associated with the user.

At step 301, the documents are classified to obtain a first set of final documents in the class according to one or more embodiments of the present invention. In other words, the classifier identifies the document and extracts the attribute values of the document corresponding to the attributes in the attribute set of the classifier. By comparing the attribute values with the attributes of the classes, the classifier is trained thereby, and the classifier determines to which class the document is to be added. The classifier repeats the process for each document. In some embodiments, the classifier operates in series as a single thread. In other embodiments, the classifier may have multiple threads or multiple application instances that classify documents in parallel.

At step 303, users corresponding to the documents are identified according to one or more embodiments of the present invention. In one or more embodiments of the present invention, each document may have one or more users associated with the document. For example, a user may have sent a message with a link to a document, may be the author of the document, may have accessed the document, or may have performed other actions on the document. By accessing profile information about the user and / or metadata information about the document, one or more users having a relationship with the document can be identified. In one or more embodiments of the invention, one or more predefined relationship criteria may exist for the document-user relationship. Rather than any user having a relationship with the document, for example, the predefined relationship criteria may specify a predefined relationship the user has to be identified at step 303. For example,

In step 305, communications are sent to each of the users corresponding to at least one document that is classified according to one or more embodiments of the present invention. In one or more embodiments of the invention, the communication is transmitted based on the classification. In other words, the communication is sent to each user having one or more predefined relationships with the document in the class in response to the user having a predefined relationship. The communications may be sent via the user interface of the executing application, via a social media message, via e-mail, or via any other technique.

At step 307, the documents are classified using a second classifier to obtain a second set of final documents in the class according to one or more embodiments of the present invention. In one or more embodiments of the present invention, the second classifier is selected based on the degree of independence for the first classifier. In particular, for arbitrary document collection, documents tend to overlap. That is, the same information is represented in many different ways throughout the document. Several ways to leverage this redundancy to create conditional independent classifiers can be used. For example, the second classifier may be configured to classify documents based on different parts of the content in the document, or based on different properties, based on different non-overlapping sets of attributes, Can be trained to classify. In other words, in one or more embodiments of the invention, the independence between the first classifier and the second classifier may not be complete. For example, some dependencies may exist between attributes of a classifier. However, in accordance with one or more embodiments of the present invention, the amount of independence between classifiers meets a threshold value. Constructing exemplary sets of classifiers is discussed below. In one or more embodiments of the present invention, classification of documents by the second classifier may be performed using the same or similar techniques as discussed above with reference to step 301 of FIG.

At step 309, the recall metric is estimated based on the first set of final documents and the second set of final documents according to one or more embodiments of the present invention. In particular, one or more embodiments of the invention perform recall estimation on the first classifier using the results of the second classifier. Recall estimation can be performed using the following equation for two independent classifiers (e.g., classifier 1, classifier 2) that perform bi-directional classification:

In the equation, r is the recall estimation, A ₁ is a set of documents that are classified by the classifier 1 in the class, and a set of documents that are classified by the A ₂ are classified in a class character 2, A ₁₂ is classified in a class P ₁ is the accuracy of classifier 1, p ₂ is the accuracy of classifier 2, and U is the set of all documents in the set. The equation may be modified for additional classifiers, multi-direction classifiers, or classifiers that do not have much independence.

At step 311, a determination is made whether the recall metric meets a threshold value in accordance with one or more embodiments of the present invention. If the recall metric meets the threshold, then the flow can proceed to termination. In other words, the communication can be regarded as being transmitted to all the targeted users. In one or more embodiments of the invention, if the amount does not meet the threshold, the flow may proceed to step 313.

In step 313, the first classifier is revised based on the quantity to obtain the classifier revised in accordance with one or more embodiments of the present invention. The step of revising the first classifier may include adding attributes of the second classifier to the first classifier, retraining the first classifier using additional training samples, or performing other actions.

At step 315, the documents are classified using the classifier revised in accordance with one or more embodiments of the present invention. In one or more embodiments of the invention, the step of classifying the documents by the revised classifier may be performed using the same or similar techniques as discussed above with reference to step 301 of FIG. Rather than the original classifier classifying the documents, however, the revised classifier classifies the documents. For example, the revised classifier may use additional attributes or attribute values.

At step 317, any additional users are identified based on the documents classified according to one or more embodiments of the present invention. Identifying additional users may be performed using the same or similar techniques as discussed above with reference to step 303 of FIG. In other words, using the modified classification results, additional users are identified.

At step 319, communication is sent to additional users according to one or more embodiments of the present invention. The step of transmitting the communication may be performed using the same or similar technology as discussed above with reference to step 305 of FIG. In one or more embodiments of the invention, rather than sending the communication to all users, the communication is sent only to additional users that are added by revising the classifier.

Figure 4 shows a flow chart for classifying documents, for example for indexing purposes. In Fig. 4, the corpus of the document may be virtually any type of document, e.g. the documents discussed above with reference to Fig. At step 401, the documents are sorted in an index according to one or more embodiments of the present invention to obtain a first set of final documents in the class. In other words, the classifier identifies the document and extracts the attribute values of the document corresponding to the attributes in the attribute set of the classifier. By comparing the attribute values with the attributes of the classes, the classifier is trained thereby, and the classifier determines to which class the document is to be added. The classifier can then add the document to the index as appropriate. The classifier repeats the process for each document. In some embodiments, the classifier operates in series as a single thread. In other embodiments, the classifier may have multiple threads or multiple application instances that classify documents in parallel.

In step 403, the documents are classified using a second classifier to obtain a second set of final documents in the class in the index according to one or more embodiments of the present invention. The step of selecting a second classifier may be performed using the techniques described above with reference to step 307 of FIG. In one or more embodiments of the invention, the classification of documents by the second classifier may be performed using the same or similar techniques as discussed above with reference to step 401 of FIG.

At step 405, the recall metric is estimated based on the first set of final documents and the second set of final documents according to one or more embodiments of the present invention. At step 407, a determination is made whether the recall metric meets a threshold value in accordance with one or more embodiments of the present invention. If the recall metric meets the threshold, then the flow can proceed to termination. In other words, the communication can be regarded as being transmitted to all the targeted users. In one or more embodiments of the invention, if the amount does not meet the threshold, the flow may proceed to step 409. [ In step 413, the first classifier is revised based on the quantity to obtain the classifier revised in accordance with one or more embodiments of the present invention. At step 415, the documents are classified using the classifier revised in accordance with one or more embodiments of the present invention. Steps to perform steps 405, 407, 409, and 411 may be performed using techniques described above with reference to steps 309, 311, 313, and 315 of FIG. 3, respectively.

At step 413, a request for communication is received from the user in accordance with one or more embodiments of the present invention. For example, a user may request a set of documents corresponding to a class. In other words, the user can request documents that are indexed according to a particular topic. As another example, the user may request that the analysis be performed based on the classes associated with the documents. For example, the analysis may be the percentages and types of documents in the class.

In step 417, the classification results are obtained in response to the request. In one or more embodiments of the invention, an analysis requested by the user is performed, identifiers of documents and / or documents indexed with the class are obtained, or another set of classification results is obtained.

At step 419, the communication is transmitted to the user in accordance with one or more embodiments of the present invention. In other words, the communication is generated by the requested information and transmitted to the user. The step of transmitting the communication can be performed using various techniques, for example, via an electronic mail message, in a web page, as a page in a web application, or using any other techniques. In one or more embodiments of the invention, the step of transmitting the communication may be performed by dividing the message into packets and transmitting on the network connecting the various physical devices.

The following examples for generating independent classifiers are for illustrative purposes only and are not intended to limit the scope of the invention. In the following example, we consider a pair of state keyword filters that tend to exhibit the conditional independence properties. A keyword filter is a special type of classifier in which a document is displayed as an on-topic when any of the words in the filter appear in the document. A seed term is a single word or phrase that captures the topic of interest in the widest generality possible. Some examples of seed terms would be Sports, Mars or Obama. Classifiers are defined as follows: a first classifier in a pair, C _seed , filters on terms seed; #seed (in the case of social media messages), while filters for similar neighbors of the second filter, C _neighbors , and seed terms. The number of different scores can be used to measure the overlap. If A _seed and A _y denote documents found by each of the seed term filter and candidate keyword _y , then one such similarity score is jacquard similarity using the equation below.

The following is an exemplary set of seed terms and high jacquard similarities. The first example of seed terms is [apple; #apple] and similar neighbors are [# ios6, # ipad3, #iphone, hack, macintosh, iphones, #siri, ios, macbook, icloud, ipad, samsung, #ipodtouch, 4s , ITunes, Sidia, Sider, #gadget, #tech, #tablet, app, connector, #mac, ...]. A second example of seed terms is [mars; #mars] and similar neighbors [rover, screw, #curiosity, #curiousity, image, mission, surface, #curiosityrover, Bruno, Milky, budget, @marscuriosity, Orbit, successfully, land, landing, breathing ...]. A third example of seed terms is [obama; #obama] and similar neighbors are [@barackobama, Barack, bush, #mitt2012, # obama2012, obamas, #dems, #gop, #military, romney, #idontsupportobama, president, administration, press, #politics, vote ...]. The fourth example of seed terms is [olympics; #olympics] and similar neighbors [medalists, gold, london, gb, keylani, gymnast, kate, sprinter, win, won, #boxing, football, watch, watching ), #usa, #teamgb, #canada, #london, pitcher, nbc, competition, 2012, 400m ...].

Another similarity score is an asymmetric overlap measure using the following equation.

.

.

PMI, bus key similarity, or other similarity measure values may be used. Regardless of which measurement is used, the top scoring neighbors (excluding seed terms) are included in the filter defining the C _neighbors . Although some terms appearing in the list of neighbors may not be on-topics, both filters have "reasonable" overall accuracy. In some embodiments, the accuracy in the denominators of the above equations is only required to be not so small as not to make the calculations unstable in addition to the conditional independent assumptions.

Seed terms may occur independently of other terms, with respect to well-defined topics, given an on-topic document. For example, for a topic olympics, taking into account that a document is drawn from a set of on-topic documents, it is expected that occurrences of the terms 100m, swimming and pitching will occur independently of the seed term olympics.

As another example, threshold-based classifiers and social media classifier pairs that satisfy conditional independence properties can be considered. In a second example, classifiers derived through training corpus that produce a real-valued score when evaluated through out-of-sample data are considered. For scores above the set threshold, the document is labeled on-topic. Many commonly used machine learning techniques are divided into these categories, including logistic regression, SVM or Naive Bayes. Additionally, the population of documents U is also a set of documents that contain links to other documents. In the case of social media, the classifier may also be restricted to social media messages that contain links to documents on the web. Using threshold-based classifiers, a conditional independent pair can be constructed as follows: one classifier is wholly trained and executed on the social media message text, while the other is the web documents to which the social media message links Lt; / RTI > In training, the processes that generate each description / web document pair are independent of each other. In some embodiments, only conditional independence is maintained in close proximity.

In the example, as the thresholds for the classifiers C1 and C2 decrease, the conditional independent assumptions become more and more valid. However, the tradeoff of selecting thresholds at a lower level is to make it more expensive to obtain accurate estimates of each of the accuracies pi and p2.

As another example, upon returning to the original population of documents U, a classifier pair is considered that allows the first classifier C1 to be filter-based or threshold-based. The second classifier C2 is based solely on the social signals. The social signals may consist of copyright (the author, which is mainly known for technical papers, will have all of his / her social media messages categorized by C2 as on-topic to topic technology). On social media platforms, @mentions can provide additional social signals. For example, recognizing that a social media message was sent to @ barackobama as a reply, it can provide a signal that the social media message is about politics. Due to the fact that the signals used by each of these classifiers have independent sources, conditional independence can also be maintained here.

In one or more embodiments of the present invention, although the classifier is being used, two new classifiers may be constructed using the following. Having recalls of each of those classifiers, an estimate, i.e., a nest, of the total positive numbers can be formed. That is, the estimated value nest can be calculated as follows:

,

,

Where r ₁ is the recall of the conditional independent classifier, p ₁ is the corresponding accuracy and A 1 is the set of documents retrieved by this classifier. Using the estimates currently being addressed, an estimate of the recall for the classifier of interest can be obtained since the accuracy can be computed inexpensively as discussed above. The estimate for the recall of interest classifiers (i.e., n _new-est ) can be obtained using the following equations.

The following examples are for illustrative purposes only and are not intended to limit the scope of the invention. In the following example, consider a scenario in which classification is performed to classify whether users are appropriate for targeting for a particular advertising campaign. An ad campaign is meant to reach a target based on a small number of users, e.g., niche products. One classifier may, for example, be based on the corpus of generated social media messages, while the other may be based on users whose accounts are followed. The two classifiers can be used together to determine the recall metric of each classifier (e.g., a portion of all appropriate targeting candidates that each classifier can discover).

As a more specific example, consider a scenario in which a targeted advertisement is sent for an advertisement distribution device premium automobile. Specifically, the ad distributor targets rich users. In the example, the classifier X may be classified based on user profiles. That is, the classifier X searches for specific attributes in the user profile, which may be an indication of how rich each user is. The classifier X stores user identifiers of identified users having at least a certain revenue or net asset level in the classifier X results. The classifier Y searches for specific keywords in user profiles, which may be an indication of how rich each user is, and stores the identified users with at least a certain income or net asset level in the classifier Y results.

The classifier X results may be used by the analysis module with classifier Y results to estimate the recall of classifier Y results and thus the percentage of target users lost by classifier Y. [ In the example, if the recall percentage is 30%, then the ad distributor may lose 70% of the users who may be interested in the premium automobile. Because the classifier Y has lost more than the threshold percentage of the targeted users, the estimated recall may be used to revise the classifier Y that generates the revised classifier. For example, the classifier Y may be retrained by an additional training set, may have additional keywords to be added, have additional or alternative types of documents to be added, and / . If classifier X and classifier Y are combined to produce a revised classifier, the recall and accuracy can be estimated based on existing classifier X results and classifier Y results in accordance with one or more embodiments of the present invention. have.

The quality of the estimated recall may depend on the degree of independence between classifier X and classifier Y and / or the accuracy of classifier X results and classifier Y results. In other words, the quality of the estimate can be calculated by calculating the degree of independence between the classifier and / or the results and / or by estimating the accuracy. For a greater degree of independence (i. E., More independent) and / or greater accuracy, the quality increases.

One or more embodiments of the present invention may be applied to social media analysis as follows. The volume and / or availability of the creation and updates of social media updates (e.g., social media messages, status posts) has resulted in the use of various predictive tasks. For example, prediction tasks include analyzing emotions around products and brands (and ultimately potential sales), inferring spectators to sporting events, estimating the popularity of politicians, and the like . Thus, one or more embodiments may define independent classifiers that classify social media streams into topics. One or more embodiments can then estimate the recall using the techniques described above to determine the performance of one or both classifiers. Based on the estimated recall, the classifier can be updated and used to reclassify social media updates. The results of the classification may be used to generate metrics describing the social media updates in accordance with the prediction operations. The social media metrics may be described in terms of communication to one or more users, e.g., other companies that use the results to perform actions.

Figure 5 illustrates an example, in accordance with one or more embodiments of the present invention. In particular, Figure 5 illustrates an exemplary Venn diagram. In Figure 5, an area with a set of all documents is labeled as area U 500 and is a whole rectangle. The area with the set of documents that actually belongs to the class is the circularly labeled area T (502). The area having a set of documents classified by the classifier C ₁ as belonging to the class is the circularly labeled area A ₁ 504. The area having the set of documents classified by the classifier C ₂ as belonging to the class is the circularly labeled area A ₂ 506. If A1 and A2 meet the constraints, the recall r ₁ of C ₁ can be estimated using the following equation:

That is, the equations and Figure 5 shows the amount for which A ₂ can be found a classifier (C1). If C1 and C2 are sufficiently independent, the estimates using the above equations reliably carry the recall r ₁ . Rather than being independent of random draws from the entire set of documents, U's on-topic and independence through on-topic subsets validate the intuition of the equation according to one or more embodiments of the present invention.

Embodiments of the invention may be implemented on a computing system. Any combination of moving, desktop, server, embedded, or other types of hardware may be used. 6, computing system 600 may include one or more computer processor (s) 602, associated memory 604 (e.g., random access memory (RAM) (E.g., a hard disk, an optical drive such as a compact disk (CD) drive or a digital versatile disk (e.g., a hard disk, DVD) drives, flash memory sticks, etc.), and a number of other elements and functionality. The computer processor (s) 602 may be an integrated circuit that processes instructions. For example, the computer processor (s) may be one or more cores, or microcores of the processor. The computing system 600 may also include one or more input device (s) 610, such as a touch screen, keyboard, mouse, microphone, touch pad, electronic pen, or any other type of input device. In addition, computing system 600 may include one or more output device (s) 608, such as a screen (e.g., a liquid crystal display (LCD), a plasma display, a touch screen, a cathode ray tube (CRT) monitor, projector, or other display device), a printer, external storage, or any other output device. One or more of the output device (s) may be the same as or different from the input device (s). Computing system 600 may be coupled to a network 612 (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, Other types of networks). The input and output device (s) may be connected locally or remotely to computer processor (s) 602, memory 604, and storage device (s) 606 (e.g., via network 612) . There are many different types of computing systems, and the input and output device (s) described above can take other forms.

Software instructions in the form of computer readable program code for carrying out embodiments of the present invention may be stored on a non-volatile computer readable medium such as a CD, DVD, storage device, diskette, tape, flash memory, physical memory, On a removable storage medium, in whole or in part, temporarily or permanently. In particular, software instructions may correspond to computer readable program code that is configured to perform embodiments of the present invention when executed by the processor (s).

In addition, one or more elements of the computing system 600 described above may be located at a remote location and may be coupled to other elements via the network 612. In addition, embodiments of the present invention may be implemented on a distributed system of a plurality of nodes, and each portion of the invention may be located on a different node in the distributed system. In one embodiment of the invention, the node corresponds to a separate computing device. Alternatively, the node may correspond to a computer processor having associated physical memory. A node may alternatively correspond to a microcore of a computer processor or computer processor having shared memory and / or resources.

While the invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate that other embodiments may be devised which do not depart from the scope of the invention as disclosed herein I will understand. Accordingly, the scope of the present invention should be limited only by the appended claims.

Claims (20)

CLAIMS 1. A method for document retrieval,
Classifying a plurality of documents by a first classifier to obtain a first set of final documents, the first set of final documents corresponding to a plurality of users;
Sending a communication to each of the plurality of users based on each of the plurality of users corresponding to at least one document in the first set of documents;
Classifying the plurality of documents by a second classifier to obtain a second set of final documents;
Estimating, by a computer processor, an amount of the plurality of documents excluded from identification by the first classifier based on the first set of final documents and the second set of final documents;
Revising the first classifier based on the quantity to obtain a revised classifier;
Classifying the plurality of documents by the revised classifier to obtain at least one additional document excluded from the first set of documents; And
Transmitting the communication to the at least one additional user that is excluded from the plurality of users based on at least one additional user corresponding to the at least one additional document
≪ / RTI > 2. The method of claim 1, wherein the first classifier classifies the plurality of documents based on a first set of constraints, the second classifier classifies a second plurality of documents based on a second set of constraints, Wherein the first set of constraints is within a criticality of independence for the second set of constraints. 3. The method of claim 2, wherein the first set of constraints comprises a first keyword and the second set of constraints comprises a second keyword. The method of claim 1, wherein the communication is an advertisement. The method of claim 1, wherein the plurality of documents comprises a plurality of social media messages. 6. The method of claim 5 wherein the first classifier is based on at least one selected from the group consisting of the content of each social media message, the copyright of each social media message, and connections with second social media messages How to sort. 2. The method of claim 1 wherein the first classifier classifies the plurality of documents based on a first portion of each document and the second classifier classifies the second plurality of documents based on a second portion of each document, Wherein a first portion of each of the documents and a second portion of each of the documents are non-overlapping. As a system for recall estimation,
A computing processor;
Instructions, when executed on a computing device processor,
Classifying a plurality of documents by a first classifier to obtain a first set of final documents, the first set of final documents corresponding to a plurality of users;
Send communications to each of the plurality of users based on each of the plurality of users corresponding to at least one document in the first set of documents;
Classify the plurality of documents by a second classifier to obtain a second set of final documents;
Estimate by the computer processor the amount of the plurality of documents excluded from identification by the first classifier based on the first set of final documents and the second set of final documents;
Revising said first classifier based on said amount to obtain a revised classifier;
Classify the plurality of documents by the revised classifier to obtain at least one supplementary document excluded from the first set of documents;
The communication is transmitted to the at least one additional user that is excluded from the plurality of users based on at least one additional user corresponding to the at least one additional document. 9. The method of claim 8, wherein the first classifier classifies the plurality of documents based on a first set of constraints, the second classifier classifies a second plurality of documents based on a second set of constraints, Wherein the first set of constraints is within a criticality of independence for the second set of constraints. 10. The system of claim 9, wherein the first set of constraints comprises a first keyword and the second set of constraints comprises a second keyword. 9. The system of claim 8, wherein the communication is an advertisement. 9. The system of claim 8, wherein the plurality of documents comprises a plurality of social media messages. 13. The method of claim 12, wherein the first classifier is based on at least one selected from the group consisting of the content of each social media message, the copyright of each social media message, and connections with second social media messages Classification, system. 9. The method of claim 8, wherein the first classifier classifies the plurality of documents based on a first portion of each document, and the second classifier classifies the second plurality of documents based on a second portion of each document Wherein the first portion of each document and the second portion of each document are non-overlapping. 17. A non-transitory computer readable medium comprising computer readable program code,
Classifying a plurality of documents by a first classifier to obtain a first set of final documents, the first set of final documents corresponding to a plurality of users;
Send communications to each of the plurality of users based on each of the plurality of users corresponding to at least one document in the first set of documents;
Classify the plurality of documents by a second classifier to obtain a second set of final documents;
Estimate by the computer processor the amount of the plurality of documents excluded from identification by the first classifier based on the first set of final documents and the second set of final documents;
Revising said first classifier based on said amount to obtain a revised classifier;
Classify the plurality of documents by the revised classifier to obtain at least one supplementary document excluded from the first set of documents;
Wherein the communication is for transmitting the communication to the at least one additional user that is excluded from the plurality of users based on at least one additional user corresponding to the at least one additional document. 16. The apparatus of claim 15, wherein the first classifier classifies the plurality of documents based on a first set of constraints, the second classifier classifies a second plurality of documents based on a second set of constraints, Wherein the first set of constraints is within a criticality of independence for the second set of constraints. 17. The non-transitory computer readable medium of claim 16, wherein the first set of constraints comprises a first keyword, and wherein the second set of constraints comprise a second keyword. 16. The non-transitory computer readable medium of claim 15, wherein the plurality of documents comprises a plurality of social media messages. 20. The method of claim 19, wherein the first classifier is based on at least one selected from the group consisting of the content of each social media message, the copyright of each social media message, and connections with second social media messages Non-transitory computer readable medium. 16. The apparatus of claim 15 wherein the first classifier classifies the plurality of documents based on a first portion of each document and the second classifier classifies the second plurality of documents based on a second portion of each document, Wherein a first portion of each of the documents and a second portion of each of the documents are non-overlapping.

Images