JP2022552918A - Ranking Messages in Conversation Graphs in Messaging Platforms Using Prediction Results - Google Patents

Abstract

According to one aspect, a method for ranking messages of a conversation graph in a messaging platform includes receiving a conversation viewing request to retrieve conversation graph messages stored in the messaging platform; and determining a plurality of prediction results for each of the plurality of messages of the conversation graph. The determining step includes acquiring a plurality of signals related to the prediction model, including data structure related signals related to the conversation graph, and determining the plurality of prediction results. inputting the information into the prediction model. The method includes: the messaging platform ranking the plurality of messages based on the prediction results; and the messaging platform rendering at least one of the plurality of messages through the network at a client application according to the rank. and transmitting the part.

Description

The present invention relates to a computer-implemented method and system for ranking messages in a conversation graph in a messaging platform using predictive results.

Social media messaging platforms are capable of exchanging millions or hundreds of millions of social media messages between their users. Messages exchanged on the platform can often provide the user of the platform with the latest updates or reports on current events. An exchange of messages in a messaging platform may be part of a conversation between users. Some conventional messaging systems may store the reply structure of messages so that users can view parts of the conversation when viewing a particular message. However, the list of messages forming a conversation view may be relatively large, thereby causing slow load times for the user.

A messaging platform facilitates the exchange of messages between users of the messaging platform. A message may be part of a conversation conducted on a messaging platform. For example, a user may post a message on the platform. Other users may post multiple replies to that message, then replies to that reply, and so on. A reply structure may be stored as a conversation graph. A messaging platform may store any number of conversation graphs associated with conversations conducted on the messaging platform. In some examples, the conversation graph is relatively large (eg, multiple nodes in the conversation graph exceed a threshold). Generating and maintaining a conversation graph enables rapid retrieval of information when responding to requests to view messages in a reply relationship to a particular message.

When viewing a particular message associated with a conversation graph, a user may select a message to view other messages in the conversation graph, where the client application submits a conversation viewing request to the messaging platform. Generate. Rather than displaying all of the messages (which may be relatively large) that are related to the conversation graph, the Timeline Manager uses the conversation graph to rank the branches (or nodes) that are most relevant to a particular user. messages may be ranked using a unique scoring system (which incorporates engagement predictions) and then a ranked list (or a subset thereof) may be provided. In this way, messages in the conversation graph are displayed according to their level of prediction results (balancing positive prediction results against negative prediction results). Additionally, by providing a ranked list, the messaging platform allows increasing the value provided to the user while providing fewer responses. For example, the messaging platform may provide a subset of responses (eg, top 10, 15, 20, etc. responses) for each request. This can result in faster computation on the server, faster load times for the user, and almost no loss in terms of engagement.

A prediction manager (running on the messaging platform) may use one or more prediction models to determine a prediction result for each candidate node in the conversation graph. The predicted result is the predicted user engagement result for each message (or subset of messages) of the conversation graph. Prediction results include mutual engagement probabilities. A mutual engagement probability indicates a probability value that a user is expected to further develop the conversation graph (eg, to reply to a message). Incorporating mutual engagement probabilities into prediction results may encourage more conversations on the platform. In some examples, the predicted results include mutual engagement probabilities and one or more of positive engagement probabilities and negative engagement probabilities. A positive engagement probability indicates a probability value that a user is expected to engage positively with a message (eg, like or add a node to favorites). A negative engagement probability indicates a probability value that a user is expected to negatively view or engage with the message (eg, the user may think the node is a swear word). In some examples, the predicted results include mutual engagement probabilities, positive engagement probabilities, and negative engagement probabilities.

The predictive models may include a positive engagement model that predicts positive engagement probabilities, a negative engagement model that predicts negative engagement probabilities, and a cross-engagement model that predicts cross-engagement probabilities. In some examples, the predictive model is a machine learning model that is trained using training data applied to one or more machine learning algorithms. In some examples, predictive models include neural networks. Training data may be a combination of historical data obtained from client event logs and injection logs while the messaging system is active (and the prediction manager is enabled). Historical data may be obtained over a period of time (eg, a day, a week, a month, a year, etc.) and then used to train (or retrain) a predictive model in offline mode. The updated forecast model may now be sent to the forecast manager.

In response to a conversation view request to retrieve a message of the conversation graph, the prediction manager may obtain a signal to determine the prediction result and input the signal into the prediction model to determine the prediction result. The signals may include signals generated by the client application and/or signals generated by the messaging platform. In some examples, the signals include the user's and author's social graph and engagement history. However, the signals may include a variety of signals such as data structure related signals, health related signals, engagement signals, social graph signals, history aggregation signals, and/or content related signals. The number of signals and the number of different categories of signals used for prediction may improve the accuracy of the model prediction.

The prediction manager may combine prediction results to obtain an engagement value for each node. Here, the engagement value indicates the total level of predicted engagement with the messages in the conversation graph. The timeline manager may use the engagement value to greedily select the most relevant branches (or nodes) of the conversation graph for the user. In addition to locating the most relevant branches (or nodes), the timeline manager may determine the amount of branches to display based on engagement value. Additionally, the timeline manager may collapse irrelevant parts of the conversation graph, but allow the user to expand those parts to view other messages in the conversation graph.

Messages ranked according to engagement value may be unique to each user. For example, some messages in the conversation graph may be more relevant to the first user, while other messages in the conversation graph may be relevant to the second user. However, the prediction results determined by the prediction manager may be tailored to specific users. In contrast, some conventional approaches use vote-based mechanisms that can provide identical views for each. Additionally, because the messaging platform incorporates interactive engagement probabilities into its scoring algorithm, the messaging platform may encourage more conversations on the messaging platform.

In some examples, the messaging platform performs a two-level ranking system on the messages of the conversation graph. For example, a two-level ranking mechanism (eg, hard ranking and soft ranking) may provide an effective mechanism for combating swearing in conversations on messaging platforms. In some examples, hard ranking classifies (or sectioning) messages (or branches) of a conversation into multiple sections. For example, a messaging platform may include a content quality classifier that divides nodes (or branches) of a conversation graph into sections. Each section may represent a different categorization (or classification) of quality levels. In some examples, a content quality classifier is a classification model (e.g., , neural networks).

A content quality classifier may determine that the first message is of relatively low quality and the second message is of relatively high quality. Here, the content quality classifier may assign the first message (or the branch containing the first message) to the first section (eg, the first section considered a low quality section), 2 messages (or the branch containing the second message) may be assigned to the second section (eg, the second section considered the high quality section). Soft ranking, a two-level ranking mechanism, may involve ranking messages (or branches) within each section using the engagement values described above. A message may then be presented to the user to provide a ranked list of high quality sections followed by a ranked list of low quality sections.

According to one aspect, a method for ranking conversation graph messages in a messaging platform using prediction results receives, over a network, a conversation view request to retrieve conversation graph messages stored in the messaging platform. and determining, where the messaging platform uses a predictive model to determine a plurality of predicted outcomes for each of a plurality of messages of the conversation graph. The plurality of prediction results includes mutual engagement probabilities. The method comprises the steps of: the messaging platform ranking the plurality of messages based on the predicted results; and the messaging platform rendering at least a subset of the plurality of messages in a client application according to the rankings over the network. and transmitting. According to further aspects, a non-transitory computer-readable medium storing a corresponding system and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more of the following features (or any combination thereof). The plurality of predicted outcomes also includes a positive engagement probability and a negative engagement probability. The method comprises calculating an engagement value for each of the plurality of messages using the mutual engagement probability, the positive engagement probability, and the negative engagement probability, wherein the plurality of messages are matched according to the engagement value. Ranked. The method comprises the steps of: the messaging platform generating the conversation graph based on a reply structure of messages exchanged on the messaging platform, the conversation graph comprising a tree data structure of messages associated with a conversation; A process may be provided. The method comprises the steps of: the messaging platform obtaining signals generated by one or more of the client application and the messaging platform; and the messaging platform determining the plurality of predicted results, and inputting said signal to said prediction model. The method may comprise the messaging platform training the predictive model based on a machine learning algorithm fed with training data. The method comprises the steps of: the messaging platform, in response to a number of messages in the conversation graph exceeding a threshold level, selecting a candidate subset from the plurality of messages; is determined for each message of said candidate subset. The method comprises the steps of: the messaging platform sending a first subset of the plurality of messages rendered in the client application according to the rank; and the messaging platform displaying additional messages from the conversation graph. It may further comprise receiving a request from the client application, and the messaging platform sending a second subset of the plurality of messages rendered in the client application according to the rank. The conversation graph may include a tree data structure having an arrangement of nodes representing the messages of the conversation graph. The method comprises the steps of: the messaging platform calculating an engagement value for each of the plurality of messages based on the plurality of predicted results; or selecting a plurality of branches.

According to one aspect, a system for ranking messages of a conversation graph in a messaging platform using prediction results is configured to generate a conversation graph based on reply structures of messages exchanged in the messaging platform. a conversation graph manager, said conversation graph including a data structure of messages of a conversation; a timeline manager configured to receive, over the network, a request to view a conversation from the client application; An engagement predictor configured to determine a plurality of predicted outcomes for each of a plurality of messages, the plurality of predicted outcomes being a mutual engagement probability and one of a positive engagement probability and a negative engagement probability. and an engagement scorer configured to calculate an engagement value for each of the plurality of messages using the prediction results for each message. The timeline manager is configured to rank the plurality of messages using the engagement value and provide over the network at least a subset of the plurality of messages rendered in the timeline according to the ranking. there is According to further aspects, a non-transitory computer-readable medium storing corresponding methods and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The mutual engagement probability includes a probability value that the messaging platform expects to receive a reply to the message in the conversation graph, and the positive engagement probability indicates that the messaging platform will receive an acknowledgment to the message in the conversation graph. said negative engagement probability comprises a probability value that said messaging platform is expected to receive negative user engagement to said message of said conversation graph. . The predictive models include mutual engagement models, positive engagement models, and negative engagement models. The engagement predictor obtains signals generated by one or more of the client application and the messaging platform and applies the signals to the mutual engagement model, the positive engagement model, and the negative engagement model. configured to input and obtain the mutual engagement probability, the positive engagement probability, and the negative engagement probability, respectively. The system is a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with training data, the trained predictive model being: A predictive model trainer configured to provide to the predictive manager may further comprise. The system may further comprise a candidate message selector configured to select a subset of candidates from a plurality of sources, wherein the plurality of sources are selected from the conversation graph ranked according to the most recently posted. A first source with messages and a second source with said messages of said conversation graph ranked according to a relevance algorithm. The candidate subset includes a number of messages from the first source and a number of messages from the second source, and the plurality of prediction results is determined for each message of the candidate subset. . The data structure of the conversation graph includes a tree data structure having an arrangement of a plurality of nodes representing the messages of the conversation graph, and the timeline manager selects a branch of the tree data structure using the engagement value. is configured to The engagement scorer determines the mutual engagement probability, the positive engagement probability, and the negative engagement probability such that the weighting applied to the mutual engagement probability is higher than the weighting applied to the negative engagement probability. It is configured to apply a weighting to each.

According to one aspect, a non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, communicates with the one or more processors over a network to a messaging platform. selecting a candidate subset of messages from the conversation graph; and determining a plurality of predicted outcomes for each message of the candidate subset using a predictive model, wherein the plurality of predicted outcomes comprises a cross-engagement probability, the cross-engagement probability is associated with the messaging platform. includes a predicted probability value of receiving a reply to each message of the candidate subset; calculating an engagement value for each message of the candidate subset using the prediction result; ranking the plurality of messages using a value; and sending at least a subset of the plurality of messages rendered in a client application according to the ranking. computer readable medium. According to further aspects, corresponding systems and methods for storing corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The candidate subset includes a number of recently posted messages and a number of top ranked messages from the conversation graph. The data structure of the conversation graph includes a tree data structure having an arrangement of a plurality of nodes representing the messages of the conversation graph. The act may comprise selecting a branch of the tree data structure using the engagement value. The operations comprise periodically training the prediction model based on one or more machine learning algorithms input with training data, and providing the trained prediction model to a prediction manager. you can

According to one aspect, a method for ranking conversation graph messages in a messaging platform using prediction results receives, over a network, a conversation view request to retrieve conversation graph messages stored in the messaging platform. and determining, where the messaging platform uses a predictive model to determine a plurality of predicted outcomes for each of a plurality of messages of the conversation graph. The determining step includes obtaining a plurality of signals associated with the prediction model, including data structure-related signals associated with the conversation graph; into the prediction model. The method comprises the steps of: the messaging platform ranking the plurality of messages based on the predicted results; and the messaging platform rendering at least a subset of the plurality of messages in a client application according to the rankings over the network. and transmitting. According to further aspects, a non-transitory computer-readable medium storing a corresponding system and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . the conversation graph includes a tree data structure having a plurality of nodes representing the messages of the conversation graph and edges representing links between the nodes, the tree data structure defining one or more branches of nodes; The data structure related signals include signals representing the number of nodes and the number of branches within the conversation graph. The data structure related signals include the number of replies within a branch, the number of conversations within a branch, and the number of unique authors within a branch or the conversation graph. The plurality of signals includes a signal indicative of whether a user of the client application has restricted creators of messages of the conversation graph. The plurality of signals includes engagement signals representing user engagement data associated with the messages of the conversation graph, including engagement of users following users of the client application in a connection graph. The plurality of signals includes historical aggregated signals including engagement data associated with users of the client application on the messaging platform. The plurality of signals includes content-related signals associated with messages of the conversation graph. At least a subset of the plurality of signals are obtained from one or more data services executing on the messaging platform. The method obtains training data from a client event log in which the messaging platform stores information received from the client application and an injection log in which information from a timeline manager executing on the messaging platform is stored. and said messaging platform training said predictive model based on a machine learning algorithm input with said training data.

According to one aspect, a system for ranking messages of a conversation graph in a messaging platform using prediction results is configured to generate a conversation graph based on reply structures of messages exchanged in the messaging platform. a conversation graph manager, said conversation graph including a data structure of messages of a conversation; a timeline manager configured to receive a conversation viewing request from the client application over the network; and an engagement predictor configured to determine a plurality of predicted outcomes for each of the plurality of messages. The engagement predictor configured to obtain a plurality of signals relevant to the predictive model from one or more data services stored on the messaging platform to determine the plurality of prediction outcomes. configured to input the plurality of signals to the predictive model so as to The system comprises an engagement scorer configured to calculate an engagement value for each of the plurality of messages using the predicted results for each message, the timeline manager using the engagement value ranks the plurality of messages in the network and provides over the network at least a subset of the plurality of messages rendered in the timeline according to the ranking. According to further aspects, a non-transitory computer-readable medium storing corresponding methods and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The plurality of signals includes data structure related signals associated with the conversation graph. The engagement predictor is configured to obtain one or more first signals from a first data service and one or more second signals from a second data service. The plurality of signals includes one or more of health-related signals, engagement signals, social graph signals, history aggregation signals, or content-related signals. The prediction model includes a mutual engagement model, a positive engagement model, and a negative engagement model, and the engagement predictor obtains mutual engagement probability, positive engagement probability, and negative engagement probability, respectively. configured to input signals into the mutual engagement model, the positive engagement model and the negative engagement model. The system is a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with training data, the trained predictive model being: A predictive model trainer configured to provide to the predictive manager may further comprise. The plurality of signals includes signals generated by the client application and signals generated by the messaging platform. The plurality of signals includes similarity signals.

According to one aspect, a non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, communicates with the one or more processors over a network to a messaging platform. selecting a candidate subset of messages from the conversation graph; determining a plurality of predicted outcomes for each message of said candidate subset using a prediction model, obtaining a plurality of signals including data structure related signals associated with said conversation graph; inputting the plurality of signals into the prediction model to determine a plurality of prediction outcomes; and using the prediction outcomes to calculate an engagement value for each message of the candidate subset; ranking the plurality of messages using the engagement value; and sending at least a subset of the plurality of messages rendered in a client application according to the ranking. Temporary computer readable medium. According to further aspects, corresponding systems and methods may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The actions include obtaining training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform; training the predictive model based on a data-inputted machine learning algorithm. The plurality of signals includes one or more of health-related signals, engagement signals, social graph signals, history aggregation signals, or content-related signals.

According to one aspect, a method for ranking conversation graph messages in a messaging platform, wherein the messaging platform classifies conversation graph messages into a plurality of sections based on content quality of said messages. and the plurality of sections comprises a sorting step including a first section and a second section. The first section has messages from the conversation graph determined to be of higher quality than the messages of the second section. The method comprises the steps of: the messaging platform determining a plurality of predicted outcomes for each of the plurality of messages of the conversation graph using a predictive model; ranking the messages of the first section based on prediction results; and the messaging platform sending at least a subset of the messages of the first section rendered in a client application according to the ranking. , provided. According to further aspects, a non-transitory computer-readable medium storing a corresponding system and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The method comprises the messaging platform ranking the messages of the second section based on the plurality of prediction results for the messages of the second section. In some examples, the method comprises the steps of: the messaging platform receiving a request to render additional messages of the conversation graph in the client application; and the messaging platform rendered in the client application according to the rank. and sending at least a subset of said messages of said second section. The classifying step may include receiving a message identifier for a message of the conversation graph and using a classification model to determine that the message has a content quality corresponding to the first section. . The method may comprise assigning a branch of the conversation graph containing the message to the first section. The method comprises calculating an engagement value for each message of the conversation graph using the plurality of prediction results for each message, wherein the messages of the first quality section are are ranked according to the engagement value of said messages in . The prediction results include mutual engagement probabilities, positive engagement probabilities, and negative engagement probabilities. The method comprises the steps of: the messaging platform generating the conversation graph based on a reply structure of messages exchanged on the messaging platform, the conversation graph comprising a tree data structure of messages associated with a conversation; Further steps may be included. The plurality of predictive models includes a mutual engagement model, a positive engagement model, and a negative engagement model, wherein one or more of the mutual engagement model, the positive engagement model, and the negative engagement model are neural networks. including.

According to one aspect, a system for ranking messages in a conversation graph in a messaging platform is a conversation graph manager configured to generate a conversation graph based on reply structures of messages exchanged in the messaging platform. a conversation graph manager, wherein the conversation graph includes a data structure of messages of a conversation; and a timeline manager configured to provide a stream of messages to a user's timeline on a client application over a network. a timeline manager configured to receive conversation view requests from the client application over a network; and to classify the messages of the conversation graph into a plurality of sections based on content quality of the messages. wherein the plurality of sections includes a first section and a second section. The first section has messages from the conversation graph determined to be of higher quality than the messages of the second section. The system includes an engagement predictor configured to determine a plurality of predicted outcomes for each of a plurality of messages of the conversation graph using a predictive model; and using the plurality of predicted outcomes for each message. and an engagement scorer configured to calculate an engagement value for each of the plurality of messages. The timeline manager ranks the messages of the first section using the engagement value of the first section, and at least of the messages of the first section rendered in the timeline according to the rank. A subset is configured to be provided over the network. According to further aspects, a non-transitory computer-readable medium storing corresponding methods and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The engagement predictor is configured to obtain a plurality of signals relevant to the predictive model from one or more data services stored on the messaging platform, and extracts the plurality of prediction results. configured to input the plurality of signals to the predictive model to determine. The plurality of signals includes data structure related signals associated with the conversation graph. The timeline manager is configured to separately rank the messages of the second section using the engagement value of the second section. The content quality classifier receives a message identifier for a message of the conversation graph and uses a classification model to determine that the message has a content quality corresponding to the first quality section. It is configured. The plurality of prediction results includes mutual engagement probabilities, wherein the mutual engagement probabilities include probability values that the messaging platform is expected to receive a reply to each message of the conversation graph. The system is a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with training data, the trained predictive model being: A predictive model trainer configured to provide to the predictive manager may further comprise.

According to one aspect, a non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, cause the one or more processors to use a classification model to: , classifying messages of a conversation graph into a plurality of sections based on content quality of said messages, said plurality of sections including a first section and a second section, said first section said for each of the plurality of messages of the conversation graph using one or more predictive models, having messages from the conversation graph determined to be of higher quality than the messages of the second section; wherein the plurality of prediction results includes a mutual engagement probability, a positive engagement probability, and a negative engagement probability; and ranking the messages in the second section based on the plurality of prediction results for the messages in the second section. and transmitting at least a subset of said messages of said first quality section rendered in a client application according to said rank. According to further aspects, corresponding systems and methods may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The actions include obtaining training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform; and training the predictive model based on a machine learning algorithm input by . The act comprises receiving a message identifier for a first message of the conversation graph and using the classification model to determine that the first message has a content quality corresponding to the first section. and receiving a message identifier of a second message of the conversation graph, and using the classification model to determine that the second message has a content quality corresponding to the second section; may be provided. The act comprises receiving a request to render additional messages of the conversation graph in the client application and sending at least a subset of the messages of the second section to be rendered in the client application according to the rank. , may be provided. These and other features are explained further in the detailed description.

1 illustrates a messaging system for ranking messages in a conversation graph using predictive results from predictive models, according to one aspect. FIG. 1 illustrates a candidate message selector of a messaging system, according to one aspect; FIG. FIG. 4 illustrates an example of signals used to determine a prediction result, according to one aspect. 1 illustrates a prediction manager of a messaging system, according to one aspect; FIG. FIG. 4 illustrates an example of generating training data for a predictive model, according to one aspect. 1 illustrates an example of a two-level ranking mechanism including a content quality classifier, according to one aspect; FIG. FIG. 3 illustrates an example predictive model as a neural network, according to one aspect. 4 is a flowchart illustrating exemplary operation of a messaging system, according to one aspect. 4 is a flowchart illustrating exemplary operation of a messaging system, according to one aspect. 4 is a flowchart illustrating exemplary operation of a messaging system, according to one aspect.

1A-1E illustrate a messaging system 100 for ranking messages in a conversation graph 126 using predictive results 118 from predictive models 112 according to one aspect. Messaging system 100 , in one aspect, comprises a messaging platform 104 executable by server computer 102 and a client application 154 executable by computing device 152 . Client application 154 communicates with messaging platform 104 over network 150 to send (and receive) messages to (and from) other users of messaging platform 104 .

Client application 154 may be a social media messaging application. Social media messaging applications allow users to post and interact with messages. In some examples, client application 154 is a native application that runs on the operating system of computing device 152, or in conjunction with a browser-based application of computing device 152, to server computer 102 (or other It may be a web-based application running on a server of the Computing device 152 may be connected via network 150 using any type of network connection and/or application programming interface (API) to enable client application 154 and messaging platform 104 to communicate with each other. to access messaging platform 104 .

Computing device 152 may be a mobile computing device (eg, smart phone, PDA, tablet, or laptop computer) or a non-mobile computing device (eg, desktop computing device). The computing device 152 has, for example, a mobile network interface through which the computing device 152 can communicate with a cellular network, a Wi-Fi based station with which the computing device 152 can communicate; A Wi-Fi network interface, a Bluetooth network interface with which computing device 152 can communicate with other Bluetooth® devices, and/or access by computing device 152 to network 150 It also includes various network interface circuitry, such as an Ethernet connection or other wired connection, that allows for

Server computer 102 may be a single computing device or a representation of two or more distributed computing devices communicatively coupled to share workload and resources. The server computer 102 stores one or more processors and executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations described herein. , and a non-transitory computer-readable medium.

Messaging platform 104 is a computing platform for communication (eg, real-time communication) between user devices (one of which is shown as computing device 152). The messaging platform 104 may store millions of accounts 141 (eg, pseudonym accounts, novelty accounts, etc.) of individuals, businesses, and/or entities. One or more users of each account 141 may use messaging platform 104 to send messages to other accounts 141 internal and/or external to messaging platform 104 . In some examples, messaging platform 104 enables users to communicate in "real time," e.g., converse with other users with minimal delay, and to communicate with one or more other users during concurrent sessions. It may be possible to have a conversation. In other words, messaging platform 104 may allow users to broadcast messages to one or more users within a reasonable time frame (eg, less than two seconds) to conduct live conversations between users. may display messages to other users of In some examples, recipients of a message may have a predetermined graphical relationship in the connection graph 134 with the account of the user who broadcast the message.

The connection graph 134 indicates which accounts 141 within the messaging platform 104 are associated with (e.g., followed, friends with, subscribed to, etc.) a particular account 141 and therefore messages from a particular account 141 . Contains a data structure that indicates whether you are subscribed to receive For example, connection graph 134 may link a first account with a second account. This indicates that the first account is related to the second account. The second account user may view messages posted on messaging platform 104 by the first account user (and/or the first account user may view messages posted by the second account user). may view messages posted on the platform 104). Relationships may include unidirectional (eg, follower/follower) and/or bidirectional (eg, friendship). Messages can be any of a variety of lengths, which may be limited by a particular messaging system or protocol.

In some examples, a user interested in viewing messages written by a particular user may choose to follow the particular user. A first user can follow a second user by identifying the second user as a user that the first user wishes to follow. After the first user indicates that they want to follow the second user, the connection graph 134 is updated to reflect the relevance and the first user receives messages written by the second user. is provided. A user can choose to follow multiple users. Users can also respond to messages, thereby allowing them to converse with each other. In addition to this, users can engage with messages, such as sharing messages with their followers, or favouriting (or "liking") messages whose engagement is shared with their followers. good.

Messages exchanged on messaging platform 104 are stored in message repository 138 . Message repository 138 may comprise one or more tables that store records. In some examples, each record corresponds to a separately stored message. For example, a record may include a message identifier for a message posted to messaging platform 104, an author identifier identifying the author of the message (e.g., @tristan), message content (e.g., text, image, video, and/or web content URL), one or more participant account identifiers identified in the body of the message, and/or reply information identifying the parent message to which the message is replied (if the message is a reply to a message) can be identified.

The messaging platform 104 comprises a conversation graph manager 136 that generates a conversation graph 126 and a timeline manager 142 that injects a timeline 156 of messages into the client application 154 . Messaging platform 104 comprises candidate message selector 108 that selects candidate subset 133 of messages from conversation graph 126 . The messaging platform 104 comprises a prediction manager 110 that obtains signals 106 related to predictions and inputs the signals 106 to one or more prediction models 112 to determine prediction results 118 . In some examples, prediction manager 110 calculates engagement value 116 based on prediction results 118 . The timeline manager 142 ranks (and selects) messages in the conversation graph 126 using the engagement values 116 and provides a ranked list 158 of messages to the client application 154 . Messaging platform 104 includes predictive model trainer 140 that obtains predictive data 148 and trains predictive model 112 using one or more machine learning algorithms 149 input by training data 148 .

More specifically, conversation graph manager 136 generates (and updates) one or more conversation graphs 126 as messages are exchanged on messaging platform 104 . In some examples, conversation graph 126 is stored in a data storage device associated with messaging platform 104 . In some examples, conversation graph 126 is stored at timeline manager 142 . The messaging platform 104 may store multiple conversation graphs 126 (eg, hundreds, thousands, or millions of conversation graphs 126). Each conversation graph 126 may represent the structure of replies to messages that were not original replies (eg, root messages). For example, each time a user creates and posts a message to messaging platform 104 that is not an original reply, a potential new conversation can be started. Others can then reply to the original or "root" message, generating their own reply branches. Over time, when the number of replies to messages that are not original replies (and/or replies to replies to messages that are not original replies) exceeds a threshold level, conversation graph manager 136 adds conversation identifiers to conversation graph 126. A conversation identifier may be assigned and may uniquely identify the conversation graph 126 . In some examples, conversation graph manager 136 may assign a conversation identifier to each message that has a reply. For example, if a messaging platform has message A and someone responds to message A with message B, message A is assigned a conversation identifier that can be used to identify the conversation. This yields conversation graph 126, as described in detail below. In some instances, a conversation exists if there is a reply to a message.

Conversation graph 126 may be a hierarchical data structure that represents the messages in a conversation. In some examples, conversation graph 126 includes a non-linear or linear data structure. In some examples, conversation graph 126 includes a tree data structure. Conversation graph 126 may comprise nodes 128 (or vertices) representing messages and edges 130 (or arcs) representing links between nodes 128 . Conversation graph 126 may store a message identifier for each message at each node 128 . In some examples, conversation graph 126 stores the user identifier of the creator of each message at each node 128 . Conversation graph 126 may define one or more branches 132 of nodes 128 . In some examples, branch 132 is a portion (eg, subtree) of conversation graph 126 that includes one or more nodes 128 . In some examples, branch 132 may be two or more nodes 128 connected by edge 130 . Here, one of the nodes 128 is a leaf node. In some examples, branch 132 may be a message connected in a single line (e.g., a leaf message, a first parent message connected to the leaf message, a first parent message connected to the first parent message). (until the parent message does not have another parent message, such as a second parent message that has another parent message).

The term “node” may be referred to as a message within conversation graph 126 and the term “message” may be referred to as node 128 whose message is included as part of conversation graph 126 . A particular node 128 may be linked to another node 128 via an edge 130, the direction of edge 130 identifying the parent message. Node 128 may represent a root message, a message in reply to a root message, a message in reply to a message in reply to a root message, and so on.

The conversation graph manager 136 may generate the conversation graph 126 based on the message's reply structure. A reply structure may be identified based on metadata associated with each message and/or reply information identified from within the message content. In some examples, reply structures are identified based on metadata associated with each message received from client application 154 to compose the message. For example, the user may click on a reply link displayed below a message displayed on the user interface of client application 154 . Client application 154 may then display a message composition box for drafting a reply message. Client application 154 may publish metadata that includes the reply relationship with the reply message (eg, the message identifier of the parent message). In some examples, the reply relationship is explicitly defined by the user in the message content (eg, identifying user account 141 (eg, @tristan) in the body of the message). In this example, the reply structure may be identified by identifying one or more account identifiers and/or message identifiers mentioned within the body of the message.

Timeline manager 142 may transmit digital information over network 150 to enable client application 154 to render and display social content timeline 156 in the user interface of client application 154 . Timeline 156 includes a stream of messages (eg, message A, message B, message C). In some examples, the stream of messages is organized in reverse chronological order. In some examples, the stream of messages is organized in chronological order. In some examples, timeline 156 is a timeline of social content specific to a particular user. In some examples, timeline 156 includes streams curated (eg, generated and assembled) by messaging platform 104 . In some examples, timeline 156 includes a list of messages resulting from searches on messaging platform 104 . In some examples, the timeline 156 is a stream of messages posted by the user from the account 141 associated with the user's account 141 of the client application 154 (e.g., messages the user has chosen to follow on the messaging platform 104). stream of messages from account 141). In some examples, the stream of messages includes promoted or re-shared messages.

When viewing messages in timeline 156, a user may select one of the messages from timeline 156 (eg, message B). This may cause client application 154 to generate and send conversation view request 121 for messaging over network 150 . In some examples, the selected message (eg, message B) may be referred to as a content message or focal message, which may serve as an entry point or point of reference within conversation graph 126 . View conversation request 121 may be a request to retrieve a message from conversation graph 126 . In some examples, view conversation request 121 includes the message identifier of the selected message and the user identifier of the user of client application 154 . In some examples, the conversation view request 121 includes the time of the request, the device the user has turned on, the operating system (OS), and/or other metadata associated with the request.

In response to conversation view requests 121, timeline manager 142 may control prediction manager 110 to generate prediction results 118 for each message (or a subset thereof) of conversation graph 126, where prediction results 118 An engagement value 116 may be calculated for each message based on. For example, prediction manager 110 comprises engagement predictor 125 and engagement scorer 114 . In response to conversation view request 121, engagement predictor 125 obtains prediction-related signal 106 and uses that signal 106 to determine prediction outcome 118 for each message (or subset of messages) in conversation graph 126. It may be input to predictive model 112 . In some examples, prediction results 118 include cross-engagement probabilities 124 . In some examples, predicted results 118 include mutual engagement probabilities 124 and one or more of positive engagement probabilities 120 and negative engagement probabilities 122 . In some examples, predicted results 118 include mutual engagement probability 124 , positive engagement probability 120 , and negative engagement probability 122 .

Engagement scorer 114 may use prediction results 118 (eg, combine prediction results 118 to produce engagement value 116) to calculate engagement value 116 for each message. The engagement value 116 is used by the timeline manager 142 to rank messages in the conversation graph. Timeline manager 142 may provide messages (or a subset of messages in conversation graph 126 ) as ranked lists 158 in user's timeline 156 over network 150 . Ranked list 158 (eg, message 1, message 2, message 3) includes some or all of the messages of conversation graph 126 ranked according to engagement value 116 determined by prediction results 118. OK.

In some examples, candidate message selector 108 selects a portion of messages in conversation graph 126 to be parsed by prediction manager 110 . In other words, candidate message selector 108 selects candidate messages from a number of messages contained in conversation graph 126 . Here, candidate messages are those messages that are amenable to predictive analysis by the predictive manager 110 . In some examples, the conversation graph 126 is relatively large, and if the number of nodes 128 included in the conversation graph 126 exceeds a threshold level (eg, more than 3000 or 4000 messages), the candidate message selector 108 uses the prediction manager A subset of nodes 128 (rather than all of nodes 128) may be selected for further analysis by 110. FIG. In other words, when the number of nodes 128 exceeds a threshold level, candidate message selector 108 identifies a subset of nodes 128, and the identification of those nodes 128 is provided to prediction manager 110 to make a prediction. . This may reduce (eg, significantly reduce) technical resource consumption and allow messaging platform 104 to deliver more quality responses. For example, if conversation graph 126 has 30,000 responses (or nodes 128), using the techniques described above, messaging platform 104 selects the top X volume messages (eg, X may be 3000). , collect signals for them, score them, and select the top Y amount of messages (eg, Y may be 10) to display to the user. This allows the amount of computational resources to be reduced (eg, significantly reduced) without affecting the quality of the response delivered to the user.

For example, referring to FIG. 1B, candidate message selector 108 may select subset 133 of candidates from conversation graph 126 in response to a number of messages in conversation graph 126 exceeding a threshold level. Here, a prediction result 118 is determined for each message of the candidate subset 133 . In some examples, candidate subset 133 includes a plurality of messages that are less than the total number of messages in conversation graph 126 . In some examples, candidate message selector 108 selects a subset of candidates 133 from one or more candidate message sources 131 . In some examples, candidate message selector 108 may merge candidates for conversation viewing from multiple sources. For example, candidate message selector 108 selects a first set of messages contained in conversation graph 126 from first source 131-1 as a second set of messages contained in conversation graph 126 from second source 131-2. select a set of messages. Here, candidate subset 133 includes the first set of messages and the second set of messages.

In some examples, first source 131-1 includes messages of conversation graph 126 ranked according to most recently posted. In some examples, candidate message selector 108 selects a certain number (n) of the most recent (eg, chronological order) messages in conversation graph 126 from first source 131-1. In some examples, the number (n) is in the range of 2000-4500. In some examples, the number (n) is in the range of 2500-3500. In some examples, second source 131-2 includes messages of conversation graph 126 ranked according to a relevance algorithm (eg, a heuristic algorithm). In some examples, the relevance ranking for the second source 131-2 is based on whether the message in the conversation graph 126 is from a user account 141 linked to the user account 141 of the user of the client application 154. . In some examples, the relevance algorithm may be a message, such as the amount of engagement received, the number of likes, comments, and/or signals such as re-shares, whether the message is a photo, video, and/or link. Signals representing metadata and/or health-related metadata such as whether the message is poisonous, workplace warnings (NSFW), or whether the creator of the message has been recently reported. In some examples, candidate message selector 108 selects a certain number (p) of top ranked messages in conversation graph 126 from second source 131-2. In some examples, the number (p) is in the range of 200-1500. In some examples, the number (p) is in the range of 500-1000. In some examples, the number (p) is less than the number (n). The candidate subset 133 may include a number (n) of messages from the first source 131-1 and a number (p) of messages from the second source 131-2. Here, a prediction result 118 is determined for each message of the candidate subset 133 .

At least some of the nodes 128 (or all of the nodes 128) in the conversation graph 126 (or those identified by the candidate message selector 108), the engagement predictor 125 uses the predictive model 112 to predict a positive engagement probability 120, negative engagement probability 122, and/or mutual engagement probability may be predicted. For example, in response to a conversation view request 121, the prediction manager 110 controls the engagement predictor 125 to obtain the signal 106, apply the signal 106 to the prediction model 112, and select nodes of the conversation graph 126. A prediction result 118 may be determined for 128 .

Prediction manager 110 may obtain signals 106 from one or more data services 165 . Data service 165 may be a component on messaging platform 104 that computes or derives data obtained by messaging platform 104 and/or client application 154 . In some examples, prediction manager 110 may communicate with data service 165 through a server communication interface. In some examples, prediction manager 110 may obtain at least some of signals 106 from data service 165 via one or more APIs. In some examples, in response to view conversation request 121, prediction manager 110 sends a thrift call or remote procedure call (RPC) to data service 165 and then sends at least some of signals 106 to the associated data service. 165 may be received. In some examples, prediction manager 110 may send a representable state transfer (REST) request to data service 165 and then receive at least some of signals 106 from associated data service 165 . In some examples, prediction manager 110 communicates with data service 165 via GraphQL requests. In some examples, prediction manager 110 obtains some of signals 106 from other components of messaging platform 104 including conversation graph manager 136 and/or timeline manager 142 .

Signals 106 may include signals generated by messaging platform 104 and/or generated by client application 154 related to predicting user outcomes for displaying messages to client application 154 . For example, signals 106 may include signals generated by client application 154 based on user interactions with client application 154 . Signals generated by client application 154 may be sent to messaging platform 104 for storage on messaging platform 104 . Signals generated by client application 154 may be engagement information, such as positive user engagement with the message (e.g., favorite, like, reshare) and/or negative user engagement with the message (e.g., report of swear content). In some examples, signals 106 may include signals generated by messaging platform 104 . In some examples, the signals generated by the messaging platform 104 may be data generated from the user's connection graph 134, data generated from the conversation graph 126, data generated from user behavior on the platform (e.g., if the user and/or data generated from the content of the message, such as the results of semantic analysis to predict user sentiment or topical analysis to determine the topic of one or more messages. May contain signals.

As shown in FIG. 1C, signals 106 include a data structure related signal 101 associated with conversation graph 126, a health related signal 103 providing messages from conversation graph 126 to the user of client application 154, a conversation Engagement signal 105 related to user engagement with messages in graph 126; social graph signal 107 related to data from user's connection graph 134; historical aggregation signal 109 related to data aggregated by messaging platform 104; Content-related signals 111 related to the content of the message and/or how similar the message is to other messages that the user has favorited or liked and/or how the user has engaged with the message. may include a similarity signal 113 representing how similar the user is to the user. However, signal 106 may include any type of categorical or granular signal relevant to predicting user outcomes from displayed messages.

Data structure related signals 101 may include signals related to data from conversation graph 126 . In some examples, the data structure related signals 101 include the number of nodes 128, the number of edges 130, the number of branches 132, the length or size of each branch 132, the number of parent nodes, the number of child nodes, the number of leaf nodes. , the height of the conversation graph 126 (e.g., the length of the longest path to a leaf node), and/or the depth of the nodes (e.g., the depth of a node is the length of the path to the root node) may include a signal representing In some examples, data structure related signals 101 include one or more signals representing the number of unique authors in conversation graph 126 or subsets of conversation graph 126 , such as branches 132 . In some examples, data structure-related signals 101 include signals representing locations of messages having some type of data (eg, images, videos, links to videos, etc.) within conversation graph 126 . In some examples, for a particular message in conversation graph 126, data structure related signal 101 indicates whether the message is a child node, whether the message is a parent node, whether the message is a leaf node, whether the conversation Signals representing the location of the message within graph 126, the location of branch 132 containing the message, the size of branch 132 containing the message, and the depth of the message within conversation graph 126 may be included.

Data structure related signals 101 may include branch context features. In some examples, data structure related signals 101 may include the number of replies within branch 132, the number of conversations within branch 132, the number of conversations within branch 132 between the user of client application 154 and the originator of the root message. number, the number of conversations within branch 132 between users of application 154 and users mentioned in new messages, and/or the number of conversations between a particular node (e.g., a focal message) and a leaf node. contains a signal that represents In some examples, for branch context features, a conversation may be defined as a round trip between two or more users. In some examples, a conversation may be defined as a message posted by user A, a reply posted by user B, and then a reply posted by user A.

In some examples, the conversation graph manager 136 may receive the conversation identifier from the prediction manager 110, derive or determine the data structure related signal 101 from the conversation graph 126 according to the conversation identifier, and transfer the data structure related signal 101 to the messaging platform. It may be stored in data storage at 104 . For example, in response to conversation view request 121, prediction manager 110 controls conversation graph manager 136 to derive or determine data structure related signals 101, which are then used by prediction models 112 to determine prediction results 118. As such, data structure related signals 101 may be received from conversation graph manager 136 . In some examples, prediction manager 110 may derive or determine data structure-related signals 101 from conversation graph 126 . In some examples, in response to view conversation request 121, prediction manager 110 may send a conversation identifier to conversation graph manager 136, which then derives or determines data structure related signals 101 from conversation graph 126. Conversation graph 126 may be received as follows.

Health-related signals 103 may include signals representing the health of presenting messages of conversation graph 126 to a user of client application 154 . In some examples, health-related signal 103 may include a signal representing whether a user of client application 154 has restricted (eg, blocked, muted, etc.) authors of messages in past conversation graph 126 . Health-related signals 103 may be stored in data storage in messaging platform 104 . In some examples, prediction manager 110 may send a request to data service 165 (eg, health data service) to obtain health-related signal 103 . Here, the request may include the message identifier of the message of conversation graph 126 and/or the user identifier of the user of client application 154 .

Engagement signals 105 may represent user engagement data associated with messages in conversation graph 126 . In some examples, the engagement signal 105 is the number of engagements for a message in the conversation graph 126 (e.g., the number of times the message has been favorited or liked, the number of replies to the message, the number of times the message has been re-shared). ). In some examples, the engagement signal 105 is the engagement of the user following the user of the client application 154 in the user's connection graph 134 (e.g., the message follows the user of the client application 154 in the user's connection graph 134). has one or more engagements provided by a user who is In some examples, prediction manager 110 obtains engagement signal 105 from data service 168 that stores engagement data. In some examples, prediction manager 110 may send a request that may include a message identifier for conversation graph 126 . Prediction manager 110 may receive engagement signal 105 from data service 165 .

Social graph signals 107 may include signals representing information from connection graph 134 . In some examples, the social graph signal 107 indicates the number of times a user of the client application 154 has favorited or liked the message's creator's message over time, the user's link to the message's creator in the connection graph 134 . and/or a signal representing the number of times the user has re-shared or replied to the message's author's message over a period of time. In some examples, prediction manager 110 obtains social graph signals 107 from data service 168 that stores social graph signals. In some examples, prediction manager 110 may send a request that may include the user identifier of the user of client application 154 . Prediction manager 110 may receive social graph signals 107 from data service 165 .

Historical aggregated signals 109 may include signals representative of user behavior on messaging platform 104 . In some examples, history aggregation signal 109 may be the number of times the user of client application 154 favorited a message on messaging platform 104 over a period of time, the number of times the user of client application 154 favorited a message on messaging platform 104 over a period of time, A signal representing the number of times it has been re-shared and/or the number of times a user of client application 154 has replied to a message on messaging platform 104 over a period of time may be included. The period may be the previous day, the last month, the last year, or the like. In some examples, history aggregation signal 109 counts the number of times a user of client application 154 favorited, liked, re-shared, and/or replied to a message containing an image or video. may include a signal representing

In some examples, the history aggregation signal 109 indicates that the user of the client application 154 has favorited, liked, re-shared, and/or messages from accounts 141 linked to the user in the connection graph 134 . or the number of times the message was replied to, and/or the user favorited, liked, re-shared, and/or messages from accounts 141 not linked to the user in connection graph 134 A signal representing the number of times the message has been replied to may be included. In some examples, prediction manager 110 obtains historical aggregated signal 109 from data storage in messaging platform 104 . In some examples, prediction manager 110 sends a request to data service 165 to obtain historical aggregated signal 109 . In some examples, the request includes the user identifier of the user of client application 154 . In some examples, historical aggregated signal 109 includes batch aggregated information and real-time aggregated information. Batch aggregate information may include a relatively long history (eg, over 50 days). In some examples, batch aggregate information may not include interactions from the previous day (or the last few days). Real-time aggregate information may include relatively recent interaction history (eg, within the last 30 minutes, etc.).

Content-related signals 111 may include signals representing one or more aspects of the content of messages in conversation graph 126 . In some examples, content-related signals 111 may include signals that indicate the length of the message and/or whether the content includes text, video, or images. In some examples, prediction manager 110 obtains content-related signals 111 from data storage in messaging platform 104 . In some examples, prediction manager 110 sends a request to data service 165 to obtain content-related signal 111 . In some examples, the request includes the message identifier of the message in conversation graph 126 .

Similarity signals 113 may include one or more signals representing how similar a message is to other messages that a user has favorited or liked. For example, similarity signal 113 may represent the level of similarity between a particular message and one or more messages that the user has favorited or liked. If the level of similarity is relatively high, the similarity signal 113 provides an indication of potential positive engagement. In some examples, similarity signal 113 may include one or more signals representing how similar the user is to other users engaged with the message. For example, determining that a user's user profile is relatively similar to a user profile that has engaged with a message may provide an indication of potential positive engagement. In some examples, prediction manager 110 may obtain affinity signal 113 from data storage in messaging platform 104 . In some examples, prediction manager 110 sends a request to data service 165 to obtain affinity signal 113 . In some examples, the request may include the message identifier and/or the user's user identifier.

In some instances, there are technical difficulties or hurdles to obtain at least some of the signals 106 used for prediction (e.g., the relationship between viewers and creators, especially if the message is spread). for signals associated with ). Popular messages may have a relatively high volume of responses (eg, over 80,000 in some cases). This means that many users can try and view popular messages at the same time. For each viewer, messaging platform 104 may obtain its relationship with all authors who have replied to the popular message. Using the techniques described above for candidate message selector 108, messaging platform 104 may be able to filter a total number of messages from 80,000 to 4,000. This may also mean that there may be 4,000 viewer-author pairs to obtain a relationship signal.

Also, in some instances, a viewer-author relationship may not exist because the viewer does not follow the author. To handle these kinds of situations, instead of querying by viewers and creators as keys to data service 166, messaging platform 104 queries by viewer identifiers and their relationships with all other creators. can be obtained at once. The messaging platform 104 can then determine overlap with the creator of the creator's reply, if any, and maintain associated signals. This reduces calls through the network to a relatively large extent because the prediction manager 110 can generate and send one call instead of 4,000 calls per viewer.

Another technical difficulty can exist with message level signals. For example, in a large conversation, messaging platform 104 may query other data services 166 with 4,000 queries per viewer. This can create a "hotkey" problem where the data service 166 receives too many queries for the same message identifier. To overcome the difficulties identified above, messaging platform 104 may employ in-memory caching. The service caches the feature in memory if the underlying data service 166 indicates a hotkey. For example, a message T is spread with responses R1, R2...R4000 and the message feature is the number of characters in the message. Then 1000 users send requests for the same message at the same time (or near the same time). If the data service 166 indicates a hotkey, the messaging platform 104 can store the character values for R1, R2 . , it is possible to just use them.

Predictive model 112 is a predictive model trained by one or more machine learning algorithms 149 input with training data 148 . Machine learning algorithms 149 may include one or more of Markov models, logistic regression, decision tree analysis, random forest analysis, neural nets, and combinations thereof. In general, machine learning is the field in which a computer learns to perform a class of tasks using feedback generated from the experience or data that the machine learning process acquires while performing those tasks on the computer. Supervised machine learning allows a computer to learn one or more roles or functions to map between exemplary inputs and desired outputs in a manner predetermined by an operator or program. be. The labeled data points can then be used in training a computer. Unsupervised machine learning can involve using unlabeled data. The computer can then identify implicit relationships in the data, for example, by reducing the dimensionality of the data set.

As shown in FIG. 1D, predictive models 112 may include positive engagement models 115, negative engagement models 117, and mutual engagement models 119. As shown in FIG. Positive engagement model 115 is configured to calculate a positive engagement probability 120, negative engagement model 117 is configured to calculate a negative engagement probability 122, and mutual engagement model 119 is configured to: It is configured to calculate mutual engagement probability 124 . For example, in response to conversation view request 121 (eg, user selects message B), engagement predictor 125 obtains signal 106 and assigns signal 106 (which also includes user identifier and message identifier) to a positive engagement model 115, negative engagement model 117, and/or mutual engagement model 119 to determine positive engagement probability 120, negative engagement probability 122, and/or mutual engagement probability 124, respectively.

A positive engagement probability 120 indicates a probability value that a user is expected to positively view or engage with the message. In some examples, the probability value for positive engagement probability 120 is a number (x) between the first value and the second value. Here, the first value represents a probability of 0 that the user is predicted to view or engage with the message positively, and the second value represents the probability that the user will view the message positively. Or represents a 100% chance of being predicted to engage. In some examples, the probability value for positive engagement probability 120 is a positive number. In some examples, the first value is 0 and the second value is 1. However, the first value and the second value may define any kind of range (eg, 0-1, 0-50, 0-100, etc.). In other words, the positive engagement probability 120 indicates the level of likelihood that a user is expected to favorite, like, or share the message.

Negative engagement probability 122 indicates the probability value that a user is expected to negatively view or engage with the message. In some examples, the probability value for negative engagement probability 122 is a number (y) between the first value and the second value. Here, the first value represents a probability of 0 that the user is predicted to view or engage with the message negatively, and the second value represents the probability that the user will view the message negatively. Or represents a 100% chance of being predicted to engage. In some examples, the probability value for negative engagement probability 122 is a negative number. In some examples, the first value is 0 and the second value is negative one. However, the first and second values may define any kind of range (eg, 0 to -1, 0 to -50, 0 to -100, etc.). In other words, negative engagement probability 122 indicates the level of likelihood that a user will block the message creator, unfollow the message creator, and/or report the message as abusive.

Mutual engagement probability 124 indicates a probability value that the user is expected to continue to develop conversation graph 126 . In some examples, the probability value for mutual engagement probability 124 is a number (z) between the first value and the second value. Here, the first value represents a probability of 0 that the user is expected to continue to develop the conversation graph 126 and the second value is 100% that the user is expected to continue to develop the conversation graph 126. represents the possibility of In some examples, the probability value for mutual engagement probability 124 is a positive number. In some examples, the first value is 0 and the second value is 1. However, the first value and the second value may define any kind of range (eg, 0-1, 0-50, 0-100, etc.). In some examples, mutual engagement probability 124 indicates the level of likelihood that users are expected to further develop conversation graph 126 by replying to messages.

For a particular candidate node 128, engagement predictor 125 uses positive engagement model 115 to determine that the message has a certain probability of receiving positive engagement by the user, and negative engagement model 117 to , may determine that the message has a certain probability of receiving negative engagement by the user, and using the mutual engagement model 119, determine that the message has a certain probability of receiving mutual engagement by the user. In some examples, for each candidate node 128, engagement predictor 125 determines all three prediction outcomes 118. In some examples, for each node 128 , engagement predictor 125 predicts mutual engagement probability 124 and one or more of positive engagement probability 120 and negative engagement probability 122 .

Engagement scorer 114 uses prediction results 118 to calculate engagement values 116 for messages in conversation graph 126 . Engagement value 116 may provide an overall engagement value for each node 128 (eg, indicating a level of relevance for the user of client application 154). This encourages healthier conversations on messaging platform 104 . For example, for a particular candidate node 128, engagement scorer 114 combines positive engagement probability 120, negative engagement probability 122, and cross-engagement probability 124 to select the most relevant node 128 for the user. An engagement value 116 may be provided that may be used. For example, engagement scorer 114 may combine the values of predicted results 118 to determine engagement value 116 for a particular message. If the probability value of negative engagement probability 122 is relatively high (eg, has a larger negative value), this value may offset the positive values of positive engagement probability 120 and mutual engagement probability 124 . In a simple example, if the positive engagement probability 120 is +10, the negative engagement probability 122 is -10, and the mutual engagement probability 124 is +10, then the engagement value 116 for the message is +10.

In some examples, the engagement scorer 114 may apply weightings to the predicted results 118 and then determine the engagement value 116 based on the weighted positive engagement probabilities 120, negative engagement probabilities 122, and mutual engagement probabilities 124. can be calculated. In some examples, the weighting applied to mutual engagement probability 124 is greater than the weighting applied to negative engagement probability 122 .

The engagement value 116 is used to select relevant messages or branches of messages within the conversation graph 126 to be rendered to the user. For example, timeline manager 142 receives engagement values 116 from prediction manager 110 and uses engagement values 116 to rank messages in conversation graph 126 (eg, highest to lowest). Timeline manager 142 may provide at least a subset of messages of conversation graph 126 rendered in timeline 156 according to rank over network 150 . In some examples, timeline manager 142 only provides a subset of the messages of conversation graph 126 rendered in timeline 156 . Here, the subset contains the higher ranked messages of conversation graph 126 . Timeline manager 142 may then receive a request from client application 154 for additional messages in conversation graph 126 (eg, select a user affordance to view more messages in conversation graph 126). Timeline manager 142 may select the next group of messages from conversation graph 126 to be sent to client application 154 . In this manner, messaging system 100 may collapse portions of conversation graph 126 that are less likely to provide positive engagement, but present those messages when requested by a user.

In some examples, timeline manager 142 selects one or more branches (or a subset of branches 132) of conversation graph 126 to be rendered in the timeline using engagement value 116. FIG. For example, branch 132 includes one or more nodes with high engagement values 116 (or engagement values 116 above a threshold level). The timeline manager 142 provides the user with more context about the conversation, despite the fact that the branch 132 may contain one or more nodes 128 with low engagement values 116 (or engagement values 116 below the threshold level). The entire branch 132 may be selected to be rendered as part of the message delivered to the client application 154 to provide the . In some examples, a particular branch 132 is associated with a total engagement value, which may be an average of engagement values 116 for nodes 128 within the particular branch 132 . The timeline manager 142 may then rank the branches 132 according to their total engagement value.

In some examples, timeline manager 142 selects nodes 128 with high engagement values (or engagement values 116 above a threshold level) for inclusion in the set of messages provided to client application 154 . In some examples, the timeline manager 142 ranks the selected branches 132 and/or nodes 128 according to engagement value 116 from highest to lowest (e.g., here the branch 132 with the highest engagement value 116 or node 128 is presented to the user first). Because messaging platform 104 incorporates (or predicts) mutual engagement probabilities 124 within its scoring algorithm, messaging platform 104 encourages more conversations on messaging platform 104 .

In some examples, ranked list 158 represents a subset of messages in conversation graph 126 determined to be relevant to the user. For example, some messages in conversation graph 126 may be relevant to a first user, while other messages in conversation graph 126 may be relevant to a second user. In contrast, some conventional approaches use vote-based mechanisms that can provide identical views for each. More specifically, the engagement predictor 125 may obtain signals 106 (e.g., engagement history, connection graph data, etc.) associated with the first user, and for each message in the conversation graph 126, then the engagement value 116. Predicted results 118 used to compute may be obtained. The timeline manager 142 may receive the engagement value 116 from the prediction manager 110 and then apply the engagement value 116 (which may increase the likelihood that the user will continue the conversation (eg, by incorporating mutual engagement probability 124)). may be used to rank messages in conversation graph 126 .

However, for a second user, the engagement predictor 125 may obtain the signal 106 associated with the second user and then predictive results tailored to the second user used to calculate the engagement value 116. 118 may be obtained. Timeline manager 142 may then rank messages in conversation graph 126 using engagement value 116 . Thus, the ranked list 158 displayed in the client application 154 for the second user is different from the messages in the ranked list 158 displayed in the client application 154 for the first user. can be different.

Messaging platform 104 includes a predictive model trainer 140 that trains predictive models 112 (eg, positive engagement model 115, negative engagement model 117, and mutual engagement model 119). Messaging platform 104 provides trained predictive model 112 to engagement predictor 125 so that engagement predictor 125 can determine predicted outcome 118 when determining conversation viewing. In some examples, predictive model trainer 140 is configured to run periodically (eg, daily, weekly, monthly) to retrain (and thereby update) predictive model 112 . In some examples, predictive model trainer 140 operates in offline mode to train predictive model 112 using training data 148, and is then employed in online mode when predictive manager 110 is active. As such, the prediction model 112 is sent to the prediction manager 110 . In some examples, predictive model trainer 140 sends the weightings and biases of positive engagement model 115 , negative engagement model 117 , and cross-engagement model 119 to prediction manager 110 .

In some examples, predictive model trainer 140 executes on server computer 102 . In some examples, predictive model trainer 140 executes on a computing device separate from server computer 102 . Here, predictive model trainer 140 and engagement predictor 125 communicate with each other over a networking interface. As shown in FIG. 1D, predictive model trainer 140 trains predictive model 112 using training data 148 according to one or more machine learning algorithms 149 . Training data 148 may include one or more (or any combination) of the data described for signal 106 . The data may be historical data from a previous period of time, eg, the last day, the last month, the last year, and so on. For example, training data 148 includes data structure related signals 101 , health related signals 103 , engagement signals 105 , social graph signals 107 , history aggregation signals 109 , and content related signals 111 .

FIG. 1E illustrates an example of obtaining training data 148 for training predictive model 112 according to one aspect. For example, timeline manager 142 may inject a stream of messages to client application 154 . As part of one or more stream injections, timeline manager 142 may store information about message injections in injection log 127 . Injection log 127 may identify which messages were provided to client application 154 and may identify certain features associated with the injection. In some examples, the features may include information related to any of the signals 106 described above. Additionally, injection log 127 may include details about prediction manager 110 , such as prediction results 118 and/or engagement values 116 associated with one or more conversation graph 126 messages. As the user interacts with the message in the client application 154, the client event log 123 captures user engagement information such as whether the user has replied to the message, favorited or liked the message, and re-shared the message. Remember. In some examples, client event log 123 includes information such as whether the message has a photo or video and/or whether the message came from some depth in conversation graph 126 . Information in client event log 123 may be sent to messaging platform 104 for storage in messaging platform 104 . Training data 148 may include information from client event log 123 and information from injection log 127 . Training data 148 is used to train predictive model 112 . In some examples, predictive model trainer 140 periodically trains predictive model 112 by offline analysis and sends training results (eg, weights/biases) to predictive manager 110 .

In some examples, messaging platform 104 is configured to perform a two-level ranking mechanism for messages in conversation graph 126 . For example, a two-level ranking mechanism may provide an effective mechanism for dealing with swearing in conversations on messaging platform 104 . In some examples, the two-level ranking includes hard ranking and soft ranking. In some examples, hard ranking divides messages in conversation graph 126 into different sections (or groups) based on the level of confidence that the message is profanity and/or that the author is profane. including sectioning with In some examples, hard rankings are not personalized from the perspective of the user requesting the View Conversation Request 121 . Rather, hard ranking may be done using signals about the content itself and/or signals about the creator of the message. In some examples, soft ranking includes computing predicted results 118, generating engagement values 116 for messages in conversation graph 126, and then ranking messages in each section according to engagement values 116. including. In some examples, soft rankings are personalized from the perspective of the user requesting the View Conversation Request 121, as described above. In this way, messages from higher quality sections are presented to users ranked according to engagement value 116 . In some examples, hard ranking is performed first, followed by soft ranking. However, in some examples, hard rankings can be personalized by using the output of predictive model 112 to do hard rankings.

As shown with respect to FIGS. 1A and 1F, in some examples, messaging platform 104 may classify (eg, section) messages of conversation graph 126 into multiple sections 160 based on the content quality of the messages. may include a content quality classifier 147 configured as: The content quality of a message may include the probability (or belief) of being abusive (or poisonous) at the message level or author level. In some examples, content quality classifier 147 may determine (or receive) a probability of being an abusive for a particular message. (For example, this may be based on the probability that the content itself may be abusive or the creator or account generating the message may be linked to abusive behavior). If the probability of being an abusive is relatively high (or the level of confidence that the message and/or creator is considered an abusive is relatively high), the content quality classifier 147 classifies the message into one section 160 (e.g., low quality). or swearing section). Alternatively, if the probability of being an abusive is relatively low (or the level of confidence that the message and/or creator is considered an abusive is relatively low), the content quality classifier 147 classifies the message into another section 160 (e.g., high quality or non-swear section). In other words, content quality classifier 147 is configured to divide node 128 (or branch 132 ) of conversation graph 126 into multiple sections 160 . Each section 160 may refer to a different categorization (or classification) of quality level (or cursed (or poisonous) level). Content quality classifier 147 may determine that the first message is of relatively low quality and the second message is of relatively high quality. Here, content quality classifier 147 assigns the first message (or branch 132 containing the first message) to the first section (eg, the first section is considered a low quality section). Well, the second message (or branch 132 containing the second message) may be assigned to the second section (eg, the second section is considered a high quality section).

The classification or sectioning performed by content quality classifier 147 may be considered the first part (eg, hard ranking) of a two-level ranking mechanism. In some examples, section 160 may include two or more different sections, each representing a separate quality category or classification. In some examples, section 160 includes low quality section 162 , medium quality section 164 , and high quality section 166 . In some examples, the term section may refer to a category or taxonomy. Although three sections 160 are shown in FIG. 1F, sections 160 may include any number of sections, including two sections or any number greater than three. After being classified by content quality classifier 147, low quality section 162 contains or identifies nodes 128 (or branches 132) from conversation graph 126 that have been determined to be of low quality, and medium quality section 164 contains or identifies nodes of medium quality. and the high quality section 166 contains or identifies nodes 128 (or branches 132) from the conversation graph 126 that have been determined to be of high quality. contains or identifies In some examples, medium quality section 164 includes messages that may be abusive. Alternatively, there is a probability that the account posting the message could be linked with swearing behavior with moderate confidence. In some examples, a low quality section 162 may be reported and reviewed by a bystander, or a low quality section 162 may contain untrustworthy links, or an account posting a message may have a high probability of abusive behavior. Contains messages that would likely be judged to be profanity if linked to.

In some examples, content quality classifier 147 includes a classification model 161 (or models) that classifies nodes 128 (or branches 132 ) of conversation graph 126 into sections 160 . In some examples, classification model 161 is a machine learning model that is trained using training data applied to one or more machine learning algorithms. In some examples, classification model 161 includes a neural network. Training data may be obtained over a period of time (eg, a day, a week, a month, a year, etc.) and then used to train (or retrain) the classification model 161 in offline mode. . The updated classification model 161 is now sent to the content quality classifier 147 .

Content quality classifier 147 obtains one or more signals 163 associated with classification model 161 to determine which messages in conversation graph 126 are classified into low quality section 162, medium quality section 164, or high quality section 166. That signal 163 may be input to the classification model 161 so as to determine whether the For a particular message (e.g., node 128) of conversation graph 126, the output of classification model 161 may identify a quality classification, e.g., either low quality section 162, medium quality section 164, or high quality section 166. . In some examples, nodes 128 (or branches 132 ) of conversation graph 126 are annotated with sections 160 determined by classification model 161 . In some examples, content quality classifier 147 stores the classification of node 128 (or branch 132 ) in a data store on messaging platform 104 .

To classify a particular message in conversation graph 126, in some examples, signal 163 is one or more signals generated by client application 154 and/or one or more signals generated by messaging platform 104. signal. In some examples, signal 163 includes one or more signals that are different from signal 106 used to determine prediction result 118 . In some examples, signal 163 includes one or more signals representing the content of the message. In some examples, signal 163 includes one or more signals representing the user history of the author who generated the content. In some examples, signals 163 include one or more signals that model the user profile of the author who generated the content. In some examples, signals 163 include one or more signals that model author behavior on messaging platform 104 . In some examples, signals 163 may include signals representing message text, user behavior history such as logging history, type of device the user is using, how many times the user has been reported as swearing, and the like. Based on these signals 163, the content quality classifier 147 may predict whether the message is profanity and/or whether the user is prone to profanity. Once those probabilities (or beliefs or scores) are determined, the content quality classifier 147 can make sectioning decisions. For example, if a message is abusive with high confidence and the user who wrote the message is also abusive with high confidence, content quality classifier 147 classifies the message into the lowest quality section, e.g. you can

In some examples, content quality classifier 147 may receive the message identifier of the message and/or the user identifier of the author who generated the content. In response to the message identifier and/or user identifier, content quality classifier 147 may obtain signal 163 that is input to classification model 161 . In some examples, content quality classifier 147 may use the message identifier to retrieve the content of the message from message repository 138 . In some examples, instead of receiving a message identifier, content quality classifier 147 may receive the content of the message. In some examples, the content quality classifier 147 applies one or more categorizations to authors by communicating with one or more data services 165 through any of the techniques described with reference to FIG. 1C. A user identifier may be used to acquire the signal. In some examples, the categorization of a message into one of multiple sections 160 may occur at the time of message generation. In some examples, message categorization occurs at (or near) receipt of the view conversation request 121 . In some examples, content quality classifier 147 may convert signal 163 into a format compatible with classification model 161 .

In some examples, timeline manager 142 and/or prediction manager 110 may communicate with content quality classifier 147 through a server communication interface (eg, Thrift calls, REST calls, GraphQL requests, etc.). Here, the message identifier and/or user identifier are sent to content quality classifier 147 . The content quality classifier 147 annotates the node 128 (or branch 132) of the conversation graph 126 with the determined classification and/or passes the classification to the service requesting the classification information via the server communication interface. return. Content quality classifier 147 provides signal 163 to predictive model 161 to determine whether the message has a content quality corresponding to low quality section 162, medium quality section 164, or high quality section 166. good. In some examples, unlike predicted results 118 , the categorization of messages into sections 160 is not personalized to the user associated with view conversation request 121 .

In response to view conversation request 121, timeline manager 142 controls prediction manager 110 to generate prediction results 118 for each message (or subset of messages) of conversation graph 126, as already described above. and an engagement value 116 may be calculated for each message based on the predicted results 118 . For example, engagement predictor 125 obtains prediction-related signal 106 and inputs signal 106 into predictive model 112 to determine predicted outcome 118 for each message (or subset of messages) in conversation graph 126. good. In some examples, prediction results 118 include cross-engagement probabilities 124 . In some examples, predicted results 118 include mutual engagement probabilities 124 and one or more of positive engagement probabilities 120 and negative engagement probabilities 122 . In some examples, predicted results 118 include mutual engagement probability 124 , positive engagement probability 120 , and negative engagement probability 122 .

Engagement scorer 114 may use prediction results 118 (eg, combine prediction results 118 to produce engagement value 116) to calculate engagement value 116 for each message. The engagement value 116 is then used by the timeline manager 142 to rank the messages into low quality section 162, medium quality section 164, and high quality section 166 separately.

In some examples, the timeline manager provides a list of messages classified as low quality section 162 from content quality classifier 147, a list of messages classified as medium quality section 164 from content quality classifier 147, and h Content quality classifier 147 may receive a list of messages classified as high quality section 166 . The timeline manager may also receive engagement values 116 from the forecast manager 110 . Ranking messages within a particular section 160 using engagement values 116 for messages within the respective section 160 may be considered a second part of the two-level ranking mechanism of FIG. 1F. In some examples, the second part of the two-level ranking mechanism may be considered a soft ranking of messages.

In some examples, timeline manager 142 generates first ranked list 158-1 of messages categorized in low quality section 162, ranked using engagement value 116 from prediction manager 110. may be generated. In some examples, first ranked list 158-1 includes branches 132 of conversation graph 126 classified as low quality. Branches 132 are ranked within the poor quality section 162 according to their engagement value 116 . Timeline manager 142 may generate a second ranked list 158 - 2 of messages categorized in medium quality section 164 ranked using engagement values 116 from prediction manager 110 . In some examples, second ranked list 158-2 includes branches 132 of conversation graph 126 classified as medium quality. Branches 132 are ranked within medium quality section 164 according to their engagement value 116 . Timeline manager 142 may generate a third ranked list 158 - 3 of messages categorized in high quality section 166 ranked using engagement values 116 from prediction manager 110 . In some examples, third ranked list 158-3 includes branches 132 of conversation graph 126 classified as high quality. Branches 132 are ranked within high quality section 166 according to their engagement value 116 .

Timeline manager 142 transmits at least a subset of messages in high quality section 166 (eg, at least a portion of third ranked list 158-3) according to rank rendered at client application 154. you can In this manner, a user of client application 154 may view messages that are considered high quality that are also ranked according to the message's engagement value 116 . Timeline manager 142 may request subsequent requests to view additional messages from conversation graph 126 . Here, timeline manager 142 may send at least a portion of second ranked list 158-2 followed by at least a portion of first ranked list 158-1.

In some examples, the timeline manager 142 selects the third ranked list 158-3, the second ranked list 158-2, followed by the first ranked list 158-1. Rank the messages in the conversation graph 126 according to order. Thus, client application 154 first renders messages from high quality section 166 ranked according to engagement value 116, renders messages from medium quality section 164 ranked according to engagement value 116, and finally , render messages from the low quality section 162 ranked according to engagement value 116 . If the high quality section 166 contains 10 messages, the medium quality section 164 contains 10 messages, and the low quality section 162 contains 10 messages, and the threshold of displayed messages is 15. , the client application 154 renders the entire third ranked list 158-3 and the five messages of the second ranked list 158-2. Timeline manager 142 may receive requests to display additional messages in conversation graph 126 . The timeline manager 142 may now send the second ranked list 158-2, followed by the other five messages of the first ranked list 158-1.

In some examples, hard rankings can be personalized by using the output of predictive model 112 to do hard rankings. For example, prediction manager 110 may determine prediction results 118 and engagement values 116 as described above. Content Quality Classifier 147 may use the output of Prediction Manager 110 to classify (and/or rank) messages according to a hard ranking technique, which is from the perspective of the user who initiated View Conversation Request 121. Personalized.

FIG. 2 illustrates a neural network 219 according to one aspect. Neural network 219 may be an example of mutual engagement model 119 of messaging system 100 of FIGS. 1A-1D. However, the features described with reference to FIG. 2 may be applied to any of predictive models 112 including positive engagement model 115 and negative engagement model 117 . In some examples, neural network 219 may be an example of classification model 161 of FIG. 1F. Neural network 219 is configured to output mutual engagement probability 224 . Cross-engagement probability 224 may be an example of cross-engagement probability 124 in FIG. 1A. Neural network 219 may be an interconnected group of nodes 260, where each node 260 represents an artificial neuron. Nodes 260 are connected to each other by layers, with the output of one layer being the input of the next layer. Neural network 219 transforms the input X ₁ , X ₂ -X _n (eg, signal 106) received by input layer 262 and feeds it to one or more hidden layers 264 (eg, FIG. ) to produce an output Y ₁ (eg, mutual engagement probability 124 ) by output layer 266 . Each layer consists of a subset of the set of nodes 260 .

Using neural network 219 to obtain mutual engagement probabilities 224 includes applying weighted and biased numerical inputs to interconnected nodes 260 in neural network 219 and calculating their outputs. good. The weightings and biases applied to each node 260 in neural network 219 are obtained by training neural network 219, for example, using machine learning algorithm 149 (eg, by predictive model trainer 140 of FIG. 1A). good. Nodes 260 in neural network 219 may be organized into two or more layers, including at least an input layer 262 and an output layer 266 . In a multi-layer neural network 219, the output from one layer may serve as the input to the next layer. Layers with no external output connections may be referred to as hidden layers 264 . The output of each node 260 is a function of the weighted sum of its inputs and bias.

A vector of feature values (X ₁ . . . X _N ) is applied as input to each node 260 in input layer 262 to obtain mutual engagement probabilities 224 . In some examples, the vector of feature values (X ₁ . . . X _N ) includes the values of signal 106 described above. Input layer 262 distributes values to each of the nodes 260 in hidden layer 264 . Upon reaching node 260 in hidden layer 264, the value from each input node is multiplied by a weight, and the resulting weighted values are summed and added to the weighted bias value to produce a combined value. The bound value is passed through a transformation or activation function that outputs the value. The output from hidden layer 264 is then distributed to nodes 260 in output layer 266 of neural network 219 . Upon reaching node 260 in output layer 266, the values from each hidden layer node are multiplied by a weight and the resulting weighted values are summed and added to the weighted bias value to produce the combined value. . The combined values are passed through a transformation or activation function that outputs Y ₁ (eg, mutual engagement probability 224).

FIG. 3 depicts a flowchart 300 illustrating exemplary operation of the messaging platform 104 for ranking conversation graph messaging according to an aspect.
Operation 302 includes receiving conversation view request 121 to retrieve messages of conversation graph 126 stored in messaging platform 104 executable by server computer 102 over network 150 . Act 304 includes messaging platform 104 using predictive model 112 to determine a plurality of predicted outcomes for each of the plurality of messages in conversation graph 126 . Here, multiple prediction results 118 include mutual engagement probabilities 124 . Act 306 includes messaging platform 104 ranking the plurality of messages based on predicted results 118 . Act 308 includes messaging platform 104 sending at least a subset of the plurality of messages rendered in client application 154 according to rank over network 150 .

FIG. 4 illustrates exemplary operation of a messaging platform for ranking messages of a conversation graph according to one aspect.
Operation 402 includes receiving conversation view request 121 to retrieve messages of conversation graph 126 stored in messaging platform 104 executable by server computer 102 over network 150 . Operation 404 includes determining, in which messaging platform 104 uses predictive model 112 to determine a plurality of predicted outcomes for each of the plurality of messages in conversation graph 126 . The determining step may include obtaining a plurality of signals 106 that are relevant to predictive model 112 . Here, the plurality of signals includes data structure related signals 101 associated with the conversation graph 126 and inputs the plurality of signals 106 to the predictive model 112 to determine the plurality of predicted outcomes 118 . Act 406 includes messaging platform 104 ranking the plurality of messages based on predicted results 118 . Act 408 includes messaging platform 104 sending at least a subset of the plurality of messages rendered in client application 154 according to rank over network 150 .

FIG. 5 depicts a flowchart 500 illustrating exemplary operation of a messaging platform for ranking conversation graph messaging in accordance with an aspect.
Operation 502 includes messaging platform 104 classifying messages in conversation graph 126 into multiple sections 160 based on the content quality of the messages. Here, multiple sections 160 include a first section and a second section. The first section has messages from conversation graph 126 determined to be of higher quality than the messages of the second section. Operation 504 includes determining, in which messaging platform 104 uses predictive model 112 to determine a plurality of predicted outcomes for each of the plurality of messages in conversation graph 126 . Act 506 includes messaging platform 104 ranking the first section messages based on predicted results 118 for the first section messages. Act 508 includes messaging platform 104 sending at least a subset of the messages of the first section rendered in client application 154 according to rank.

Numerous details have been described above. However, it will be apparent to one skilled in the art having the benefit of this disclosure that implementations of the disclosure may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the description.

Some portions of the detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally thought here to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

Note, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, it should be apparent from the discussion above that throughout the description "identify", "determine", "calculate", "update", "transmit", " Discussions using terms such as "receive", "generate", and "modify" refer to data represented as (e.g., electronic) physical quantities in computer system registers and memory as computer system memory or registers. the action of a computer system or similar electronic computing device that manipulates and transforms it into other data similarly represented as physical quantities within or in other such information storage, transmission, or display devices; It is understood that reference is made to processes.

Implementations of the present disclosure also relate to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such computer programs may be stored on any kind of disk, including floppy disk, optical disk, CD-ROM and magneto-optical disk, read only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, flash It may be stored in a non-transitory computer-readable storage medium, such as, but not limited to, memory or any type of medium suitable for storing electronic instructions.

The words "example" or "exemplary" are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "example" or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word "example" or "exemplary" is intended to present concepts in a concrete fashion. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise, or clear from context, "X comprises A or B" is intended to mean any of its natural inclusive permutations. That is, if X contains A, X contains B, or X contains both A and B, then "X contains A or B" is satisfied under any of the above cases. . In addition, as used in this application and the appended claims, the articles "a" and "an" are used unless otherwise specified or clear from the context that they are directed to the singular , should generally be taken to mean "one or more". Further, use of the terms "implementation" or "one embodiment" or "implementation" or "one implementation" throughout is intended to mean the same embodiment or implementation unless so stated. not Moreover, as used herein, the terms “first,” “second,” “third,” “fourth,” etc. are meant as labels to distinguish different elements. and may not necessarily have an ordinal meaning according to their numerical designation.

The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. I know something. The required structure for a variety of these systems will appear from the description below. Additionally, the present disclosure is described without reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the disclosure described herein.

The above description reveals numerous specific details, such as examples of specific systems, components, methods, etc., in order to provide a good understanding of some implementations of the present disclosure. However, it will be apparent to those skilled in the art that at least some implementations of this disclosure may be practiced without these specific details. In other instances, well-known components or methods are not described in detail, or are presented in simple block diagram format, in order to avoid unnecessarily obscuring the present disclosure. Accordingly, the specific details disclosed above are only examples. Particular implementations may vary from these detailed examples and still be contemplated within the scope of the present disclosure.

The present invention relates to a computer-implemented method and system for ranking messages in a conversation graph in a messaging platform using predictive results.

Social media messaging platforms are capable of exchanging millions or hundreds of millions of social media messages between their users. Messages exchanged on the platform can often provide the user of the platform with the latest updates or reports on current events. An exchange of messages in a messaging platform may be part of a conversation between users. Some conventional messaging systems may store message reply structures so that users can view parts of the conversation when viewing a particular message. However, the list of messages forming a conversation view may be relatively large, thereby causing slow load times for the user.

A messaging platform facilitates the exchange of messages between users of the messaging platform. A message may be part of a conversation conducted on a messaging platform. For example, a user may post a message on the platform. Other users may post multiple replies to that message, then replies to that reply, and so on. A reply structure may be stored as a conversation graph. A messaging platform may store any number of conversation graphs associated with conversations conducted on the messaging platform. In some examples, the conversation graph is relatively large (eg, multiple nodes in the conversation graph exceed a threshold). Generating and maintaining a conversation graph enables rapid retrieval of information when responding to requests to view messages in a reply relationship to a particular message.

When viewing a particular message associated with a conversation graph, a user may select a message to view other messages in the conversation graph, where the client application submits a conversation viewing request to the messaging platform. Generate. Rather than displaying all of the messages (which may be relatively large) that are relevant to the conversation graph, the timeline manager uses the conversation graph to rank the branches (or nodes) that are most relevant to a particular user. messages may be ranked using a unique scoring system (which incorporates engagement predictions) and then a ranked list (or a subset thereof) may be provided. In this way, messages in the conversation graph are displayed according to their level of prediction results (balancing positive prediction results against negative prediction results). In addition to this, by providing a ranked list, the messaging platform allows increasing the value provided to the user while providing fewer responses. For example, the messaging platform may provide a subset of responses (eg, top 10, 15, 20, etc. responses) for each request. This can result in faster computation on the server, faster load times for the user, and almost no loss in terms of engagement.

A prediction manager (running on the messaging platform) may use one or more prediction models to determine a prediction result for each candidate node in the conversation graph. The predicted result is the predicted user engagement result for each message (or subset of messages) of the conversation graph. Prediction results include mutual engagement probabilities. A mutual engagement probability indicates a probability value that a user is expected to further develop the conversation graph (eg, to reply to a message). Incorporating mutual engagement probabilities into prediction results may encourage more conversations on the platform. In some examples, the predicted results include mutual engagement probabilities and one or more of positive engagement probabilities and negative engagement probabilities. A positive engagement probability indicates a probability value that a user is expected to engage positively with a message (eg, like or add a node to favorites). A negative engagement probability indicates a probability value that a user is expected to negatively view or engage with the message (eg, the user may think the node is a swear word). In some examples, the predicted results include mutual engagement probabilities, positive engagement probabilities, and negative engagement probabilities.

The predictive models may include a positive engagement model that predicts positive engagement probabilities, a negative engagement model that predicts negative engagement probabilities, and a cross-engagement model that predicts cross-engagement probabilities. In some examples, the predictive model is a machine learning model that is trained using training data applied to one or more machine learning algorithms. In some examples, predictive models include neural networks. Training data may be a combination of historical data obtained from client event logs and injection logs while the messaging system is active (and the prediction manager is enabled). Historical data may be obtained over a period of time (eg, a day, a week, a month, a year, etc.) and then used to train (or retrain) a predictive model in offline mode. The updated forecast model may now be sent to the forecast manager.

In response to a conversation view request to retrieve a message of the conversation graph, the prediction manager may obtain a signal for determining the prediction result and input the signal into the prediction model to determine the prediction result. The signals may include signals generated by the client application and/or signals generated by the messaging platform. In some examples, the signals include the user's and author's social graph and engagement history. However, the signals may include a variety of signals such as data structure related signals, health related signals, engagement signals, social graph signals, history aggregation signals, and/or content related signals. The number of signals and the number of different categories of signals used for prediction may improve the accuracy of the model prediction.

The prediction manager may combine prediction results to obtain an engagement value for each node. Here, the engagement value indicates the total level of predicted engagement with the messages in the conversation graph. The timeline manager may use the engagement value to greedily select the most relevant branches (or nodes) of the conversation graph for the user. In addition to locating the most relevant branches (or nodes), the timeline manager may determine the amount of branches to display based on engagement value. Additionally, the timeline manager may collapse irrelevant parts of the conversation graph, but allow the user to expand those parts to view other messages in the conversation graph.

Messages ranked according to engagement value may be unique to each user. For example, some messages in the conversation graph may be more relevant to the first user, while other messages in the conversation graph may be relevant to the second user. However, the prediction results determined by the prediction manager may be tailored to specific users. In contrast, some conventional approaches use vote-based mechanisms that can provide the same view for each. Additionally, because the messaging platform incorporates interactive engagement probabilities into its scoring algorithm, the messaging platform may encourage more conversations on the messaging platform.

In some examples, the messaging platform performs a two-level ranking system on the messages of the conversation graph. For example, a two-level ranking mechanism (eg, hard ranking and soft ranking) may provide an effective mechanism for combating swearing in conversations on messaging platforms. In some examples, hard ranking classifies (or sectioning) messages (or branches) of a conversation into multiple sections. For example, a messaging platform may include a content quality classifier that divides nodes (or branches) of a conversation graph into sections. Each section may represent a different categorization (or classification) of quality levels. In some examples, a content quality classifier is a classification model (e.g., , neural networks).

A content quality classifier may determine that the first message is of relatively low quality and the second message is of relatively high quality. Here, the content quality classifier may assign the first message (or the branch containing the first message) to the first section (eg, the first section considered a low quality section), 2 messages (or the branch containing the second message) may be assigned to the second section (eg, the second section considered the high quality section). Soft ranking, a two-level ranking mechanism, may involve ranking messages (or branches) within each section using the engagement values described above. A message may then be presented to the user to provide a ranked list of high quality sections followed by a ranked list of low quality sections.

According to one aspect, a method for ranking conversation graph messages in a messaging platform using prediction results receives, over a network, a conversation view request to retrieve conversation graph messages stored in the messaging platform. and determining, where the messaging platform uses a predictive model to determine a plurality of predicted outcomes for each of a plurality of messages of the conversation graph. The plurality of prediction results includes mutual engagement probabilities. The method comprises the steps of: the messaging platform ranking the plurality of messages based on the predicted results; and the messaging platform rendering at least a subset of the plurality of messages in a client application according to the rankings over the network. and transmitting. According to further aspects, a non-transitory computer-readable medium storing a corresponding system and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more of the following features (or any combination thereof). The plurality of predicted outcomes also includes a positive engagement probability and a negative engagement probability. The method comprises calculating an engagement value for each of the plurality of messages using the mutual engagement probability, the positive engagement probability, and the negative engagement probability, wherein the plurality of messages are matched according to the engagement value. Ranked. The method comprises the steps of: the messaging platform generating the conversation graph based on a reply structure of messages exchanged on the messaging platform, the conversation graph comprising a tree data structure of messages associated with a conversation; A process may be provided. The method comprises the steps of: the messaging platform obtaining signals generated by one or more of the client application and the messaging platform; and the messaging platform determining the plurality of predicted results, and inputting said signal to said prediction model. The method may comprise the messaging platform training the predictive model based on a machine learning algorithm fed with training data. The method comprises the steps of: the messaging platform, in response to a number of messages in the conversation graph exceeding a threshold level, selecting a candidate subset from the plurality of messages; is determined for each message of said candidate subset. The method comprises the steps of: the messaging platform sending a first subset of the plurality of messages rendered in the client application according to the rank; and the messaging platform displaying additional messages from the conversation graph. It may further comprise receiving a request from the client application, and the messaging platform sending a second subset of the plurality of messages rendered in the client application according to the rank. The conversation graph may include a tree data structure having an arrangement of nodes representing the messages of the conversation graph. The method comprises the steps of: the messaging platform calculating an engagement value for each of the plurality of messages based on the plurality of predicted results; or selecting a plurality of branches.

According to one aspect, a system for ranking messages of a conversation graph in a messaging platform using prediction results is configured to generate a conversation graph based on reply structures of messages exchanged in the messaging platform. a conversation graph manager, said conversation graph including a data structure of messages of a conversation; a timeline manager configured to receive a conversation viewing request from the client application over the network; An engagement predictor configured to determine a plurality of predicted outcomes for each of a plurality of messages, the plurality of predicted outcomes being a mutual engagement probability and one of a positive engagement probability and a negative engagement probability. and an engagement scorer configured to calculate an engagement value for each of the plurality of messages using the prediction results for each message. The timeline manager is configured to rank the plurality of messages using the engagement value and provide over the network at least a subset of the plurality of messages rendered in the timeline according to the ranking. there is According to further aspects, a non-transitory computer-readable medium storing corresponding methods and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The mutual engagement probability includes a probability value that the messaging platform expects to receive a reply to the message in the conversation graph, and the positive engagement probability indicates that the messaging platform will receive an acknowledgment to the message in the conversation graph. said negative engagement probability comprises a probability value that said messaging platform is expected to receive negative user engagement to said message of said conversation graph. . The predictive models include mutual engagement models, positive engagement models, and negative engagement models. The engagement predictor obtains signals generated by one or more of the client application and the messaging platform and applies the signals to the mutual engagement model, the positive engagement model, and the negative engagement model. configured to input and obtain the mutual engagement probability, the positive engagement probability, and the negative engagement probability, respectively. The system is a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with training data, the trained predictive model being: A predictive model trainer configured to provide to the predictive manager may further comprise. The system may further comprise a candidate message selector configured to select a subset of candidates from a plurality of sources, wherein the plurality of sources are selected from the conversation graph ranked according to the most recently posted. A first source with messages and a second source with said messages of said conversation graph ranked according to a relevance algorithm. The candidate subset includes a number of messages from the first source and a number of messages from the second source, and the plurality of prediction results is determined for each message of the candidate subset. . The data structure of the conversation graph includes a tree data structure having an arrangement of a plurality of nodes representing the messages of the conversation graph, and the timeline manager selects a branch of the tree data structure using the engagement value. is configured to The engagement scorer determines the mutual engagement probability, the positive engagement probability, and the negative engagement probability such that the weighting applied to the mutual engagement probability is higher than the weighting applied to the negative engagement probability. It is configured to apply a weighting to each.

According to one aspect, a non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, communicates with the one or more processors over a network to a messaging platform. selecting a candidate subset of messages from the conversation graph; and determining a plurality of predicted outcomes for each message of the candidate subset using a predictive model, wherein the plurality of predicted outcomes comprises a cross-engagement probability, the cross-engagement probability is determined by the messaging platform. includes a probability value predicted to receive a reply to each message of the candidate subset; calculating an engagement value for each message of the candidate subset using the prediction result; ranking the plurality of messages using a value; and sending at least a subset of the plurality of messages rendered in a client application according to the ranking. computer readable medium. According to further aspects, corresponding systems and methods for storing corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The candidate subset includes a number of recently posted messages and a number of top ranked messages from the conversation graph. The data structure of the conversation graph includes a tree data structure having an arrangement of a plurality of nodes representing the messages of the conversation graph. The act may comprise selecting a branch of the tree data structure using the engagement value. The operations comprise periodically training the prediction model based on one or more machine learning algorithms input with training data, and providing the trained prediction model to a prediction manager. you can

According to one aspect, a method for ranking conversation graph messages in a messaging platform using prediction results receives, over a network, a conversation view request to retrieve conversation graph messages stored in the messaging platform. and determining, where the messaging platform uses a predictive model to determine a plurality of predicted outcomes for each of a plurality of messages of the conversation graph. The determining step includes obtaining a plurality of signals associated with the prediction model, including data structure-related signals associated with the conversation graph; into the prediction model. The method comprises the steps of: the messaging platform ranking the plurality of messages based on the predicted results; and the messaging platform rendering at least a subset of the plurality of messages in a client application according to the rankings over the network. and transmitting. According to further aspects, a non-transitory computer-readable medium storing a corresponding system and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . the conversation graph includes a tree data structure having a plurality of nodes representing the messages of the conversation graph and edges representing links between the nodes, the tree data structure defining one or more branches of nodes; The data structure related signals include signals representing the number of nodes and the number of branches within the conversation graph. The data structure related signals include the number of replies within a branch, the number of conversations within a branch, and the number of unique authors within a branch or the conversation graph. The plurality of signals includes a signal indicative of whether a user of the client application has restricted creators of messages of the conversation graph. The plurality of signals includes engagement signals representing user engagement data associated with the messages of the conversation graph, including engagement of users following users of the client application in a connection graph. The plurality of signals includes historical aggregated signals including engagement data associated with users of the client application on the messaging platform. The plurality of signals includes content-related signals associated with messages of the conversation graph. At least a subset of the plurality of signals are obtained from one or more data services executing on the messaging platform. The method obtains training data from a client event log in which the messaging platform stores information received from the client application and an injection log in which information from a timeline manager executing on the messaging platform is stored. and said messaging platform training said predictive model based on a machine learning algorithm input with said training data.

According to one aspect, a system for ranking messages of a conversation graph in a messaging platform using prediction results is configured to generate a conversation graph based on reply structures of messages exchanged in the messaging platform. a conversation graph manager, said conversation graph including a data structure of messages of a conversation; a timeline manager configured to receive, over the network, a request to view a conversation from the client application; and an engagement predictor configured to determine a plurality of predicted outcomes for each of the plurality of messages. The engagement predictor configured to obtain a plurality of signals relevant to the predictive model from one or more data services stored on the messaging platform to determine the plurality of prediction outcomes. configured to input the plurality of signals to the predictive model so as to The system comprises an engagement scorer configured to calculate an engagement value for each of the plurality of messages using the predicted results for each message, the timeline manager using the engagement value ranks the plurality of messages by a method and provides over the network at least a subset of the plurality of messages rendered in the timeline according to the ranking. According to further aspects, a non-transitory computer-readable medium storing corresponding methods and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The plurality of signals includes data structure related signals associated with the conversation graph. The engagement predictor is configured to obtain one or more first signals from a first data service and one or more second signals from a second data service. The plurality of signals includes one or more of health-related signals, engagement signals, social graph signals, history aggregation signals, or content-related signals. The prediction model includes a mutual engagement model, a positive engagement model, and a negative engagement model, and the engagement predictor obtains mutual engagement probability, positive engagement probability, and negative engagement probability, respectively. configured to input signals into the mutual engagement model, the positive engagement model and the negative engagement model. The system is a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with training data, the trained predictive model being: A predictive model trainer configured to provide to the predictive manager may further comprise. The plurality of signals includes signals generated by the client application and signals generated by the messaging platform. The plurality of signals includes similarity signals.

According to one aspect, a non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, communicates with the one or more processors over a network to a messaging platform. selecting a candidate subset of messages from the conversation graph; determining a plurality of predicted outcomes for each message of said candidate subset using a prediction model, obtaining a plurality of signals including data structure related signals associated with said conversation graph; inputting the plurality of signals into the prediction model to determine a plurality of prediction outcomes; and using the prediction outcomes to calculate an engagement value for each message of the candidate subset; ranking the plurality of messages using the engagement value; and sending at least a subset of the plurality of messages rendered in a client application according to the ranking. Temporary Computer Readable Medium. According to further aspects, corresponding systems and methods may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The actions include obtaining training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform; training the predictive model based on a data-inputted machine learning algorithm. The plurality of signals includes one or more of health-related signals, engagement signals, social graph signals, history aggregation signals, or content-related signals.

According to one aspect, a method for ranking conversation graph messages in a messaging platform, wherein the messaging platform classifies conversation graph messages into a plurality of sections based on content quality of said messages. and the plurality of sections comprises a sorting step including a first section and a second section. The first section has messages from the conversation graph determined to be of higher quality than the messages of the second section. The method comprises the steps of: the messaging platform determining a plurality of predicted outcomes for each of the plurality of messages of the conversation graph using a predictive model; ranking the messages of the first section based on prediction results; and the messaging platform sending at least a subset of the messages of the first section rendered in a client application according to the ranking. , provided. According to further aspects, a non-transitory computer-readable medium storing a corresponding system and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The method comprises the messaging platform ranking the messages of the second section based on the plurality of prediction results for the messages of the second section. In some examples, the method comprises the steps of: the messaging platform receiving a request to render additional messages of the conversation graph in the client application; and the messaging platform rendered in the client application according to the rank. and sending at least a subset of said messages of said second section. The classifying step may include receiving a message identifier for a message of the conversation graph and using a classification model to determine that the message has a content quality corresponding to the first section. . The method may comprise assigning a branch of the conversation graph containing the message to the first section. The method comprises calculating an engagement value for each message of the conversation graph using the plurality of prediction results for each message, wherein the messages of the first quality section are are ranked according to the engagement value of said messages in . The prediction results include mutual engagement probabilities, positive engagement probabilities, and negative engagement probabilities. The method comprises the steps of: the messaging platform generating the conversation graph based on a reply structure of messages exchanged on the messaging platform, the conversation graph comprising a tree data structure of messages associated with a conversation; Further steps may be included. The plurality of predictive models includes a mutual engagement model, a positive engagement model, and a negative engagement model, wherein one or more of the mutual engagement model, the positive engagement model, and the negative engagement model are neural networks. including.

According to one aspect, a system for ranking messages in a conversation graph in a messaging platform is a conversation graph manager configured to generate a conversation graph based on reply structures of messages exchanged in the messaging platform. a conversation graph manager, wherein the conversation graph includes a data structure of messages of a conversation; and a timeline manager configured to provide a stream of messages to a user's timeline on a client application over a network. a timeline manager configured to receive conversation view requests from the client application over a network; and to classify the messages of the conversation graph into a plurality of sections based on content quality of the messages. wherein the plurality of sections includes a first section and a second section. The first section has messages from the conversation graph determined to be of higher quality than the messages of the second section. The system includes an engagement predictor configured to determine a plurality of predicted outcomes for each of a plurality of messages of the conversation graph using a predictive model; and using the plurality of predicted outcomes for each message. and an engagement scorer configured to calculate an engagement value for each of the plurality of messages. The timeline manager ranks the messages of the first section using the engagement value of the first section, and at least of the messages of the first section rendered in the timeline according to the rank. A subset is configured to be provided over the network. According to further aspects, a non-transitory computer-readable medium storing corresponding methods and corresponding instructions may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The engagement predictor is configured to obtain a plurality of signals relevant to the predictive model from one or more data services stored on the messaging platform, and extracts the plurality of prediction results. configured to input the plurality of signals to the predictive model to determine. The plurality of signals includes data structure related signals associated with the conversation graph. The timeline manager is configured to separately rank the messages of the second section using the engagement value of the second section. The content quality classifier receives a message identifier for a message of the conversation graph and uses a classification model to determine that the message has a content quality corresponding to the first quality section. It is configured. The plurality of prediction results includes mutual engagement probabilities, wherein the mutual engagement probabilities include probability values that the messaging platform is expected to receive a reply to each message of the conversation graph. The system is a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with training data, the trained predictive model being: A predictive model trainer configured to provide to the predictive manager may further comprise.

According to one aspect, a non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, cause the one or more processors to use a classification model to: , classifying messages of a conversation graph into a plurality of sections based on content quality of said messages, said plurality of sections including a first section and a second section, said first section said for each of the plurality of messages of the conversation graph using one or more predictive models, having messages from the conversation graph determined to be of higher quality than the messages of the second section; wherein the plurality of prediction results includes a mutual engagement probability, a positive engagement probability, and a negative engagement probability; and ranking the messages in the second section based on the plurality of prediction results for the messages in the second section. and transmitting at least a subset of said messages of said first quality section rendered in a client application according to said rank. According to further aspects, corresponding systems and methods may be provided.

According to some aspects, methods, systems, and/or non-transitory computer-readable media may include one or more (or any combination thereof) of the following features above/below: . The actions include obtaining training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform; and training the predictive model based on a machine learning algorithm input by . The act comprises receiving a message identifier for a first message of the conversation graph and using the classification model to determine that the first message has a content quality corresponding to the first section. and receiving a message identifier of a second message of the conversation graph, and using the classification model to determine that the second message has a content quality corresponding to the second section; may be provided. The act comprises receiving a request to render additional messages of the conversation graph in the client application and sending at least a subset of the messages of the second section to be rendered in the client application according to the rank. , may be provided. These and other features are explained further in the detailed description.

1 illustrates a messaging system for ranking messages in a conversation graph using predictive results from predictive models, according to one aspect. FIG. 1 illustrates a candidate message selector of a messaging system, according to one aspect; FIG. FIG. 4 illustrates an example of signals used to determine a prediction result, according to one aspect. 1 illustrates a prediction manager of a messaging system, according to one aspect; FIG. FIG. 4 illustrates an example of generating training data for a predictive model, according to one aspect. 1 illustrates an example of a two-level ranking mechanism including a content quality classifier, according to one aspect; FIG. FIG. 3 illustrates an example predictive model as a neural network, according to one aspect. 4 is a flowchart illustrating exemplary operation of a messaging system, according to one aspect. 4 is a flowchart illustrating exemplary operation of a messaging system, according to one aspect. 4 is a flowchart illustrating exemplary operation of a messaging system, according to one aspect.

1A-1E illustrate a messaging system 100 for ranking messages in a conversation graph 126 using predictive results 118 from predictive models 112 according to one aspect. Messaging system 100 , in one aspect, comprises a messaging platform 104 executable by server computer 102 and a client application 154 executable by computing device 152 . Client application 154 communicates with messaging platform 104 over network 150 to send (and receive) messages to (and from) other users of messaging platform 104 .

Client application 154 may be a social media messaging application. Social media messaging applications allow users to post and interact with messages. In some examples, client application 154 is a native application that runs on the operating system of computing device 152, or in conjunction with a browser-based application of computing device 152, to server computer 102 (or other It may be a web-based application running on a server of Computing device 152 may be connected via network 150 using any type of network connection and/or application programming interface (API) to enable client application 154 and messaging platform 104 to communicate with each other. to access messaging platform 104 .

Computing device 152 may be a mobile computing device (eg, smart phone, PDA, tablet, or laptop computer) or a non-mobile computing device (eg, desktop computing device). The computing device 152 has, for example, a mobile network interface through which the computing device 152 can communicate with a cellular network, a Wi-Fi based station with which the computing device 152 can communicate; A Wi-Fi network interface, a Bluetooth network interface with which computing device 152 can communicate with other Bluetooth® devices, and/or access by computing device 152 to network 150 It also includes various network interface circuitry, such as an Ethernet connection or other wired connection, that allows for

Server computer 102 may be a single computing device or a representation of two or more distributed computing devices communicatively coupled to share workload and resources. The server computer 102 stores one or more processors and executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations described herein. , and a non-transitory computer-readable medium.

Messaging platform 104 is a computing platform for communication (eg, real-time communication) between user devices (one of which is shown as computing device 152). The messaging platform 104 may store millions of accounts 141 (eg, pseudonym accounts, novelty accounts, etc.) of individuals, businesses, and/or entities. One or more users of each account 141 may use messaging platform 104 to send messages to other accounts 141 internal and/or external to messaging platform 104 . In some examples, messaging platform 104 enables users to communicate in "real time," e.g., converse with other users with minimal delay, and to communicate with one or more other users during concurrent sessions. It may be possible to have a conversation. In other words, messaging platform 104 may allow users to broadcast messages to one or more users within a reasonable time frame (eg, less than two seconds) to conduct live conversations between users. may display messages to other users of In some examples, recipients of a message may have a predetermined graphical relationship in the connection graph 134 with the account of the user who broadcast the message.

The connection graph 134 indicates which accounts 141 within the messaging platform 104 are associated with (e.g., followed, friends with, subscribed to, etc.) a particular account 141 and therefore messages from a particular account 141 . Contains a data structure that indicates whether you are subscribed to receive For example, connection graph 134 may link a first account with a second account. This indicates that the first account is related to the second account. The second account user may view messages posted on messaging platform 104 by the first account user (and/or the first account user may view messages posted by the second account user). may view messages posted on the platform 104). Relationships may include unidirectional (eg, follower/follower) and/or bidirectional (eg, friendship). Messages can be any of a variety of lengths, which may be limited by a particular messaging system or protocol.

In some examples, a user interested in viewing messages written by a particular user may choose to follow the particular user. A first user can follow a second user by identifying the second user as a user that the first user wishes to follow. After the first user indicates that they want to follow the second user, the connection graph 134 is updated to reflect the relevance and the first user receives messages written by the second user. is provided. A user can choose to follow multiple users. Users can also respond to messages, thereby allowing them to converse with each other. In addition to this, users can engage with messages, such as sharing messages with their followers, or favouriting (or "liking") messages whose engagement is shared with their followers. good.

Messages exchanged on messaging platform 104 are stored in message repository 138 . Message repository 138 may comprise one or more tables that store records. In some examples, each record corresponds to a separately stored message. For example, a record may include a message identifier for a message posted to messaging platform 104, an author identifier identifying the author of the message (e.g., @tristan), message content (e.g., text, image, video, and/or web content URL), one or more participant account identifiers identified in the body of the message, and/or reply information identifying the parent message to which the message is replied (if the message is a reply to a message) can be identified.

The messaging platform 104 comprises a conversation graph manager 136 that generates a conversation graph 126 and a timeline manager 142 that injects a timeline 156 of messages into the client application 154 . Messaging platform 104 includes candidate message selector 108 that selects candidate subset 133 of messages from conversation graph 126 . The messaging platform 104 comprises a prediction manager 110 that obtains signals 106 related to predictions and inputs the signals 106 to one or more prediction models 112 to determine prediction results 118 . In some examples, prediction manager 110 calculates engagement value 116 based on prediction results 118 . The timeline manager 142 ranks (and selects) messages in the conversation graph 126 using the engagement values 116 and provides a ranked list 158 of messages to the client application 154 . Messaging platform 104 includes predictive model trainer 140 that obtains predictive data 148 and trains predictive model 112 using one or more machine learning algorithms 149 input by training data 148 .

More specifically, conversation graph manager 136 generates (and updates) one or more conversation graphs 126 as messages are exchanged on messaging platform 104 . In some examples, conversation graph 126 is stored in a data storage device associated with messaging platform 104 . In some examples, conversation graph 126 is stored at timeline manager 142 . The messaging platform 104 may store multiple conversation graphs 126 (eg, hundreds, thousands, or millions of conversation graphs 126). Each conversation graph 126 may represent the structure of replies to messages that were not original replies (eg, root messages). For example, each time a user creates and posts a message to messaging platform 104 that is not an original reply, a potential new conversation can be started. Others can then reply to the original or "root" message, generating their own reply branches. Over time, when the number of replies to messages that are not original replies (and/or replies to replies to messages that are not original replies) exceeds a threshold level, conversation graph manager 136 adds conversation identifiers to conversation graph 126. A conversation identifier may be assigned and may uniquely identify the conversation graph 126 . In some examples, conversation graph manager 136 may assign a conversation identifier to each message that has a reply. For example, if a messaging platform has message A and someone responds to message A with message B, message A is assigned a conversation identifier that can be used to identify the conversation. This yields conversation graph 126, as described in detail below. In some instances, a conversation exists if there is a reply to a message.

Conversation graph 126 may be a hierarchical data structure that represents the messages in a conversation. In some examples, conversation graph 126 includes a non-linear or linear data structure. In some examples, conversation graph 126 includes a tree data structure. Conversation graph 126 may comprise nodes 128 (or vertices) representing messages and edges 130 (or arcs) representing links between nodes 128 . Conversation graph 126 may store a message identifier for each message at each node 128 . In some examples, conversation graph 126 stores the user identifier of the creator of each message at each node 128 . Conversation graph 126 may define one or more branches 132 of nodes 128 . In some examples, branch 132 is a portion (eg, subtree) of conversation graph 126 that includes one or more nodes 128 . In some examples, branch 132 may be two or more nodes 128 connected by edge 130 . Here, one of the nodes 128 is a leaf node. In some examples, branch 132 may be a message connected in a single line (e.g., a leaf message, a first parent message connected to that leaf message, a first parent message connected to that first parent message). (until the parent message does not have another parent message, such as a second parent message that has another parent message).

The term “node” may be referred to as a message within conversation graph 126 and the term “message” may be referred to as node 128 whose message is included as part of conversation graph 126 . A particular node 128 may be linked to another node 128 via an edge 130, the direction of edge 130 identifying the parent message. Node 128 may represent a root message, a message in reply to a root message, a message in reply to a message in reply to a root message, and so on.

The conversation graph manager 136 may generate the conversation graph 126 based on the message's reply structure. A reply structure may be identified based on metadata associated with each message and/or reply information identified from within the message content. In some examples, reply structures are identified based on metadata associated with each message received from client application 154 to compose the message. For example, the user may click on a reply link displayed below a message displayed on the user interface of client application 154 . Client application 154 may then display a message composition box for drafting a reply message. Client application 154 may publish metadata that includes the reply relationship with the reply message (eg, the message identifier of the parent message). In some examples, the reply relationship is explicitly defined by the user in the message content (eg, identifying user account 141 (eg, @tristan) in the body of the message). In this example, the reply structure may be identified by identifying one or more account identifiers and/or message identifiers mentioned within the body of the message.

Timeline manager 142 may transmit digital information over network 150 to enable client application 154 to render and display social content timeline 156 in the user interface of client application 154 . Timeline 156 includes a stream of messages (eg, message A, message B, message C). In some examples, the stream of messages is organized in reverse chronological order. In some examples, the stream of messages is organized in chronological order. In some examples, timeline 156 is a timeline of social content specific to a particular user. In some examples, timeline 156 includes streams curated (eg, generated and assembled) by messaging platform 104 . In some examples, timeline 156 includes a list of messages resulting from searches on messaging platform 104 . In some examples, the timeline 156 is a stream of messages posted by the user from the account 141 associated with the user's account 141 of the client application 154 (e.g., messages the user has chosen to follow on the messaging platform 104). stream of messages from account 141). In some examples, the stream of messages includes promoted or re-shared messages.

When viewing messages in timeline 156, a user may select one of the messages from timeline 156 (eg, message B). This may cause client application 154 to generate and send conversation view request 121 for messaging over network 150 . In some examples, the selected message (eg, message B) may be referred to as a content message or focal message, which may serve as an entry point or point of reference within conversation graph 126 . View conversation request 121 may be a request to retrieve a message from conversation graph 126 . In some examples, view conversation request 121 includes the message identifier of the selected message and the user identifier of the user of client application 154 . In some examples, the conversation view request 121 includes the time of the request, the device the user has turned on, the operating system (OS), and/or other metadata associated with the request.

In response to conversation view requests 121, timeline manager 142 may control prediction manager 110 to generate prediction results 118 for each message (or a subset thereof) of conversation graph 126, where prediction results 118 An engagement value 116 may be calculated for each message based on. For example, prediction manager 110 comprises engagement predictor 125 and engagement scorer 114 . In response to conversation view request 121, engagement predictor 125 obtains prediction-related signal 106 and uses that signal 106 to determine prediction result 118 for each message (or subset of messages) in conversation graph 126. It may be input to predictive model 112 . In some examples, prediction results 118 include mutual engagement probabilities 124 . In some examples, predicted results 118 include mutual engagement probability 124 and one or more of positive engagement probability 120 and negative engagement probability 122 . In some examples, predicted results 118 include mutual engagement probability 124 , positive engagement probability 120 , and negative engagement probability 122 .

Engagement scorer 114 may use prediction results 118 (eg, combine prediction results 118 to produce engagement value 116) to calculate engagement value 116 for each message. The engagement value 116 is used by the timeline manager 142 to rank messages in the conversation graph. Timeline manager 142 may provide messages (or a subset of messages in conversation graph 126 ) as ranked lists 158 in user's timeline 156 over network 150 . Ranked list 158 (eg, message 1, message 2, message 3) includes some or all of the messages of conversation graph 126 ranked according to engagement value 116 determined by prediction results 118. OK.

In some examples, candidate message selector 108 selects a portion of messages in conversation graph 126 to be parsed by prediction manager 110 . In other words, candidate message selector 108 selects candidate messages from a number of messages contained in conversation graph 126 . Here, candidate messages are those messages that are amenable to predictive analysis by the predictive manager 110 . In some examples, the conversation graph 126 is relatively large, and if the number of nodes 128 included in the conversation graph 126 exceeds a threshold level (eg, more than 3000 or 4000 messages), the candidate message selector 108 uses the prediction manager A subset of nodes 128 (rather than all of nodes 128) may be selected for further analysis by 110. FIG. In other words, when the number of nodes 128 exceeds a threshold level, candidate message selector 108 identifies a subset of nodes 128, and the identification of those nodes 128 is provided to prediction manager 110 to make a prediction. . This may reduce (eg, significantly reduce) technical resource consumption and allow messaging platform 104 to deliver more quality responses. For example, if conversation graph 126 has 30,000 responses (or nodes 128), using the techniques described above, messaging platform 104 selects the top X volume messages (eg, X may be 3000). , collect signals for them, score them, and select the top Y amount of messages (eg, Y may be 10) to display to the user. This allows the amount of computational resources to be reduced (eg, significantly reduced) without affecting the quality of the response delivered to the user.

For example, referring to FIG. 1B, candidate message selector 108 may select subset 133 of candidates from conversation graph 126 in response to a number of messages in conversation graph 126 exceeding a threshold level. Here, a prediction result 118 is determined for each message of the candidate subset 133 . In some examples, candidate subset 133 includes a plurality of messages that are less than the total number of messages in conversation graph 126 . In some examples, candidate message selector 108 selects a subset of candidates 133 from one or more candidate message sources 131 . In some examples, candidate message selector 108 may merge candidates for conversation viewing from multiple sources. For example, candidate message selector 108 selects a first set of messages contained in conversation graph 126 from first source 131-1 as a second set of messages contained in conversation graph 126 from second source 131-2. select a set of messages. Here, candidate subset 133 includes the first set of messages and the second set of messages.

In some examples, first source 131-1 includes messages of conversation graph 126 ranked according to most recently posted. In some examples, candidate message selector 108 selects a certain number (n) of the most recent (eg, chronological order) messages in conversation graph 126 from first source 131-1. In some examples, the number (n) is in the range of 2000-4500. In some examples, the number (n) is in the range of 2500-3500. In some examples, second source 131-2 includes messages of conversation graph 126 ranked according to a relevance algorithm (eg, a heuristic algorithm). In some examples, the relevance ranking for the second source 131-2 is based on whether the message in the conversation graph 126 is from a user account 141 linked to the user account 141 of the user of the client application 154. . In some examples, the relevance algorithm may be a message such as the amount of engagement received, the number of likes, comments, and/or signals such as re-shares, whether the message is a photo, video, and/or link. Signals representing metadata and/or health-related metadata such as whether the message is poisonous, workplace warnings (NSFW), or whether the creator of the message has been recently reported. In some examples, candidate message selector 108 selects a certain number (p) of top ranked messages in conversation graph 126 from second source 131-2. In some examples, the number (p) is in the range of 200-1500. In some examples, the number (p) is in the range of 500-1000. In some examples, the number (p) is less than the number (n). The candidate subset 133 may include a number (n) of messages from the first source 131-1 and a number (p) of messages from the second source 131-2. Here, a prediction result 118 is determined for each message of the candidate subset 133 .

At least some of the nodes 128 (or all of the nodes 128) in the conversation graph 126 (or those identified by the candidate message selector 108), the engagement predictor 125 uses the predictive model 112 to predict a positive engagement probability 120, negative engagement probability 122, and/or mutual engagement probability may be predicted. For example, in response to View Conversation Request 121, Prediction Manager 110 controls Engagement Predictor 125 to obtain Signal 106 and apply Signal 106 to Predictive Model 112 to select nodes of Conversation Graph 126. A prediction result 118 may be determined for 128 .

Prediction manager 110 may obtain signals 106 from one or more data services 165 . Data service 165 may be a component on messaging platform 104 that computes or derives data obtained by messaging platform 104 and/or client application 154 . In some examples, prediction manager 110 may communicate with data service 165 through a server communication interface. In some examples, prediction manager 110 may obtain at least some of signals 106 from data service 165 via one or more APIs. In some examples, in response to view conversation request 121, prediction manager 110 sends a thrift call or remote procedure call (RPC) to data service 165 and then transmits at least some of signals 106 to the associated data service. 165 may be received. In some examples, prediction manager 110 may send a representable state transfer (REST) request to data service 165 and then receive at least some of signals 106 from associated data service 165 . In some examples, prediction manager 110 communicates with data service 165 via GraphQL requests. In some examples, prediction manager 110 obtains some of signals 106 from other components of messaging platform 104 including conversation graph manager 136 and/or timeline manager 142 .

Signals 106 may include signals generated by messaging platform 104 and/or generated by client application 154 related to predicting user outcomes for displaying messages to client application 154 . For example, signals 106 may include signals generated by client application 154 based on user interactions with client application 154 . Signals generated by client application 154 may be sent to messaging platform 104 for storage on messaging platform 104 . Signals generated by client application 154 may be engagement information, such as positive user engagement with the message (e.g., favorite, like, reshare) and/or negative user engagement with the message (e.g., report of swear content). In some examples, signals 106 may include signals generated by messaging platform 104 . In some examples, the signals generated by the messaging platform 104 may be data generated from the user's connection graph 134, data generated from the conversation graph 126, data generated from user behavior on the platform (e.g., if the user and/or data generated from the content of the message, such as the results of semantic analysis to predict user sentiment or topical analysis to determine the topic of one or more messages. May contain signals.

As shown in FIG. 1C, signals 106 include a data structure related signal 101 associated with conversation graph 126, a health related signal 103 providing messages from conversation graph 126 to the user of client application 154, a conversation Engagement signal 105 related to user engagement with messages in graph 126; social graph signal 107 related to data from user's connection graph 134; historical aggregation signal 109 related to data aggregated by messaging platform 104; Content-related signals 111 related to the content of the message and/or how similar the message is to other messages that the user has favorited or liked and/or how the user has engaged with the message. may include a similarity signal 113 representing how similar the user is to the user. However, signal 106 may include any type of categorical or granular signal relevant to predicting user outcomes from displayed messages.

Data structure related signals 101 may include signals related to data from conversation graph 126 . In some examples, data structure related signals 101 include the number of nodes 128, the number of edges 130, the number of branches 132, the length or size of each branch 132, the number of parent nodes, the number of child nodes, the number of leaf nodes. , the height of the conversation graph 126 (e.g., the length of the longest path to a leaf node), and/or the depth of the nodes (e.g., the depth of a node is the length of the path to the root node) may include a signal representing In some examples, data structure related signals 101 include one or more signals representing the number of unique authors in conversation graph 126 or subsets of conversation graph 126 , such as branches 132 . In some examples, data structure-related signals 101 include signals representing locations of messages having some type of data (eg, images, videos, links to videos, etc.) within conversation graph 126 . In some examples, for a particular message in conversation graph 126, data structure related signal 101 indicates whether the message is a child node, whether the message is a parent node, whether the message is a leaf node, whether the conversation Signals representing the location of the message within graph 126, the location of branch 132 containing the message, the size of branch 132 containing the message, and the depth of the message within conversation graph 126 may be included.

Data structure related signals 101 may include branch context features. In some examples, data structure related signals 101 may include the number of replies within branch 132, the number of conversations within branch 132, the number of conversations within branch 132 between the user of client application 154 and the originator of the root message. number, the number of conversations within branch 132 between users of application 154 and users mentioned in new messages, and/or the number of conversations between a particular node (e.g., a focal message) and a leaf node. contains a signal that represents In some examples, for branch context features, a conversation may be defined as a round trip between two or more users. In some examples, a conversation may be defined as a message posted by user A, a reply posted by user B, and then a reply posted by user A.

In some examples, the conversation graph manager 136 may receive the conversation identifier from the prediction manager 110, derive or determine the data structure related signal 101 from the conversation graph 126 according to the conversation identifier, and transfer the data structure related signal 101 to the messaging platform. It may be stored in data storage at 104 . For example, in response to conversation view request 121, prediction manager 110 controls conversation graph manager 136 to derive or determine data structure related signals 101, which are then used by prediction models 112 to determine prediction results 118. As such, data structure related signals 101 may be received from conversation graph manager 136 . In some examples, prediction manager 110 may derive or determine data structure-related signals 101 from conversation graph 126 . In some examples, in response to view conversation request 121, prediction manager 110 may send a conversation identifier to conversation graph manager 136, which then derives or determines data structure related signals 101 from conversation graph 126. Conversation graph 126 may be received as follows.

Health-related signals 103 may include signals representing the health of presenting messages of conversation graph 126 to a user of client application 154 . In some examples, health-related signal 103 may include a signal representing whether a user of client application 154 has restricted (eg, blocked, muted, etc.) authors of messages in past conversation graph 126 . Health-related signals 103 may be stored in data storage in messaging platform 104 . In some examples, prediction manager 110 may send a request to data service 165 (eg, health data service) to obtain health-related signal 103 . Here, the request may include the message identifier of the message of conversation graph 126 and/or the user identifier of the user of client application 154 .

Engagement signals 105 may represent user engagement data associated with messages in conversation graph 126 . In some examples, the engagement signal 105 is the number of engagements for a message in the conversation graph 126 (e.g., the number of times the message has been favorited or liked, the number of replies to the message, the number of times the message has been re-shared). ). In some examples, the engagement signal 105 is the engagement of the user following the user of the client application 154 in the user's connection graph 134 (e.g., the message follows the user of the client application 154 in the user's connection graph 134). has one or more engagements provided by a user who is In some examples, prediction manager 110 obtains engagement signal 105 from data service 168 that stores engagement data. In some examples, prediction manager 110 may send a request that may include a message identifier for conversation graph 126 . Prediction manager 110 may receive engagement signal 105 from data service 165 .

Social graph signals 107 may include signals representing information from connection graph 134 . In some examples, the social graph signal 107 indicates the number of times a user of the client application 154 has favorited or liked the message's creator's message over time, the user's link to the message's creator in the connection graph 134 . and/or a signal representing the number of times the user has re-shared or replied to the message's author's message over a period of time. In some examples, prediction manager 110 obtains social graph signals 107 from data service 168 that stores social graph signals. In some examples, prediction manager 110 may send a request that may include the user identifier of the user of client application 154 . Prediction manager 110 may receive social graph signals 107 from data service 165 .

Historical aggregated signals 109 may include signals representative of user behavior on messaging platform 104 . In some examples, history aggregation signal 109 may be the number of times the user of client application 154 favorited a message on messaging platform 104 over a period of time, the number of times the user of client application 154 favorited a message on messaging platform 104 over a period of time, A signal representing the number of times it has been re-shared and/or the number of times a user of client application 154 has replied to a message on messaging platform 104 over a period of time may be included. The period may be the previous day, the last month, the last year, or the like. In some examples, history aggregation signal 109 counts the number of times a user of client application 154 favorited, liked, re-shared, and/or replied to a message containing an image or video. may include a signal representing

In some examples, the history aggregation signal 109 indicates that the user of the client application 154 has favorited, liked, re-shared, and/or messages from accounts 141 linked to the user in the connection graph 134 . or the number of times the message was replied to, and/or the user favorited, liked, re-shared, and/or messages from accounts 141 not linked to the user in connection graph 134 A signal may be included that indicates the number of times the message has been replied to. In some examples, prediction manager 110 obtains historical aggregated signal 109 from data storage in messaging platform 104 . In some examples, prediction manager 110 sends a request to data service 165 to obtain historical aggregated signal 109 . In some examples, the request includes the user identifier of the user of client application 154 . In some examples, historical aggregated signal 109 includes batch aggregated information and real-time aggregated information. Batch aggregate information may include a relatively long history (eg, over 50 days). In some examples, batch aggregate information may not include interactions from the previous day (or the last few days). Real-time aggregate information may include relatively recent interaction history (eg, within the last 30 minutes, etc.).

Content-related signals 111 may include signals representing one or more aspects of the content of messages in conversation graph 126 . In some examples, content-related signals 111 may include signals that indicate the length of the message and/or whether the content includes text, video, or images. In some examples, prediction manager 110 obtains content-related signals 111 from data storage in messaging platform 104 . In some examples, prediction manager 110 sends a request to data service 165 to obtain content-related signal 111 . In some examples, the request includes the message identifier of the message in conversation graph 126 .

Similarity signals 113 may include one or more signals representing how similar a message is to other messages that a user has favorited or liked. For example, similarity signal 113 may represent the level of similarity between a particular message and one or more messages that the user has favorited or liked. If the level of similarity is relatively high, the similarity signal 113 provides an indication of potential positive engagement. In some examples, similarity signal 113 may include one or more signals representing how similar a user is to other users engaged with the message. For example, determining that a user's user profile is relatively similar to a user profile that has engaged with a message may provide an indication of potential positive engagement. In some examples, prediction manager 110 may obtain affinity signal 113 from data storage in messaging platform 104 . In some examples, prediction manager 110 sends a request to data service 165 to obtain affinity signal 113 . In some examples, the request may include the message identifier and/or the user's user identifier.

In some instances, there are technical difficulties or hurdles to obtain at least some of the signals 106 used for prediction (e.g., the relationship between viewers and authors, especially if the message is spread). for signals associated with ). Popular messages may have a relatively high volume of responses (eg, over 80,000 in some cases). This means that many users can try and view popular messages at the same time. For each viewer, messaging platform 104 may obtain its relationship with all authors who have replied to the popular message. Using the techniques described above for candidate message selector 108, messaging platform 104 may be able to filter a total number of messages from 80,000 to 4,000. This may also mean that there may be 4,000 viewer-author pairs to obtain a relationship signal.

Also, in some instances, a viewer-author relationship may not exist because the viewer does not follow the author. To handle these kinds of situations, instead of querying by viewers and creators as keys to data service 166, messaging platform 104 queries by viewer identifiers and their relationships with all other creators. can be obtained at once. The messaging platform 104 can then determine overlap with the creator of the creator's reply, if any, and maintain associated signals. This reduces calls through the network to a relatively large extent because the prediction manager 110 can generate and send one call instead of 4,000 calls per viewer.

Another technical difficulty can exist with message level signals. For example, in a large conversation, messaging platform 104 may query other data services 166 with 4,000 queries per viewer. This can create a "hotkey" problem where the data service 166 receives too many queries for the same message identifier. To overcome the difficulties identified above, messaging platform 104 may employ in-memory caching. The service caches the feature in memory if the underlying data service 166 indicates a hotkey. For example, a message T is spread and has responses R1, R2...R4000 and the message feature is the number of characters in the message. Then 1000 users send requests for the same message at the same time (or near the same time). If the data service 166 indicates a hotkey, the messaging platform 104 can store the character values for R1, R2 . , it is possible to just use them.

Predictive model 112 is a predictive model trained by one or more machine learning algorithms 149 input with training data 148 . Machine learning algorithms 149 may include one or more of Markov models, logistic regression, decision tree analysis, random forest analysis, neural nets, and combinations thereof. In general, machine learning is the field in which a computer learns to perform a class of tasks using feedback generated from the experience or data that the machine learning process acquires while performing those tasks on the computer. Supervised machine learning allows a computer to learn one or more roles or functions to map between exemplary inputs and desired outputs in a manner predetermined by an operator or program. be. The labeled data points can then be used in training a computer. Unsupervised machine learning can involve using unlabeled data. The computer can then identify implicit relationships in the data, for example, by reducing the dimensionality of the data set.

As shown in FIG. 1D, predictive models 112 may include positive engagement models 115, negative engagement models 117, and mutual engagement models 119. As shown in FIG. Positive engagement model 115 is configured to calculate positive engagement probability 120, negative engagement model 117 is configured to calculate negative engagement probability 122, and mutual engagement model 119 is configured to: It is configured to calculate mutual engagement probability 124 . For example, in response to conversation view request 121 (eg, user selects message B), engagement predictor 125 obtains signal 106 and assigns signal 106 (which also includes user identifier and message identifier) to a positive engagement model. 115, negative engagement model 117, and/or mutual engagement model 119 to determine positive engagement probability 120, negative engagement probability 122, and/or mutual engagement probability 124, respectively.

A positive engagement probability 120 indicates a probability value that a user is expected to positively view or engage with the message. In some examples, the probability value for positive engagement probability 120 is a number (x) between the first value and the second value. Here, the first value represents a probability of 0 that the user is predicted to view or engage with the message positively, and the second value represents the probability that the user will view the message positively. Or represents a 100% chance of being predicted to engage. In some examples, the probability value for positive engagement probability 120 is a positive number. In some examples, the first value is 0 and the second value is 1. However, the first and second values may define any kind of range (eg, 0-1, 0-50, 0-100, etc.). In other words, the positive engagement probability 120 indicates the level of likelihood that a user is expected to favorite, like, or share the message.

Negative engagement probability 122 indicates the probability value that a user is expected to negatively view or engage with the message. In some examples, the probability value for negative engagement probability 122 is a number (y) between the first value and the second value. Here, the first value represents a probability of 0 that the user is predicted to view or engage with the message negatively, and the second value represents the probability that the user will view the message negatively. Or represents a 100% chance of being predicted to engage. In some examples, the probability value for negative engagement probability 122 is a negative number. In some examples, the first value is 0 and the second value is negative one. However, the first and second values may define any kind of range (eg, 0 to -1, 0 to -50, 0 to -100, etc.). In other words, negative engagement probability 122 indicates the level of likelihood that a user will block the message creator, unfollow the message creator, and/or report the message as abusive.

Mutual engagement probability 124 indicates a probability value that the user is expected to continue to develop conversation graph 126 . In some examples, the probability value for mutual engagement probability 124 is a number (z) between the first value and the second value. Here, the first value represents a probability of 0 that the user is expected to continue to develop the conversation graph 126 and the second value is 100% that the user is expected to continue to develop the conversation graph 126. represents the possibility of In some examples, the probability value for mutual engagement probability 124 is a positive number. In some examples, the first value is 0 and the second value is 1. However, the first and second values may define any kind of range (eg, 0-1, 0-50, 0-100, etc.). In some examples, mutual engagement probability 124 indicates the level of likelihood that users are expected to further develop conversation graph 126 by replying to messages.

For a particular candidate node 128, engagement predictor 125 uses positive engagement model 115 to determine that the message has a certain probability of receiving positive engagement by the user, and negative engagement model 117 to determine , may determine that the message has a certain probability of receiving negative engagement by the user, and using the mutual engagement model 119, determine that the message has a certain probability of receiving mutual engagement by the user. In some examples, for each candidate node 128, engagement predictor 125 determines all three prediction outcomes 118. In some examples, for each node 128 , engagement predictor 125 predicts mutual engagement probability 124 and one or more of positive engagement probability 120 and negative engagement probability 122 .

Engagement scorer 114 uses prediction results 118 to calculate engagement values 116 for messages in conversation graph 126 . Engagement value 116 may provide an overall engagement value for each node 128 (eg, indicating a level of relevance for the user of client application 154). This encourages healthier conversations on messaging platform 104 . For example, for a particular candidate node 128, engagement scorer 114 combines positive engagement probability 120, negative engagement probability 122, and cross-engagement probability 124 to select the most relevant node 128 for the user. An engagement value 116 may be provided that may be used. For example, engagement scorer 114 may combine the values of predicted results 118 to determine engagement value 116 for a particular message. If the probability value of negative engagement probability 122 is relatively high (eg, has a larger negative value), this value may offset the positive values of positive engagement probability 120 and mutual engagement probability 124 . In a simple example, if the positive engagement probability 120 is +10, the negative engagement probability 122 is -10, and the mutual engagement probability 124 is +10, the engagement value 116 for the message is +10.

In some examples, the engagement scorer 114 may apply weightings to the predicted results 118 and then determine the engagement value 116 based on the weighted positive engagement probabilities 120, negative engagement probabilities 122, and mutual engagement probabilities 124. can be calculated. In some examples, the weighting applied to mutual engagement probability 124 is greater than the weighting applied to negative engagement probability 122 .

The engagement value 116 is used to select relevant messages or branches of messages within the conversation graph 126 to be rendered to the user. For example, timeline manager 142 receives engagement values 116 from prediction manager 110 and uses engagement values 116 to rank messages in conversation graph 126 (eg, highest to lowest). Timeline manager 142 may provide at least a subset of the messages of conversation graph 126 rendered in timeline 156 according to rank over network 150 . In some examples, timeline manager 142 only provides a subset of the messages of conversation graph 126 rendered in timeline 156 . Here, the subset contains the higher ranked messages of conversation graph 126 . Timeline manager 142 may then receive a request from client application 154 for additional messages in conversation graph 126 (eg, select a user affordance to view more messages in conversation graph 126). Timeline manager 142 may select the next group of messages from conversation graph 126 to be sent to client application 154 . In this manner, messaging system 100 may collapse portions of conversation graph 126 that are less likely to provide positive engagement, but present those messages when requested by a user.

In some examples, timeline manager 142 selects one or more branches (or a subset of branches 132) of conversation graph 126 to be rendered in the timeline using engagement value 116 . For example, branch 132 includes one or more nodes with high engagement values 116 (or engagement values 116 above a threshold level). The timeline manager 142 provides the user with more context about the conversation, despite the fact that the branch 132 may contain one or more nodes 128 with low engagement values 116 (or engagement values 116 below the threshold level). The entire branch 132 may be selected to be rendered as part of the message delivered to the client application 154 so as to provide . In some examples, a particular branch 132 is associated with a total engagement value, which may be an average of engagement values 116 for nodes 128 within the particular branch 132 . The timeline manager 142 may then rank the branches 132 according to their overall engagement value.

In some examples, timeline manager 142 selects nodes 128 with high engagement values (or engagement values 116 above a threshold level) for inclusion in the set of messages provided to client application 154 . In some examples, the timeline manager 142 ranks the selected branches 132 and/or nodes 128 according to engagement value 116 from highest to lowest (e.g., here the branch 132 with the highest engagement value 116 or node 128 is presented to the user first). Because messaging platform 104 incorporates (or predicts) mutual engagement probabilities 124 within its scoring algorithm, messaging platform 104 encourages more conversations on messaging platform 104 .

In some examples, ranked list 158 represents a subset of messages in conversation graph 126 determined to be relevant to the user. For example, some messages in conversation graph 126 may be relevant to a first user, while other messages in conversation graph 126 may be relevant to a second user. In contrast, some conventional approaches use vote-based mechanisms that can provide identical views for each. More specifically, the engagement predictor 125 may obtain signals 106 (e.g., engagement history, connection graph data, etc.) associated with the first user, and for each message in the conversation graph 126, then the engagement value 116. Predicted results 118 used to compute may be obtained. The timeline manager 142 may receive the engagement value 116 from the prediction manager 110 and then apply the engagement value 116 (which may increase the likelihood that the user will continue the conversation (eg, by incorporating mutual engagement probability 124)). may be used to rank messages in conversation graph 126 .

However, for a second user, the engagement predictor 125 may obtain the signal 106 associated with the second user and then predictive results tailored to the second user used to calculate the engagement value 116. 118 may be obtained. Timeline manager 142 may then rank messages in conversation graph 126 using engagement value 116 . Thus, the ranked list 158 displayed in the client application 154 for the second user is different from the messages in the ranked list 158 displayed in the client application 154 for the first user. can be different.

Messaging platform 104 includes a predictive model trainer 140 that trains predictive models 112 (eg, positive engagement model 115, negative engagement model 117, and mutual engagement model 119). Messaging platform 104 provides trained predictive model 112 to engagement predictor 125 so that engagement predictor 125 can determine predicted outcome 118 when determining conversation viewing. In some examples, predictive model trainer 140 is configured to run periodically (eg, daily, weekly, monthly) to retrain (and thereby update) predictive model 112 . In some examples, predictive model trainer 140 operates in offline mode to train predictive model 112 using training data 148, and is then employed in online mode when predictive manager 110 is active. As such, the prediction model 112 is sent to the prediction manager 110 . In some examples, predictive model trainer 140 sends the weightings and biases of positive engagement model 115 , negative engagement model 117 , and cross-engagement model 119 to prediction manager 110 .

In some examples, predictive model trainer 140 executes on server computer 102 . In some examples, predictive model trainer 140 executes on a computing device separate from server computer 102 . Here, predictive model trainer 140 and engagement predictor 125 communicate with each other over a networking interface. As shown in FIG. 1D, predictive model trainer 140 trains predictive model 112 using training data 148 according to one or more machine learning algorithms 149 . Training data 148 may include one or more (or any combination) of the data described for signal 106 . The data may be historical data from a previous period of time, eg, the last day, the last month, the last year, and so on. For example, training data 148 includes data structure related signals 101 , health related signals 103 , engagement signals 105 , social graph signals 107 , history aggregation signals 109 , and content related signals 111 .

FIG. 1E illustrates an example of obtaining training data 148 for training predictive model 112 according to one aspect. For example, timeline manager 142 may inject a stream of messages to client application 154 . As part of one or more stream injections, timeline manager 142 may store information about message injections in injection log 127 . Injection log 127 may identify which messages were provided to client application 154 and may identify certain features associated with the injection. In some examples, the features may include information related to any of the signals 106 described above. Additionally, injection log 127 may include details about prediction manager 110 , such as prediction results 118 and/or engagement values 116 associated with one or more conversation graph 126 messages. As the user interacts with the message in the client application 154, the client event log 123 captures user engagement information such as whether the user has replied to the message, favorited or liked the message, and re-shared the message. Remember. In some examples, client event log 123 includes information such as whether the message has a photo or video and/or whether the message came from some depth in conversation graph 126 . Information in client event log 123 may be sent to messaging platform 104 for storage in messaging platform 104 . Training data 148 may include information from client event log 123 and information from injection log 127 . Training data 148 is used to train predictive model 112 . In some examples, predictive model trainer 140 periodically trains predictive model 112 by offline analysis and sends training results (eg, weights/biases) to predictive manager 110 .

In some examples, messaging platform 104 is configured to perform a two-level ranking mechanism for messages in conversation graph 126 . For example, a two-level ranking mechanism may provide an effective mechanism for dealing with swearing in conversations on messaging platform 104 . In some examples, the two-level ranking includes hard ranking and soft ranking. In some examples, hard ranking divides messages in conversation graph 126 into different sections (or groups) based on the level of confidence that the message is profanity and/or that the author is profane. including sectioning with In some examples, hard rankings are not personalized from the perspective of the user requesting the View Conversation Request 121 . Rather, hard ranking may be done using signals about the content itself and/or signals about the creator of the message. In some examples, soft ranking includes computing predicted results 118, generating engagement values 116 for messages in conversation graph 126, and then ranking messages in each section according to engagement values 116. including. In some examples, soft rankings are personalized from the perspective of the user requesting the View Conversation Request 121, as described above. In this way, messages from higher quality sections are presented to users ranked according to engagement value 116 . In some examples, hard ranking is performed first, followed by soft ranking. However, in some examples, hard rankings can be personalized by using the output of predictive model 112 to do hard rankings.

As shown with respect to FIGS. 1A and 1F, in some examples, messaging platform 104 may classify (eg, section) messages of conversation graph 126 into multiple sections 160 based on the content quality of the messages. may comprise a content quality classifier 147 configured as: The content quality of a message may include the probability (or belief) of being abusive (or poisonous) at the message level or author level. In some examples, content quality classifier 147 may determine (or receive) a probability of being an abusive for a particular message. (For example, this may be based on the probability that the content itself may be abusive or the creator or account generating the message may be linked to abusive behavior). If the probability of being an abusive is relatively high (or the level of confidence that the message and/or creator is considered an abusive is relatively high), the content quality classifier 147 classifies the message into one section 160 (e.g., low quality). or swearing section). Alternatively, if the probability of being an abusive is relatively low (or the level of confidence that the message and/or creator is considered an abusive is relatively low), the content quality classifier 147 classifies the message into another section 160 (e.g., high quality or non-swear section). In other words, content quality classifier 147 is configured to divide node 128 (or branch 132 ) of conversation graph 126 into multiple sections 160 . Each section 160 may refer to a different categorization (or classification) of quality level (or cursed (or poisonous) level). Content quality classifier 147 may determine that the first message is of relatively low quality and the second message is of relatively high quality. Here, content quality classifier 147 assigns the first message (or branch 132 containing the first message) to the first section (eg, the first section is considered a low quality section). Well, the second message (or branch 132 containing the second message) may be assigned to the second section (eg, the second section is considered a high quality section).

The classification or sectioning performed by content quality classifier 147 may be considered the first part (eg, hard ranking) of a two-level ranking mechanism. In some examples, section 160 may include two or more different sections, each representing a separate quality category or classification. In some examples, section 160 includes low quality section 162 , medium quality section 164 , and high quality section 166 . In some examples, the term section may refer to a category or taxonomy. Although three sections 160 are shown in FIG. 1F, sections 160 may include any number of sections, including two sections or any number greater than three. After being classified by content quality classifier 147, low quality section 162 contains or identifies nodes 128 (or branches 132) from conversation graph 126 that have been determined to be of low quality, and medium quality section 164 contains or identifies nodes of medium quality. and high quality section 166 contains or identifies nodes 128 (or branches 132) from conversation graph 126 that have been determined to be of high quality. contains or identifies In some examples, medium quality section 164 includes messages that may be abusive. Alternatively, there is a probability that the account posting the message could be linked with swearing behavior with moderate confidence. In some examples, a low quality section 162 may be reported and reviewed by a bystander, or a low quality section 162 may contain untrustworthy links, or an account posting a message may have a high probability of abusive behavior. Contains messages that would likely be judged to be profanity if linked to.

In some examples, content quality classifier 147 includes a classification model 161 (or models) that classifies nodes 128 (or branches 132 ) of conversation graph 126 into sections 160 . In some examples, classification model 161 is a machine learning model that is trained using training data applied to one or more machine learning algorithms. In some examples, classification model 161 includes a neural network. Training data may be obtained over a period of time (eg, a day, a week, a month, a year, etc.) and then used to train (or retrain) the classification model 161 in offline mode. . The updated classification model 161 is now sent to the content quality classifier 147 .

Content quality classifier 147 obtains one or more signals 163 associated with classification model 161 to determine which messages in conversation graph 126 are classified into low quality section 162, medium quality section 164, or high quality section 166. That signal 163 may be input to the classification model 161 so as to determine whether the For a particular message (e.g., node 128) of conversation graph 126, the output of classification model 161 may identify a quality classification, e.g., either low quality section 162, medium quality section 164, or high quality section 166. . In some examples, nodes 128 (or branches 132 ) of conversation graph 126 are annotated with sections 160 determined by classification model 161 . In some examples, content quality classifier 147 stores the classification of node 128 (or branch 132 ) in a data store on messaging platform 104 .

To classify a particular message in conversation graph 126, in some examples, signal 163 is one or more signals generated by client application 154 and/or one or more signals generated by messaging platform 104. signal. In some examples, signal 163 includes one or more signals that are different from signal 106 used to determine prediction result 118 . In some examples, signal 163 includes one or more signals representing the content of the message. In some examples, signal 163 includes one or more signals representing the user history of the author who generated the content. In some examples, signals 163 include one or more signals that model the user profile of the author who generated the content. In some examples, signals 163 include one or more signals that model author behavior on messaging platform 104 . In some examples, signals 163 may include signals representing message text, user behavior history such as logging history, type of device the user is using, how many times the user has been reported as swearing, and the like. Based on these signals 163, the content quality classifier 147 may predict whether the message is profanity and/or whether the user is prone to profanity. Once those probabilities (or beliefs or scores) are determined, the content quality classifier 147 can make sectioning decisions. For example, if a message is abusive with high confidence and the user who wrote the message is also abusive with high confidence, content quality classifier 147 classifies the message into the lowest quality section, e.g. you can

In some examples, content quality classifier 147 may receive the message identifier of the message and/or the user identifier of the author who generated the content. In response to the message identifier and/or user identifier, content quality classifier 147 may obtain signal 163 that is input to classification model 161 . In some examples, content quality classifier 147 may use the message identifier to retrieve the content of the message from message repository 138 . In some examples, instead of receiving a message identifier, content quality classifier 147 may receive the content of the message. In some examples, the content quality classifier 147 applies one or more categorizations to authors by communicating with one or more data services 165 through any of the techniques described with reference to FIG. 1C. A user identifier may be used to acquire the signal. In some examples, the categorization of a message into one of multiple sections 160 may occur at the time of message generation. In some examples, message categorization occurs at (or near) receipt of the view conversation request 121 . In some examples, content quality classifier 147 may convert signal 163 into a format compatible with classification model 161 .

In some examples, timeline manager 142 and/or prediction manager 110 may communicate with content quality classifier 147 through a server communication interface (eg, Thrift calls, REST calls, GraphQL requests, etc.). Here, the message identifier and/or user identifier are sent to content quality classifier 147 . The content quality classifier 147 annotates the node 128 (or branch 132) of the conversation graph 126 with the determined classification and/or passes the classification to the service requesting the classification information via the server communication interface. return. Content quality classifier 147 provides signal 163 to predictive model 161 to determine whether the message has a content quality corresponding to low quality section 162, medium quality section 164, or high quality section 166. good. In some examples, unlike predicted results 118 , the categorization of messages into sections 160 is not personalized to the user associated with view conversation request 121 .

In response to view conversation request 121, timeline manager 142 controls prediction manager 110 to generate prediction results 118 for each message (or subset of messages) of conversation graph 126, as already described above. and an engagement value 116 may be calculated for each message based on the predicted results 118 . For example, engagement predictor 125 obtains prediction-related signal 106 and inputs signal 106 into predictive model 112 to determine predicted outcome 118 for each message (or subset of messages) in conversation graph 126. good. In some examples, prediction results 118 include cross-engagement probabilities 124 . In some examples, predicted results 118 include mutual engagement probabilities 124 and one or more of positive engagement probabilities 120 and negative engagement probabilities 122 . In some examples, predicted results 118 include mutual engagement probability 124 , positive engagement probability 120 , and negative engagement probability 122 .

Engagement scorer 114 may use prediction results 118 (eg, combine prediction results 118 to produce engagement value 116) to calculate engagement value 116 for each message. The engagement value 116 is then used by the timeline manager 142 to rank the messages into low quality section 162, medium quality section 164, and high quality section 166 separately.

In some examples, the timeline manager provides a list of messages classified as low quality section 162 from content quality classifier 147, a list of messages classified as medium quality section 164 from content quality classifier 147, and h Content quality classifier 147 may receive a list of messages classified as high quality section 166 . The timeline manager may also receive engagement values 116 from the forecast manager 110 . Ranking messages within a particular section 160 using engagement values 116 for messages within the respective section 160 may be considered a second part of the two-level ranking mechanism of FIG. 1F. In some examples, the second part of the two-level ranking mechanism may be considered a soft ranking of messages.

In some examples, timeline manager 142 generates first ranked list 158-1 of messages categorized in low quality section 162, ranked using engagement value 116 from prediction manager 110. may be generated. In some examples, first ranked list 158-1 includes branches 132 of conversation graph 126 classified as low quality. Branches 132 are ranked within the poor quality section 162 according to their engagement value 116 . Timeline manager 142 may generate a second ranked list 158 - 2 of messages categorized in medium quality section 164 ranked using engagement values 116 from prediction manager 110 . In some examples, second ranked list 158-2 includes branches 132 of conversation graph 126 classified as medium quality. Branches 132 are ranked within medium quality section 164 according to their engagement value 116 . Timeline manager 142 may generate a third ranked list 158 - 3 of messages categorized in high quality section 166 ranked using engagement values 116 from prediction manager 110 . In some examples, third ranked list 158-3 includes branches 132 of conversation graph 126 classified as high quality. Branches 132 are ranked within high quality section 166 according to their engagement value 116 .

Timeline manager 142 transmits at least a subset of messages in high quality section 166 (eg, at least a portion of third ranked list 158-3) according to rank rendered at client application 154. you can In this manner, a user of client application 154 may view messages that are considered high quality that are also ranked according to the message's engagement value 116 . Timeline manager 142 may request subsequent requests to view additional messages from conversation graph 126 . Here, timeline manager 142 may send at least a portion of second ranked list 158-2 followed by at least a portion of first ranked list 158-1.

In some examples, the timeline manager 142 selects the third ranked list 158-3, the second ranked list 158-2, followed by the first ranked list 158-1. Rank the messages in the conversation graph 126 according to order. Thus, client application 154 first renders messages from high quality section 166 ranked according to engagement value 116, renders messages from medium quality section 164 ranked according to engagement value 116, and finally , render messages from the low quality section 162 ranked according to engagement value 116 . If the high quality section 166 contains 10 messages, the medium quality section 164 contains 10 messages, and the low quality section 162 contains 10 messages, and the threshold of displayed messages is 15. , the client application 154 renders the entire third ranked list 158-3 and the five messages of the second ranked list 158-2. Timeline manager 142 may receive requests to display additional messages in conversation graph 126 . The timeline manager 142 may now send the second ranked list 158-2, followed by the other five messages of the first ranked list 158-1.

In some examples, hard rankings can be personalized by using the output of predictive model 112 to do hard rankings. For example, prediction manager 110 may determine prediction results 118 and engagement values 116 as described above. Content Quality Classifier 147 may use the output of Prediction Manager 110 to classify (and/or rank) messages according to a hard ranking technique, which is from the perspective of the user who initiated View Conversation Request 121. Personalized.

FIG. 2 illustrates a neural network 219 according to one aspect. Neural network 219 may be an example of mutual engagement model 119 of messaging system 100 of FIGS. 1A-1D. However, the features described with reference to FIG. 2 may be applied to any of predictive models 112 including positive engagement model 115 and negative engagement model 117 . In some examples, neural network 219 may be an example of classification model 161 of FIG. 1F. Neural network 219 is configured to output mutual engagement probability 224 . Cross-engagement probability 224 may be an example of cross-engagement probability 124 in FIG. 1A. Neural network 219 may be an interconnected group of nodes 260, where each node 260 represents an artificial neuron. Nodes 260 are connected to each other by layers, with the output of one layer being the input of the next layer. Neural network 219 transforms the input X ₁ , X ₂ -X _n (eg, signal 106) received by input layer 262 and feeds it to one or more hidden layers 264 (eg, FIG. ) to produce an output Y ₁ (eg, mutual engagement probability 124 ) by output layer 266 . Each layer consists of a subset of the set of nodes 260 .

Using neural network 219 to obtain mutual engagement probabilities 224 includes applying weighted and biased numerical inputs to interconnected nodes 260 in neural network 219 and calculating their outputs. good. The weightings and biases applied to each node 260 in neural network 219 are obtained by training neural network 219, for example, using machine learning algorithm 149 (eg, by predictive model trainer 140 of FIG. 1A). good. Nodes 260 in neural network 219 may be organized into two or more layers, including at least an input layer 262 and an output layer 266 . In a multi-layer neural network 219, the output from one layer may serve as the input to the next layer. Layers with no external output connections may be referred to as hidden layers 264 . The output of each node 260 is a function of the weighted sum of its inputs and bias.

A vector of feature values (X ₁ . . . X _N ) is applied as input to each node 260 in input layer 262 to obtain mutual engagement probabilities 224 . In some examples, the vector of feature values (X ₁ . . . X _N ) includes the values of signal 106 described above. Input layer 262 distributes values to each of the nodes 260 in hidden layer 264 . Upon reaching node 260 in hidden layer 264, the value from each input node is multiplied by a weight, and the resulting weighted values are summed and added to the weighted bias value to produce a combined value. The bound value is passed through a transformation or activation function that outputs the value. The output from hidden layer 264 is then distributed to nodes 260 in output layer 266 of neural network 219 . Upon reaching node 260 in output layer 266, the values from each hidden layer node are multiplied by a weight and the resulting weighted values are summed and added to the weighted bias value to produce the combined value. . The combined values are passed through a transformation or activation function that outputs Y ₁ (eg, mutual engagement probability 224).

FIG. 3 depicts a flowchart 300 illustrating exemplary operation of the messaging platform 104 for ranking conversation graph messaging according to an aspect.
Operation 302 includes receiving conversation view request 121 to retrieve messages of conversation graph 126 stored in messaging platform 104 executable by server computer 102 over network 150 . Act 304 includes messaging platform 104 using predictive model 112 to determine a plurality of predicted outcomes for each of the plurality of messages in conversation graph 126 . Here, multiple prediction results 118 include mutual engagement probabilities 124 . Act 306 includes messaging platform 104 ranking the plurality of messages based on predicted results 118 . Act 308 includes messaging platform 104 sending at least a subset of the plurality of messages rendered in client application 154 according to rank over network 150 .

FIG. 4 illustrates exemplary operation of a messaging platform for ranking messages of a conversation graph according to one aspect.
Operation 402 includes receiving conversation view request 121 to retrieve messages of conversation graph 126 stored in messaging platform 104 executable by server computer 102 over network 150 . Operation 404 includes determining, in which messaging platform 104 uses predictive model 112 to determine a plurality of predicted outcomes for each of the plurality of messages in conversation graph 126 . The determining step may include obtaining a plurality of signals 106 that are relevant to predictive model 112 . Here, the plurality of signals includes data structure related signals 101 associated with the conversation graph 126 and inputs the plurality of signals 106 to the predictive model 112 to determine the plurality of predicted outcomes 118 . Act 406 includes messaging platform 104 ranking the plurality of messages based on predicted results 118 . Act 408 includes messaging platform 104 sending at least a subset of the plurality of messages rendered in client application 154 according to rank over network 150 .

FIG. 5 depicts a flowchart 500 illustrating exemplary operation of a messaging platform for ranking conversation graph messaging in accordance with an aspect.
Operation 502 includes messaging platform 104 classifying messages in conversation graph 126 into multiple sections 160 based on the content quality of the messages. Here, multiple sections 160 include a first section and a second section. The first section has messages from conversation graph 126 determined to be of higher quality than the messages of the second section. Operation 504 includes determining, in which messaging platform 104 uses predictive model 112 to determine a plurality of predicted outcomes for each of the plurality of messages in conversation graph 126 . Act 506 includes messaging platform 104 ranking the first section messages based on predicted results 118 for the first section messages. Act 508 includes messaging platform 104 sending at least a subset of the messages of the first section rendered in client application 154 according to rank.

Numerous details have been described above. However, it will be apparent to one skilled in the art having the benefit of this disclosure that implementations of the disclosure may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the description.

Some portions of the detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally thought here to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

Note, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, it should be apparent from the discussion above that throughout the description "identify", "determine", "calculate", "update", "transmit", " Discussions using terms such as "receive", "generate", and "modify" refer to data represented as (e.g., electronic) physical quantities in computer system registers and memory as computer system memory or registers. the action of a computer system or similar electronic computing device that manipulates and transforms it into other data similarly represented as physical quantities within or in other such information storage, transmission, or display devices; It is understood that reference is made to processes.

Implementations of the disclosure also relate to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such computer programs may be stored on any kind of disk, including floppy disk, optical disk, CD-ROM and magneto-optical disk, read only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, flash It may be stored in a non-transitory computer-readable storage medium, such as, but not limited to, memory or any type of medium suitable for storing electronic instructions.

The words "example" or "exemplary" are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "example" or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word "example" or "exemplary" is intended to present concepts in a concrete fashion. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise, or clear from context, "X comprises A or B" is intended to mean any of its natural inclusive permutations. That is, if X contains A, X contains B, or X contains both A and B, then "X contains A or B" is satisfied under any of the above cases. . In addition, as used in this application and the appended claims, the articles "a" and "an" are used unless otherwise specified or clear from the context that they are directed to the singular , should generally be taken to mean "one or more". Further, use of the terms "implementation" or "one embodiment" or "implementation" or "one implementation" throughout is intended to mean the same embodiment or implementation unless so stated. not Moreover, as used herein, the terms “first,” “second,” “third,” “fourth,” etc. are meant as labels to distinguish different elements. and may not necessarily have an ordinal meaning according to their numerical designation.

The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. I know something. The required structure for a variety of these systems will appear from the description below. Additionally, the present disclosure is described without reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the disclosure described herein.

The above description reveals numerous specific details, such as examples of specific systems, components, methods, etc., in order to provide a good understanding of some implementations of the disclosure. However, it will be apparent to those skilled in the art that at least some implementations of this disclosure may be practiced without these specific details. In other instances, well-known components or methods are not described in detail, or are presented in simple block diagram format, in order to avoid unnecessarily obscuring the present disclosure. Accordingly, the specific details disclosed above are only examples. Particular implementations may vary from these detailed examples and still be contemplated within the scope of the present disclosure.

1. A computer-implemented method for ranking messages in a conversation graph in a messaging platform using predicted results, comprising:
receiving, over a network, a conversation view request to retrieve messages in a conversation graph stored in the messaging platform;
the messaging platform determining a plurality of prediction outcomes for each of a plurality of messages in the conversation graph using a prediction model, the plurality of prediction outcomes being a mutual engagement probability, a positive engagement probability, and a negative engagement probability; a decision process, including an engagement probability of
calculating an engagement value for each of the plurality of messages using the plurality of prediction results;
the messaging platform ranking the plurality of messages based on the engagement value;
said messaging platform transmitting at least a portion of said plurality of messages rendered in a client application according to said rank over said network.
2. 2. The computer-implemented method of 1, wherein the mutual engagement probability includes a probability value that the messaging platform is expected to receive a reply to each message.
3. 3. The computer-implemented method of 1 or 2, wherein the predictive model comprises one or more neural networks.
4. Further comprising the step of said messaging platform generating said conversation graph based on a reply structure of messages exchanged on said messaging platform, said conversation graph comprising a tree data structure of messages associated with a conversation. , 1-3.
5. The determining step includes:
the messaging platform acquiring signals generated by one or more of the client application and the messaging platform;
and the messaging platform inputting the signal to the predictive model to determine the plurality of predictive outcomes.
6. 6. The computer-implemented method of any one of clauses 1-5, further comprising the messaging platform training the predictive model based on a machine learning algorithm input with training data.
7. the messaging platform selecting a subset of candidates from the plurality of messages in response to a number of messages in the conversation graph exceeding a threshold level, wherein the plurality of predicted results are selected from the 7. The computer-implemented method of any one of clauses 1-6, further comprising: determining for each candidate subset of messages.
8. the messaging platform sending a first portion of the plurality of messages to be rendered in the client application according to the rank;
the messaging platform receiving a request from the client application to display additional messages from the conversation graph;
8. The computer-implemented method of any one of claims 1-7, further comprising: the messaging platform sending a second portion of the plurality of messages to be rendered in the client application according to the rank.
9. said conversation graph comprising a tree data structure having an arrangement of a plurality of nodes representing said messages of said conversation graph;
9. The computer-implemented method of any one of claims 1-8, the method further comprising the messaging platform using the engagement value to select one or more branches of the tree data structure.
10. A system for ranking messages in a conversation graph in a messaging platform using predicted results, comprising:
a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged in a messaging platform, said conversation graph including a data structure of messages of a conversation;
A timeline manager configured to provide a stream of messages to a user's timeline on a client application over a network, and configured to receive conversation view requests from the client application over the network. , the timeline manager, and
an engagement predictor configured, in response to the view conversation request, to determine a plurality of predicted outcomes for each of a plurality of messages of the conversation graph using a predictive model, the plurality of predicted outcomes being , an engagement predictor including mutual engagement probability, positive engagement probability, and negative engagement probability;
an engagement scorer configured to calculate an engagement value for each of the plurality of messages using the predicted results for each message;
The timeline manager is configured to rank the plurality of messages using the engagement value and provide through the network at least a portion of the plurality of messages rendered in the timeline according to the ranking. system.
11. The mutual engagement probability includes a probability value that the messaging platform expects to receive a reply to the message in the conversation graph, and the positive engagement probability indicates that the messaging platform will receive an acknowledgment to the message in the conversation graph. said negative engagement probability comprises a probability value that said messaging platform is expected to receive negative user engagement to said message of said conversation graph. , 10.
12. The predictive models include a mutual engagement model, a positive engagement model, and a negative engagement model, and the engagement predictor obtains signals generated by one or more of the client application and the messaging platform. , to input the signals into the cross-engagement model, the positive engagement model, and the negative engagement model, and to obtain the cross-engagement probability, the positive engagement probability, and the negative engagement probability, respectively. 12. The system of claim 10 or 11, wherein the system is configured to:
13. a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with training data, the trained predictive model being provided to the predictive manager; 13. The system of any one of claims 10-12, further comprising a predictive model trainer configured to provide to.
14. first, further comprising a candidate message selector configured to select a portion of the candidates from a plurality of sources, the plurality of sources having the messages of the conversation graph ranked according to the most recently posted; and a second source having said messages of said conversation graph ranked according to a relevance algorithm, wherein some of said candidates are a number of messages from said first source and said first 14. The system of any one of claims 10-13, comprising a number of messages from two sources, and wherein said plurality of prediction results is determined for each message of said candidate subset.
15. The data structure of the conversation graph includes a tree data structure having an arrangement of a plurality of nodes representing the messages of the conversation graph, and the timeline manager selects a branch of the tree data structure using the engagement value. 15. The system of any one of claims 10-14, configured to:
16. The engagement scorer determines the mutual engagement probability, the positive engagement probability, and the negative engagement probability such that the weighting applied to the mutual engagement probability is higher than the weighting applied to the negative engagement probability. 16. The system of any one of clauses 10-15, configured to apply a weighting to each.
17. A non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, cause the one or more processors to:
receiving, over a network, a conversation view request for messages in a conversation graph stored in a messaging platform, wherein the conversation graph includes a data structure of messages in a conversation;
selecting a portion of candidate messages from the conversation graph;
determining a plurality of predicted outcomes for each of said candidate subset messages using a predictive model, said plurality of predicted outcomes comprising a mutual engagement probability, a positive engagement probability and a negative engagement probability; said mutual engagement probability comprises a probability value that said messaging platform is expected to receive a reply to each message of said candidate portion;
calculating an engagement value for each of the candidate subset messages using the prediction results;
ranking the plurality of messages using the engagement value;
and sending at least a portion of said plurality of messages to be rendered in a client application according to said rank.
18. 18. The non-transitory computer-readable medium of 17, wherein the portion of the candidates includes a number of recently posted messages and a number of top ranked messages from the conversation graph.
19. said data structure of said conversation graph comprising a tree data structure having an arrangement of a plurality of nodes representing said messages of said conversation graph;
19. The non-transitory computer readable medium of 17 or 18, further comprising using the engagement value to select a branch of the tree data structure.
20. periodically training the predictive model based on one or more machine learning algorithms input with training data;
and providing the trained prediction model to a prediction manager.
21. A computer-implemented method for ranking messages in a conversation graph in a messaging platform using predicted results, comprising:
receiving, over a network, a conversation view request to retrieve messages in a conversation graph stored in the messaging platform;
the messaging platform determining a plurality of predicted outcomes for each of a plurality of messages of the conversation graph using a predictive model;
obtaining a plurality of signals relevant to the predictive model, the plurality of signals including data structure-related signals relevant to the conversation graph;
inputting said plurality of signals into said prediction model to determine said plurality of prediction results;
the messaging platform ranking the plurality of messages based on the predicted results;
said messaging platform transmitting at least a portion of said plurality of messages rendered in a client application according to said rank over said network.
22. the conversation graph includes a tree data structure having a plurality of nodes representing the messages of the conversation graph and edges representing links between the nodes, the tree data structure defining one or more branches of nodes; 22. The computer-implemented method of Claim 21, wherein the data structure related signals include signals representing a number of nodes and a number of branches within the conversation graph.
23. 23. The computer-implemented method of 21 or 22, wherein the data structure related signals include a number of replies within a branch, a number of conversations within a branch, and a number of unique authors within a branch or the conversation graph.
24. 24. The computer-implemented method of any one of claims 21-23, wherein the plurality of signals includes a signal indicative of whether a user of the client application has restricted creators of messages in the conversation graph.
25. 21-24, wherein the plurality of signals includes engagement signals representing user engagement data associated with the messages in the conversation graph, including engagement of users following users of the client application in a connection graph; A computer-implemented method according to any one of the preceding paragraphs.
26. 26. The computer-implemented method of any one of claims 21-25, wherein the plurality of signals includes historical aggregated signals including engagement data associated with users of the client application on the messaging platform.
27. 27. The computer-implemented method of any one of claims 21-26, wherein the plurality of signals includes content-related signals associated with messages of the conversation graph.
28. 28. The computer-implemented method of any one of clauses 21-27, wherein at least some of the plurality of signals are obtained from one or more data services executing on the messaging platform.
29. the messaging platform obtaining training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform;
29. The computer-implemented method of any one of clauses 21-28, further comprising: the messaging platform training the predictive model based on a machine learning algorithm input with the training data.
30. A system for ranking messages in a conversation graph in a messaging platform using predicted results, comprising:
a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged in a messaging platform, said conversation graph including a data structure of messages of a conversation;
A timeline manager configured to provide a stream of messages to a user's timeline on a client application over a network, and configured to receive conversation view requests from the client application over the network. , the timeline manager, and
an engagement predictor configured, in response to the view conversation request, to determine a plurality of predicted outcomes for each of a plurality of messages of the conversation graph using a predictive model;
configured to obtain a plurality of signals relevant to the predictive model from one or more data services stored in the messaging platform; and to determine the plurality of prediction results. an engagement predictor configured to input signals into the predictive model, the plurality of signals including data structure related signals associated with the conversation graph;
an engagement scorer configured to calculate an engagement value for each of the plurality of messages using the predicted results for each message;
The timeline manager is configured to rank the plurality of messages using the engagement value and provide through the network at least a portion of the plurality of messages rendered in the timeline according to the ranking. There is a system.
31. The engagement predictor obtains one or more first signals from a first data service in the messaging platform and obtains one or more second signals from a second data service in the messaging platform. 31. The system according to 30, configured to:
32. 32. The system of 30 or 31, wherein the plurality of signals includes one or more of engagement signals, social graph signals, history aggregation signals, and content related signals.
33. The prediction model includes a mutual engagement model, a positive engagement model, and a negative engagement model, and the engagement predictor obtains mutual engagement probability, positive engagement probability, and negative engagement probability, respectively. 33. The system of any one of claims 30-32, configured to input signals to the mutual engagement model, the positive engagement model and the negative engagement model.
34. configured to periodically train the prediction model based on one or more machine learning algorithms input with training data, and configured to provide the trained prediction model to the prediction manager; 34. The system of any one of claims 30-33, further comprising a predictive model trainer.
35. 35. The system of any one of claims 30-34, wherein the plurality of signals includes signals generated by the client application and signals generated by the messaging platform.
36. 36. The system of any one of claims 30-35, wherein the plurality of signals comprises similarity signals.
37. A non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, cause the one or more processors to:
receiving, over a network, a conversation view request for messages in a conversation graph stored in a messaging platform, wherein the conversation graph includes a data structure of messages in a conversation;
selecting a portion of candidate messages from the conversation graph;
determining a plurality of prediction outcomes for each of the candidate subsets of messages using a prediction model, comprising:
obtaining a plurality of signals including data structure related signals associated with the conversation graph;
inputting said plurality of signals into said prediction model to determine said plurality of prediction results;
calculating an engagement value for each of the candidate subset messages using the prediction results;
ranking the plurality of messages using the engagement value;
and sending at least a portion of said plurality of messages to be rendered in a client application according to said rank.
38. obtaining training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform;
38. The non-transitory computer-readable medium of 37, further comprising training the predictive model based on a machine learning algorithm input with the training data.
39. A computer-implemented method for ranking messages in a conversation graph in a messaging platform, comprising:
A messaging platform classifying messages of a conversation graph into a plurality of sections based on content quality of said messages, said plurality of sections including a first section and a second section; a classification step, wherein one section comprises messages from the conversation graph determined to be of higher quality than the messages of the second section;
the messaging platform determining a plurality of predicted outcomes for each of a plurality of messages of the conversation graph using a predictive model;
the messaging platform ranking the messages of the first section based on the predicted results for the messages of the first section;
said messaging platform sending at least a portion of said message of said first section rendered in a client application according to said rank.
40. 40. The computer-implemented method of 39, further comprising the messaging platform ranking the messages of the second section based on the plurality of prediction results for the messages of the second section.
41. the messaging platform receiving a request to render additional messages of the conversation graph in the client application;
41. The computer-implemented method of 40, further comprising the messaging platform sending at least a portion of the message of the second section rendered in a client application according to the rank.
42. The classification step includes:
receiving a message identifier for a message of said conversation graph;
and using a classification model to determine that the message has a content quality corresponding to the first section.
43. 43. The computer-implemented method of any one of claims 39-42, further comprising assigning a branch of the conversation graph containing the message to the first section.
44. calculating an engagement value for each message of the conversation graph using the plurality of prediction results for each message, wherein the message of the first quality section is the message of the first quality section; 44. The computer-implemented method of any one of clauses 39-43, wherein the computer-implemented method is ranked according to an engagement value of .
45. 45. The computer-implemented method of any one of clauses 39-44, wherein the predicted outcome comprises mutual engagement probabilities, positive engagement probabilities, and negative engagement probabilities.
46. Further comprising the step of said messaging platform generating said conversation graph based on a reply structure of messages exchanged on said messaging platform, said conversation graph comprising a tree data structure of messages associated with a conversation. , 39-45.
47. The plurality of predictive models includes a mutual engagement model, a positive engagement model, and a negative engagement model, wherein one or more of the mutual engagement model, the positive engagement model, and the negative engagement model are neural networks. 47. The computer-implemented method of any one of 39-46, comprising:
48. A system for ranking messages of a conversation graph in a messaging platform, comprising:
a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged in a messaging platform, said conversation graph including a data structure of messages of a conversation;
A timeline manager configured to provide a stream of messages to a user's timeline on a client application over a network, and configured to receive conversation view requests from the client application over the network. , the timeline manager, and
A content quality classifier configured to classify the messages of the conversation graph into a plurality of sections based on content quality of the messages, the plurality of sections comprising a first section and a second section. a content quality classifier comprising sections, the first section having messages from the conversation graph determined to be of higher quality than the messages of the second section;
an engagement predictor configured to determine a plurality of predicted outcomes for each of a plurality of messages of said conversation graph using a predictive model;
an engagement scorer configured to calculate an engagement value for each of the plurality of messages using the plurality of predicted results for each message;
The timeline manager ranks the messages of the first section using the engagement value of the first section, and at least of the messages of the first section rendered in the timeline according to the rank. A system configured to provide a portion through said network.
49. The engagement predictor is configured to obtain a plurality of signals relevant to the predictive model from one or more data services stored on the messaging platform, and extracts the plurality of prediction results. 49. The system of 48, configured to input the plurality of signals into the predictive model to determine.
50. 50. The system of Claim 49, wherein the plurality of signals includes data structure related signals associated with the conversation graph.
51. 51. The system of any one of clauses 48-50, wherein the timeline manager is configured to separately rank the messages of the second section using engagement values of the second section. .
52. The content quality classifier receives a message identifier for a message of the conversation graph and uses a classification model to determine that the message has a content quality corresponding to the first quality section. 52. The system of any one of clauses 48-51, wherein:
53. 53. any one of 48 to 52, wherein said plurality of prediction results comprises mutual engagement probabilities, said mutual engagement probabilities comprising probability values that said messaging platform is expected to receive a reply to each message of said conversation graph. A system as described in .
54. a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with training data, the trained predictive model being provided to the predictive manager; 54. The system of any one of clauses 48-53, further comprising a predictive model trainer configured to provide to.
55. A non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, cause the one or more processors to:
classifying messages of a conversation graph into a plurality of sections based on content quality of said messages using a classification model, said plurality of sections including a first section and a second section; one section having messages from the conversation graph determined to be of higher quality than messages of the second section;
determining a plurality of prediction outcomes for each of a plurality of messages of the conversation graph using one or more prediction models, the plurality of prediction outcomes being a mutual engagement probability, a positive engagement probability, and a process comprising a negative engagement probability;
ranking the messages of the first section based on the plurality of prediction results for the messages of the first section;
ranking the messages of the second section based on the plurality of prediction results for the messages of the second section;
and sending at least a portion of said message of said first quality section rendered in a client application according to said rank.
56. obtaining training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform;
56. The non-transitory computer-readable medium of 55, further comprising training the predictive model based on a machine learning algorithm input with the training data.
57. receiving a message identifier of a first message of the conversation graph;
determining, using the classification model, that the first message has a content quality corresponding to the first section;
receiving a message identifier of a second message of the conversation graph;
57. The non-transitory computer-readable medium of 55 or 56, further comprising using the classification model to determine that the second message has a content quality corresponding to the second section.
58. receiving a request to render additional messages of the conversation graph at the client application;
58. The non-transitory computer-readable medium of any one of claims 55-57, further comprising sending at least a portion of the message of the second section rendered in a client application according to the rank.

Claims (58)

A computer-implemented method for ranking messages in a conversation graph in a messaging platform using predicted results, comprising:
receiving, over a network, a conversation view request to retrieve messages in a conversation graph stored in the messaging platform;
the messaging platform determining a plurality of prediction outcomes for each of a plurality of messages in the conversation graph using a prediction model, the plurality of prediction outcomes being a mutual engagement probability, a positive engagement probability, and a negative engagement probability; a decision process, including an engagement probability of
calculating an engagement value for each of the plurality of messages using the plurality of prediction results;
the messaging platform ranking the plurality of messages based on the engagement value;
said messaging platform transmitting at least a portion of said plurality of messages rendered in a client application according to said rank over said network. 2. The computer-implemented method of claim 1, wherein the mutual engagement probability comprises a probability value that the messaging platform is expected to receive a reply to each message. 3. The computer-implemented method of claim 1 or 2, wherein the predictive model comprises one or more neural networks. Further comprising the step of said messaging platform generating said conversation graph based on a reply structure of messages exchanged on said messaging platform, said conversation graph comprising a tree data structure of messages associated with a conversation. The computer-implemented method of any one of claims 1-3. The determining step includes:
the messaging platform acquiring signals generated by one or more of the client application and the messaging platform;
and the messaging platform inputting the signal to the predictive model to determine the plurality of predictive outcomes. The computer-implemented method of any of claims 1-5, further comprising the messaging platform training the predictive model based on machine learning algorithms input with training data. the messaging platform selecting a subset of candidates from the plurality of messages in response to a number of messages in the conversation graph exceeding a threshold level, wherein the plurality of predicted results are selected from the The computer-implemented method of any one of claims 1-6, further comprising the step of being determined for each candidate partial message. the messaging platform sending a first portion of the plurality of messages to be rendered in the client application according to the rank;
the messaging platform receiving a request from the client application to display additional messages from the conversation graph;
The computer-implemented method of any one of claims 1-7, further comprising: the messaging platform sending a second portion of the plurality of messages rendered in the client application according to the rank. . said conversation graph comprising a tree data structure having an arrangement of a plurality of nodes representing said messages of said conversation graph;
The computer-implemented method of any one of claims 1-8, wherein the method further comprises the messaging platform using the engagement value to select one or more branches of the tree data structure. Method. A system for ranking messages in a conversation graph in a messaging platform using predicted results, comprising:
a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged in a messaging platform, said conversation graph including a data structure of messages of a conversation;
A timeline manager configured to provide a stream of messages to a user's timeline on a client application over a network, and configured to receive conversation view requests from the client application over the network. , the timeline manager, and
an engagement predictor configured, in response to the view conversation request, to determine a plurality of predicted outcomes for each of a plurality of messages of the conversation graph using a predictive model, the plurality of predicted outcomes being , an engagement predictor including mutual engagement probability, positive engagement probability, and negative engagement probability;
an engagement scorer configured to calculate an engagement value for each of the plurality of messages using the predicted results for each message;
The timeline manager is configured to rank the plurality of messages using the engagement value and provide through the network at least a portion of the plurality of messages rendered in the timeline according to the rank. system. The mutual engagement probability includes a probability value that the messaging platform expects to receive a reply to the message in the conversation graph, and the positive engagement probability indicates that the messaging platform will receive an acknowledgment to the message in the conversation graph. said negative engagement probability comprises a probability value that said messaging platform is expected to receive negative user engagement to said message of said conversation graph. 11. The system of claim 10. The predictive models include a mutual engagement model, a positive engagement model, and a negative engagement model, and the engagement predictor obtains signals generated by one or more of the client application and the messaging platform. , to input the signals into the cross-engagement model, the positive engagement model, and the negative engagement model, and to obtain the cross-engagement probability, the positive engagement probability, and the negative engagement probability, respectively. 12. A system according to claim 10 or 11, configured for: a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with training data, the trained predictive model being provided to the predictive manager; 13. The system of any one of claims 10-12, further comprising a predictive model trainer configured to provide to. first, further comprising a candidate message selector configured to select a portion of the candidates from a plurality of sources, the plurality of sources having the messages of the conversation graph ranked according to the most recently posted; and a second source having said messages of said conversation graph ranked according to a relevance algorithm, wherein some of said candidates are a number of messages from said first source and said first and a certain number of messages from two sources, wherein said plurality of prediction results is determined for each message of said candidate subset. The data structure of the conversation graph includes a tree data structure having an arrangement of a plurality of nodes representing the messages of the conversation graph, and the timeline manager selects a branch of the tree data structure using the engagement value. The system of any one of claims 10-14, wherein the system is configured to: The engagement scorer determines the mutual engagement probability, the positive engagement probability, and the negative engagement probability such that the weighting applied to the mutual engagement probability is higher than the weighting applied to the negative engagement probability. A system according to any one of claims 10 to 15, arranged to apply a weighting to each. A non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, cause the one or more processors to:
receiving, over a network, a conversation view request for messages in a conversation graph stored in a messaging platform, wherein the conversation graph includes a data structure of messages in a conversation;
selecting a portion of candidate messages from the conversation graph;
determining a plurality of predicted outcomes for each of said candidate subset messages using a predictive model, said plurality of predicted outcomes comprising a mutual engagement probability, a positive engagement probability and a negative engagement probability; said mutual engagement probability comprises a probability value that said messaging platform is expected to receive a reply to each message of said candidate portion;
calculating an engagement value for each of the candidate subset messages using the prediction results;
ranking the plurality of messages using the engagement value;
and sending at least a portion of said plurality of messages to be rendered in a client application according to said rank. 18. The non-transitory computer-readable medium of claim 17, wherein some of the candidates include a number of recently posted messages and a number of top ranked messages from the conversation graph. said data structure of said conversation graph comprising a tree data structure having an arrangement of a plurality of nodes representing said messages of said conversation graph;
19. The non-transitory computer-readable medium of claim 17 or 18, further comprising using the engagement value to select a branch of the tree data structure. periodically training the predictive model based on one or more machine learning algorithms input with training data;
and providing the trained prediction model to a prediction manager. A computer-implemented method for ranking messages in a conversation graph in a messaging platform using predicted results, comprising:
receiving, over a network, a conversation view request to retrieve messages in a conversation graph stored in the messaging platform;
the messaging platform determining a plurality of predicted outcomes for each of a plurality of messages of the conversation graph using a predictive model;
obtaining a plurality of signals relevant to the predictive model, the plurality of signals including data structure-related signals relevant to the conversation graph;
inputting said plurality of signals into said prediction model to determine said plurality of prediction results;
the messaging platform ranking the plurality of messages based on the predicted results;
said messaging platform transmitting at least a portion of said plurality of messages rendered in a client application according to said rank over said network. the conversation graph includes a tree data structure having a plurality of nodes representing the messages of the conversation graph and edges representing links between the nodes, the tree data structure defining one or more branches of nodes; 22. The computer-implemented method of claim 21, wherein said data structure related signals include signals representing a number of nodes and a number of branches within said conversation graph. 23. The computer-implemented method of claim 21 or 22, wherein the data structure related signals include a number of replies within a branch, a number of conversations within a branch, and a number of unique authors within a branch or the conversation graph. 24. The computer-implemented method of any of claims 21-23, wherein the plurality of signals includes a signal representing whether a user of the client application has restricted creators of messages in the conversation graph. 21-, wherein the plurality of signals includes engagement signals representing user engagement data associated with the messages in the conversation graph, including engagement of users following users of the client application in a connection graph. 25. The computer-implemented method of any one of Clause 24. 26. The computer-implemented method of any one of claims 21-25, wherein the plurality of signals comprises historical aggregated signals containing engagement data associated with users of the client application on the messaging platform. 27. The computer-implemented method of any one of claims 21-26, wherein the plurality of signals comprises content-related signals associated with messages of the conversation graph. 28. The computer-implemented method of any of claims 21-27, wherein at least some of the plurality of signals are obtained from one or more data services executing on the messaging platform. the messaging platform obtaining training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform;
The computer-implemented method of any one of claims 21-28, further comprising the messaging platform training the predictive model based on a machine learning algorithm input with the training data. A system for ranking messages in a conversation graph in a messaging platform using predicted results, comprising:
a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged in a messaging platform, said conversation graph including a data structure of messages of a conversation;
A timeline manager configured to provide a stream of messages to a user's timeline on a client application over a network, and configured to receive conversation view requests from the client application over the network. , the timeline manager, and
an engagement predictor configured, in response to the view conversation request, to determine a plurality of predicted outcomes for each of a plurality of messages of the conversation graph using a predictive model;
configured to obtain a plurality of signals relevant to the predictive model from one or more data services stored in the messaging platform; and to determine the plurality of prediction results. an engagement predictor configured to input signals into the predictive model, the plurality of signals including data structure related signals associated with the conversation graph;
an engagement scorer configured to calculate an engagement value for each of the plurality of messages using the predicted results for each message;
The timeline manager is configured to rank the plurality of messages using the engagement value and provide through the network at least a portion of the plurality of messages rendered in the timeline according to the ranking. There is a system. The engagement predictor obtains one or more first signals from a first data service in the messaging platform and obtains one or more second signals from a second data service in the messaging platform. 31. The system of claim 30, configured to. 32. The system of claim 30 or 31, wherein the plurality of signals comprises one or more of engagement signals, social graph signals, history aggregation signals, and content related signals. The prediction model includes a mutual engagement model, a positive engagement model, and a negative engagement model, and the engagement predictor obtains mutual engagement probability, positive engagement probability, and negative engagement probability, respectively. 33. The system of any one of claims 30-32, configured to input signals into the mutual engagement model, the positive engagement model and the negative engagement model. configured to periodically train the prediction model based on one or more machine learning algorithms input with training data, and configured to provide the trained prediction model to the prediction manager; 34. The system of any one of claims 30-33, further comprising a predictive model trainer. 35. The system of any of claims 30-34, wherein the plurality of signals includes signals generated by the client application and signals generated by the messaging platform. The system of any one of claims 30-35, wherein the plurality of signals comprises similarity signals. A non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, cause the one or more processors to:
receiving, over a network, a conversation view request for messages in a conversation graph stored in a messaging platform, wherein the conversation graph includes a data structure of messages in a conversation;
selecting a portion of candidate messages from the conversation graph;
determining a plurality of prediction outcomes for each of the candidate subsets of messages using a prediction model, comprising:
obtaining a plurality of signals including data structure related signals associated with the conversation graph;
inputting said plurality of signals into said prediction model to determine said plurality of prediction results;
calculating an engagement value for each of the candidate subset messages using the prediction results;
ranking the plurality of messages using the engagement value;
and sending at least a portion of said plurality of messages to be rendered in a client application according to said rank. obtaining training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform;
38. The non-transitory computer-readable medium of claim 37, further comprising training the predictive model based on a machine learning algorithm input with the training data. A computer-implemented method for ranking messages in a conversation graph in a messaging platform, comprising:
A messaging platform classifying messages of a conversation graph into a plurality of sections based on content quality of said messages, said plurality of sections including a first section and a second section; a classification step, wherein one section comprises messages from the conversation graph determined to be of higher quality than the messages of the second section;
the messaging platform determining a plurality of predicted outcomes for each of a plurality of messages of the conversation graph using a predictive model;
the messaging platform ranking the messages of the first section based on the predicted results for the messages of the first section;
said messaging platform sending at least a portion of said message of said first section rendered in a client application according to said rank. 40. The computer-implemented method of claim 39, further comprising the messaging platform ranking the messages of the second section based on the plurality of prediction results for the messages of the second section. the messaging platform receiving a request to render additional messages of the conversation graph in the client application;
41. The computer-implemented method of claim 40, further comprising the messaging platform sending at least a portion of the message of the second section rendered in a client application according to the rank. The classification step includes:
receiving a message identifier for a message of said conversation graph;
and using a classification model to determine that the message has a content quality corresponding to the first section. 43. The computer-implemented method of any one of claims 39-42, further comprising assigning a branch of the conversation graph containing the message to the first section. calculating an engagement value for each message of the conversation graph using the plurality of prediction results for each message, wherein the message of the first quality section is the message of the first quality section; 44. The computer-implemented method of any one of claims 39-43, ranked according to an engagement value of . 45. The computer-implemented method of any one of claims 39-44, wherein the prediction results include mutual engagement probabilities, positive engagement probabilities, and negative engagement probabilities. The messaging platform further comprising generating the conversation graph based on the reply structure of messages exchanged on the messaging platform, the conversation graph comprising a tree data structure of messages associated with a conversation. The computer-implemented method of any one of claims 39-45. The plurality of predictive models includes a mutual engagement model, a positive engagement model, and a negative engagement model, wherein one or more of the mutual engagement model, the positive engagement model, and the negative engagement model are neural networks. 47. The computer-implemented method of any one of claims 39-46, comprising: A system for ranking messages of a conversation graph in a messaging platform, comprising:
a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged in a messaging platform, said conversation graph including a data structure of messages of a conversation;
A timeline manager configured to provide a stream of messages to a user's timeline on a client application over a network, and configured to receive conversation view requests from the client application over the network. , the timeline manager, and
A content quality classifier configured to classify the messages of the conversation graph into a plurality of sections based on content quality of the messages, the plurality of sections comprising a first section and a second section. a content quality classifier comprising sections, the first section having messages from the conversation graph determined to be of higher quality than the messages of the second section;
an engagement predictor configured to determine a plurality of predicted outcomes for each of a plurality of messages of said conversation graph using a predictive model;
an engagement scorer configured to calculate an engagement value for each of the plurality of messages using the plurality of predicted results for each message;
The timeline manager ranks the messages of the first section using the engagement value of the first section, and at least of the messages of the first section rendered in the timeline according to the rank. A system configured to provide a portion through said network. The engagement predictor is configured to obtain a plurality of signals relevant to the predictive model from one or more data services stored on the messaging platform, and extracts the plurality of prediction results. 49. The system of claim 48, configured to input said plurality of signals into said predictive model to determine. 50. The system of claim 49, wherein said plurality of signals includes data structure related signals associated with said conversation graph. 51. The timeline manager of any one of claims 48-50, wherein the timeline manager is configured to separately rank the messages of the second section using the engagement value of the second section. system. The content quality classifier receives a message identifier for a message of the conversation graph and uses a classification model to determine that the message has a content quality corresponding to the first quality section. 52. The system of any one of claims 48-51 configured. 53. Any one of claims 48 to 52, wherein said plurality of prediction results comprise mutual engagement probabilities, said mutual engagement probabilities comprising probability values that said messaging platform is expected to receive a reply to each message of said conversation graph. or the system according to item 1. a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with training data, the trained predictive model being provided to the predictive manager; 54. The system of any one of claims 48-53, further comprising a predictive model trainer configured to provide to. A non-transitory computer-readable medium storing executable instructions that, when executed by one or more processors, cause the one or more processors to:
classifying messages of a conversation graph into a plurality of sections based on content quality of said messages using a classification model, said plurality of sections including a first section and a second section; one section having messages from the conversation graph determined to be of higher quality than messages of the second section;
determining a plurality of prediction outcomes for each of a plurality of messages of the conversation graph using one or more prediction models, the plurality of prediction outcomes being a mutual engagement probability, a positive engagement probability, and a process comprising a negative engagement probability;
ranking the messages of the first section based on the plurality of prediction results for the messages of the first section;
ranking the messages of the second section based on the plurality of prediction results for the messages of the second section;
and sending at least a portion of said message of said first quality section rendered in a client application according to said rank. obtaining training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform;
56. The non-transitory computer-readable medium of claim 55, further comprising training the predictive model based on a machine learning algorithm input with the training data. receiving a message identifier of a first message of the conversation graph;
determining, using the classification model, that the first message has a content quality corresponding to the first section;
receiving a message identifier of a second message of the conversation graph;
57. The non-transitory computer-readable medium of claim 55 or 56, further comprising using the classification model to determine that the second message has a content quality corresponding to the second section. . receiving a request to render additional messages of the conversation graph at the client application;
sending at least a portion of said message of said second section rendered in a client application according to said rank. medium.

Images