KR20220123663A - Ranking Messages in Conversation Graphs in Messaging Platform Using Prediction Results - Google Patents

Abstract

According to one aspect, a method for ranking messages in a conversation graph in a messaging platform includes receiving a conversation view request to retrieve messages of a conversation graph stored on the messaging platform and using predictive models to represent a plurality of messages in the conversation graph using predictive models. and determining a plurality of prediction results for each of the messages. The determining includes obtaining a plurality of signals associated with the predictive models, the plurality of signals including data structure-related signals associated with the dialog graph, and sending the plurality of signals to the predictive models to determine a plurality of predictive results. inputting them. The method includes ranking, by the messaging platform, a plurality of messages based on prediction results, and at least a subset of the plurality of messages to be rendered on a client application according to the ranking, via a network, by the messaging platform. sending it.

Description

Ranking Messages in Conversation Graphs in Messaging Platform Using Prediction Results

[0001] This application is filed on December 20, 2019 in U.S. Regular Patent Application No. 16/723,981 and in U.S. Patent Application Serial No. 16/723,984, filed on December 20, 2019 and filed on December 20, 2019 is a continuation of U.S. Patent Application Serial No. 16/723,987, which claims priority to the applications, the disclosures of which are incorporated herein by reference in their entirety.

[0002] A social media messaging platform may facilitate the exchange of millions or hundreds of millions of social media messages among its users. Messages exchanged on the platform can often provide platform users with the latest updates or reporting on current events. The exchange of messages on a messaging platform may be part of a conversation between users. Some conventional messaging systems may store a reply structure of messages so that a user can view parts of a conversation when viewing a particular message. However, the list of messages forming the conversation view can be relatively large, which can lead to slower load times for the user.

[0003] A messaging platform facilitates the exchange of messages between users of the messaging platform. Messages may be part of a conversation that takes place on a messaging platform. For example, a user can post a message on the platform and other users post the number of replies to the message and then replies to the replies, and so on. The reply structure may be stored as a conversation graph and the messaging platform may store any number of conversation graphs related to conversations occurring on the messaging platform. In some examples, the dialog graph can be relatively large (eg, the number of nodes in the dialog graph exceeds a threshold number). The creation and maintenance of conversation graphs may enable faster retrieval of information when responding to requests to view messages in a reply relationship with a particular message.

[0004] When viewing a particular message associated with the conversation graph, the user can select the message to view other messages in the conversation graph, where the client application makes a conversation view request to the messaging platform. Instead of displaying all the messages related to the conversation graph (which can be relatively large), the timeline manager uses its own scoring system (which incorporates engagement predictions) to rank the messages in the conversation graph. It ranks (and possibly selects) the most relevant branches (or nodes) for a particular user, and then presents the ranked list (or a subset thereof) on the user interface of the client application. In this way, the messages of the conversation graph are displayed according to the level of their prediction results (positive prediction results are balanced against negative prediction results). Also, providing a ranked list allows the messaging platform to serve fewer responses while increasing the value provided to users. For example, a messaging platform may provide a subset of responses (eg, top 10, 15, 20, etc. responses) to each request, which speeds up computation on the server and The load times are faster and there can be virtually no loss to engagement (eg, no loss at all).

[0005] A prediction manager (running on the messaging platform) may use one or more prediction models to determine prediction results for each candidate node of the conversation graph. The prediction results are the predicted user engagement results for each message (or a subset of messages) of the conversation graph. Prediction results include mutual engagement probabilities. The mutual engagement probability indicates a probability value at which the user is predicted to further develop the conversation graph (eg, to reply to a message). The incorporation of mutual engagement probabilities into prediction results can encourage more conversations on the platform. In some examples, the prediction result includes a mutual engagement probability and at least one of a positive engagement probability or a negative engagement probability. Positive engagement probability indicates a probability value that the user is expected to positively engage with the message (eg, like or favorite a node). The negative engagement probability indicates a probability value that the user is expected to negatively view or engage with the message (eg, the user may discover that the node is abusive). In some examples, the prediction results include a mutual engagement probability, a positive engagement probability, and a negative engagement probability.

[0006] The predictive models may include a positive engagement model predicting a positive engagement probability, a negative engagement model predicting a negative engagement probability, and a mutual engagement model predicting a mutual engagement probability. In some examples, the predictive models are machine learning models trained using training data applied to one or more machine learning algorithms. In some examples, predictive models include neural networks. The training data may be a combination of historical data obtained from the client event log and the injection log while the messaging system is active (and the predictive manager is enabled). Historical data may be acquired over a period of time (eg, days, weeks, months, years, etc.) and then used to train (or retrain) predictive models in an offline mode.

[0007] In response to the conversation view request to retrieve messages of the conversation graph, the prediction manager may obtain signals to determine prediction results and input signals to the prediction model to determine prediction results. The signals may include signals generated by a client application and/or signals generated by a messaging platform. In some examples, the signals include a social graph and engagement history of the user and author. However, signals may include a wide variety of signals, such as data structure-related signals, health-related signals, engagement signals, social graph signals, historical aggregate signals, and/or content-related signals. The number of signals used for prediction and the number of different categories of signals can improve the accuracy of model predictions.

[0008] The prediction manager may combine the prediction results to obtain an engagement value for each node, where the engagement value indicates the overall level of predicted engagement with the message in the dialog graph. The timeline manager can use the engagement values to greedily select the most relevant branch (or node) of the dialog graph for the user. In addition to locating the most relevant branch(s) (or nodes), the timeline manager can determine how many of the branches to display based on engagement values. The timeline manager can also collapse irrelevant parts of the dialog graph, but allows the user to expand those parts to view other messages in the dialog graph.

[0009] Messages ranked according to engagement values may be specific to each user. For example, some messages in the conversation graph may be more relevant to a first user, while other messages in the conversation graph may be more relevant to a second user. However, the prediction results determined by the prediction manager are tailored to the specific user. In contrast, some conventional approaches use a vote-based mechanism that can provide the same view for each. Also, because the messaging platform incorporates mutual engagement probabilities within its scoring algorithm, the messaging platform can encourage more conversations on the messaging platform.

[0010] In some examples, the messaging platform performs a two-level ranking system with respect to the messages in the conversation graph. For example, a two-level ranking mechanism (eg, hard ranking and soft ranking) may provide an effective mechanism for handling profanity in conversations on a messaging platform. In some examples, hard ranking includes categorizing (or sectioning) the messages (or branches) of the conversation into a plurality of sections. For example, the messaging platform may include a content quality classifier that divides the nodes (or branches) of the conversation graph into sections. Each section may refer to a different categorization (or classification) of quality levels. In some examples, the content quality classifier includes a classification model (eg, a neural network) that determines the quality classification of messages based on the message's content and the author's behavior, profile, or other signals related to engagements on the messaging platform. .

[0011] The content quality classifier may determine that the first message has relatively low quality and the second message has relatively high quality, wherein the content quality classifier divides the first message (or branch containing the first message) into a first section (eg, A first section may be assigned to a low quality section, and a second message (or a branch containing the second message) may be assigned to a second section (eg, the second section is considered a high quality section). have. Soft ranking of the two-level ranking mechanism may include ranking the messages (or branches) within each section using the engagement values described above. The messages are then surfaced to the user in such a way that they provide a ranked list of high quality sections followed by a ranked list of low quality sections.

[0012] According to an aspect, a method for ranking messages in a conversation graph in a messaging platform using prediction results comprises: receiving, via a network, a conversation view request to retrieve messages of a conversation graph stored on the messaging platform; determining, by the platform, a plurality of prediction results for each of a plurality of messages of the conversation graph using the prediction models. The plurality of prediction results include a mutual engagement probability. The method includes ranking, by the messaging platform, a plurality of messages based on prediction results, and at least a subset of the plurality of messages to be rendered on a client application according to the ranking, via a network, by the messaging platform. sending it. According to other aspects, a non-transitory computer-readable medium storing a corresponding system and corresponding instructions may be provided.

[0013] According to some aspects, a method, system, and/or non-transitory computer-readable medium may include one or more of the following features (or any combination thereof). The plurality of prediction results also include a positive engagement probability and a negative engagement probability. The method may further comprise computing an engagement value for each of the plurality of messages using the mutual engagement probability, the positive engagement probability, and the negative engagement probability, wherein the plurality of messages are engaged They are ranked according to the easement values. The method may include generating, by the messaging platform, a conversation graph based on a reply structure of messages exchanged on the messaging platform, the conversation graph including a tree data structure of messages related to the conversation. The method includes obtaining, by the messaging platform, signals generated by at least one of a client application or the messaging platform, and inputting, by the messaging platform, the signals into predictive models to determine a plurality of predictive results. can do. The method may include training the predictive model based on the machine learning algorithm input with the training data by the messaging platform. The method may include selecting, by the messaging platform, a candidate subset from the plurality of messages in response to a number of messages in the conversation graph being greater than a threshold level, wherein the plurality of prediction results are of the candidate subset. It is determined for each message. The method includes sending, by the messaging platform, a first subset of the plurality of messages to be rendered on a client application according to a ranking, receiving, by the messaging platform, a request from the client application to display additional messages from the conversation graph. and sending, by the messaging platform, a second subset of the plurality of messages to be rendered on the client application according to the ranking. The dialog graph may include a tree data structure having an arrangement of a plurality of nodes representing messages of the dialog graph. The method includes, by the messaging platform, computing, by the messaging platform, an engagement value for each of a plurality of messages based on the plurality of prediction results, and, by the messaging platform, one or more of the tree data structure using the engagement values. may include selecting branches.

[0014] According to an aspect, a system for ranking messages of conversation graphs in a messaging platform using prediction results comprises: a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged on the messaging platform, the conversation graph comprising: comprising a data structure of messages of the conversation, in a timeline of a user on the client application, a timeline manager configured to provide a stream of messages over a network, the timeline manager receiving, from the client application, a request to view the conversation over the network an engagement predictor configured to determine, in response to a conversation view request, a plurality of prediction results for each of a plurality of messages of the conversation graph using the predictive models, wherein the plurality of prediction results are a mutual engagement probability , comprising at least one of a positive engagement probability and a negative engagement probability, and an engagement scorer configured to compute an engagement value for each of the plurality of messages using the prediction results for the individual message. includes The timeline manager is configured to rank the plurality of messages using the engagement values and provide, via the network, at least a subset of the plurality of messages to be rendered on the timeline according to the ranking. According to other aspects, a non-transitory computer-readable medium storing a corresponding method and corresponding instructions may be provided.

[0015] In accordance with some aspects, a method, system, and/or non-transitory computer-readable medium may include one or more of the features above/below (or any combination thereof). The mutual engagement probability includes a value of the probability that the messaging platform is expected to receive a reply to a message in the conversation graph, and the positive engagement probability includes the probability value that the messaging platform receives a positive user engagement with the message in the conversation graph. contains a probability value predicted to do, and the negative engagement probability includes a probability value that the messaging platform is predicted to receive a negative user engagement with a message in the conversation graph. Predictive models include a mutual engagement model, a positive engagement model, and a negative engagement model. The engagement predictor obtains signals generated by at least one of a client application or a messaging platform, and inputs the signals to a mutual engagement model, a positive engagement model and a negative engagement model to determine the mutual engagement probability; configured to obtain a positive engagement probability and a negative engagement probability, respectively. The system may include a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with the training data, the predictive model trainer configured to provide the trained predictive model to the predictive manager. . The system can include a candidate message selector configured to select a candidate subset from a plurality of sources, wherein the plurality of sources are of a first source and a conversation graph ranked according to most recently published messages in the conversation graph. The messages include a second source ranked according to a relevance algorithm. The candidate subset includes a plurality of messages from a first source and a plurality of messages from a second source, wherein a plurality of prediction results are determined for each message of the candidate subset. The data structure of the dialog graph includes a tree data structure having an arrangement of a plurality of nodes representing messages of the dialog graph, wherein the timeline manager is configured to select a branch of the tree data structure using engagement values. the engagement scorer is configured to apply weights to each of the mutual engagement probability, the positive engagement probability and the negative engagement probability such that the weight applied to the mutual engagement probability is higher than the weight applied to the negative engagement probability .

[0016] According to one aspect, there is a non-transitory computer-readable medium having stored thereon executable instructions that, when executed by at least one processor, cause the at least one processor to cause: via a network, a messaging platform; receive a conversation view request for messages of the conversation graph stored on, the conversation graph including a data structure of the messages of the conversation, select a candidate subset of messages from the conversation graph, and use the predictive model to serve the candidate sub determine a plurality of prediction results for each message in the set, wherein the plurality of prediction results include a mutual engagement probability, wherein the mutual engagement probability causes the messaging platform to receive a reply to the respective message in the candidate subset. comprising a probability value predicted to do—compute an engagement value for each message in the candidate subset using the prediction results, rank the plurality of messages using the engagement values, and rank transmit at least a subset of the plurality of messages to be rendered on the client application according to According to other aspects, a corresponding system and method of storing corresponding instructions may be provided.

[0017] In accordance with some aspects, a method, system, and/or non-transitory computer-readable medium may include one or more of the features above/below (or any combination thereof). The candidate subset includes a number of most recently posted messages and a number of top-ranked messages from the conversation graph. The data structure of the dialog graph includes a tree data structure having an arrangement of a plurality of nodes representing messages of the dialog graph. The operations may include selecting a branch of the tree data structure using the engagement values. The operations may include periodically training the predictive model based on one or more machine learning algorithms input with the training data and providing the trained predictive model to the predictive manager.

[0018] According to an aspect, a method for ranking messages in a conversation graph in a messaging platform using prediction results comprises: receiving, via a network, a conversation view request to retrieve messages of a conversation graph stored on the messaging platform; determining, by the platform, a plurality of prediction results for each of a plurality of messages of the conversation graph using the prediction models. The determining comprises obtaining a plurality of signals associated with the predictive models, the plurality of signals comprising data structure-related signals associated with the dialog graph, and sending the plurality of signals to the predictive models to determine a plurality of prediction results. may include entering The method includes ranking, by a messaging platform, a plurality of messages based on prediction results, and at least a subset of the plurality of messages to be rendered on a client application according to the ranking, via a network, by the messaging platform. sending it. According to other aspects, a non-transitory computer-readable medium storing a corresponding system and corresponding instructions may be provided.

[0019] In accordance with some aspects, a method, system, and/or non-transitory computer-readable medium may include one or more of the features above/below (or any combination thereof). The dialog graph includes a tree data structure having a plurality of nodes representing messages of the dialog graph and edges representing links between nodes, the tree data structure defining one or more branches of the nodes, and a data structure-related signal These include signals representing the number of nodes and the number of branches within the dialog graph. Data structure-related signals include the number of replies in a branch, the number of conversations in the branch, and the number of unique authors in the branch or conversation graph. The plurality of signals includes signals indicative of whether a user of the client application has restricted a creator of a message of the conversation graph. The plurality of signals includes engagement signals representing user engagement data associated with messages of the dialog graph including engagements of users following the user of the client application in the connection graph. The plurality of signals includes historical aggregate signals comprising engagement data associated with a user of a client application on the messaging platform. The plurality of signals includes content-related signals associated with messages of the conversation graph. At least a subset of the plurality of signals is obtained from one or more data services running on the messaging platform. The method includes obtaining, by the messaging platform, training data from a client event log storing information received from a client application and an injection log storing information from a timeline manager running on the messaging platform, and by the messaging platform; It may include training the predictive model based on the machine learning algorithm input along with the training data.

[0020] According to an aspect, a system for ranking messages of conversation graphs in a messaging platform using prediction results comprises: a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged on the messaging platform, the conversation graph comprising: comprising a data structure of messages of the conversation, in a timeline of a user on the client application, a timeline manager configured to provide a stream of messages over a network, the timeline manager receiving, from the client application, a request to view the conversation over the network and an engagement predictor configured to determine, in response to the conversation view request, a plurality of prediction results for each of a plurality of messages of the conversation graph using the predictive models. The engagement predictor is configured to obtain a plurality of signals associated with the predictive models from one or more data services stored on the messaging platform, wherein the engagement predictor sends the plurality of signals to the predictive models to determine a plurality of predictive results. are configured to input The system includes an engagement scorer configured to compute an engagement value for each of a plurality of messages using the prediction results for the individual messages, wherein the timeline manager uses the engagement values to compute a plurality of engagement values. rank messages; and provide, via the network, at least a subset of the plurality of messages to be rendered on the timeline according to the ranking. According to other aspects, a non-transitory computer-readable medium storing a corresponding method and corresponding instructions may be provided.

[0021] In accordance with some aspects, a method, system, and/or non-transitory computer-readable medium may include one or more of the features above/below (or any combination thereof). The plurality of signals includes data structure-related signals associated with the dialog graph. The engagement predictor is configured to obtain one or more first signals from a first data service and obtain one or more second signals from a second data service. The plurality of signals includes at least one of health-related signals, engagement signals, social graph signals, historical aggregation signals, or content-related signals. The predictive models include a mutual engagement model, a positive engagement model and a negative engagement model, and the engagement predictor inputs signals to the mutual engagement model, positive engagement model and negative engagement model. to obtain a mutual engagement probability, a positive engagement probability, and a negative engagement probability, respectively. The system includes a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with the training data, the predictive model trainer configured to provide the trained predictive model to the predictive manager. The plurality of signals includes signals generated by a client application and signals generated by a messaging platform. The plurality of signals includes similarity signals.

[0022] According to one aspect, there is a non-transitory computer-readable medium having stored thereon executable instructions that, when executed by at least one processor, cause the at least one processor to cause: via a network, a messaging platform; receive a conversation view request for messages of the conversation graph stored on the dialog graph, wherein the conversation graph includes a data structure of messages of the conversation; select a candidate subset of messages from the conversation graph, and use the predictive model to determine a plurality of prediction results for each message in the candidate subset - a plurality of signals comprising data structure-related signals associated with the conversation graph and inputting a plurality of signals to the predictive models to determine a plurality of prediction results, using the prediction results to compute an engagement value for each message in the candidate subset; and rank the plurality of messages using the engagement values, and transmit at least a subset of the plurality of messages to be rendered on the client application according to the ranking. According to other aspects, a corresponding system and method may be provided.

[0023] In accordance with some aspects, a method, system, and/or non-transitory computer-readable medium may include one or more of the features above/below (or any combination thereof). The operations include obtaining training data from a client event log, which stores information received from a client application, and an injection log, which stores information from a timeline manager running on the messaging platform, and a machine learning algorithm input with the training data. training a predictive model based on the The plurality of signals includes at least one of health-related signals, engagement signals, social graph signals, historical aggregation signals, or content-related signals.

[0024] According to an aspect, a method for ranking messages in conversation graphs in a messaging platform includes classifying, by the messaging platform, messages into a plurality of sections based on content quality of messages in the conversation graph, wherein The plurality of sections includes a first section and a second section. The first section has messages from the conversation graph determined to be of higher quality than the messages of the second section. The method includes, by the messaging platform, determining, by the messaging platform, a plurality of predictive results for each of a plurality of messages of the conversation graph using the predictive models, based on the predictive results for the messages of the first section, by the messaging platform ranking the messages of the first section, and transmitting, by the messaging platform, at least a subset of the messages of the first section to be rendered on the client application according to the ranking. According to other aspects, a non-transitory computer-readable medium storing a corresponding system and corresponding instructions may be provided.

[0025] In accordance with some aspects, a method, system, and/or non-transitory computer-readable medium may include one or more of the features above/below (or any combination thereof). The method includes ranking, by the messaging platform, the messages of the second section based on a plurality of prediction results for the messages of the second section. In some examples, the method includes receiving, by the messaging platform, a request to render additional messages of the conversation graph on the client application, and the message of at least the second section to be rendered by the messaging platform on the client application according to the ranking. transmitting a subset of Classifying may include receiving a message identifier of the message of the conversation graph and determining that the message has a content quality corresponding to the first quality section using the classification model. The method may include assigning a branch of the conversation graph comprising the message to the first section. The method may include computing an engagement value for each message of the conversation graph using the plurality of prediction results of the respective message, wherein the messages of the first quality section are engaged of the messages of the first quality section. They are ranked according to the easement values. The prediction results include a mutual engagement probability, a positive engagement probability and a negative engagement probability. The method may include generating, by the messaging platform, a conversation graph based on a reply structure of messages exchanged on the messaging platform, the conversation graph including a tree data structure of messages related to the conversation. The plurality of predictive models includes a mutual engagement model, a positive engagement model, and a negative engagement model, and at least one of the mutual engagement model, the positive engagement model, or the negative engagement model includes a neural network. do.

[0026] According to an aspect, a system for ranking messages of conversation graphs in a messaging platform comprises a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged on the messaging platform, the conversation graph comprising data of messages of the conversation. comprising: a timeline manager configured to provide a stream of messages over a network, in a timeline of a user on the client application, the timeline manager configured to receive, via the network, a conversation view request from the client application, and and a content quality classifier configured to classify the messages into a plurality of sections based on content quality of the messages of the conversation graph, the plurality of sections including a first section and a second section. The first section has messages from the conversation graph determined to be of higher quality than the messages of the second section. The system includes an engagement predictor configured to determine a plurality of prediction results for each of a plurality of messages of the conversation graph using the prediction models, and a plurality of prediction results for each of the plurality of messages using the plurality of prediction results for the respective message. and an engagement scorer configured to compute an engagement value. the timeline manager is configured to rank the messages of the first section using the engagement values of the first section, and provide at least a subset of the messages of the first section over the network to be rendered on the timeline according to the ranking . According to other aspects, a non-transitory computer-readable medium storing a corresponding method and corresponding instructions may be provided.

[0027] In accordance with some aspects, a method, system, and/or non-transitory computer-readable medium may include one or more of the features above/below (or any combination thereof). The engagement predictor is configured to obtain a plurality of signals associated with the predictive models from one or more data services stored on the messaging platform, wherein the engagement predictor sends the plurality of signals to the predictive models to determine a plurality of predictive results. are configured to input The plurality of signals includes data structure-related signals associated with the dialog graph. The timeline manager is configured to separately rank the messages of the second section using the engagement values of the second section. The content quality classifier is configured to receive a message identifier of the message of the conversation graph, and determine that the message has a content quality corresponding to the first quality section using the classification model. The plurality of prediction results includes a mutual engagement probability, wherein the mutual engagement probability includes a probability value at which the messaging platform is predicted to receive a reply to an individual message of the conversation graph. The system may include a predictive model trainer configured to periodically train the predictive model based on one or more machine learning algorithms input with the training data, the predictive model trainer configured to provide the trained predictive model to the predictive manager. .

[0028] According to one aspect, there is a non-transitory computer-readable medium having stored thereon executable instructions, which, when executed by at least one processor, cause the at least one processor to: classify the messages into a plurality of sections based on the content quality of the messages of the conversation graph, the plurality of sections comprising a first section and a second section, the first section being of higher quality than the messages of the second section with messages from the determined dialog graph; determine a plurality of prediction results for each of a plurality of messages of the conversation graph, using the one or more prediction models, the plurality of prediction results including a mutual engagement probability, a positive engagement probability, and a negative engagement probability Ham ― ; rank the messages in the first section based on the plurality of prediction results for the messages in the first section, and rank the messages in the second section based on the plurality of prediction results for the messages in the second section and transmit at least a subset of the messages of the first quality section to be rendered on the client application according to the ranking. According to other aspects, a corresponding system and method may be provided.

[0029] In accordance with some aspects, a method, system, and/or non-transitory computer-readable medium may include one or more of the features above/below (or any combination thereof). The operations include obtaining training data from a client event log that stores information received from the client application and an injection log that stores information from a timeline manager running on the messaging platform, and a machine learning algorithm input with the training data. training predictive models based on The operations include receiving a message identifier of a first message of the conversation graph, determining, using the classification model, that the first message has a content quality corresponding to the first section, a message identifier of a second message of the conversation graph and determining, using the classification model, that the second message has a content quality corresponding to the second section. The operations may include receiving a request to render additional messages of the conversation graph on the client application and transmitting at least a subset of the messages of the second section to be rendered on the client application according to the ranking. These and other features are further discussed in the detailed disclosure.

1A illustrates a messaging system for ranking messages in conversation graphs using prediction results from predictive models according to an aspect.
1B illustrates a candidate message selector of a messaging system in accordance with an aspect.
1C illustrates examples of signals used to determine prediction results according to an aspect.
1D illustrates a predictive manager of a messaging system according to an aspect.
1E illustrates an example of generating training data for predictive models according to an aspect.
1F illustrates an example of a two-level ranking mechanism including a content quality classifier according to an aspect.
2 illustrates an example of a predictive model as a neural network according to an aspect.
3 illustrates a flow diagram illustrating example operations of a messaging system in accordance with an aspect;
4 illustrates a flow diagram illustrating example operations of a messaging system in accordance with an aspect.
5 illustrates a flow diagram illustrating example operations of a messaging system in accordance with an aspect.

[0040] 1A-1E illustrate a messaging system 100 for ranking messages in conversation graphs 126 using prediction results 118 from predictive models 112 in accordance with an aspect. The messaging system 100 includes a messaging platform 104 executable by a server computer 102 , and a client application 154 executable by a computing device 152 , in accordance with one aspect. The client application 154 communicates with the messaging platform 104 to send (and receive) messages to (and from) other users of the messaging platform 104 over the network 150 .

[0041] The client application 154 may be a social media messaging application through which users post and interact with messages. In some examples, client application 154 is a native application running on the operating system of computing device 152 or server computer 102 (or other server) along with a browser-based application of computing device 152 . It may be a web-based application running on it. Computing device 152 may be configured over network 150 using any type of network connections and/or application programming interfaces (APIs) in a manner that allows client application 154 and messaging platform 104 to communicate with each other. A messaging platform 104 may be accessed.

[0042] Computing device 152 may be a mobile computing device (eg, a smart phone, PDA, tablet, or laptop computer) or a non-mobile computing device (eg, a desktop computing device). Computing device 152 may also include, for example, a mobile network interface that enables computing device 152 to communicate with a cellular network, a Wi-Fi network interface that enables computing device 152 to communicate with a Wi-Fi base station, computing Various network interfaces, such as a Bluetooth network interface that enables device 152 to communicate with other Bluetooth devices, and/or an Ethernet connection or other wired connection that enables computing device 152 to access network 150 . includes a circuit.

[0043] Server computer 102 may be a single computing device or may be a representation of two or more distributed computing devices communicatively coupled to share workloads and resources. The server computer 102 is at least one processor and non-transitory computer-readable storing executable instructions that, when executed by the at least one processor, cause the at least one processor to perform the operations discussed herein. media may be included.

[0044] Messaging platform 104 is a computing platform for facilitating communication (eg, real-time communication) between user devices, one of which is shown as computing device 152 . The messaging platform 104 may store millions of accounts 141 of individuals, businesses, and/or entities (eg, pseudonymous accounts, new accounts, etc.). One or more users of each account 141 may use the messaging platform 104 to send messages to other accounts 141 within and/or external to the messaging platform 104 . In some examples, the messaging platform 104 may enable users to communicate “in real time,” eg, to chat with other users with minimal delay and to conduct a conversation with one or more other users during concurrent sessions. . In other words, the messaging platform 104 may allow a user to broadcast messages and send messages to one or more other users within a reasonable time frame (eg, less than two seconds) to facilitate live conversations between users. can be displayed. In some examples, recipients of the message may have a predefined graph relationship in the connection graph 134 with the account of the user broadcasting the message.

[0045] The connection graph 134 shows that certain accounts 141 in the messaging platform 104 are associated with a particular account 141 (eg, follow, be friends with, subscribe to, and the like, the particular account 141 ). ) and thus includes a data structure indicating whether it is subscribed to receive messages from a particular account 141 . For example, the connection graph 134 may link a first account with a second account, indicating that the first account is related to a second account. A user of the second account can view messages posted on the messaging platform 104 by the user of the first account (and/or vice versa). Relationships may include one-way (eg, follower/followee) and/or two-way (eg, friendship). Messages may be of any of a variety of lengths, which may be limited by a particular messaging system or protocol.

[0046] In some examples, users interested in viewing messages created by a particular user may choose to follow the particular user. The first user may follow the second user by identifying the second user as a user the first user wants to follow. After the first user indicates that he wants to follow the second user, the connection graph 134 will be updated to reflect the relationship, and the first user will be provided with messages created by the second user. Users may choose to follow multiple users. In addition, users can respond to messages and thus interact with each other. Users can also engage with messages, such as by sharing the message with their followers or by liking (or "like") the message the engagement is shared with their followers. can do.

[0047] Messages exchanged on the messaging platform 104 are stored in a message store 138 . Message store 138 may include one or more tables that store records. In some examples each record corresponds to a separately stored message. For example, the record may include a message identifier for a message posted to the messaging platform 104 , an author identifier identifying the author of the message (eg, @tristan), and the message content (eg, a URL to text, image, video, and/or web content). ), one or more participant account identifiers identified in the body of the message, and/or reply information identifying the parent message to which the message replies (if the message is a reply to a message).

[0048] The messaging platform 104 includes a conversation graph manager 136 that creates conversation graphs 126 , and a timeline manager 142 that injects a timeline 156 of messages into a client application 154 . The messaging platform 104 includes a candidate message selector 108 that selects a candidate subset 133 of messages from the conversation graph 126 . The messaging platform 104 includes a prediction manager 110 that obtains signals 106 related to prediction and inputs the signals 106 into one or more prediction models 112 to determine prediction results 118 . do. In some examples, prediction manager 110 calculates engagement values 116 based on prediction results 118 , and timeline manager 142 uses engagement values 116 to graph the conversation ( It ranks (and selects) the messages of 126 , and provides a ranked list 158 of messages to the client application 154 . The messaging platform 104 obtains training data 148 and trains the predictive models 112 using one or more machine learning algorithms 149 input with the training data 148 . includes

[0049] More specifically, the conversation graph manager 136 creates (and updates) one or more conversation graphs 126 as messages are exchanged on the messaging platform 104 . In some examples, conversation graphs 126 are stored in a data storage device associated with messaging platform 104 . In some examples, conversation graphs 126 are stored in timeline manager 142 . Messaging platform 104 may store a number of conversation graphs 126 (eg, hundreds, thousands, or millions of conversation graphs 126 ). Each conversation graph 126 may represent a structure of replies to an original non-reply message (eg, a root message). For example, whenever a user creates and posts an original non-reply message on the messaging platform 104 , a potential new conversation may be initiated. Others can then reply to the original or "root" messages and create their own reply branches. Over time, if the number of replies to the original non-reply message (and/or replies to replies to the original non-reply message) exceeds a threshold level, the conversation graph manager 136 displays the conversation A conversation identifier may be assigned to the graph 126 , and the conversation identifier may uniquely identify the conversation graph 126 . In some examples, conversation graph manager 136 may assign a conversation identifier to each message with a reply. For example, if the messaging platform has message A and then someone responds to it with message B, message A is assigned a conversation identifier that can be used to identify the conversation, which is discussed in detail below in conversation graph 126 . ) leads to In some examples, if there is a reply to the message, there is a conversation.

[0050] Conversation graph 126 may be a hierarchical data structure representing messages within a conversation. In some examples, dialog graph 126 includes a non-linear or linear data structure. In some examples, dialog graph 126 includes a tree data structure. Conversation graph 126 may include nodes 128 (or vertices) representing messages and edges 130 (or arcs) representing links between nodes 128 . Conversation graph 126 may store the message identifier of each message at each node 128 . In some examples, conversation graph 126 stores in each node 128 the user identifier of the author of the respective message. Conversation graph 126 may define one or more branches 132 of nodes 128 . In some examples, branch 132 is a portion (eg, a sub-tree) of dialog graph 126 that includes one or more nodes 128 . In some examples, branch 132 may be at least two nodes 128 connected by edge 130 , where one of nodes 128 is a leaf node. In some examples, branch 132 is a single-line concatenated message (eg, a leaf message, a first parent message coupled to a leaf message, a second parent message coupled to a first parent message, etc.) ) can be defined as

[0051] Note that the term “node” may refer to a message within the conversation graph 126 , or the term “message” may refer to a node 128 if the message is included as part of the conversation graph 126 . do. A particular node 128 may be linked to another node 128 via an edge 130 , and the direction of the edge 130 identifies the parent message. Nodes 128 may represent a root message, a message in a reply to a root message, messages in a reply to a message in a reply to a root message, and the like.

[0052] The conversation graph manager 136 may generate the conversation graph 126 based on the reply structure of the messages. A reply structure may be identified based on metadata associated with each message and/or reply information identified from within the message content. In some examples, a reply structure is identified based on metadata associated with each message received from the client application 154 to compose the message. For example, the user may click a reply link displayed below the message displayed on the user interface of the client application 154 . The client application 154 may then display a compose message box to draft a reply message. The client application 154 may submit metadata including a reply relationship with the reply message (eg, the message identifier of the parent message). In some examples, the reply relationship may be explicitly defined by the user within the message content (eg, identifying the user account 141 (eg, @tristan) within the body of the message). In this example, the reply structure may be identified by identifying one or more account identifiers and/or message identifiers mentioned within the body of the message.

[0053] Timeline manager 142 sends digital information, via network 150 , to enable client application 154 to render and display timeline 156 of social content on a user interface of client application 154 . make it Timeline 156 includes a stream of messages (eg, message A, message B, message C). In some examples, the stream of messages is arranged in reverse chronological order. In some examples, the message stream is arranged chronologically. In some examples, timeline 156 is a timeline of social content specific to a particular user. In some examples, timeline 156 includes a stream of messages curated (eg, created and assembled) by messaging platform 104 . In some examples, timeline 156 includes a list of messages resulting from a search on messaging platform 104 . In some examples, timeline 156 is a stream of messages posted by users from accounts 141 that are associated with user's account 141 of client application 154 (eg, when a user 104) a stream of messages from accounts 141 that it has chosen to follow. In some examples, the stream of messages includes a promoted message or re-shared messages.

[0054] When viewing messages on timeline 156 , the user may select one of the messages from timeline 156 (eg, message B), which causes client application 154 to generate conversation view request 121 and to the messaging platform 104 over the network 150 . In some examples, the selected message (eg, message B) may be referred to as a context message or focus message, which may serve as an entry point or reference point within the conversation graph 126 . The conversation view request 121 may be a request to retrieve messages from the conversation graph 126 . In some examples, the conversation view request 121 includes a message identifier of the selected messages and a user identifier of a user of the client application 154 . In some examples, the conversation view request 121 also includes the time of the request, which device the user is on, the operating system (OS) version, and/or other metadata associated with the request.

[0055] In response to the conversation view request 121 , the timeline manager 142 controls the prediction manager 110 to display the prediction results 118 for each message (or a subset thereof) of the conversation graph 126 . and compute an engagement value 116 for each message based on the prediction results 118 . For example, the prediction manager 110 includes an engagement predictor 125 and an engagement scorer 114 . In response to the conversation view request 121 , the engagement predictor 125 obtains signals 106 related to the prediction and inputs the signals 106 into the predictive models 112 to generate a conversation graph 126 . may determine prediction results 118 for each message (or a subset of messages) in . In some examples, the prediction results 118 include a mutual engagement probability 124 . In some examples, the prediction result 118 includes a mutual engagement probability 124 , and at least one of a positive engagement probability 120 or a negative engagement probability 122 . In some examples, the prediction results 118 include a mutual engagement probability 124 , a positive engagement probability 120 , and a negative engagement probability 122 .

[0056] Engagement scorer 114 uses prediction results 118 (eg, combines prediction results 118 to produce engagement value 116 ) for an engagement value ( 116) can be computed. Engagement values 116 are used by timeline manager 142 to rank messages in conversation graph 126 , which timeline manager 142 uses to rank list 158 in user's timeline 156 . ) via network 150 (or a subset of messages in conversation graph 126 ). Ranked list 158 (eg, message 1, message 2, message 3) is some of the messages of conversation graph 126 ranked according to engagement values 116 determined by prediction results 118 . or all of them.

[0057] In some examples, the candidate message selector 108 selects a portion of the messages of the conversation graph 126 to be analyzed by the prediction manager 110 . In other words, the candidate message selector 108 may select candidate messages from a larger number of messages included in the conversation graph 126 , where the candidate messages are the message to be subjected to predictive analysis by the predictive manager 110 . admit. In some examples, conversation graph 126 is relatively large, and when the number of nodes 128 included in conversation graph 126 exceeds a threshold level (eg, more than 3000 or 4000 messages), a candidate message selector ( 108 may select a subset of nodes 128 (instead of all nodes 128 ) for further analysis by prediction manager 110 . In other words, if the number of nodes 128 exceeds a threshold level, the candidate message selector 108 can identify a subset of nodes 128 , and identification of such nodes 128 performs the prediction. It is provided to the prediction manager 110 to do so. This may reduce (eg, significantly reduce) technical resource consumption and may allow the messaging platform 104 to deliver better quality responses. For example, if the conversation graph 126 has 30,000 responses (or nodes 128), but using the technique described above, the messaging platform 104 will display the top X amount of messages (eg, where X is 3000 days). can), collect signals for them, and then score these signals to select the top Y quantity of messages to display to the user (eg, Y could be 10), which gives the user It can reduce (eg, significantly reduce) the amount of computational resources without affecting the quality of the delivered responses.

[0058] For example, referring to FIG. 1B , the candidate message selector 108 may select a candidate subset 133 from the conversation graph 126 in response to the number of messages in the conversation graph 126 being greater than a threshold level, where A prediction result 118 is determined for each message in the candidate subset 133 . In some examples, the candidate subset 133 includes a number of messages that are less than the total number of messages in the conversation graph 126 . In some examples, the candidate message selector 108 selects the candidate subset 133 from one or more candidate message sources 131 . In some examples, candidate message selector 108 may merge candidates for a conversation view from multiple sources. For example, candidate message selector 108 may include a first set of messages included in conversation graph 126 from first source 131-1 and messages included in conversation graph 126 from second source 131-2. select a second set of , where the candidate subset 133 includes the first and second sets of messages.

[0059] In some examples, first source 131-1 includes messages in conversation graph 126 ranked according to most recently posted. In some examples, candidate message selector 108 selects a number n (eg, chronological order) of most recent messages in conversation graph 126 from first source 131-1. In some examples, the number n is in the range of 2000 to 4500. In some examples, the number n is in the range of 2500 to 3500. In some examples, second source 131 - 2 includes messages in conversation graph 126 ranked according to a relevance algorithm (eg, a heuristic algorithm). In some examples, the relevance ranking for the second source 131 - 2 is based on whether the messages in the conversation graph 126 are from user accounts 141 linked to the user account 141 of the user of the client application 154 . based on In some examples, the relevance algorithm may include signals such as the amount of engagements received, the number of likes, comments and/or re-shares, and message meta such as whether the message has a photo, video and/or link. Signals representing data and/or signals representing health-related metadata are used, such as when the message is toxic and the NSFW or the author of the message has recently been reported. The candidate message selector 108 selects the number p of top-ranked messages of the conversation graph 126 from the second source 131 - 2 . In some examples, the number p is in the range of 200 to 1500. In some examples, the number p is in the range of 500 to 1000. In some examples, number p is less than number n. The candidate subset 133 may include a number n of messages from the first source 131-1 and a number p of messages from the second source 131-2, where the prediction results are 118 is determined for each message in the candidate subset 133 .

[0060] For at least some (or all of the nodes 128 ) of the nodes 128 (or those identified by the candidate message selector 108 ) in the conversation graph 126 , the engagement predictor 125 is a predictive model Elements 112 may be used to predict a positive engagement probability 120 , a negative engagement probability 122 , and/or a mutual engagement probability 124 for an individual message. For example, in response to the conversation view request 121 , the prediction manager 110 controls the engagement predictor 125 to obtain the signals 106 and the signals 106 to the predictive models 112 . applied to determine prediction results 118 for the selected nodes 128 of the dialog graph 126 .

[0061] Prediction manager 110 may obtain signals 106 from one or more data services 165 . The data service(s) 165 may be components on the messaging platform 104 that compute or otherwise derive data obtained by the messaging platform 104 and/or the client application 154 . In some examples, prediction manager 110 may communicate with data services 165 via a server communication interface. In some examples, prediction manager 110 may obtain at least a portion of signals 106 from data service(s) 165 via one or more APIs. In some examples, in response to the conversation view request 121 , the predictive manager 110 sends a thrift call or remote procedure call (RPC) to the data service(s) 165 , after which the appropriate at least some of the signals 106 from the data service(s) 165 . In some examples, the prediction manager 110 sends a representational state transfer (REST) request to the data service(s) 165 and then receives at least some of the signals 106 from the appropriate data service(s) 165 . can receive In some examples, prediction manager 110 communicates with data service(s) 165 via a GraphQL request. In some examples, prediction manager 110 obtains a portion of signals 106 from other components of messaging platform 104 , including conversation graph manager 136 and/or timeline manager 142 .

[0062] Signals 106 may include signals generated by the messaging platform 104 and/or generated by the client application 154 related to predictive user results to display messages on the client application 154 . can For example, signals 106 may include signals generated by client application 154 based on user interaction with client application 154 . Signals generated by the client application 154 may be sent to the messaging platform 104 for storage. Signals generated by the client application 154 may include positive user engagement with messages (eg, favorites, likes, reshares) and/or negative user engagement with messages (eg, reporting of profane content). ) may include signals representing engagement information, such as. In some examples, signals 106 may include signals generated by messaging platform 104 . In some examples, signals generated by messaging platform 104 may include data generated from a user's connection graph 134 , data generated from conversation graph 126 , data generated from user actions on the platform (eg, data generated by a user data generated from the content of messages, such as the result of a semantic analysis that predicts user sentiment or the result of a topic analysis that determines the topic of one or more messages) and/or may include signals representing

[0063] 1C , signals 106 are data structure-related signals 101 associated with conversation graph 126 , health providing messages from conversation graph 126 to a user of client application 154 . health-related signals 103 associated with, engagement signals 105 associated with user engagement for messages in conversation graph 126 , and social graph associated with data from user's connection graph 134 . Signals 107 , historical aggregate signals 109 associated with data aggregated by messaging platform 104 , content-related signals 111 associated with the content of messages in conversation graph 126 , and/or a user may include similarity signals 113 representing how similar the message is to other messages it has liked or liked and/or how similar the user is to other users with whom it has engaged with the message. However, signals 106 may include any type of categorization or granularity of signals related to predicting user results from displaying messages.

[0064] The data structure-related signals 101 may include signals related to data from the conversation graph 126 . In some examples, the data structure-related signals 101 include the number of nodes 128 , the number of edges 130 , the number of branches 132 , the length or size of each branch 132 , the parent The number of nodes, the number of child nodes, the number of leaf nodes, the height of the dialog graph 126 (eg, the length of the longest path to the leaf node) and/or the depth of the node (eg, the depth of the node is up to the root node) is the path length of ). In some examples, data structure-related signals 101 include one or more signals representing a subset of dialog graph 126 , such as branch 132 , or a number of unique authors within dialog graph 126 . In some examples, data structure-related signals 101 include signals representing a location of a message having a particular type of data (eg, image, video, link to a video, etc.) within conversation graph 126 . In some examples, with respect to a particular message in the conversation graph 126 , the data structure-related signals 101 determine whether the message is a child node, whether the message is a parent node, whether the message is a leaf node, or whether the message is a leaf node. It may include signals representing the location of the message, the location of the branch containing the message 132 , the size of the branch containing the message 132 , and the depth of the message in the conversation graph 126 .

[0065] The data structure-related signals 101 may include branch context characteristics. In some examples, the data structure-related signals 101 may be the number of replies in branch 132 , the number of conversations in branch 132 , the number of conversations in branch 132 between the user of the client application 154 and the originator of the root message. contains signals representing the number of conversations, the number of conversations in branch 132 between the user of the application 154 and the user mentioned in the new message, and/or the number of conversations between a particular node (eg, focus message) and a leaf node. do. In some examples, with respect to branching context features, a conversation may be defined back and forth between at least two users. In some examples, a conversation may be defined as a message posted by user A, a reply posted by user B, and then a reply posted by user A.

[0066] In some examples, the conversation graph manager 136 may receive the conversation identifier from the prediction manager 110 , and then derive or determine the data structure-related signals 101 from the conversation graph 126 according to the conversation identifier and , store the data structure-related signals 101 in a data store on the messaging platform 104 . In some examples, in response to the conversation view request 121 , the prediction manager 110 derives or determines the data structure-related signals 101 and then determines the prediction results 118 . The dialog graph manager 136 may be controlled to receive the data structure related signal 101 to be used together with the 112 from the dialog graph manager 136 . In some examples, prediction manager 110 may derive or determine data structure-related signals 101 from conversation graph 126 . In some examples, in response to the conversation view request 121 , the prediction manager 110 sends a conversation identifier to the conversation graph manager 136 , and then receives the conversation graph 126 from the conversation graph 126 . The data structure-related signals 101 may be derived or determined.

[0067] Health-related signals 103 may include health-representing signals presenting a message of conversation graph 126 to a user of client application 154 . In some examples, the health-related signals 103 are a signal representing whether the user of the client application 154 has restricted (eg, blocked, muted, etc.) the author of a message in the conversation graph 126 in the past. may include The health-related signals 103 may be stored in a data store on the messaging platform 104 . In some examples, predictive manager 110 may send a request to data service 165 (eg, health data service) to obtain health-related signals 103 , where the request is conversation graph 126 . message identifiers of the messages of , and/or the user identifier of the user of the client application 154 .

[0068] Engagement signals 105 may represent user engagement data associated with a message in conversation graph 126 . In some examples, engagement signals 105 may be the number of engagements for a message in conversation graph 126 (eg, the number of times messages were favorited or liked, number of replies to the message, number of replies to the message, shared number of times). In some examples, the engagement signals 105 are the engagements of users following the user of the client application 154 in the user's connection graph 134 (eg, the message is in the user's connection graph 134 ). one or more signals indicative of having one or more engagements provided by users following the user of the client application 154 . In some examples, prediction manager 110 obtains engagement signals 105 from data service 168 that stores engagement data. In some examples, prediction manager 110 may send a request that may include message identifiers of conversation graph 126 , and prediction manager 110 may send engagement signals 105 from data service 165 . can receive

[0069] Social graph signals 107 may include signals representing information from connection graph 134 . In some examples, social graph signals 107 include the number of times the user of the client application 154 has liked or liked the message author's messages over a period of time, the user's link with the message author in the connection graph 134 . and/or signals representing the number of times the user has reshared or replied to the message author's message over a period of time. In some examples, prediction manager 110 obtains social graph signals 107 from data service 168 that stores social graph signals. In some examples, prediction manager 110 can send a request that can include a user identifier of a user of client application 154 , and prediction manager 110 can send social graph signals 107 from data service 165 . ) can be received.

[0070] The historical aggregation signals 109 may include signals representing user behavior on the messaging platform 104 . In some examples, historical aggregation signals 109 include the number of times the user of the client application 154 has favorited messages on the messaging platform 104 over a period of time, the number of times the user of the client application 154 has viewed the messaging platform during the period of time. signals representing the number of times messages on 104 have been reshared and/or the number of times a user of the client application 154 has responded to messages on the messaging platform 104 over a period of time. The time period may be within the past day, last month, last year, etc. In some examples, the historical aggregation signals 109 include signals representing the number of times a user of the client application 154 has liked, liked, reshared, and/or replied to messages containing an image or video. can do.

[0071] In some examples, historical aggregation signals 109 indicate that a user of client application 154 may bookmark, like, reshare messages from accounts 141 linked to the user in connection graph 134 and/or A signal representing the number of times it has responded, and/or the number of times the user has bookmarked, liked, reshared and/or replied to messages from accounts 141 not linked to the user in the connection graph 134 . may include In some examples, prediction manager 110 obtains historical aggregate signals 109 from a data store on messaging platform 104 . In some examples, prediction manager 110 sends a request to data service 165 to obtain historical aggregation signals 109 . In some examples, the request includes a user identifier of a user of the client application 154 . In some examples, historical aggregation signals 109 include batch aggregation information and real-time aggregation information. Batch aggregation information may include a relatively long history (eg, more than 50 days). In some examples, the batch aggregate information may not include interactions from the past day (or past few days). The real-time aggregate information may include a history of relatively recent interactions (eg, within the last 30 minutes or so).

[0072] The content-related signals 111 may include signals representing one or more aspects of the contents of the message of the conversation graph 126 . In some examples, content-related signals 111 may include signals representing the length of the message and/or whether the content includes text, video, or image. In some examples, prediction manager 110 obtains content-related signals 111 from a data store on messaging platform 104 . In some examples, prediction manager 110 sends a request to data service 165 to obtain content-related signals 111 . In some examples, the request includes message identifiers of messages of conversation graph 126 .

[0073] Similarity signals 113 may include one or more signals representing how similar the message is to other messages that the user has liked or liked. For example, similarity signals 113 may represent a level of similarity between a particular message and one or more other messages that the user has liked or liked, and if the level of similarity is relatively high, this is an indication of potential positive gauge. can provide In some examples, similarity signals 113 may include one or more signals representing how similar the user is to other users engaged in the message. For example, if the user's user profile is determined to be relatively similar to the user profiles engaged in the message, this may provide an indication of potential positive engagement. In some examples, prediction manager 110 may obtain similarity signals 113 from a data store on messaging platform 104 . In some examples, prediction manager 110 may send a request to data service 165 to obtain similarity signals 113 . In some examples, the request may include the user's message identifiers and/or user identifiers.

[0074] In some examples, it is technically possible to obtain at least some of the signals 106 used for prediction (eg, particularly for signals related to viewer-author relationships whenever the message goes viral). Difficulties or difficulties exist. Popular messages may have a relatively large amount of responses (eg, in some cases greater than 80K). This also means that multiple users can try and view popular messages at the same time. For each viewer, the messaging platform 104 may obtain a relationship with all authors who have responded to popular messages. Using the techniques described above with respect to the candidate message selector 108, the messaging platform 104 may filter the total number of messages from 80K to 4K, which is still 4K for obtaining relationship signals. It may mean that there may be viewer-writer pairs.

[0075] Also, in some examples, the viewer author relationship may not even exist because the viewer does not follow the author. To handle these types of situations, instead of querying by viewer-author as a key to data service 166, messaging platform 104 is queried by viewer identifier at once and all other authors You can get his relationship with them. Messaging platform 104 may then determine if any authors overlap with the authors of replies and maintain signals if appropriate. This reduces network calls by a relatively large amount because prediction manager 110 can make and send one call instead of making 4K calls per viewer.

[0076] Message level signals may present other technical difficulties. For example, for large conversations, the messaging platform 104 may query other data services 166 with 4K queries for each viewer. This can cause “hot-key” problems in which data service 166 receives too many queries for the same message identifier. To overcome the difficulties identified above, the messaging platform 104 may employ in-memory caching. If the underlying data service 166 indicates a hot-key, the service will cache the features in memory. For example, message T is word of mouth and has responses R1, R2...R4000, the message characteristics being the number of characters in the message. 1000 users then send requests for the same message simultaneously (or nearly simultaneously). When data service 166 presents a hot-key, instead of invoking data service 166 for each user, messaging platform 104 stores in memory for R1,R2..R4000 for a very short duration. You can store character values individually and just use them.

[0077] Predictive models 112 are predictive models trained by one or more machine learning algorithms 149 input with training data 148 . Machine learning algorithms 149 may include one or more of Markov models, logistic regression, decision tree analysis, random forest analysis, neural nets, and combinations thereof. In general, machine learning is the field in which a computer learns to perform classes of tasks while using feedback generated from experience or data that a machine learning process acquires while performing classes of tasks. In supervised machine learning, a computer may learn one or more rules or functions that map between exemplary inputs and desired outputs as predetermined by an operator or programmer. The labeled data points can then be used to train the computer. Unsupervised machine learning can involve using unlabeled data, and a computer can then identify implicit relationships in the data, for example, by reducing the dimensionality of the data set.

[0078] As shown in FIG. 1D , the predictive models 112 may include a positive engagement model 115 , a negative engagement model 117 , and a mutual engagement model 119 . The positive engagement model 115 is configured to compute a positive engagement probability 120 , and the negative engagement model 117 is configured to compute a negative engagement probability 122 , The model 119 is configured to compute the mutual engagement probability 124 . For example, in response to the conversation view request 121 (eg, the user selects message B), the engagement predictor 125 obtains signals 106 and signals 106 (which are also user identifiers). and message identifier) to the positive engagement model 115, negative engagement model 117 and/or mutual engagement model 119 to obtain positive engagement probability 120, negative engagement An engagement probability 122 and/or a mutual engagement probability 124 may be determined, respectively.

[0079] The positive engagement probability 120 indicates a probability value that the user is expected to positively view or engage with the message. In some examples, the probability value for positive engagement probability 120 is a number (x) between a first value and a second value, wherein the first value is a value at which the user is expected to positively view or engage the message. 0 represents a chance, and a second value represents a 100% likelihood that the user will positively view or engage the message. In some examples, the probability value for positive engagement probability 120 is positive. In some examples, the first value is 0 and the second value is 1. However, the values for the first value and the second value may define any type of range (eg, 0 to 1, 0 to 50, 0 to 100, etc.). In other words, the positive engagement probability 120 indicates the level of likelihood that the user is predicted to like, like, or share the message.

[0080] Negative engagement probability 122 indicates a probability value that the user is expected to negatively view or engage with the message. In some examples, the probability value for negative engagement probability 122 is a number y between a first value and a second value, wherein the first value is a value at which the user is expected to negatively view or engage the message. 0 represents a likelihood, and a second value represents a 100% likelihood that the user will negatively view or engage the message. In some examples, the probability value for negative engagement probability 122 is negative. In some examples, the first value is 0 and the second value is negative. However, the values for the first value and the second value may define any type of range (eg, 0 to -1, 0 to -50, 0 to -100, etc.). In some examples, negative engagement probability 122 indicates a level of likelihood that a user is expected to block the author of the message, unfollow the author of the message, and/or report the message as abusive.

[0081] Mutual engagement probability 124 indicates a probability value at which the user is predicted to continue developing dialog graph 126 . In some examples, the probability value for the mutual engagement probability 124 is a number z between a first value and a second value, wherein the first value predicts that the user will continue to develop the conversation graph 126 . represents a zero probability of being, and the second value represents a 100% probability that the user is expected to continue developing the conversation graph 126 . In some examples, the probability value for mutual engagement probability 124 is positive. In some examples, the first value is 0 and the second value is 1. However, the values for the first value and the second value may define any type of range (eg, 0 to 1, 0 to 50, 0 to 100, etc.). In some examples, the mutual engagement probability 124 indicates a level of likelihood that the user is expected to reply to the message and thus further develop the conversation graph 126 .

[0082] With respect to a particular candidate node 128, the engagement predictor 125 uses the positive engagement model 115 to determine that the message has a certain probability of receiving positive engagement by the user; Using the negative engagement model 117, it is determined that the message has a certain probability of receiving a negative engagement by the user, and using the mutual engagement model 119, the message is mutually engaged by the user. It may be determined that it has a certain probability of receiving an engagement. In some examples, for each candidate node 128 , the engagement predictor 125 determines all three prediction results 118 . In some examples, for each node 128 , engagement predictor 125 determines mutual engagement probability 124 , and at least one of positive engagement probability 120 or negative engagement probability 122 . predict one.

[0083] Engagement scorer 114 uses prediction results 118 to compute engagement values 116 for messages in conversation graph 126 . Engagement value 116 may provide an overall engagement value (eg, indicative of a level of relevance for a user of client application 154 ) for an individual node 128 , which may include messaging platform 104 . ) encourages healthier conversations. For example, with respect to a particular candidate node 128, the engagement scorer 114 combines a positive engagement probability 120, a negative engagement probability 122, and a mutual engagement probability 124 to An engagement value 116 may be provided, which may be used to select the most relevant nodes 128 for the user. For example, engagement scorer 114 may combine values of prediction results 118 to determine engagement value 116 for a particular message. If the probability value of negative engagement probability 122 is relatively high (eg, having a greater negative value), then this value is the positive value of mutual engagement probability 124 and positive engagement probability 120 . values can be offset. In a simple example, if the positive engagement probability 120 is +10, the negative engagement probability 122 is -10, and the mutual engagement probability 124 is +10, the engagement value for the message ( 116) is +10.

[0084] In some examples, engagement scorer 114 weights prediction results 118 and then weights positive engagement probability 120 , negative engagement probability 122 , and mutual engagement An engagement value 116 may be computed based on the probability 124 . In some examples, the weight applied to the mutual engagement probability 124 is greater than the weight applied to the negative engagement probability 122 .

[0085] Engagement values 116 are used to select a relevant message or branch of messages in conversation graph 126 to be rendered to the user. For example, timeline manager 142 receives engagement values 116 from prediction manager 110 and uses engagement values 116 to rank messages in conversation graph 126 (eg, best from the lowest). The timeline manager 142 may provide at least a subset of the messages of the conversation graph 126 over the network 150 to be rendered on the timeline 156 according to a ranking. In some examples, timeline manager 142 provides only a subset of the messages of conversation graph 126 to be rendered on timeline 156 , where the subset includes higher ranked messages in conversation graph 126 . do. Timeline manager 142 may then receive a request from client application 154 for additional messages in conversation graph 126 (eg, a user to view more messages in conversation graph 126 ). selecting a user affordance), timeline manager 142 may select the next group of messages from conversation graph 126 to be sent to client application 154 . In this way, the messaging system 100 may collapse portions of the conversation graph 126 that are less likely to provide positive engagement, but may then surface these messages when requested by the user.

[0086] In some examples, timeline manager 142 uses engagement values 116 to render on timeline 156 on one or more branches 132 (or branch 132 ) of dialog graph 126 . subset) is selected. For example, if branch 132 includes one or more nodes 128 with high engagement values 116 (or engagement values 116 above a threshold level), then branch 132 to the user Notwithstanding the fact that it may include one or more nodes 128 with a low engagement value 116 (or engagement values 116 below a threshold level) to provide more context for the conversation. , the timeline manager 142 may select the entire branch 132 to be rendered as part of the messages delivered to the client application 154 . In some examples, a particular branch 132 is associated with an overall combined value, which may be the average of the engagement values 116 for nodes 128 within the particular branch 132 . The timeline manager 142 may then rank the branches 132 according to their overall engagement values.

[0087] In some examples, the timeline manager 142 provides nodes with high engagement values (or engagement values 116 above a threshold level) for inclusion in the set of messages provided to the client application 154 . Choose (128). In some examples, timeline manager 142 determines that according to highest-lowest engagement values 116 (eg, where branches 132 or nodes 128 with highest engagement values 116 are user first), the selected branches 132 and/or nodes 128 are ranked. Because the messaging platform 104 incorporates (or predicts) the mutual engagement probability 124 within its scoring algorithm, the messaging platform 104 encourages more conversations on the messaging platform 104 .

[0088] In some examples, ranked list 158 represents a subset of messages of conversation graph 126 that are determined to be relevant to the user. For example, some messages in conversation graph 126 may relate to a first user, while other messages in conversation graph 126 may relate to a second user. In contrast, some conventional approaches use a vote-based mechanism that can provide the same view for each. More specifically, the engagement predictor 125 obtains the signals 106 (eg, engagement history, connection graph data, etc.) associated with the first user and for each message in the conversation graph 126 . Prediction results 118 may be obtained, which are then used to compute engagement values 116 . Timeline manager 142 may receive engagement values 116 from prediction manager 110 , and then use engagement values 116 to rank messages in conversation graph 126 ( This may increase the likelihood that the user will continue the conversation (eg, by incorporating the mutual engagement probability 124 ).

[0089] However, with respect to the second user, the engagement predictor 125 may obtain signals 106 associated with the second user, and prediction results 118 tailored to the second user, This is then used to compute the engagement values 116 . The timeline manager 142 may then use the engagement values 116 to rank the messages in the conversation graph 126 . Accordingly, the messages in the ranked list 158 displayed on the client application 154 for the second user may be different from the messages in the ranked list 158 displayed on the client application 154 for the first user. have.

[0090] The messaging platform 104 trains predictive models 112 (eg, positive engagement model 115 , negative engagement model 117 , and mutual engagement model 119 ) and uses the trained predictive models. and a predictive model trainer 140 that provides 112 to engagement predictor 125 to enable engagement predictor 125 to determine predictive results 118 when determining conversation views. In some examples, predictive model trainer 140 is configured to run periodically (eg, daily, weekly, monthly) to retrain (and thereby update) predictive models 112 . In some examples, predictive model trainer 140 operates in offline mode to train predictive models 112 using training data 148 , and then uses in online mode when predictive manager 110 is active. The prediction models 112 are transmitted to the prediction manager 110 as much as possible. In some examples, predictive model trainer 140 assigns weights and biases of positive engagement model 115 , negative engagement model 117 , and mutual engagement model 119 to predictive manager 110 . send.

[0091] In some examples, predictive model trainer 140 runs on server computer 102 . In some examples, predictive model trainer 140 runs on a computing device separate from server computer 102 , where predictive model trainer 140 and engagement predictor 125 communicate with each other via a networking interface. 1D , predictive model trainer 140 trains predictive models 112 using training data 148 in accordance with one or more machine learning algorithms 149 . Training data 148 may include one or more (or any combination) of data discussed with respect to signals 106 , which data may include historical data from a previous time period, eg, last day, last month. , data from last year, etc. For example, training data 148 may include data structure-related signals 101 , health-related signals 103 , engagement signals 105 , social graph signals 107 , historical aggregation signals 109 . ) and content-related signals 111 .

[0092] 1E illustrates an example of obtaining training data 148 to train a predictive model 112 in accordance with an aspect. For example, timeline manager 142 can inject a stream of messages into client application 154 . As part of one or more stream injections, timeline manager 142 may store information about message injections in injection log 127 . The injection log 127 may identify which messages were provided to the client application 154 and may identify specific characteristics associated with injections. In some examples, the features may include information related to any of the signals 106 described above. The injection log 127 also includes details about the prediction manager 110 , such as engagement values 116 associated with messages in the prediction results 118 and/or one or more conversation graphs 126 . can do. As users interact with messages on the client application 154 , the client event log 123 displays user engagement such as whether the user has responded to messages, liked or liked certain messages, re-shared messages, and the like. Save the information. In some examples, the client event log 123 includes information such as whether the message had a photo or video, and/or whether the message was from a particular depth of the conversation graph 126 . The information in the client event log 123 may be sent to the messaging platform 104 for storage. Training data 148 may include information from client event log 123 and information from injection log 127 , and training data 148 is used to train predictive models 112 . In some examples, predictive model trainer 140 periodically trains predictive models 112 in offline analysis and sends the results of the training (eg, weight/bias) to predictive manager 110 .

[0093] In some examples, the messaging platform 104 is configured to perform a two-level ranking mechanism with respect to the messages of the conversation graph 126 . For example, two-level ranking may provide an effective mechanism for handling profanity in conversations on the messaging platform 104 . In some examples, the two-level ranking includes hard ranking and soft ranking. In some examples, hard ranking includes sectioning the messages of conversation graph 126 into different sections (or groups) based on author and/or confidence level at which the message is considered profane. In some examples, the hard ranking is not personalized from the perspective of the user requesting the conversation view request 121 . Rather, hard ranking may be performed using signals to the content itself and/or signals to the author of the message. In some examples, soft ranking computes the prediction results 118 and generates engagement values 116 for messages in the conversation graph 126 and then ranks each according to the engagement values 116 . Includes ranking the messages in a section. In some examples, as described above, the soft ranking is personalized from the perspective of the user requesting the conversation view request 121 . In this way, a higher quality section of messages ranked according to engagement values 116 is presented to the user. In some examples, hard ranking is performed first, followed by soft ranking. However, in some examples, hard ranking may be personalized by using the output of predictive models 112 to perform hard ranking.

[0094] As shown in relation to FIGS. 1A and 1F , in some examples, the messaging platform 104 classifies (eg, categorizes the messages of the conversation graph 126 into a plurality of sections 160 based on the content quality of the messages (eg, and a content quality classifier 147 configured to section). The content quality of messages may include profanity (or toxic) probabilities (or beliefs) at the message level or at the author level. In some examples, the content quality classifier 147 may be based on the proficiency of profanity for a particular message (e.g., it may be based on the probability that the content itself is profane or that the author or account generating the message may be linked to profanity on the platform) can be determined (or received). If the probability of profanity is relatively high (or if the author and/or the message has a relatively high level of confidence that the message is considered profane), the content quality classifier 147 divides the message into a section 160 (eg, low quality or profane section). If it is possible to classify, or if the probability of profanity is relatively low (or if the message and/or author has a relatively low level of confidence that the message and/or author is considered profane), the content quality classifier 147 may classify the message into another section 160 (e.g., high quality or non-profanity section). In other words, the content quality classifier 147 is configured to partition the nodes 128 (or branches 132 ) of the conversation graph 126 into sections 160 . Each section 160 may refer to a different categorization (or classification) of quality levels (or profanity (or toxic) levels). The content quality classifier 147 may determine that the first message has a relatively low quality and the second message has a relatively high quality, where the content quality classifier 147 may determine the first message (or branch 132 containing the first message). )) to the first section (eg, the first section is considered a low quality section), and the second message (or branch 132 containing the second message) to the second section (eg, the second section) Sections are considered high quality sections).

[0095] The classification or sectioning performed by the content quality classifier 147 may be considered the first part of a two-level ranking mechanism (eg, hard ranking). In some examples, sections 160 may include at least two different sections, each representing a distinct quality category or classification. In some examples, sections 160 include low quality section 162 , medium quality section 164 , and high quality section 166 . In some examples the term section may refer to a category or classification. Although three sections 160 are shown in FIG. 1F , sections 160 may include any number of sections including two sections or any number of more than three sections. After classification by the content quality classifier 147, the low-quality section 162 identifies or contains nodes 128 (or branch 132) from the dialog graph 126 that has been determined to be of low quality, and the medium-quality section ( 164 identifies or includes nodes 128 (or branch 132 ) from dialog graph 126 determined to be of medium quality, and high quality section 166 is a node from dialog graph 126 determined to be of high quality Identify or include s 128 (or branch 132 ). In some examples, the medium quality section 164 contains messages that may be abusive or there is a probability that the account posting the message may be linked to abusive behavior with medium confidence. In some examples, the low-quality section 162 contains messages that are likely to be judged as abusive if reported and reviewed by bystanders, contain untrusted links, or link to abusive behavior with a high probability that the account posting the message is do.

[0096] In some examples, the content quality classifier 147 classifies the nodes 128 (or branches 132 ) of the conversation graph 126 into sections 160 , the classification model 161 (or multiple models). ) is included. In some examples, classification model 161 is a machine learning model that is trained using training data applied to one or more machine learning algorithms. In some examples, the classification model 161 comprises a neural network. Training data is acquired over a period of time (eg, days, weeks, months, years, etc.) and then the classification model ( 161) can be used to train (or retrain).

[0097] The content quality classifier 147 obtains one or more signals 163 associated with the classification model 161 and inputs the signals 163 to the classification model 161 so that any messages in the conversation graph 126 are identified as low quality sections ( 162 ), medium quality section 164 , or high quality section 166 . For a particular message (eg, node 128 ) of conversation graph 126 , the output of classification model 161 is a quality classification, eg, low-quality section 162 , medium-quality section 164 , or high-quality section 166 . ) can be identified. In some examples, nodes 128 (or branches 132 ) of dialog graph 126 are annotated with section 160 (or classification) determined by classification model 161 . In some examples, the content quality classifier 147 stores the classification of nodes 128 (or branches 132 ) in a data store on the messaging platform 104 .

[0098] To classify a particular message in the conversation graph 126 , in some examples, the signals 163 may include one or more signals generated by the client application 154 and/or one or more signals generated by the messaging platform 104 . includes signals. In some examples, signals 163 include one or more signals that are different from signals 106 used to determine prediction results 118 . In some examples, signals 163 include one or more signals representing content of the message. In some examples, signals 163 include one or more signals representing a user history of an author who created the content. In some examples, signals 163 include one or more signals that model a user profile of an author who created the content. In some examples, signals 163 include one or more signals that model an author's behavior on messaging platform 104 . In some examples, signals 163 may include signals representing message text, a history of the user's actions, such as logging history, the type of device the user is using, the number of times they have been reported as abusive, and the like. Based on these signals 163 , the content quality classifier 147 can predict whether the message is abusive and/or the user is abusive. Once the probabilities (or beliefs or scores) have been determined, the content quality classifier 147 can make sectioning decisions. For example, if the message is abusive with high confidence and the user who wrote it is also violent with high confidence, content quality classifier 147 may classify the message in a worst quality section, eg, low quality section 162 .

[0099] In some examples, content quality classifier 147 may receive a message identifier of the message and/or a user identifier of an author who created the content, and in response to the message identifier and/or user identifier, content quality classifier 147 Signals 163 to be input to the classification model 161 may be obtained. In some examples, the content quality classifier 147 may use the message identifier to obtain the content of the message from the message store 138 . In some examples, instead of receiving the message identifier, the content quality classifier 147 may receive the content of the message. In some examples, the content quality classifier 147 may use the user identifier to obtain one or more signals belonging to the author by communicating with one or more data services 165 in any manner as described with reference to FIG. 1C . have. In some examples, classification of the message into one of sections 160 may be performed upon message creation. In some examples, the classification of the message is performed at (or about) the time of receipt of the conversation view request 121 . In some examples, the content quality classifier 147 can convert the signals 163 into a format compatible with the classification model 161 .

[00100] In some examples, timeline manager 142 and/or prediction manager 110 may communicate with content quality classifier 147 via a server communication interface (eg, thrift call, REST call, GraphQL request, etc.), where The message identifier and/or user identifier are sent to the content quality classifier 147, which annotates the node 128 (or branch 132) of the conversation graph 126 as determined by the classification; and / or return the classification to the service requesting the classification information through the server communication interface. The content quality classifier 147 evaluates the signals 163 to predictive model(s) ( 161) can be provided. In some examples, unlike the prediction results 118 , the message classification into sections 160 is not personalized for the user associated with the conversation view request 121 .

[00101] In response to the conversation view request 121 , as previously discussed above, the timeline manager 142 controls the prediction manager 110 to display each message (or a subset thereof) in the conversation graph 126 . generate prediction results 118 for each message and compute an engagement value 116 for each message based on the prediction results 118 . For example, engagement predictor 125 obtains signals 106 associated with prediction and inputs signals 106 to predictive models 112 for each message (or sub-messages) in conversation graph 126 . set) for prediction results 118 . In some examples, the prediction results 118 include a mutual engagement probability 124 . In some examples, the prediction result 118 includes a mutual engagement probability 124 , and at least one of a positive engagement probability 120 or a negative engagement probability 122 . In some examples, the prediction results 118 include a mutual engagement probability 124 , a positive engagement probability 120 , and a negative engagement probability 122 .

[00102] Engagement scorer 114 uses prediction results 118 (eg, combines prediction results 118 to produce engagement value 116 ) an engagement value for an individual message. (116) can be computed. Engagement values 116 are then used by timeline manager 142 to rank messages in low quality section 162 , medium quality section 164 , and high quality section 166 individually.

[00103] In some examples, the timeline manager includes lists of messages classified as low quality section 162 from content quality classifier 147 , lists of messages classified as medium quality section 164 from content quality classifier 147 , and Lists of messages classified as high quality section 166 may be received from content quality classifier 147 . The timeline manager may also receive engagement values 116 from the prediction manager 110 . Ranking of messages within a particular section 160 using the engagement values 116 for messages within the respective section 160 may be considered a second part of the two-level ranking mechanism of FIG. 1F . In some examples, the second part of the two-level ranking mechanism may be considered soft ranking of messages.

[00104] In some examples, the timeline manager 142 is a first ranked list 158 - 1 of messages classified in the low quality section 162 , ranked using the engagement values 116 from the prediction manager 110 . ) can be created. In some examples, the first ranked list 158 - 1 includes branches 132 of the conversation graph 126 classified as low quality, the branches 132 being low quality according to their engagement values 116 . Within section 162 are ranked. The timeline manager 142 uses the engagement values 116 from the prediction manager 110 to generate a second ranked list 158-2 of messages classified in the medium quality section 164, ranked. can do. In some examples, second ranked list 158 - 2 includes branches 132 of conversation graph 126 classified as medium quality, branches 132 according to their engagement values 116 . They are ranked within the medium quality section 164 . The timeline manager 142 may use the engagement values 116 from the prediction manager 110 to generate a third ranked list 158-3 of messages classified in the high quality section 166, ranked. can In some examples, the third ranked list 158 - 3 includes branches 132 of the conversation graph 126 classified as high quality, the branches 132 being high quality according to their engagement values 116 . Within section 166 are ranked.

[00105] The timeline manager 142 may transmit at least a subset of the messages of the high quality section 166 according to the rank to be rendered on the client application 154 (eg, at least a portion of the third ranked list 158-3). sent). In this way, the user of the client application 154 can see messages that are considered high quality, which are also ranked according to their engagement values 116 as well. The timeline manager 142 may request subsequent request(s) to view additional messages from the conversation graph 126 , where the timeline manager 142 selects at least a portion of the second ranked list 158-2. After transmitting, at least a portion of the first ranked list 158 - 1 may be transmitted.

[00106] In some examples, the timeline manager 142 arranges the conversation graph 126 according to the order of the third ranked list 158-3, the second ranked list 158-2, then the first ranked list 158-1. ) to rank the messages. Accordingly, the client application 154 first renders the messages from the high quality section 166 ranked according to the engagement values 116 , and the medium quality section 164 ranked according to the engagement values 116 . Render messages from , and finally from low quality section 162 ranked according to engagement values 116 . If high quality section 166 contains 10 messages, medium quality section 164 contains 10 messages, low quality section 162 contains 10 messages, and the displayed message threshold is 15; The client application 154 renders the entire third ranked list 158-3 and five messages of the second ranked list 158-2. Timeline manager 142 may receive a request to display additional messages in conversation graph 126 , where timeline manager 142 sends the remaining five messages of second ranked list 158-2. Then, the first priority list 158-1 may be transmitted.

[00107] In some examples, the hard ranking may be personalized by using the output of the predictive models 112 to perform the hard ranking. For example, prediction manager 110 may determine prediction results 118 and engagement values 116 as discussed above. The content quality classifier 147 is configured to classify (and/or rank) messages according to hard ranking techniques such that the hard ranking is personalized from the perspective of the user who initiated the conversation view request 121 . outputs can be used.

2 illustrates a neural network 219 according to an aspect. The neural network 219 may be an example of the mutual engagement model 119 of the messaging system 100 of FIGS. 1A-1D . However, the features discussed with respect to FIG. 2 can be applied to any of the predictive models 112 including the positive engagement model 115 and the negative engagement model 117 . In some examples, neural network 219 may be an example of classification model 161 of FIG. 1F . Neural network 219 is configured to output mutual engagement probability 224 . Mutual engagement probability 224 may be an example of mutual engagement probability 124 of FIG. 1A . Neural network 219 may be an interconnected group of nodes 260 , where each node 260 represents an artificial neuron. The nodes 260 are interconnected by layers, and the output of one layer becomes the input of the next layer. Neural network 219 transforms the input X ₁ , X ₂ through X _N (eg, signals 106 ) received by input layer 262 , and transforms it through one or more hidden layers 264 , and (eg, FIG. 2 illustrates one hidden layer 264 ) produces an output Y ₁ (eg, mutual engagement probability 124 ) via output layer 266 . Each layer consists of a subset of the set of nodes 260 .

[00109] Using the neural network 219 to obtain the mutual engagement probability 224 may involve applying a weighted and biased numeric input to the interconnected nodes 260 of the neural network 219 and computing their outputs. can The weights and biases applied to each node 260 of the neural network 219 are computed using, for example, the machine learning algorithm 149 (eg, by the predictive model trainer 140 of FIG. 1A ). It can be obtained by training. The nodes 260 of the neural network 219 may be organized into two or more layers including at least an input layer 262 and an output layer 266 . In the case of a multi-layer neural network 219 , the output from one layer may serve as an input to the next layer. Layers without external output connections may be referred to as hidden layers 264 . The output of each node 260 is a function of the weighted sum of its inputs plus the bias.

To obtain a mutual engagement probability 224 , a vector of feature values X ₁ ...X _N is applied as input to each node 260 of the input layer 262 . In some examples, the vector of feature values X ₁ ...X _N includes values of the signals 106 described above. The input layer 262 distributes values to each of the nodes 260 of the hidden layer 264 . Upon reaching node 260 of hidden layer 264, the value from each input node is multiplied by a weight, and the resulting weighted values are summed and added to the weighted bias value to produce a combined value. . The combined value is passed through a pass or activation function that outputs a value. Next, the outputs from the hidden layer 264 are distributed to the node 260 of the output layer 266 of the neural network 219 . Upon reaching node 260 of output layer 266, the value from each hidden layer node is multiplied by a weight, and the resulting weighted values are summed and added to the weighted bias value to produce a combined value. do. The combined value is passed through a transfer or activation function that outputs Y ₁ (eg, mutual engagement probability 224 ).

[00111] 3 illustrates a flow diagram 300 illustrating example operations of a messaging platform for ranking messaging of a conversation graph in accordance with an aspect.

[00112] The operation 302 includes receiving, via the network 150 , the conversation view request 121 for retrieving the messages of the conversation graph 126 stored on the messaging platform 104 executable by the server computer 102 . include The operation 304 includes determining, by the messaging platform 104 , a plurality of predictive results 118 for each of a plurality of messages of the conversation graph 126 using the predictive models 112 , Here, the plurality of prediction results 118 include a mutual engagement probability 124 . Act 306 includes ranking, by messaging platform 104 , a plurality of messages based on prediction results 118 . Operation 308 includes transmitting, by the messaging platform 104 , over the network 150 , at least a subset of the plurality of messages to be rendered on the client application 154 according to a ranking.

[00113] 4 illustrates a flow diagram 400 illustrating example operations of a messaging platform for ranking messages in a conversation graph in accordance with an aspect.

[00114] Operation 402 involves receiving, via network 150 , a conversation view request 121 for retrieving messages of the conversation graph 126 stored on the messaging platform 104 executable by the server computer 102 . include Operation 404 includes determining, by the messaging platform 104 , a plurality of predictive results 118 for each of a plurality of messages of the conversation graph 126 using the predictive models 112 . Determining includes obtaining a plurality of signals 106 associated with the predictive models 112 , the plurality of signals including data structure-related signals 101 associated with the dialog graph 126 , and a plurality of inputting the plurality of signals 106 to the prediction models 112 to determine prediction results 118 . Operation 406 includes ranking, by the messaging platform 104 , a plurality of messages based on the prediction results 118 . Act 408 includes sending, by the messaging platform 104 , over the network 150 , at least a subset of the plurality of messages to be rendered on the client application 154 according to a ranking.

[00115] 5 illustrates a flow diagram 500 illustrating example operations of a messaging platform for ranking messaging of a conversation graph in accordance with an aspect.

[00116] Act 502 includes categorizing, by messaging platform 104 , messages of conversation graph 126 into a plurality of sections 160 based on content quality of the messages, where plurality of sections 160 . includes a first section and a second section. The first section has messages from the conversation graph 126 determined to be of higher quality than the messages in the second section. Operation 504 includes determining, by the messaging platform 104 , a plurality of predictive results 118 for each of a plurality of messages of the conversation graph 126 using the predictive models 112 . Operation 506 includes ranking, by the messaging platform 104 , the messages of the first section based on the prediction results 118 for the messages of the first section. Act 508 includes sending, by the messaging platform 104 , the message subset of at least the first section to be rendered on the client application 154 according to the ranking.

[00117] In the above description, numerous details are set forth. However, it will be apparent to one skilled in the art having the benefit of this disclosure that implementations of the disclosure may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form rather than in detail in order to avoid obscuring the description.

[00118] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions or representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here and generally considered to be a self-consistent sequence of steps leading to a desired result. Steps are those that require physical manipulations of physical quantities. Generally, though not necessarily, such quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.

[00119] It should be borne in mind, however, that all of these and similar terms will be associated with the appropriate physical quantities, and are merely convenient labels applied to these quantities. As apparent from the above discussions, unless specifically stated otherwise, throughout the description, "identifying", "determining", "calculating", "updating", "sending", "receiving", Discussions utilizing terms such as “generating,” “altering,” and the like refer to computer system memories or registers or other such data represented as physical (eg, electronic) quantities within registers and memories of a computer system. It is recognized that reference to actions and processes of a computer system, or similar electronic computing device, that manipulates and/or transforms other data similarly represented as physical quantities within information storage, transmission, or display devices.

[00120] Implementations of the present disclosure also relate to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purpose, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may include floppy disks, optical disks, CD-ROMs and magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards. may be stored in a non-transitory computer-readable storage medium such as, but not limited to, flash memory, or any tangible medium suitable for storing electronic instructions.

[00121] The word “example” or “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the words "example" or "exemplary" is intended to present concepts in a robust manner. The term “or” as used herein is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from context, "X comprises A or B" is intended to mean any of the original inclusive permutations. That is, X comprises A; X comprises B; or when X includes both A and B, then "X includes A or B" is satisfied under any of the above cases. Additionally, the singular expressions as used in this application and the appended claims are to be construed generally to mean “one or more” unless otherwise specified or clear from context to be indicated in the singular form. Also, use of the terms “embodiment” or “an embodiment” or “implementation” or “an implementation” throughout is not intended to mean the same embodiment or implementation unless otherwise described. Also, as used herein, terms such as “first,” “second,” “third,” “fourth,” etc. are intended as labels to distinguish between different elements, and not necessarily according to their numerical designation. It doesn't have to have an ordinal meaning.

[00122] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may be convenient to construct a more specialized apparatus to perform the required method steps. The required structure for these various systems will appear in the description below. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present disclosure as described herein.

[00123] The above description sets forth numerous specific details, such as examples of specific systems, components, methods, etc., in order to provide a good understanding of various implementations of the present disclosure. It will be apparent, however, to one skilled in the art, that at least some implementations of the present disclosure may be practiced without these specific details. In other instances, well-known components or methods are not described in detail or are presented in simple block diagram format in order to avoid unnecessarily obscuring the present disclosure. Accordingly, the specific details set forth above are merely examples. Particular implementations may vary in these exemplary details and are still considered to be within the scope of the present disclosure.

Claims (58)

A computer-implemented method for ranking messages in conversation graphs in a messaging platform using prediction results, comprising:
receiving, via a network, a conversation view request to retrieve messages of a conversation graph stored on the messaging platform;
determining, by the messaging platform, a plurality of prediction results for each of a plurality of messages of the conversation graph using predictive models, wherein the plurality of prediction results are a mutual engagement probability, positive engagement including probabilities and negative engagement probabilities;
computing an engagement value for each of the plurality of messages using the plurality of prediction results;
ranking, by the messaging platform, the plurality of messages based on engagement values; and
sending, by the messaging platform, over the network, at least a subset of the plurality of messages to be rendered on a client application according to the ranking. The method of claim 1,
wherein the mutual engagement probability comprises a value of a probability that the messaging platform is expected to receive a reply to the respective message. 3. The method according to claim 1 or 2,
wherein the predictive models comprise one or more neural networks. 4. The method according to any one of claims 1 to 3,
generating, by the messaging platform, the conversation graph based on a reply structure of messages exchanged on the messaging platform;
wherein the conversation graph comprises a tree data structure of messages related to the conversation. 5. The method according to any one of claims 1 to 4,
The determining step is
obtaining, by the messaging platform, signals generated by at least one of the client application or the messaging platform; and
and inputting, by the messaging platform, the signals into the predictive models to determine the plurality of predictive results. 6. The method according to any one of claims 1 to 5,
and training the predictive models based on a machine learning algorithm input with training data by the messaging platform. 7. The method according to any one of claims 1 to 6,
selecting, by the messaging platform, a candidate subset from the plurality of messages in response to a number of messages in the conversation graph being greater than a threshold level;
wherein the plurality of prediction results are determined for each message in the candidate subset. 8. The method according to any one of claims 1 to 7,
sending, by the messaging platform, a first subset of the plurality of messages to be rendered on the client application according to the ranking;
receiving, by the messaging platform, a request from the client application to display additional messages from the conversation graph; and
sending, by the messaging platform, a second subset of the plurality of messages to be rendered on the client application according to the ranking. 9. The method according to any one of claims 1 to 8,
the dialog graph comprises a tree data structure having an arrangement of a plurality of nodes representing messages of the dialog graph;
wherein the computer-implemented method further comprises selecting, by the messaging platform, one or more branches of the tree data structure using the engagement values. A system for ranking messages in conversation graphs in a messaging platform using prediction results, comprising:
a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged on a messaging platform, the conversation graph comprising a data structure of messages of the conversation;
a timeline manager configured to provide, in a timeline of a user on a client application, a stream of messages over a network, wherein the timeline manager is configured to receive, from the client application, a conversation view request via the network;
an engagement predictor configured to, in response to the conversation view request, determine a plurality of prediction results for each of a plurality of messages of the conversation graph using predictive models, wherein the plurality of prediction results include a mutual engagement probability; including positive engagement probability and negative engagement probability; and
an engagement scorer configured to compute an engagement value for each of the plurality of messages using the prediction results for the respective message;
wherein the timeline manager is configured to rank the plurality of messages using engagement values and provide, via the network, at least a subset of the plurality of messages for rendering on the timeline according to the ranking; system. 11. The method of claim 10,
wherein the mutual engagement probability comprises a probability value that the messaging platform is expected to receive a reply to the message in the conversation graph, and the positive engagement probability is that the messaging platform is positive with the message in the conversation graph. a probability value that is predicted to receive user engagement, wherein the negative engagement probability includes a probability value that the messaging platform is predicted to receive a negative user engagement with a message in the conversation graph. , system. 12. The method of claim 10 or 11,
wherein the predictive models include a mutual engagement model, a positive engagement model and a negative engagement model, wherein the engagement predictor obtains signals generated by at least one of the client application or the messaging platform; inputting the signals to the mutual engagement model, the positive engagement model, and the negative engagement model to obtain the mutual engagement probability, the positive engagement probability, and the negative engagement probability, respectively constituted, system. 13. The method according to any one of claims 10 to 12,
a predictive model trainer configured to periodically train the predictive models based on one or more machine learning algorithms input with training data;
and the predictive model trainer is configured to provide the trained predictive models to the predictive manager. 14. The method according to any one of claims 10 to 13,
further comprising a candidate message selector configured to select a candidate subset from the plurality of sources;
The plurality of sources includes a first source ranked according to most recently published messages in the conversation graph and a second source ranked messages in the conversation graph according to a relevance algorithm, wherein the candidate subset comprises: a plurality of messages from the first source and a plurality of messages from the second source, wherein the plurality of prediction results are determined for each message in the candidate subset. 15. The method according to any one of claims 10 to 14,
The data structure of the dialog graph includes a tree data structure having an arrangement of a plurality of nodes representing messages of the dialog graph, and the timeline manager uses the engagement values to select a branch of the tree data structure a system configured to 16. The method according to any one of claims 10 to 15,
The engagement scorer is configured to calculate each of the mutual engagement probability, the positive engagement probability, and the negative engagement probability such that the weight applied to the mutual engagement probability is higher than the weight applied to the negative engagement probability. A system configured to apply weights. A non-transitory computer-readable medium having stored thereon executable instructions, comprising:
The executable instructions, when executed by at least one processor, cause the at least one processor to:
receive, via a network, a conversation view request for messages of a conversation graph stored on a messaging platform, wherein the conversation graph includes a data structure of messages of the conversation;
select a candidate subset of messages from the conversation graph;
determine a plurality of prediction results for each message of the candidate subset using a predictive model, wherein the plurality of prediction results include a mutual engagement probability, a positive engagement probability and a negative engagement probability; the mutual engagement probability comprises a probability value at which the messaging platform is predicted to receive a reply to an individual message in the candidate subset;
compute an engagement value for each message in the candidate subset using the prediction results;
rank the plurality of messages using engagement values; and
and transmit at least a subset of the plurality of messages to be rendered on a client application according to the ranking. 18. The method of claim 17,
wherein the candidate subset comprises a plurality of most recently posted messages and a plurality of top-ranked messages from the conversation graph. 19. The method according to claim 17 or 18,
The data structure of the dialog graph includes a tree data structure having an arrangement of a plurality of nodes representing messages of the dialog graph,
store the executable instructions, wherein the executable instructions are further configured to, when executed by the at least one processor, cause the at least one processor to select a branch of the tree data structure using the engagement values. a non-transitory computer-readable medium. 20. The method according to any one of claims 17 to 19,
The executable instructions, when executed by at least one processor, cause the at least one processor to:
periodically train the predictive models based on one or more machine learning algorithms input with training data; and
A non-transitory computer-readable medium having stored thereon executable instructions, further configured to cause the trained predictive models to be provided to a predictive manager. A computer-implemented method for ranking messages in conversation graphs in a messaging platform using prediction results, comprising:
receiving, via a network, a conversation view request to retrieve messages of a conversation graph stored on the messaging platform;
determining, by the messaging platform, a plurality of prediction results for each of a plurality of messages of the conversation graph using predictive models, wherein determining the plurality of prediction results comprises:
obtaining a plurality of signals associated with the predictive models; and
inputting the plurality of signals to the prediction models to determine the plurality of prediction results, the plurality of signals including data structure-related signals associated with the dialog graph;
ranking, by the messaging platform, the plurality of messages based on the prediction results; and
sending, by the messaging platform, over the network, at least a subset of the plurality of messages to be rendered on a client application according to the ranking. 22. The method of claim 21,
wherein the dialog graph comprises a tree data structure having a plurality of nodes representing messages of the dialog graph and edges representing links between the nodes, the tree data structure defining one or more branches of the nodes; wherein the data structure-related signals include signals representing a number of nodes and a number of branches within the dialog graph. 23. The method of claim 21 or 22,
wherein the data structure-related signals include a number of replies in a branch, a number of conversations in a branch, and a number of unique authors within a branch or conversation graph. 24. The method according to any one of claims 21 to 23,
wherein the plurality of signals include signals representing whether a user of the client application has restricted a creator of a message of the conversation graph. 25. The method according to any one of claims 21 to 24,
wherein the plurality of signals comprises engagement signals representing user engagement data associated with messages of the conversation graph comprising engagements of users following a user of the client application in the connection graph; How to implement. 26. The method according to any one of claims 21 to 25,
wherein the plurality of signals comprises historical aggregate signals comprising engagement data associated with a user of the client application on the messaging platform. 27. The method according to any one of claims 21 to 26,
wherein the plurality of signals include content-related signals associated with messages in the conversation graph. 28. The method according to any one of claims 21 to 27,
at least the subset of the plurality of signals is obtained from one or more data services executing on the messaging platform. 29. The method according to any one of claims 21 to 28,
obtaining, by the messaging platform, training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform; and
and training, by the messaging platform, the predictive models based on a machine learning algorithm input with the training data. A system for ranking messages in conversation graphs in a messaging platform using prediction results, comprising:
a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged on a messaging platform, the conversation graph comprising a data structure of messages of the conversation;
a timeline manager configured to provide, in a timeline of a user on a client application, a stream of messages over a network, wherein the timeline manager is configured to receive, from the client application, a conversation view request via the network;
an engagement predictor configured to determine, in response to the conversation view request, a plurality of prediction results for each of a plurality of messages of the conversation graph using predictive models, the engagement predictor being stored on the messaging platform and obtain a plurality of signals associated with the predictive models from one or more data services, wherein the engagement predictor is configured to input the plurality of signals to the predictive models to determine the plurality of predictive results; , wherein the plurality of signals include data structure-related signals associated with the dialog graph; and
an engagement scorer configured to compute an engagement value for each of the plurality of messages using the prediction results for the respective message;
wherein the timeline manager is configured to rank the plurality of messages using the engagement values and provide, via the network, at least a subset of the plurality of messages for rendering on the timeline according to the ranking. becoming a system. 31. The method of claim 30,
wherein the engagement predictor is configured to obtain one or more first signals from a first data service on the messaging platform and obtain one or more second signals from a second data service on the messaging platform. 32. The method of claim 30 or 31,
wherein the plurality of signals include engagement signals, social graph signals, historical aggregation signals, and/or content-related signals. 33. The method according to any one of claims 30 to 32,
The predictive models include a mutual engagement model, a positive engagement model and a negative engagement model, wherein the engagement predictor includes the mutual engagement model, the positive engagement model and the negative engagement model. and input the signals into a model to obtain a mutual engagement probability, a positive engagement probability, and a negative engagement probability, respectively. 34. The method according to any one of claims 30 to 33,
a predictive model trainer configured to periodically train the predictive models based on one or more machine learning algorithms input with the training data;
and the predictive model trainer is configured to provide the trained predictive models to the predictive manager. 35. The method according to any one of claims 30 to 34,
wherein the plurality of signals include signals generated by the client application and signals generated by the messaging platform. 36. The method according to any one of claims 30 to 35,
wherein the plurality of signals comprises similarity signals. A non-transitory computer-readable medium having stored thereon executable instructions, comprising:
The executable instructions, when executed by at least one processor, cause the at least one processor to:
receive, via a network, a conversation view request for messages of a conversation graph stored on a messaging platform, wherein the conversation graph includes a data structure of messages of the conversation;
select a candidate subset of messages from the conversation graph;
determining a plurality of prediction results for each message of the candidate subset using a predictive model, wherein determining the plurality of prediction results comprises:
obtaining a plurality of signals comprising data structure-related signals associated with the dialog graph; and
inputting the plurality of signals to the predictive models to determine the plurality of prediction results;
compute an engagement value for each message in the candidate subset using the prediction results;
rank the plurality of messages using the engagement values; and
and transmit at least a subset of the plurality of messages to be rendered on a client application according to the ranking. 38. The method of claim 37,
The executable instructions, when executed by at least one processor, cause the at least one processor to:
obtain training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on the messaging platform; and
A non-transitory computer-readable medium having stored thereon executable instructions, further configured to train the predictive models based on a machine learning algorithm input with the training data. A computer-implemented method for ranking messages in conversation graphs in a messaging platform, comprising:
classifying, by a messaging platform, the messages into a plurality of sections based on content quality of the messages in a conversation graph, the plurality of sections comprising a first section and a second section, the first section comprising the first section having messages from the dialog graph determined as higher quality than the messages in section 2;
determining, by the messaging platform, a plurality of prediction results for each of a plurality of messages of the conversation graph using predictive models;
ranking, by the messaging platform, the messages of the first section based on the prediction results for the messages of the first section; and
sending, by the messaging platform, at least a subset of the messages of the first section to be rendered on a client application according to the ranking. 40. The method of claim 39,
ranking, by the messaging platform, the messages of the second section based on the plurality of prediction results for the messages of the second section. 41. The method of claim 40,
receiving, by the messaging platform, a request to render additional messages of the conversation graph on the client application; and
sending, by the messaging platform, at least a subset of the messages of the second section to be rendered on the client application according to the ranking. 42. The method according to any one of claims 39 to 41,
The classification step is
receiving a message identifier of a message of the conversation graph; and
determining, using a classification model, that the message has a content quality corresponding to the first section. 43. The method according to any one of claims 39 to 42,
and assigning a branch of the conversation graph containing the message to the first section. 44. The method according to any one of claims 39 to 43,
computing an engagement value for each message in the conversation graph using the plurality of prediction results of the respective message;
and the messages of the first quality section are ranked according to engagement values of the messages of the first quality section. 45. The method according to any one of claims 39 to 44,
wherein the prediction results include a mutual engagement probability, a positive engagement probability, and a negative engagement probability. 46. The method according to any one of claims 39 to 45,
generating, by the messaging platform, the conversation graph based on a reply structure of messages exchanged on the messaging platform;
wherein the conversation graph comprises a tree data structure of messages related to the conversation. 47. The method according to any one of claims 39 to 46,
The plurality of predictive models include a mutual engagement model, a positive engagement model, and a negative engagement model, and at least one of the mutual engagement model, the positive engagement model, or the negative engagement model. is a computer-implemented method comprising a neural network. A system for ranking messages in conversation graphs in a messaging platform, comprising:
a conversation graph manager configured to generate a conversation graph based on a reply structure of messages exchanged on a messaging platform, the conversation graph comprising a data structure of messages of the conversation;
a timeline manager configured to provide, in a timeline of a user on a client application, a stream of messages over a network, wherein the timeline manager is configured to receive, from the client application, a conversation view request via the network;
a content quality classifier configured to classify the messages into a plurality of sections based on content quality of the messages of the conversation graph, the plurality of sections comprising a first section and a second section, the first section comprising the second section having messages from the dialog graph determined to be of higher quality than the messages in the section;
an engagement predictor configured to determine a plurality of prediction results for each of a plurality of messages of the conversation graph using predictive models; and
an engagement scorer configured to compute an engagement value for each of the plurality of messages using the plurality of prediction results for the respective message;
The timeline manager ranks the messages of the first section using the engagement values of the first section and selects at least a subset of the messages of the first section to be rendered on the timeline according to the ranking. A system configured to serve over a network. 49. The method of claim 48,
the engagement predictor is configured to obtain a plurality of signals associated with the predictive models from one or more data services stored on the messaging platform, wherein the engagement predictor is configured to obtain the plurality of signals to determine the plurality of predictive results. and input the plurality of signals to models. 50. The method of claim 49,
wherein the plurality of signals include data structure-related signals associated with the dialog graph. 51. The method according to any one of claims 48 to 50,
and the timeline manager is configured to separately rank the messages of the second section using the engagement values of the second section. 52. The method according to any one of claims 48 to 51,
wherein the content quality classifier is configured to receive a message identifier of a message of the conversation graph and to determine that the message has a content quality corresponding to the first quality section using a classification model. 53. The method according to any one of claims 48 to 52,
wherein the plurality of prediction results include a mutual engagement probability, wherein the mutual engagement probability includes a probability value at which the messaging platform is predicted to receive a reply to an individual message in the conversation graph. 54. The method according to any one of claims 48 to 53,
a predictive model trainer configured to periodically train the predictive models based on one or more machine learning algorithms input with the training data;
and the predictive model trainer is configured to provide the trained predictive models to the predictive manager. A non-transitory computer-readable medium having stored thereon executable instructions, comprising:
The executable instructions, when executed by at least one processor, cause the at least one processor to:
Classify the messages of the conversation graph into a plurality of sections based on the content quality of the messages, using a classification model, the plurality of sections comprising a first section and a second section, the first section comprising the second section having messages from the dialog graph determined to be of higher quality than the messages in the section;
using one or more predictive models to determine a plurality of prediction results for each of a plurality of messages of the conversation graph, wherein the plurality of prediction results are a mutual engagement probability, a positive engagement probability and a negative engagement probability. including ― ;
rank the messages of the first section based on the plurality of prediction results for the messages of the first section;
rank the messages of the second section based on the plurality of prediction results for the messages of the second section; and
and transmit at least a subset of the messages of the first quality section to be rendered on a client application according to the ranking. 56. The method of claim 55,
The executable instructions, when executed by at least one processor, cause the at least one processor to:
obtain training data from a client event log storing information received from the client application and an injection log storing information from a timeline manager running on a messaging platform; and
A non-transitory computer-readable medium having stored thereon executable instructions, further configured to train the predictive models based on a machine learning algorithm input with the training data. 57. The method of claim 55 or 56,
The executable instructions, when executed by at least one processor, cause the at least one processor to:
receive a message identifier of a first message of the conversation graph;
determine, using the classification model, that the first message has a content quality corresponding to the first section;
receive a message identifier of a second message of the conversation graph; and
and use the classification model to determine that the second message has a content quality corresponding to the second section. 58. The method according to any one of claims 55 to 57,
The executable instructions, when executed by at least one processor, cause the at least one processor to:
receive a request to render additional messages of the conversation graph on the client application; and
and transmit at least a subset of the messages of the second section to be rendered on a client application according to the ranking.

Images