CN110796263B

CN110796263B - System and method for intervention optimization

Info

Publication number: CN110796263B
Application number: CN201911010355.8A
Authority: CN
Inventors: J.伯奇; B.弗伦克尔; C.E.布蒂利尔; V.卢马; Y-L.阮; J.A.哈沙尔; H.穆萨维; S.杰汉布; V.巴莱亚; A.高塔姆
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2018-10-23
Filing date: 2019-10-23
Publication date: 2024-06-04
Anticipated expiration: 2039-10-23
Also published as: CN110796263A

Abstract

The present disclosure provides systems and methods for intervention optimization. The computing system obtains an entity history for each of a plurality of entities of the computer application. For each of the plurality of entities, the computing system may determine a respective probability that each of the plurality of available interventions will improve a target value determined based at least in part on a metric of continued use of the computer application by the entity. The computing system may provide the intervention of the plurality of available interventions to the entity of the plurality of entities based at least in part on the respective probabilities determined via the machine-learned intervention selection model. Thus, the computing system may employ a machine-learned intervention selection model to select interventions predicted to prevent entity churn from the computer application on an entity-by-entity basis.

Description

System and method for intervention optimization

Priority claim

The present application is based on and claims priority from U.S. provisional application No. 62/749,420 filed on day 10, month 23 of 2018. The applicant claims the priority and benefit of each such application and is incorporated herein by reference in its entirety.

Technical Field

The present disclosure relates generally to machine learning techniques. More particularly, the present disclosure relates to systems and methods for intervention optimization.

Background

Application developers (e.g., website developers, mobile application developers, game developers, etc.) typically have the goal of maximizing the number of entities that use their applications. As an example, for various reasons, a game developer may seek to maximize the number of users playing her developed games per day.

Thus, one concern for application developers is the user "churn". In particular, "churn" refers to the situation where an existing user of a computer application stops or otherwise significantly reduces the use of this application. User churn is typically measured by a "churn rate" or a backout rate (attribute rate) (e.g., the percentage of existing users that leave the game over a period of time (e.g., one month). Accordingly, a game developer attempting to maximize the number of users playing her game will conversely attempt to minimize user churn.

Churn is highly correlated to "user hold". Churn may include events when a user stops interacting with an application, but remains in the application for how long the user stays in the application. Thus, churn is an event that affects user retention. Preventing user churn is a technique that increases user retention. In addition to the user reservation and the user engagement indicators, there are various other user engagement indicators, and each of these indicators may measure some aspect of the user's continued use of the application.

One technique to prevent user churn or otherwise increase user engagement is through the use of interventions. Interventions may include actions and/or operational changes taken by a developer and/or application for one or more users (e.g., for the purpose of preventing user churn). Identifying which users should be subjected to interventions and which interventions should be performed for the users so identified is a challenging task.

Disclosure of Invention

Aspects and advantages of embodiments of the disclosure will be set forth in part in the description which follows, or may be learned from the description, or may be learned by practice of the embodiments.

One example aspect of the present disclosure is directed to a computing system. The computing system may include one or more processors; one or more non-transitory computer-readable media that collectively store: a machine-learned intervention selection model configured to select an intervention on an entity-by-entity basis based at least in part on a respective entity history associated with the entity; and instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining an entity history for each of a plurality of entities using a computer application. The operations may further include: for each entity of the plurality of entities, determining, via the machine-learned intervention selection model, a respective probability that each of a plurality of available interventions will improve a target value, the target value determined based at least in part on a measure of sustained use of the computer application by the entity, based at least in part on an entity history of each entity. The operations may further include: one or more interventions of the plurality of available interventions are provided to one or more entities of the plurality of entities based at least in part on respective probabilities determined via the machine-learned intervention selection model.

Another example aspect of the present disclosure is directed to a computer-implemented method. The computer-implemented method may include obtaining, by one or more computing devices, entity history data associated with an entity associated with a computer application. The method may further include inputting, by the one or more computing devices, the entity history data into a machine-learned intervention selection model configured to process the entity history data to select one or more interventions from a plurality of available interventions. The method may further comprise: receiving, by the one or more computing devices, a selection of the one or more interventions based at least in part on the entity history data by the machine-learned intervention selection model. The method may further comprise: in response to the selection, one or more interventions are performed on the entity by the one or more computing devices.

Another example aspect of the disclosure is directed to one or more non-transitory computer-readable media storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations may include obtaining entity history data associated with an entity associated with a computer application. The operations may further include inputting the entity history data into a machine-learned intervention selection model configured to process the entity history data to select one or more interventions from a plurality of available interventions, wherein at least some of the plurality of available interventions are defined by a developer of the computer application. The operations may further include: the method further includes receiving a machine-learned intervention selection model based at least in part on the selection of the one or more interventions by the entity history data, wherein the machine-learned intervention selection model is configured to make a selection of the one or more interventions to optimize an objective function, wherein the objective function measures entity engagement with the computer application. The operations may further include: in response to the selection, the one or more interventions are performed on the entity within the computing application.

Other aspects of the disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces (interfaces), and electronic devices.

These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the disclosure and together with the description, serve to explain the principles of interest.

Drawings

A detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the accompanying drawings, in which:

FIG. 1A depicts a block diagram of an example computing system performing intervention optimization, according to an example embodiment of the disclosure.

FIG. 1B depicts a block diagram of an example computing device performing intervention optimization, according to an example embodiment of the disclosure.

FIG. 1C depicts a block diagram of an example computing device performing intervention optimization, according to an example embodiment of the disclosure.

FIG. 2 depicts a flowchart of an example method of performing intervention optimization, according to an example embodiment of the present disclosure.

FIG. 3 depicts a block diagram of an example architecture of a machine-learned intervention selection model, according to an example embodiment of the disclosure.

Repeated reference characters on the drawings are intended to represent like features in the various embodiments.

Detailed Description

Overview

Example aspects of the present disclosure are directed towards systems and methods for intervention optimization. In particular, in some implementations, to reduce user churn, the systems and methods of the present disclosure may include or otherwise utilize a machine-learned intervention selection model configured to select interventions on an entity-by-entity basis based at least in part on respective entity histories associated with the entities. In particular, a computing system (e.g., an application server) may obtain an entity history for each of a plurality of entities associated with a computer application. For each of the plurality of entities, the computing system may determine, via a machine-learned intervention selection model and based at least in part on an entity history for each entity, a respective probability that each of the plurality of available interventions will increase the target value. For example, the target value may be determined based at least in part on a metric of continued use of the computer application by the entity and/or based on other objectives. The computing system may provide one or more interventions of the plurality of available interventions to one or more entities of the plurality of entities based at least in part on the respective probabilities determined via the machine-learned intervention selection model. Thus, the computing system may employ a machine-learned intervention selection model to select one or more interventions on an entity-by-entity basis that are predicted to prevent the entity from running away from a computer application (e.g., a mobile application, a web browser application, or a website or gaming application). In some implementations, the selected intervention can be performed automatically, thereby greatly reducing the workload of the application developer, while also reducing user churn or other measures of user engagement that otherwise improve.

More particularly, aspects of the present disclosure are directed to reducing user loss from computer applications and/or addressing other objectives, including custom developer-specified objectives. As used herein, the term "application" broadly includes a variety of different computer programs, software, and/or systems. One example application is a mobile application, such as a text messaging application installed on a mobile device (e.g., a smart phone). To providing a developer or other individual or organization of a mobile application may seek to maximize the number of entities (e.g., daily users) of the mobile application, or in other words, may seek to minimize users lost from the mobile application. Another example of a computer application is a web site. For example, a website owner may seek to maximize the number of entities that "visit" or otherwise interact with its website on a regular basis (e.g., daily, weekly, etc.). Another example of a computer application is a computer game (e.g., a mobile game, a game of a dedicated game console, a massively multiplayer online game, a browser game, a game embedded in a social media platform, an augmented or virtual reality game, etc.). Another example application may be a conventional computer application executing on a desktop, laptop, tablet, or the like.

An application may have multiple entities that use the application. For example, an entity may include a particular individual user, a group of one or more users, an account associated with an application (e.g., an individual account or an account shared by one or more users (e.g., a corporate account), an organization using an application, or any other entity associated with data contained in an application (e.g., an IP address, a geographic location, a business list, and/or other entities). Although portions of the present disclosure focus for purposes of explanation on applications to various aspects of the systems and methods described herein for individual users, as described above, all such aspects are equally applicable to operating on an entity-by-entity basis rather than a particular user.

According to aspects of the disclosure, a computing system may interact with a computer application to prevent user loss from the application and/or to facilitate other goals, such as increasing allocation of user resources (e.g., time, currency (e.g., virtual currency), computing resources, etc.) to the application. To achieve these goals, a computing system may be operative to automatically intervene in a computer application on a user-by-user basis based on user histories associated with users.

In particular, a computing system may obtain a user history for each of a plurality of users of a computer application. For example, a computing system may obtain a history of events associated with a user in a computer application. These events may be user-triggered events or may be events performed by or on behalf of a user. In some implementations, the history of events can be ordered chronologically. In some implementations, the user history may also include other contextual information (e.g., associated with certain events), such as time, current application state of the user's device, and/or other information about the user that may be used for whom the application has been agreed (e.g., user location). Thus, in some implementations, the user history data for each user of the computer application may be a history of events that have been triggered or performed by the user in the computer application, which may include a "time series" of chronologically ordered events for that user.

According to one aspect of the disclosure, for a particular user and based on a user history associated with the user, the computing system may determine whether to perform an intervention for the user, and if so, which intervention should be performed. In particular, the computing system may select one or more interventions from a plurality of available interventions that the system is capable of performing. As an example, the intervention may include a change in an operating parameter of the application itself (e.g., a change in a behavior and/or appearance of the application), or may include an intervention related to the application but not necessarily changing an operating parameter of the application itself (e.g., sending a notification to the user that they have a remaining game level to play). The intervention may be binary (e.g., send a notification or not), or may be quantitative (e.g., increase the point of play requirement of the monster from 89 to 92).

In some implementations, some or all of the available interventions that the system can perform can be specified by the developer or another party associated with the application. As an example, for an example gaming application, a developer may specify a set of interventions that include: 1) provide free in-game items (e.g., hatchet, enhanced player attributes, etc.), 2) make game levels easier to pass over a period of time, 3) not display any external content, 4) display twice as much external content or other intervention. Thus, a developer may provide specific inputs that control which interventions a computing system is able to perform. The developer may also specify various rules and/or relationships that control the nature of the system's use of the intervention, such as the frequency with which the intervention may be performed, the number of interventions that may be performed for a given user, which interventions may or may not be performed at the same time, the maximum number of interventions per time period, and/or the like.

Thus, in some implementations, the computing system may include a user interface and a back-end that allow the developer to provide the set of interventions. In one example, the user interface may include an intervention name, remote configuration parameters to enable/disable intervention, a valid value, an acceptable frequency, and/or identification of a goal (e.g., revenue, reservation, customization).

In some implementations, the computing system can input the user history into a machine-learned intervention selection model. The machine-learned intervention selection model may provide a respective probability that each of the plurality of available interventions will improve a target value determined based at least in part on a measure of continued use of the computer application by the user. Thus, the machine-learned intervention selection model may calculate a respective probability that each of the plurality of available interventions will improve the retention based on a respective history of events associated with the user. In some implementations, the target value is determined based at least in part on a user's allocation of resources within the computer application in addition to a measure of continued use of the computer application by the user. For example, a machine-learned intervention selection model may calculate a respective probability that each of a plurality of available interventions will enable a user to allocate resources in a computer application based on a respective history of events associated with the user.

More particularly, in some embodiments, a machine-learned intervention selection model may provide predictions corresponding to:

P(C＝yes|T,I₀＝yes),P(C＝yes|T,I₁＝yes),P(C＝yes|T,I₂＝yes)…P(C＝yes|T,I_N＝yes),

Where C is whether the user has been eroded from the computer application, T is the user's time series, and I is the possible intervention that the developer can take to reduce the likelihood that the user will be eroded. I ₀ intervention is a "null intervention" in which the default behavior of the application is given. Once each of these probabilities is calculated, the computing system may select the intervention with the lowest churn probability for the user and may then automatically apply the selected intervention to the user. That is, in some embodiments, if the system has N possible interventions (including null interventions), the system may select and apply the invention according to the following example expression:

Argmin_i(P(C＝yes|T,I_i)),0≤i≤N。

In some implementations, the machine-learned intervention selection model may be trained and operated according to a reinforcement learning scheme. For example, the machine-learned intervention selection model may be or may include an intervention agent in a reinforcement learning scheme. For example, the intervention agent may apply a policy to select certain actions (e.g., interventions) based on the current state, and may receive a respective reward associated with the outcome of each action. The computing system may optimize the machine-learned intervention selection model based on the various rewards to improve the strategy of the intervention agent.

More particularly, in some embodiments, the intervening agent may receive a current user history of a particular user, and may treat such user history as a status. Depending on the state, the intervention may select one or more actions to be performed, where the actions describe how to apply the one or more interventions to the user. The action may be a single action or a series of actions. For example, a first action may indicate that a first intervention is applied today, a second action may indicate that a second intervention is to be applied after three days, and a third action may indicate that a third intervention is to be applied after five days. Based on the outcome of the action, the intervention agent may receive a reward. For example, rewards may be determined using an objective function that measures user churn or other characteristics of user engagement. The computing system may update or modify the policy of the intervention agent based on the received rewards, resulting in an improved policy that enables the intervention selection model to select the intervention that may give the highest rewards in the subsequent interventions.

In some implementations, supervised learning techniques may be used in addition to or instead of using reinforcement learning to train the machine-learned intervention selection model. For example, the computing system may train the intervention selection model based on the training data. The training data may include a set of user histories and interventions selected based on such user histories. The training data may include a corresponding real reference (group-truth) tag for each pair of user histories/interventions that describes known outcomes that occur after the intervention. In some implementations, the computing system may train the intervention selection model using various training or learning techniques (e.g., back propagation of errors). In some implementations, performing back-propagation of the error may include performing back-propagation of the truncation over time.

In some implementations, the computing system may be operable to perform a exploratory pair utilization of the intervention policy (exploration versus exploitation of intervention strategies). More particularly, to know the impact of an intervention, the system typically needs to have data describing the impact of the user (user IN THE WILD) to which the intervention was applied to the natural scene. Thus, to calculate the impact of all interventions, the system may operate to trigger the interventions before the impact of doing so is explicitly known. This helps learn when each intervention should be applied.

In particular, in some cases, without calculating the respective probabilities of interventions, the computing system may randomly provide one or more of a plurality of available interventions to the user during a exploration period (also referred to as an exploration phase). As one example, the computing system may randomly give the user one or more interventions in the computer application over a period of time, such as a week. As another example, the computing system may randomly give the user a first intervention of the one or more interventions over a first period of time, e.g., a few hours of a particular date. The computing system may randomly administer a second intervention of the one or more interventions to the user over a second period of time (e.g., 4 days of the week). The computing system may randomly administer a third intervention of the one or more interventions to the user for a third period of time after the first intervention and the second intervention are applied to the user, for example five days after the first intervention and the second intervention are applied to the user. By observing the results of these exploratory interventions, the computing system may better assess the subsequent opportunities for intervention in computer applications. The exploration period may be defined by a developer and/or may be performed by the computing system in accordance with various optimization techniques (e.g., one or more multi-arm robbery techniques). Example multi-arm theft techniques include optimal solutions, approximate solutions (e.g., semi-uniform strategies such as epsilon-greedy strategies, epsilon-first strategies, epsilon reduction strategies, probabilistic matching strategies, pricing strategies, etc.), contextual theft solutions, resistive theft solutions, infinite arm theft solutions, non-stationary theft solutions, and other variations.

As mentioned above, not all interventions defined in the system may be used in every instance. For example, developer-defined rules may prevent reuse of specific interventions. For example, if an intervention is sending out notifications, the system may be constrained to send only one notification per a particular time period (e.g., 48 hours). As such, in some implementations, the computing system may identify which of a plurality of defined interventions are available at a particular time. The plurality of available interventions may be a subset of the plurality of defined interventions that, when selected, meet one or more developer-provided intervention criteria and/or other rules. The machine-learned intervention selection model or system, which is performed in accordance with its predictions, may be configured to select only available interventions for use.

In some implementations, the machine-learned intervention selection model can be located within a server computing device that serves the computer application. For example, a server computing system may include an intervention selection model and provide selected interventions to computer applications that may be installed on a user computing device according to a client-server architecture.

In other implementations, the machine-learned intervention selection model may be located within a computer application on the user computing device. For example, the user computing device may include a computer application having a model for selecting interventions.

In still other embodiments, the machine-learned intervention selection model may be located on the user computing device with the computer application, but may be external to the application. For example, a machine-learned intervention selection model may provide services for a plurality of different applications on a user computing device according to a client-server architecture.

In some implementations, a computing system can include a Software Developer Kit (SDK) for sending user behavior events from a computing device executing an application (e.g., a user device such as a mobile device) to a server computing system (e.g., a production server). The server computing system may receive the events and sort the events into a time series, and may then store the events for each user. The computing system may include a machine learning system that trains a machine-learned intervention selection model based on all time sequences. The computing system may employ a trained machine-learned intervention selection model, and may perform intervention predictions for one or more users, which predictions, along with the selected intervention, may then be stored. When a computer application needs to know which intervention to apply, the computing system may obtain the stored predictions. The computing system may include a feedback system that may record when each intervention is taken and record the selected intervention to be applied to the user. The computing system may also include a user interface and a back-end that allows the developer to provide the set of interventions.

Although the computing systems described herein are primarily described herein with respect to selecting interventions on a user-by-user basis, in some implementations, the computing system may alternatively or additionally select interventions for groups of multiple users (e.g., groups of users as a single predicted positive (positives) or negative (negatives) sample). For example, a computing system may use a machine-learned intervention selection model to select one or more interventions that are predicted to leave a group of users associated with a positive sample in a computer application. In some implementations, the computing system may allow the developer to have some coarse-grained control over the predictions (e.g., using current risk conditions, or using other proposed but not yet released features), and may allow the developer to target different subsets of positive or negative samples. In some implementations, the computing system may allow a developer to, for example, potentially differently address different causes of churn.

Accordingly, the present disclosure provides systems and methods for optimizing interventions to augment users in computer applications. The provided system eliminates the need for a developer to explicitly take action with respect to predictions of churn. Instead, the developer only provides a list of interventions, and the system will learn when and to whom these interventions should be applied to maximize reservations, resource allocations, or some custom optimization function that takes into account many different objectives.

The present disclosure provides a number of technical effects and benefits. As one example technical effect and benefit, in some embodiments, the systems and methods of the present disclosure may select one or more interventions predicted to improve reservation or allocation of resources on a user-by-user basis. Thus, since the intervention is specifically selected for a particular user, the accuracy and efficacy of the intervention execution may be improved. Improved efficacy may result in overall fewer interventions, as a single intervention is more likely to achieve the desired result. Using generally less intervention may save computing resources such as processor usage, memory usage, and/or network bandwidth usage because generally less intervention actions need to be performed (which require the use of computing resources).

As another example technical effect and benefit, the systems and methods of the present disclosure may provide an exploration phase prior to applying a machine-learned intervention selection model. The exploration phase may address the difficulty of calculating probabilities in the absence of data that enables training a machine-learned intervention selection model to accurately predict probabilities. In this way, the exploration phase may improve the predictive accuracy of the machine-learned intervention selection model.

As yet another example technical effect and benefit, a computing system may allow a developer to provide a set of interventions, and the computing system may provide predictions of each intervention for the developer to make them aware of which interventions are most likely to leave a user in a computer application, and/or to have the user allocate resources for the computer application. In this way, the computing system may help the developer collect information related to the intervention provided by the developer and utilize the collected information to improve the intervention to increase the reservation and/or resource allocation of the computing application.

Referring now to the drawings, example embodiments of the present disclosure will be discussed in further detail.

Example devices and systems

FIG. 1A depicts a block diagram of an example computing system 100 that performs intervention optimization, according to an example embodiment of the disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled by a network 180.

The user computing device 102 may be any type of computing device, such as a personal computing device (e.g., a laptop computer or desktop computer), a mobile computing device (e.g., a smart phone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.), and may be one processor or operatively connected multiple processors. Memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, and the like, as well as combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

The user computing device 102 may store or include one or more machine-learned intervention selection models 120. For example, the machine-learned intervention selection model 120 may be or otherwise include various machine-learned models, such as a neural network (e.g., deep neural network) or other multi-layer nonlinear model. The neural network may include a recurrent neural network (e.g., a long and short term memory recurrent neural network), a feed-forward neural network, or other form of neural network. The machine-learned intervention selection model 120 may also include other types of models, such as decision tree-based models (e.g., random forests), support vector machines, various types of classifier models, linear models, and/or other types of models.

In some implementations, one or more machine-learned intervention selection models 120 may be received from the server computing system 130 over the network 180, stored in the memory 114, and used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of the single machine-learned intervention selection model 120 (e.g., to perform parallel intervention optimization among multiple instances of the intervention optimization).

More particularly, the machine-learned intervention selection model may provide a respective probability that each of the plurality of available interventions will improve a target value determined based at least in part on a measure of continued use of the computer application by the user. Thus, the machine-learned intervention selection model may calculate a respective probability that each of the plurality of available interventions will improve the retention based on a respective history of events associated with the user. In some implementations, the target value is determined based at least in part on a user's allocation of resources within the computer application in addition to or instead of a measure of continued use of the computer application by the user. For example, a machine-learned intervention selection model may calculate a respective probability that each of a plurality of available interventions will enable a user to allocate resources in a computer application based on a respective history of events associated with the user. Other example targets may also be specified, including custom developer-specified targets. As one example, in a gaming application having a single player gaming mode and a multiplayer gaming mode, a developer may want to encourage users to play multiplayer games. In this way, the developer can specify custom targets that weigh the use of the multiplayer gaming mode. In this way, intervention may be automatically performed, which results in increased use of the multiplayer gaming mode. In the case where a plurality of targets are specified, the respective weights between the targets may be specified by a developer or may be learned.

As one example, in some implementations, a machine-learned intervention selection model may provide predictions corresponding to the following values:

P(C＝yes|T,I₀＝yes),P(C＝yes|T,I₁＝yes),P(C＝ycs|T,I₂＝yes)…

P (c=yes|t, I _N =yes), where C is whether the user has been churned away from the computer application, T is the user's time series, and I is the possible intervention that the developer can take to reduce the likelihood that the user will churned away. I ₀ intervention is a "null intervention" in which the default behavior of the application is given. Once each of these probabilities is calculated, the computing system may select the intervention with the lowest churn probability for the user and may then automatically apply the selected intervention to the user. That is, in some embodiments, if the system has N possible interventions (including null interventions), the system may select and apply the invention according to the following example expression:

Argmin_i(P(C＝yes|T,I_i)),0≤i≤N。

In other embodiments, instead of minimizing the probability of user churn, the system may select and apply the invention based on an expression that measures the probability that each intervention will increase the target value that measures the satisfaction of multiple different targets.

In some implementations, the machine-learned intervention selection model may be trained and operated according to a reinforcement learning scheme. For example, the machine-learned intervention selection model may be or may include intervention agents in a reinforcement learning scheme. For example, the intervention agent may apply a policy to select certain actions (e.g., interventions) based on the current state, and may receive a respective reward associated with the outcome of each action. The computing system may optimize the machine-learned intervention selection model based on the respective rewards to improve the strategy of the intervention agent.

In some implementations, a supervised learning technique may be used in addition to or instead of using reinforcement learning to train the machine-learned intervention selection model, which is further described in training the computing system 150.

Additionally or alternatively, one or more machine-learned intervention selection models 140 may be included in the server computing system 130 or stored and implemented by the server computing system 130, the server computing system 130 in communication with the user computing device 102 according to a client-server relationship. For example, the machine-learned intervention selection model 140 may be implemented by the server computing system 140 as part of a web service (e.g., an intervention optimization service). Accordingly, one or more models 120 may be stored and implemented at the user computing device 102, and/or one or more models 140 may be stored and implemented at the server computing system 130.

The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to touch by a user input object (e.g., a finger or stylus). The touch sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a conventional keyboard, or other device through which a user may input communications.

The user computing device 102 may also include a computer application 124. The computer application 124 may include a variety of different computer programs, software, and/or systems. One example application is a mobile application, such as a text messaging application installed on a mobile device (e.g., a smart phone). A developer or other entity involved in providing a mobile application may seek to maximize the number of users (e.g., daily users) of the mobile application, or in other words, may seek to minimize the users lost from the mobile application. Another example of a computer application is a web site. For example, a website owner may seek to maximize the number of users that "visit" or otherwise interact with their website on a regular basis (e.g., daily, weekly, etc.). Another example of a computer application is a computer game (e.g., a mobile game, a game of a dedicated game console, a massively multiplayer online game, a browser game, a game embedded in a social media platform, an augmented or virtual reality game, etc.). Another example application may be a conventional computer application executing on a desktop, laptop, tablet, or the like.

The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.), and may be one processor or operatively connected multiple processors. Memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and the like, as well as combinations thereof. Memory 134 may store data 136 and instructions 138 executed by processor 132 to cause server computing system 130 to perform operations.

In some implementations, the server computing system 130 includes or is implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate in accordance with a sequential computing architecture, a parallel computing architecture, or some combination thereof.

As described above, the server computing system 130 may store or otherwise include one or more machine-learned intervention selection models 140. For example, the machine-learned intervention selection model 140 may be or include various machine-learned models, such as a neural network (e.g., deep-loop neural network) or other multi-layer nonlinear model. The machine-learned intervention selection model 140 may also include other types of models, such as decision tree-based models (e.g., random forests), support vector machines, various types of classifier models, linear models, and/or other types of models.

The server computing system 130 may also include a computer application 142. The computer application 142 may include a variety of different computer programs, software, and/or systems. One example application is a mobile application, such as a text messaging application installed on a mobile device (e.g., a smart phone). A developer or other entity involved in providing a mobile application may seek to maximize the number of users (e.g., daily users) of the mobile application, or in other words, may seek to minimize the users lost from the mobile application. Another example of a computer application is a web site. For example, a website owner may seek to maximize the number of users that "visit" or otherwise interact with their website on a regular basis (e.g., daily, weekly, etc.). Another example of a computer application is a computer game (e.g., a mobile game, a game of a dedicated game console, a massively multiplayer online game, a browser game, a game embedded in a social media platform, an augmented or virtual reality game, etc.). Another example application may be a conventional computer application executing on a desktop, laptop, tablet, or the like.

The server computing system 130 may train the machine-learned intervention selection model 120 or 140 via interaction with a training computing system 150 communicatively coupled through a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.), and may be one processor or operatively connected multiple processors. Memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, and the like, as well as combinations thereof. Memory 154 may store data 156 and instructions 158 that are executed by processor 152 to cause training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is implemented by one or more server computing devices.

Training computing system 150 may include a model trainer 160 that trains machine learning model 140 stored at server computing system 130 using various training or learning techniques (e.g., back propagation of errors). In some implementations, performing back-propagation of the error may include performing back-propagation of the truncation over time. Model trainer 160 may perform a variety of generalization techniques (e.g., weight decay, discard, etc.) to enhance the generalization ability of the trained model.

In particular, model trainer 160 may train a machine-learned intervention selection model 140 based on a set of training data 162. Training data 162 may include, for example, a set of user histories and interventions selected based on such user histories. The training data may include, for each pair of user histories/interventions, a respective authentic reference label describing known outcomes occurring after the intervention. The training data 162 may also include rewards determined for certain actions.

In some implementations, the training examples can be provided by the user computing device 102 if the user has provided consent (e.g., based on communications previously provided by the user of the user computing device 102). Thus, in such embodiments, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 on the user-specific communication data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

Model trainer 160 includes computer logic for providing the desired functionality. Model trainer 160 may be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some embodiments, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium (e.g., a RAM hard disk or an optical or magnetic medium).

The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 180 may be carried via any type of wired and/or wireless connection using various communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), coding or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).

FIG. 1A illustrates one example computing system that may be used to implement the present disclosure. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and training data 162. In such an embodiment, the machine-learned intervention selection model 120 may be trained and used locally at the user computing device 102. In some such implementations, the user computing device 102 can implement the model trainer 160 to personalize the machine-learned intervention selection model 120 based on user-specific data.

FIG. 1B depicts a block diagram of an example computing device 10 performing intervention optimization, according to an example embodiment of the disclosure. Computing device 10 may be a user computing device or a server computing device.

Computing device 10 includes a plurality of applications (e.g., applications 1 through N). Each application contains its own machine learning library and intervention selection model(s) of machine learning. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, and the like.

As shown in fig. 1B, each application may communicate with many other components of the computing device, such as one or more sensors, a context manager, a device status component, and/or additional components. In some implementations, each application can communicate with each device component using an API (e.g., public API). In some embodiments, the API used by each application is specific to that application.

FIG. 1C depicts a block diagram of an example computing device 50 performing intervention optimization, according to an example embodiment of the disclosure. Computing device 50 may be a user computing device or a server computing device.

Computing device 50 includes a plurality of applications (e.g., applications 1 through N). Each application communicates with a central intelligence (INTELLIGENCE) layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, and the like. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

The central intelligence layer includes a plurality of machine-learned models. For example, as shown in FIG. 1C, a corresponding machine-learned intervention selection model may be provided and managed for each application by a central intelligent layer. In other embodiments, two or more applications may share a machine-learned intervention selection model. For example, in some embodiments, the central intelligence layer may provide a single machine-learned intervention selection model for all applications. In some implementations, the central intelligence layer is included within or implemented by an operating system of the computing device 50.

The central intelligence layer may communicate with the central device data layer. The central device data layer may be a centralized data store for computing devices 50. As shown in FIG. 1C, the central device data layer may communicate with a plurality of other components of the computing device, such as one or more sensors, a context manager, a device status component, and/or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

Example method

FIG. 2 depicts a flowchart of an example method of performing intervention optimization, according to an example embodiment of the present disclosure. Although fig. 2 depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particular order or arrangement shown. The various steps of method 200 may be omitted, rearranged, combined, and/or adjusted in various ways without departing from the scope of the present disclosure.

At 202, a computing system may obtain a user history for each of a plurality of users of a computer application. For example, a computing system may obtain a history of events associated with a user in a computer application. These events may be user-triggered events or may be events performed by or on behalf of a user. In some implementations, the history of events can be ordered chronologically. In some implementations, the user history may also include other context-related information (e.g., associated with certain events), such as time, current application state of the user's device, and/or other information about the user that may be used for whom the application has been agreed (e.g., user location). Thus, in some implementations, the user history data for each user of the computer application may be a history of events that have been triggered or performed by the user in the computer application, which may include a "time series" of chronologically ordered events for that user.

At 204, the computing system may determine, for each of the plurality of users, a respective probability that each of the plurality of available interventions will improve a target value based at least in part on a measure of continued use of the computer application by the user based at least in part on a user history of each user via a machine-learned intervention selection model. In some implementations, the target value is determined based at least in part on a user's allocation of resources within the computer application in addition to a measure of continued use of the computer application by the user. For example, a machine-learned intervention selection model may calculate a respective probability that each of a plurality of available interventions will enable a user to allocate resources in a computer application based on a respective history of events associated with the user.

P(C＝yes|T,I₀＝yes),P(C＝yes|T,I₁＝yes),P(C＝yes|T,I₂＝yes)…

P(C＝yes|T,I_N＝yes)，

Here, C is whether the user has been churned from the computer application, T is the user's time series, and I is the possible intervention that the developer can take to reduce the likelihood that the user will churne. I ₀ intervention is a "null intervention" in which the default behavior of the application is given. Once each of these probabilities is calculated, the computing system may select the intervention with the lowest churn probability for the user and may then automatically apply the selected intervention to the user. That is, in some embodiments, if the system has N possible interventions (including null interventions), the system may select and apply the invention according to the following example expression:

Argmin_i(P(C＝yes|T,I_i)),0≤i≤N。

At 206, the computing system may provide one or more of the plurality of available interventions to one or more users of the plurality of users based at least in part on the respective probabilities determined via the machine-learned intervention selection model. For example, the computing system may select one or more interventions that are predicted to leave the user in the computer application (e.g., a mobile application, a web browser application, or a gaming application), or to spend money in the computer application. The computing system may provide the selected intervention to one or more users.

In some implementations, the computing system may be operable to perform exploration-versus-utilization of the intervention policy. More particularly, to know the impact of an intervention, the system typically needs to have data describing the impact of the user to whom the intervention was applied to the natural scene. Thus, to calculate the impact of all interventions, the system may operate to trigger the interventions before the impact of doing so is explicitly known. This helps learn when each intervention should be applied.

In particular, in some cases, instead of calculating the respective probabilities of interventions, the computing system may randomly provide one or more of a plurality of available interventions to the user during a exploration period (also referred to as an exploration phase). As one example, the computing system may randomly give the user one or more interventions in the computer application over a period of time, such as a week. As another example, the computing system may randomly give the user a first intervention of the one or more interventions over a first period of time, e.g., a few hours of a particular date. The computing system may randomly administer a second intervention of the one or more interventions to the user over a second period of time (e.g., 4 days of the week). The computing system may randomly administer a third intervention of the one or more interventions to the user for a third period of time after the first intervention and the second intervention are applied to the user, for example five days after the first intervention and the second intervention are applied to the user. By observing the results of these exploratory interventions, the computing system may better assess the subsequent opportunities for intervention in computer applications. The exploration period may be defined by a developer and/or may be performed by the computing system in accordance with various optimization techniques (e.g., one or more multi-arm robbery techniques).

In some implementations, the method 200 can further include determining a result of the provided intervention, determining a reward based on the result, and modifying a strategy of the machine-learned intervention selection model based at least in part on the reward according to the reinforcement learning scheme.

In some implementations, the method 200 can further include determining results of the provided intervention, tagging the user history and the provided intervention with the results to form additional training data, and retraining the machine-learned intervention selection model using a supervised learning scheme based at least in part on the additional training data.

FIG. 3 depicts a block diagram of an example architecture 400 of a system using a machine-learned intervention selection model, according to an example embodiment of the disclosure. As shown in fig. 3, a developer 402 may use a user interface 404 to input a set of interventions and corresponding preferences (e.g., intervention names, remote configuration parameters, effective values, acceptable frequencies, goals to maximize, etc.). For each intervention, the user may define the intervention name, remote configuration parameters enabling/disabling the intervention, the effective value, and the acceptable frequency. The user may also define goals (e.g., revenue, reservations, custom) to maximize.

A set of interventions and corresponding preferences may be stored in database 406 (e.g., a cloud database structure). The daily pipeline 410 may read in a set of interventions and corresponding preferences as well as other sources 418 of user events and input features from the real-time event stream 408 when training the intervention selection model. During the exploration phase, the computing system may then determine a set of random users to which to apply the intervention, and may record these selections in registry 412. The remote configuration service 414 (e.g., a cloud service that can alter the behavior and appearance of the application without requiring the user to download an application update) may then read in the set of random users to whom the sets apply, and if the users 420 log in, the remote configuration service 414 may apply interventions to the users 420. Remote configuration service 414 may then record that the intervention has been applied to user 420, which is then stored in some data set (e.g., intervention application data 416, database 406, or registry 412). Daily pipeline 410 may read in intervention application data 416 to know which interventions were applied. During the exploration phase, the computing system will select a model using machine-learned interventions described herein, and interventions selected by such model may be implemented through remote configuration service 414.

Additional disclosure

The technology discussed herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to or from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and divisions of tasks and functions between two components and between multiple components. For example, the processes discussed herein may be implemented using a single device or component or multiple devices or components working in combination. The database and applications may be implemented on a single system or may be distributed across multiple systems. The distributed components may operate sequentially or in parallel.

While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of illustration and not limitation of the present disclosure. Substitutions, modifications and equivalents will now occur to those skilled in the art upon understanding the foregoing. Accordingly, the present disclosure does not preclude inclusion of such modifications, variations and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to yield still a further embodiment. It is therefore intended that the present disclosure cover such alternatives, modifications, and equivalents.

Claims

1. A computing system, comprising:

one or more processors; and

One or more non-transitory computer-readable media that collectively store:

A machine-learned intervention selection model configured to select an intervention on an entity-by-entity basis based at least in part on a respective entity history associated with the entity; and

Instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising:

obtaining an entity history for each of a plurality of entities using a computer application;

Providing one or more of a plurality of available interventions to the plurality of entities during a exploration period, wherein the exploration period precedes a utilization period;

Determining, for each of the plurality of entities, a respective probability that each of a plurality of available interventions will improve a target value based on an entity history of each entity and data obtained during the exploration period via the machine-learned intervention selection model, the target value determined based at least in part on a measure of sustained use of computer applications by the entity; and

One or more interventions of the plurality of available interventions are provided to one or more entities of the plurality of entities during the exploration period based at least in part on respective probabilities determined via the machine-learned intervention selection model.

2. The computing system of claim 1, wherein the computer application comprises at least one of: a mobile application, a web browser application, or a gaming application.

3. The computing system of claim 1, wherein the operations further comprise: one or more of the plurality of available interventions are randomly provided to the plurality of entities during a exploration period prior to determining the respective probabilities, wherein the plurality of available interventions includes changes to operating parameters of the computer application and modifications to resources within the computer application.

4. The computing system of claim 1, wherein the machine-learned intervention selection model is trained using supervised learning techniques.

5. The computing system of claim 1, wherein the machine-learned intervention selection model includes intervention agents in a reinforcement learning scheme.

6. The computing system of claim 1, wherein the target value is based at least in part on a custom developer specified target in addition to a measure of sustained use of the computer application by the entity.

7. The computing system of claim 1, wherein at least some of the plurality of available interventions are specified by a developer of the computer application, the developer specifying a frequency of a first intervention of the plurality of available interventions being performed, a maximum number of interventions per time period, and which interventions of the plurality of available interventions cannot be performed together.

8. The computing system of claim 1, wherein the operations comprise: the method includes identifying, from a plurality of defined interventions, the plurality of available interventions specified by a developer of the computer application, the plurality of available interventions being a subset of the plurality of defined interventions that, when selected, meet one or more developer-provided intervention criteria.

9. The computing system of claim 1, wherein the machine-learned intervention selection model is located within a server computing device that services the computer application.

10. The computing system of claim 1, wherein the machine-learned intervention selection model is located at a user computing device, and wherein the machine-learned intervention selection model is capable of serving a plurality of applications on the user computing device in accordance with a client-server architecture.

11. A computer-implemented method, comprising:

obtaining, by one or more computing devices, entity history data associated with an entity, the entity being associated with a computer application;

Providing, by the one or more computing devices, one or more of a plurality of available interventions to the entity during a exploration period, wherein the exploration period precedes a utilization period;

Inputting, by the one or more computing devices, the entity history data and data obtained during the exploration time period into a machine-learned intervention selection model configured to process the entity history data and data obtained during the exploration time period to determine a respective probability that each of a plurality of available interventions will improve a target value, the target value determined based at least in part on a metric of sustained use of a computer application by an entity, and selecting one or more interventions from the plurality of available interventions;

Receiving, by the one or more computing devices, a selection of the one or more interventions based at least in part on the entity history data and data obtained during the exploration time period by the machine-learned intervention selection model; and

In response to the selection, one or more interventions are performed on the entity during the exploration period by the one or more computing devices.

12. The computer-implemented method of claim 11, wherein at least some of the plurality of available interventions are defined by a developer of the computer application.

13. The computer-implemented method of claim 11, wherein the machine-learned intervention selection model is configured to make selections of the one or more interventions to optimize an objective function, wherein the objective function measures entity churn from the computer application.

14. The computer-implemented method of claim 13, wherein the machine-learned intervention selection model is configured to determine a plurality of respective probabilities that the plurality of available interventions will improve a target value provided by the objective function, wherein the selection of the one or more interventions is based at least in part on the plurality of respective probabilities.

15. The computer-implemented method of claim 11, wherein the machine-learned intervention selection model comprises intervention agents learned via reinforcement learning.

16. The computer-implemented method of claim 11, wherein the machine-learned intervention selection model has been trained via supervised learning over a set of training data.

17. The computer-implemented method of claim 11, wherein the computer application comprises a mobile application, a gaming application, or a website.

18. The computer-implemented method of claim 11, further comprising:

Performing, by the one or more computing devices, an exploration phase in which one of the plurality of available interventions is randomly selected for one or more other entities.

19. The computer-implemented method of claim 11, wherein performing, by the one or more computing devices, the one or more interventions comprises: one or more behavior changes of the computer application are modified by the one or more computing devices.

20. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:

obtaining entity history data associated with an entity, the entity being associated with a computer application;

Providing one or more of a plurality of available interventions to the entity during a exploration period, wherein the exploration period precedes a utilization period;

Inputting the entity history data and data obtained during the exploration time period into a machine-learned intervention selection model configured to process the entity history data and data obtained during the exploration time period to determine a respective probability that each of a plurality of available interventions will improve a target value, the target value being determined based at least in part on a metric of sustained use of a computer application by an entity, and to select one or more interventions from the plurality of available interventions, wherein at least some of the plurality of available interventions are defined by a developer of the computer application;

Receiving a selection of the one or more interventions by the machine-learned intervention selection model based at least in part on the entity history data and data obtained during the exploration time period, wherein the machine-learned intervention selection model is configured to make a selection of the one or more interventions to optimize an objective function, wherein the objective function measures entity engagement with the computer application; and

In response to the selection, the one or more interventions are performed on the entity within the computer application during the exploration period.