WO2018234949A1

WO2018234949A1 - Content prefetching in the presence of a/b testing

Info

Publication number: WO2018234949A1
Application number: PCT/IB2018/054368
Authority: WO
Inventors: Daniel Yellin; Shimon Moshavi
Original assignee: Tensera Networks Ltd.
Priority date: 2017-06-19
Filing date: 2018-06-14
Publication date: 2018-12-27

Abstract

An apparatus for content delivery includes a network interface (64) and one or more processors (44, 68). The network interface is configured for communicating over a communication network (32). The one or more processors are configured to detect that a content source (28) is conducting an evaluation of reaction of users of user devices (24) to two or more alternative versions (31A, 31B) of a given content item (30), to select from among the multiple alternative versions, based on the detected evaluation, a version of the given content item to be prefetched to a given user device, and to prefetch the selected version to the given user device.

Description

CONTENT PREFETCHING IN THE PRESENCE OF A/B TESTING

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Patent Application 62/521,561, filed June 19, 2017, whose disclosure is incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates generally to communication networks, and particularly to methods and systems for prefetching of content to user devices.

BACKGROUND OF THE INVENTION

Various techniques are known in the art for prefetching content to user devices. For example, U.S. Patent Application Publication 2016/0021211, whose disclosure is incorporated herein by reference, describes a method for content delivery that includes defining a guaranteed prefetching mode, in which content is continuously prefetched from a content source to a communication terminal of a user so as to maintain the communication terminal synchronized with the content source. One or more time-of-day intervals, during which the user is expected to access given content, are identified. During the identified time-of-day intervals, the given content is prefetched from the content source to the communication terminal using the guaranteed prefetching mode.

SUMMARY OF THE INVENTION

An embodiment of the present invention that is described herein provides an apparatus for content delivery, including a network interface and one or more processors. The network interface is configured for communicating over a communication network. The one or more processors are configured to detect that a content source is conducting an evaluation of reaction of users of user devices to two or more alternative versions of a given content item, to select from among the multiple alternative versions, based on the detected evaluation, a version of the given content item to be prefetched to a given user device, and to prefetch the selected version to the given user device.

In some embodiment, the one or more processors are configured to predict which of the alternative versions will be served to the given user device by the content source, and to select and prefetch the predicted version to the given user device. In some embodiments, the one or more processors are configured to identify an assignment rule, which is used by the content source for assigning the alternative versions of the given content item to user devices, and to select the version in accordance with the assignment rule. In an embodiment, the one or more processors are configured to detect the evaluation by receiving an indication from a provider of the given content item. In another embodiment, the one or more processors are configured to detect the evaluation by crawling the content source while emulating multiple different user devices, and identifying the multiple alternative versions. In yet another embodiment, the one or more processors are configured to detect the evaluation by analyzing content-usage reports from multiple user devices.

In some embodiments, the one or more processors are further configured to identify a count of the alternative versions. In a disclosed embodiment, the one or more processors are configured to identify which content is associated with each of the alternative versions. In an example embodiment, the one or more processors are configured to identify two or more groups of user devices assigned respectively to the two or more alternative versions of the content item, and to select the version corresponding to a group of user devices to which the given user device belongs.

In some embodiments, the one or more processors are configured to define a content tracking scheme based on the detected evaluation, and to track, in accordance with the content tracking scheme, changes that occur to the alternative versions of the given content item in the content source. In an example embodiment, the one or more processors are configured to track the changes by crawling the content source while emulating multiple different user devices. In another embodiment, the one or more processors are configured to track the changes by crawling the content source while emulating an assignment rule used for assigning the alternative versions of the given content item to user devices. In yet another embodiment, the one or more processors are configured to track the changes by receiving notifications from user devices that detect the changes, and distributing the notifications to other user devices. In still another embodiment, the one or more processors are configured to track the changes by accessing content in the user device.

In some embodiments, the one or more processors are configured to distinguish between the given content item for which the evaluation is conducted, and common content that is not subject to evaluation, and to track changes to the common content differently than to the given content item. In an example embodiment, the one or more processors are configured to track changes to the common content less frequently than tracking changes to the given content item. In another embodiment, the one or more processors are configured to track changes to the common content, and not track changes to the given content item for which the evaluation is conducted. In yet another embodiment, the one or more processors are configured to track changes to the common content periodically, and track changes to the given content item for which the evaluation is conducted only in response to a predefined triggering event. In a disclosed embodiment, the one or more processors are configured to prefetch the selected version in response to a predefined triggering event.

In some embodiments, the one or more processors are configured to distinguish between the given content item for which the evaluation is conducted, and common content that is not subject to evaluation, and to prefetch the common content differently than the given content item. In an embodiment, the one or more processors are configured to prefetch only the common content, and not the given content item for which the evaluation is conducted. In another embodiment, the one or more processors are configured to prefetch the common content with a higher priority than prefetching the given content item for which the evaluation is conducted. In some embodiments, the one or more processors are configured to prefetch to the given user device two or more of the alternative versions.

There is additionally provided, in accordance with an embodiment of the present invention, a content delivery method including detecting that a content source is conducting an evaluation of reaction of users of user devices to two or more alternative versions of a given content item. Based on the detected evaluation, a version of the given content item, to be prefetched to a given user device, is selected from among the multiple alternative versions. The selected version is prefetched to the given user device.

The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which:

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a block diagram that schematically illustrates a content delivery system, in accordance with an embodiment of the present invention; and

Fig. 2 is a flow chart that schematically illustrates a method for prefetching of content in the presence of A/B testing, in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

OVERVIEW

Content providers, such as operators of application ("app") servers and electronic commerce ("e-commerce") Web-sites, sometimes test the reaction of users to alternative versions of certain content. Alternative versions of a content item may comprise, for example, alternative layouts of a Web page or app feed, alternative text or graphics of an advertisement, or alternative designs for some user-interface element, to name just a few examples. Alternative versions of a content item may be tested, for example, in order to determine the version that generates the highest revenue, the most user engagement (e.g., content "clicks", add-to-cart), highest conversion rate, fastest response time, or any other desired figure-of- merit. Such an evaluation is commonly referred to as "A/B testing."

In a typical A/B testing setup, users access a given content item on a content source, e.g., by clicking a certain hyperlink, but the content source serves different alternative versions to different users. User reaction is then evaluated per version of the content, i.e., over the responses of the users presented with each version. The desired figure-of-merit is then evaluated for each version of the content in order to determine the best-performing version.

In many practical deployments, some content is also subject to prefetching, i.e., fetched to a cache memory of a user device before the user actually requests it. Prefetching may be performed, for example, by the same content provider operating the content source, by the communication service (e.g., cellular) provider, or by some third party. An important component of the prefetching process is tracking of changes to content on the content source, and notifying user devices of detected changes, i.e., of available content updates.

Unless accounted for, however, prefetching of content can distort the A/B test results significantly. Consider, for example, a scenario in which the prefetching process prefetches the wrong version of certain content to a user device (i.e., not the version that would have been served by the content source to that user device). In such a case, the user of this user device will react to a different version of the content than assumed by the A/B testing process. As a result, the A/B test results are likely to be skewed, especially when this scenario occurs in a large number of user devices.

Embodiments of the present invention that are described herein provide improved methods and systems for prefetching content from content sources to user devices. In particular, the disclosed techniques perform prefetching in a manner that is aware of the presence of A/B testing, and therefore reduces the distortion of A/B test results.

In some embodiments, among its various tasks, the prefetching system detects which content items are currently undergoing A/B testing on the content source. Having detected the presence of A/B testing, the prefetching system attempts to prefetch to each user device the same version of content that would have been served to that user device by the content source.

In some embodiments, the prefetching system identifies, with respect to a certain content item, the group of user devices associated with each content version. Alternatively, the prefetching system may identify the rule according to which the A/B testing process selects which content version to serve to which user device. Based on this information, the prefetching system may set the manner in which it tracks changes to the content on the content source.

Assisted by the content tracking results, and based on the association between user devices and content versions, the prefetching system is able to prefetch the correct content version to a given user device. Consequently, the impact of prefetching on the results of A/B testing is kept to a minimum.

SYSTEM DESCRIPTION

Fig. 1 is a block diagram that schematically illustrates a content delivery system 20, in accordance with an embodiment of the present invention. System 20 comprises multiple user devices 24 that access and consume content items provided by a content sources 28 over a network 32. A certain content item 30 is subject to A/B testing, in which the content provider evaluates user reaction to alternative versions 31 A and 3 IB of the content item.

Each user device 24 may comprise any suitable wireless or wireline device, such as, for example, a cellular phone, car mobile phone or smartphone, a wireless-enabled laptop or tablet computer, a desktop personal computer, a smart television set (TV), a wearable device, or any other suitable type of user device that is capable of communicating over a network and presenting content to a user.

A given user device 24 may consume content using suitable software, e.g., using various user applications ("apps") 36, or using a general -purpose browser. In the present context, a browser is also considered a type of user app. The figure shows a single user device 24 for the sake of clarity. Real-life systems typically comprise a large number of user devices of various kinds.

Network 32 may comprise, for example a Wide Area Network (WAN) such as the Internet, a Local Area Network (LAN), a wireless network such as a cellular network or Wireless LAN (WLAN), or any other suitable network or combination of networks.

Content source 28 may comprise, for example, a Web content server, an e-commerce Web site, or any other suitable source of content. The disclosed techniques can be used with any suitable types of content items, such as, for example, app Web pages, browser Web pages or tabs, audio or video clips, html files, java scripts and/or CSS files, to name just a few examples. In some embodiments, A/B testing may be performed on a group of content items associated with a certain property, e.g., topic, label or keyword. For example, an app may perform A/B testing on its sports-related feed but not on its news -related feed. In some embodiments, system 20 performs prefetching of content items to user devices 24. In the present example, each user device 24 comprises a processor 44 that carries out the various processing tasks of the user device. Among other tasks, processor 44 runs user apps 36, and further runs a software component referred to as a prefetch agent 48 that handles prefetching of content items for apps 36. User device 24 further comprises a content cache 52 for caching prefetched content items. Cache 52 is typically managed by prefetch agent 48.

Typically, prefetch agent 48 receives prefetched content items and stores them in content cache 52. Prefetch agent 48 may intercept user requests to access content items, and determine whether a requested item already resides in the cache. If so, the prefetch agent may retrieve the requested content item from the content cache. Otherwise, the prefetch agent would typically retrieve the requested content item from content sources 28 over network 32. In an embodiment, prefetch agent 48 may also assist in tracking historical usage patterns, and other relevant data related to the user device, which can be used as inputs for specifying prefetch policies for content.

User device 24 typically also comprises a suitable network interface (not shown in the figure) for connecting to network 32. This network interface may be wired (e.g., an Ethernet Network Interface Controller - NIC) or wireless (e.g., a cellular modem or a Wi-Fi modem). Typically, user device 24 further comprises some internal memory (not shown in the figure) that is used for storing relevant information, such as the applicable prefetch policy.

In the embodiment of Fig. 1, system 20 further comprises a prefetching subsystem 60 that performs the various content prefetching related tasks on the network side. Subsystem 60 comprises a network interface 64 for communicating over network 32, and a processor 68 that helps carry out the various processing tasks of the prefetching subsystem. In the present example, processor 68 runs a Content Prefetching Control unit (CPC) 72 that carries out content prefetching.

In an example embodiment, CPC 72 defines a prefetch policy, which specifies how content is to be prefetched to user device 24. For example, CPC 72 may determine which content items are to be prefetched from content source 28 to content cache 52, e.g., based on the likelihood that the user will request the content items. The CPC may determine the appropriate time for prefetching content items, e.g., based on a prediction of the time the user is expected to request them, and/or availability of communication resources.

The CPC may determine how content items are to be delivered to cache 52, e.g., over a Wi-Fi or cellular connection. As yet another example, the CPC may determine the format with which content items are to be delivered to the user device, e.g., whether and how to perform compression or to deliver only changes for the case that an update has occurred for given content that has already been prefetched (i.e., differential prefetch updates instead of re- prefetching the whole given content from scratch).

In some embodiments, CPC 72 may generate Prefetch Notifications (PNs) that inform user device 24 of available content updates that are relevant for prefetching. For example, if CPC 72 decides that an identified content update is significant for a particular user, the CPC may notify prefetch agent 48 running in the corresponding user device 24 that a change has occurred by sending a PN. The PN can be sent using any suitable notification mechanism, e.g., Google Cloud Messaging (GCM) or Firebase Cloud Messaging (FCM) used for Android devices, or Apple Push Notification (APN) used for iOS devices.

In various embodiments, as part of applying the prefetch policy, CPC 72 may estimate, for each content item, the likelihood that the user of user device 24 will request access to the content item. Such likelihood metrics can be sent to user device 24, and may be used by prefetch agent 48 in ranking prefetch priorities for the different content items. The likelihood estimation in CPC 72 may take into account various factors. Some factors may be user-related (e.g., gender, geographical location, interests, and/or recent and historical Internet activity). Other factors may be environment-related (e.g., time-of-day, road traffic conditions, weather, current events, and/or sporting occasions). Yet other factors may be content-related (e.g., content topic or category, content keywords, identity of the content source, and/or the current popularity or rating of the content).

In some embodiments, CPC 72 estimates the time the user is likely to access a content item in order to help determine the prefetch priorities of the various content items and/or the timing of the prefetch. These time estimates might be separately specified as part of the prefetch policy sent to the device, or they might be incorporated into likelihood metrics themselves. For example, a content item that is likely to be accessed within one hour might be given a higher likelihood metric than a content item that will not be needed for at least two hours.

Additionally or alternatively to the likelihood of the user accessing the content, other factors that CPC 72 may consider in specifying the prefetch policy may comprise power consumption considerations (e.g., preference to prefetch while a Wi-Fi connection or a strong cellular connection is available), transmission cost considerations (e.g., preference to lower- cost data transfer times), network congestion and server load considerations (e.g., preference to prefetch during off-peak traffic hours), and/or other user-related or network-related considerations. Further additionally or alternatively, in specifying the prefetch policy, CPC 72 may associate certain times-of-day with respective prefetch priority levels. This association may be performed separately for different apps or content sources, or jointly for multiple apps or content sources. One example factor in determining the prefetch priority levels is the estimated likelihood of the user accessing the different apps or content sources during various times-of- day. Assigning a high priority to a prefetch operation typically translates to the prefetch operation being likely to occur (possibly conditioned on certain constraints or limitations).

Certain aspects of content prefetching, and content prefetching schemes that can be used by subsystem 60 and agent 48, are addressed in U.S. Patent Application Publication 2016/0021211, cited above. For example, CPC 72 may choose between various prefetching modes, e.g., a guaranteed prefetching mode and a best-effort prefetching mode. In the guaranteed prefetching mode, CPC 72 continuously tracks changes in content on content sources 28 (e.g., at predefined tracking intervals) and ensures that content cache 52 in user device 24 is regularly updated by prefetching to be synchronized with the content sources (e.g., at predefined guaranteed-mode prefetching intervals). In the best-effort mode, the CPC typically performs prefetching only as feasible using the available resources.

For example, in the best-effort mode, prefetching may be restricted to scenarios in which the user device's modem is active anyhow, scenarios in which a particularly robust network connection exists, or scenarios that involve a non-metered connection (e.g., Wi-Fi but not cellular). The guaranteed prefetching mode may be utilized during one or more time-of- day intervals in which the likelihood of a user accessing a content source has been predicted to be high. Other considerations that can affect the choice between the guaranteed and best-effort modes can be based on various prefetching policy considerations, e.g., power consumption, transmission cost, network congestion and/or server load. The choice of mode can also be made separately for different applications and/or content sources and/or even for individual content items, groups of content items, or types of content items.

In some embodiments, CPC 72 regularly monitors content sources 28 and generates a "prefetch catalog" - a catalog of content items available for prefetching. Each content item is represented in the catalog by an identifier (ID) and a version number indication. The version numbers enable CPC 72 and/or prefetch agent 48 to determine, for example, whether a certain content item has changed relative to the version cached in cache 52 of user device 24. The catalog may also comprise the likelihood metrics described above, links or addresses from which the content items can be retrieved, and/or any other relevant information. The catalog is considered part of the prefetch policy, along with any other prefetch rules, strategies, thresholds or other policy matters defined by CPC 72.

The configurations of system 20 and its various elements shown in Fig. 1 are example configurations, which are chosen purely for the sake of conceptual clarity. In alternative embodiments, any other suitable configurations can be used. For example, the functions of prefetching subsystem 60 and agent 48 can be implemented using any desired number of processors, or even in a single processor. The various functions of subsystem 60 and agent 48 can be partitioned among the processors in any suitable way. In another embodiment, some or all of the functions of subsystem 60 may be performed by agent 48 in user device 24.

As another example, prefetch agent 48 may be implemented in a software module running on processor 44, in an application running on processor 44, in a Software Development Kit (SDK) embedded in an application running on processor 44, by the Operating System (OS) running on processor 44, or in any other suitable manner. In an embodiment, processor 44 may run a proxy server, which is controlled by prefetch agent 48 and is exposed to incoming and outgoing traffic.

Further alternatively, the functionality of prefetch agent 48 can be implemented entirely on the network side without an agent on user device 24. Further alternatively, some of the functionality of prefetch agent 48 can be implemented on the user-device side, and other functionality of prefetch agent 48 can be implemented on the network side. For example, a cloud-based prefetch server may track content items on content source 28, and report changes in content to a prefetch agent in the user device (possibly residing in the user device operating system).

Generally, the functions of the different system elements described herein (e.g., prefetch agent, content sources and elements of subsystem 60) can be partitioned in any other suitable way. Thus, in the context of the present patent application and in the claims, the disclosed techniques are carried out by one or more processors. The processors may reside in user device 24, and/or on the network side such as in subsystem 60 and/or in content sources 28.

The different elements of system 20 may be implemented using suitable software, using suitable hardware, e.g., using one or more Application-Specific Integrated Circuits (ASICs) or Field-Programmable Gate Arrays (FPGAs), or using a combination of hardware and software elements. Cache 52 may be implemented using one or more memory or storage devices of any suitable type. In some embodiments, agent 48 and/or subsystem 60 may be implemented using one or more general-purpose processors, which are programmed in software to carry out the functions described herein. The software may be downloaded to the processors in electronic form, over a network, for example, or it may, alternatively or additionally, be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.

A/B-TESTING-AWARE PREFETCHING OF CONTENT

In some embodiments, content source 28 conducts A/B testing with respect to certain content items. In the present context, the term "A/B testing" refers to any testing scheme that evaluates the reaction of users of user devices 24 to two or more alternative versions of a content item.

The example of Fig. 1 depicts an example content item 30 that is subject to A/B testing. Content item 30 may comprise, for example, a Web page, a video clip, an app feed, an advertisement, a user-interface element, or any other suitable type of content. In this example, user devices 24 that request content item 30 are each served one of two alternative versions 31A and 3 IB of this content item. Alternatively, any desired number of versions may be tested. The alternative versions of a tested content item may differ from one another in any desired aspect, including, for example, visual differences such as differences in graphics or layout, and/or differences in content.

Typically, all users request content item 30 in a similar manner, e.g., click on the same hyperlink or type the same Uniform Resource Locator (URL). The users are typically unaware of the fact that they are provided with one of several alternative versions of the content item they requested.

In some embodiments, CPC 72 adapts its prefetching process to the fact that content item 30 is subject to A/B testing. CPC 72 attempts to predict which version (31 A or 3 IB) of content item 30 the A/B testing scheme will assign to which user device 24. When prefetching content item 30 to a given user device 24, CPC 72 prefetches the version it has predicted, i.e., the version that content source 28 is expected to serve to that particular user device 24.

In some embodiments, content source 28 decides which of versions 31A and 3 IB to assign and serve to which user device 24, in accordance with some assignment rule. In an embodiment, CPC 72 learns the assignment rule indirectly using techniques that are described below, so as to reliably decide which version of content item 30 to prefetch to which user device 24. In other embodiments, CPC 72 receives the assignment rule directly, e.g., from the content provider via a cloud-based API. In other embodiments, the assignment of content version to user device is carried out using a suitable API in the user device. For example, an assignment rule may specify that user devices that access content item 30 (e.g., via a user click) during odd-numbered minutes will be served version 31 A, and user devices that access the content item during even-numbered minutes will be served version 3 IB.

The method begins with CPC 72 identifying the presence of A/B testing, at a testing identification step 80. CPC 72 may identify that a certain content item is undergoing A/B testing in various ways.

In one embodiment, CPC 72 may receive an explicit indication from the content provider that a content item is subject to A/B testing. The indication may be received, for example, from content source 28, e.g., through a cloud-based Application Programming Interface (API) or via a suitable API in user device 24 (e.g., from the app).

In another embodiment, CPC 72 may identify that a content item is subject to A/B testing by crawling the content on content source 28 in a user-specific manner. For example, CPC 72 may run multiple crawlers, e.g., crawlers that emulate different user devices, and compare the crawling results of the different crawlers. If the results indicate that different crawlers were served different versions of a content item (e.g., different layouts of a Web page), CPC 72 may conclude that this content item is undergoing A/B testing. Typically, CPC 72 also verifies that the number of different versions is relatively small, e.g., two or three, before concluding that A/B testing is being conducted.

CPC 72 can use various techniques for comparing the results of different crawlers. In one embodiment, CPC 72 compares the URLs returned to the different crawlers, e.g., as seen in a HyperText Markup Language (HTML) or JavaScript Object Notation (JSON) file. In some cases, however, the content source uses the same URL for the different content versions, but serves (e.g., in real time) different content to different user devices. In another embodiment, CPC 72 may calculate and compare signatures over the content served to the different crawlers. A signature may comprise, for example, a Cyclic Redundancy Check (CRC) code, a hash value or some other suitable Error Detection Code (EDC). This technique is applicable even if the different content versions are served with the same URL. In yet another embodiment, CPC 72 may compare the content display images associated with the results of the different crawlers. Further alternatively, CPC 72 may compare the results of the different crawlers in any other suitable way, in order to decide whether A/B testing is present or not. In yet another embodiment, CPC 72 may identify that a content item is undergoing A/B testing by crowd-based analysis of content-usage reports received from different genuine user devices 24 in system 20. In some embodiments, each user device 24 sends to CPC 72 reports that indicate which content items (e.g., URLs) the user has accessed. These reports are used, for example, for setting the prefetch policy for each user. Reports of this sort can also be used for detecting that different versions of a content item are served to different user devices.

Additionally or alternatively, CPC 72 may use any other suitable technique for identifying that a given content item on content source 28 is undergoing A/B testing. In some embodiments, as part of the identification process, CPC 72 also identifies the number (i.e., count) of alternative versions that are being tested for a given content item. In some embodiments, CPC 72 also identifies the actual content that is served in each alternative version. CPC 72 may use any of the above techniques (explicit indication from the content provider, user-specific crawling, crowd-based analysis of user-device reports), or any other technique, for determining the number of alternative versions, the content associated with each alternative version, and/or the group of user devices associated with each alternative version.

At a content tracking step 84, CPC 72 sets a content tracking scheme, in which the CPC tracks changes to the content on content source 28, based on the identified A/B testing. CPC 72 then tracks changes to the content in accordance with this A/B-testing-aware content- tracking scheme.

In various embodiments, CPC 72 may track changes in content in the presence of A/B testing in various ways. For example, as explained above, CPC 72 may perform user-specific crawling, i.e., crawl the content source while emulating different user devices. In an example implementation, CPC 72 runs multiple crawlers, each emulating a respective user device associated with a respective user group (which in turn, according to the assignment rule, is served a respective version of the content item). The crawling results of a given crawler, which comprise a respective alternative version of the content, are used by the CPC for prefetching to the user devices belonging to the corresponding group. This technique is best suited for cases in which the assignment of content versions to user devices does not change over time.

In other embodiments, CPC 72 tracks changes in content while emulating the assignment rule itself. This technique is different from user-specific crawling in that, for example, it is also suitable for time-varying assignments. Consider, for example, the above- described assignment rule in which user devices that access content item 30 (e.g., via a user click) during odd-numbered minutes will be served version 31 A, and user devices that access the content item during even-numbered minutes will be served version 3 IB. In crawling the content source, CPC 72 may access content item 30 during odd-order minutes (and use the results as version 31 A), and during even-order minutes (and use the results as version 3 IB).

In other embodiments, CPC 72 tracks changes in content using a crowd-based content tracking scheme. Tracking schemes of this sort are described, for example, in PCT application PCT/IB2017/052683, entitled " Crowd- Sourced Content Tracking," which is assigned to the assignee of the present patent application and whose disclosure is incorporated herein by reference. In crowd-based content tracking, user devices that encounter changes in content report these changes to a server, e.g., CPC 72. The server records the changes in a catalog, and sends the catalog to other user devices. In some embodiments, CPC 72 carries out crowd- based content tracking per user device group (assigned to a respective alternative version of the content item). In other words, CPC 72 distributes content-change notifications received from a given user device to the other user devices in the same group. In this manner, the entire group of user devices is kept up-to-date with a catalog of content updates detected by any group member.

In yet other embodiments, tracking of content changes is performed in user devices 24, e.g., by agents 48. For example, agent 48 may access an app, or in-app content, in the background, for example by emulating user clicks. Agent 48 may then add any content changes it encounters to the content catalog, or report the encountered changes to the CPC for adding to the content catalog.

In some embodiments, CPC 72 distinguishes between content items that are subject to A/B testing, and content that is not being tested and is thus served to any requested user device 24. These two types of content are referred to herein as "A/B-tested content" and "common content," respectively. CPC 72 may process the common content differently than the tested content.

For example, in some embodiments CPC 72 tracks changes in the common content periodically at a relatively high rate (i.e., frequently), in order to achieve a high degree of freshness or synchronization between the content source and the cache memories of the user devices. In the A/B-tested content, on the other hand, CPC 72 may track changes at a lower rate, i.e., less frequently than the tracking of the common content, or even not at all.

In an embodiment, CPC 72 does not perform unconditional periodic tracking of the A/B-tested content, but rather looks for changes in response to predefined triggering events that typically occur shortly before the content is accessed. This sort of tracking is also referred to herein as "Just-In-Time" (JIT) tracking. Events that trigger CPC 72 to look for changes in the A/B-tested content may comprise, for example, user access to the parent content item (e.g., the content item from which a click leads to the A/B-tested content), or certain user actions (e.g., unlocking of the user-device screen). Other suitable triggering events can also be used. In some embodiments, although not necessarily, the triggering event occurs shortly before the user requests to access the content in question.

At a prefetching step 88, CPC 72 prefetches content to user devices 24 based on the A/B-testing-aware content tracking. In some embodiments, CPC 72 carries out prefetching differently for the common content and for the A/B-tested content.

For example, in some embodiments CPC 72 refrains entirely from prefetching A/B- tested content, and prefetches only common content. This technique might be applied, for example, in embodiments in which A/B-tested content is not tracked for changes. In another embodiment, CPC 72 prefetches A/B-tested content with a lower priority than prefetching of common content. This technique is applied, for example, in embodiments in which A/B-tested content is tracked with lower priority than common content (e.g., at a lower rate or in a JIT manner). As another example, in some embodiments the guaranteed prefetching mode, defined above, is applied only to common content and not to A/B-tested content.

Note also that the JIT concept is not limited to tracking, and can be applied to the prefetching operation itself. In other words, system 20 may decide to prefetch content (e.g., to prefetch the selected version of A/B-tested content) in response to a triggering event such as the events listed above.

In yet other embodiments, CPC 72 optimizes the scheduling of prefetch operations only for common content and not for A/B-tested content. Thus, for example, CPC 72 may prefetch common content long before the expected access time by the user, e.g., in order to save battery power or better use the user's data plan. For A/B-tested content, CPC 72 may refrain from applying such optimizations.

Yet another example of prefetching common content and A/B-tested content differently has to do with app preloading. App preloading is a technique in which an app, and/or possibly in-app content, is loaded and initialized in the user device in a background mode before the user actually requests to start the app or access the in-app content. Preloading is typically transparent to the user. When the user requests to start the app or access in-app content, the user device transfers the preloaded app content to the foreground. Due to the preloading operation, from the user's perspective the app responds with extremely low latency. In some embodiments, while a preloaded app runs in the background, the user device continues to refresh the app and/or in-app content in the background. Certain aspects of app preloading and background refreshing are addressed in PCT International Publication WO 2018/055506, entitled "An Optimized Content-Delivery Network (CDN) for the Wireless Last Mile," which is assigned to the assignee of the present patent application and whose disclosure is incorporated herein by reference.

In some embodiments, when a preloaded application uses both common content and A/B-tested content, CPC 72 is able to track the common content and prefetch content items at some optimal time, even before the preloading operation. A B-tested content, on the other hand, may be more difficult to preload in advance. In some cases the CPC may only be able to select the correct version of A/B-tested content at the time the app is activated in a preload. In such cases, the CPC will be able to prefetch the correct version of the A/B-tested content only when the app is preloaded.

CPC 72 may give preference to background loading and prefetching of A/B-tested content (which is typically tracked less accurately and is less synchronized than the common content). For common content, which is typically well synchronized between the content source and the user device in the first place, CPC 72 may defer loading of the content to the actual foreground activation of the app.

In some embodiments, CPC 72 prefetches to a given user device 24 more than one alternative version of an A/B-tested content item. In an example embodiment, CPC 72 prefetches all the alternative versions. This technique is useful, for example, when it is difficult to predict in advance which version will be assigned to the user device, or when the assignment is carried out in the user device and not in the content source. Such scenarios occur, for example, when it is difficult to identify the user-device group with which the user device in question is associated, or when the assignment of content version is performed in real-time (e.g., from the content source, via an API, or using an assignment process in the user device) when the user requests the content item.

In some of the embodiments described above, the level of synchronization between the content source and the user device is better for common content than for A/B-tested content. Thus, for A/B-tested content, there may be greater reliance on real-time fetching of the content over network 32. In some embodiments, this limitation may be offset by performing a "hit and update" process in which the cached content is first displayed, and, in parallel, its validity (i.e., freshness) is evaluated. If the displayed content is found invalid, a real-time update is performed over the network, and the display is updated. Processes of this sort are addressed, for example, in U.S. Patent Application Publication 2017/0111465, entitled "Freshness-Aware Presentation of Content in Communication Terminals," which is assigned to the assignee of the present patent application and whose disclosure is incorporated herein by reference. Although the embodiments described herein mainly address wireless user devices, i.e., user devices connected via a wireless link, the methods and systems described herein can also be used with wired user devices whose wired links suffer from impairments such as content delivery latency, connection outages, and/or variable cost/quality links.

In various embodiments, the content source may be any suitable source that is connected to the user device via a network. Thus, the content source may comprise, for example, a server, a Content Delivery Network (CDN), or even another user device. In some embodiments the content source need not be connected to the user device via a network, but rather connected directly using some suitable interface.

It will thus be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art. Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered.

Claims

1. An apparatus for content delivery, comprising:

a network interface for communicating over a communication network; and

one or more processors, configured to:

detect that a content source is conducting an evaluation of reaction of users of user devices to two or more alternative versions of a given content item;

based on the detected evaluation, select from among the multiple alternative versions a version of the given content item to be prefetched to a given user device; and

prefetch the selected version to the given user device.

2. The apparatus according to claim 1, wherein the one or more processors are configured to predict which of the alternative versions will be served to the given user device by the content source, and to select and prefetch the predicted version to the given user device.

3. The apparatus according to claim 1, wherein the one or more processors are configured to identify an assignment rule, which is used by the content source for assigning the alternative versions of the given content item to user devices, and to select the version in accordance with the assignment rule.

4. The apparatus according to claim 1, wherein the one or more processors are configured to detect the evaluation by receiving an indication from a provider of the given content item.

5. The apparatus according to claim 1, wherein the one or more processors are configured to detect the evaluation by crawling the content source while emulating multiple different user devices, and identifying the multiple alternative versions.

6. The apparatus according to claim 1, wherein the one or more processors are configured to detect the evaluation by analyzing content-usage reports from multiple user devices.

7. The apparatus according to any of claims 1-6, wherein the one or more processors are configured to identify a count of the alternative versions.

8. The apparatus according to any of claims 1-6, wherein the one or more processors are configured to identify which content is associated with each of the alternative versions.

9. The apparatus according to any of claims 1-6, wherein the one or more processors are configured to identify two or more groups of user devices assigned respectively to the two or more alternative versions of the content item, and to select the version corresponding to a group of user devices to which the given user device belongs.

10. The apparatus according to any of claims 1-6, wherein the one or more processors are configured to define a content tracking scheme based on the detected evaluation, and to track, in accordance with the content tracking scheme, changes that occur to the alternative versions of the given content item in the content source.

11. The apparatus according to claim 10, wherein the one or more processors are configured to track the changes by crawling the content source while emulating multiple different user devices.

12. The apparatus according to claim 10, wherein the one or more processors are configured to track the changes by crawling the content source while emulating an assignment rule used for assigning the alternative versions of the given content item to user devices.

13. The apparatus according to claim 10, wherein the one or more processors are configured to track the changes by receiving notifications from user devices that detect the changes, and distributing the notifications to other user devices.

14. The apparatus according to claim 10, wherein the one or more processors are configured to track the changes by accessing content in the user device.

15. The apparatus according to any of claims 1-6, wherein the one or more processors are configured to distinguish between the given content item for which the evaluation is conducted, and common content that is not subject to evaluation, and to track changes to the common content differently than to the given content item.

16. The apparatus according to claim 15, wherein the one or more processors are configured to track changes to the common content less frequently than tracking changes to the given content item.

17. The apparatus according to claim 15, wherein the one or more processors are configured to track changes to the common content, and not track changes to the given content item for which the evaluation is conducted.

18. The apparatus according to claim 15, wherein the one or more processors are configured to track changes to the common content periodically, and track changes to the given content item for which the evaluation is conducted only in response to a predefined triggering event.

19. The apparatus according to any of claims 1-6, wherein the one or more processors are configured to prefetch the selected version in response to a predefined triggering event.

20. The apparatus according to any of claims 1-6, wherein the one or more processors are configured to distinguish between the given content item for which the evaluation is conducted, and common content that is not subject to evaluation, and to prefetch the common content differently than the given content item.

21. The apparatus according to claim 20, wherein the one or more processors are configured to prefetch only the common content, and not the given content item for which the evaluation is conducted.

22. The apparatus according to claim 20, wherein the one or more processors are configured to prefetch the common content with a higher priority than prefetching the given content item for which the evaluation is conducted.

23. The apparatus according to any of claims 1-6, wherein the one or more processors are configured to prefetch to the given user device two or more of the alternative versions.

24. A content delivery method, comprising:

detecting that a content source is conducting an evaluation of reaction of users of user devices to two or more alternative versions of a given content item;

based on the detected evaluation, selecting from among the multiple alternative versions a version of the given content item to be prefetched to a given user device; and

prefetching the selected version to the given user device.

25. The method according to claim 24, wherein selecting and prefetching the version comprises predicting which of the alternative versions will be served to the given user device by the content source, and selecting and prefetching the predicted version to the given user device.

26. The method according to claim 24, wherein selecting the version comprises identifying an assignment rule, which is used by the content source for assigning the alternative versions of the given content item to user devices, and selecting the version in accordance with the assignment rule.

27. The method according to claim 24, wherein detecting the evaluation comprises receiving an indication from a provider of the given content item.

28. The method according to claim 24, wherein detecting the evaluation comprises crawling the content source while emulating multiple different user devices, and identifying the multiple alternative versions.

29. The method according to claim 24, wherein detecting the evaluation comprises analyzing content-usage reports from multiple user devices.

30. The method according to any of claims 24-29, wherein detecting the evaluation further comprises identifying a count of the alternative versions.

31. The method according to any of claims 24-29, wherein detecting the evaluation further comprises identifying which content is associated with each of the alternative versions.

32. The method according to any of claims 24-29, wherein selecting the version comprises identifying two or more groups of user devices assigned respectively to the two or more alternative versions of the content item, and selecting the version corresponding to a group of user devices to which the given user device belongs.

33. The method according to any of claims 24-29, and comprising defining a content tracking scheme based on the detected evaluation, and tracking, in accordance with the content tracking scheme, changes that occur to the alternative versions of the given content item in the content source.

34. The method according to claim 33, wherein tracking the changes comprises crawling the content source while emulating multiple different user devices.

35. The method according to claim 33, wherein tracking the changes comprises crawling the content source while emulating an assignment rule used for assigning the alternative versions of the given content item to user devices.

36. The method according to claim 33, wherein tracking the changes comprises receiving notifications from user devices that detect the changes, and distributing the notifications to other user devices.

37. The method according to claim 33, wherein tracking the changes comprises accessing content in the user device.

38. The method according to any of claims 24-29, wherein tracking the changes comprises distinguishing between the given content item for which the evaluation is conducted, and common content that is not subject to evaluation, and tracking changes to the common content differently than to the given content item.

39. The method according to claim 38, wherein tracking the changes comprises tracking changes to the common content less frequently than tracking changes to the given content item.

40. The method according to claim 38, wherein tracking the changes comprises tracking changes to the common content, and not tracking changes to the given content item for which the evaluation is conducted.

41. The method according to claim 38, wherein tracking the changes comprises tracking changes to the common content periodically, and tracking changes to the given content item for which the evaluation is conducted only in response to a predefined triggering event.

42. The method according to any of claims 24-29, wherein prefetching the selected version is performed in response to a predefined triggering event.

43. The method according to any of claims 24-29, and comprising distinguishing between the given content item for which the evaluation is conducted, and common content that is not subject to evaluation, and prefetching the common content differently than the given content item.

44. The method according to claim 43, wherein prefetching the common content differently than the given content item comprises prefetching only the common content, and not the given content item for which the evaluation is conducted.

45. The method according to claim 43, wherein prefetching the common content differently than the given content item comprises prefetching the common content with a higher priority than prefetching the given content item for which the evaluation is conducted.

46. The method according to any of claims 24-29, wherein prefetching the selected version comprises prefetching to the given user device two or more of the alternative versions.