GB2518936A - Method of optimization for an application - Google Patents

Abstract

A System includes a server, a computing device a display and a processor, and a modified native application executable on the computing device. The modified native application software is written in a reflection-based programming language and configured to display a user interface on the display of the computing device. The modification is made by applying generic code to the native application ( e.g. to source code) to produce a modified native application which is configured to communicate with the server which is external to the computing device. Wherein the modified native application is remotely configurable by the server to examine and to modify the structure and behaviour of the modified native application at run time so as to be capable of altering one or more of the: data, functions or content of a user interface generated by the modified native application. The system allows for optimisation of the user interface and A/B testing of the application.

Description

METHOD OF OPTIMIZATION FOR AN APPLICATION

BACKGROUND OF THE INVENTION

1. Field of the Invention

The field of the invention relates to methods of optimization for a native application, to systems including a computing device on which such a native application is executable, to computing devices on which such a native application is executable, and to servers configured to communicate with computing devices on which such a native application is

executable.

2. Technical Background

Optimization of applications on computing devices is challenging, as when an application is written, it is usually not possible to envision all the uses to which such an application may be desired for use in the future. hitherto, applications have been updated in response to various occurrences, snch as to hugs which are discovered after software release, to the release of new services with which it may be desirable for an apphcarion to interact, or to protect against newly discovered security flaws in the software. 1 lowever, typically the updating of applications has involved the supply of an updated sofrtvare version from the supplier of the softivare, for installation on an associated conaputing device in place of the original softtvare version. A ftirther aspect is that the developer of the application doesn't have the possibility to compare application performance side-by-side to assess which of tivo or more versions of the application has the better performance eg. for sales or actions.

3. Discussion of Related Art EP1352322(A2) and EP1352322B1 disclose tirne-optintized replacement of a software application, and in particular a method for replacing old softtvare that is in use with ne\v softivare, \vhich permits the maximum availability of the sotiware. The method is subdivided into a preparation phase and an active phase. The preparation phase takes place during the operation of the old softvare. Ihe active phase is merely characterized by the execution of a MOVE command.

FR2841014A1 and FR2841O14131) disclose a method for optimizing a computer program that causes a transition between first and second predetermined states in response to an event occurring that belongs to a group of events, in which the method has the following steps: reading of a sub-group of events chosen from a group of events; modeling of a second decision graph using registers and equivalent to a first decision graph when the events of the event group do not belong do a self generating group; and generation of a software function for said transition modeling. There is further disclosed a device for optimizing a computer program.

However, it is desirable to provide greater speed or flexibility in optimizing a software application. Tt is also desirable to achieve a precise measurement of the response of users to a change in an application.

SUMMARY OF TIlE INVENTION

According to a first aspect of the invention, there is provided a method of optimization for a native application whose software is written in a reflection-based programming language, the native application executable on a computing device including a display and a processor, the native application conflgured to display a user intertace on the display of the computing device, the method including the steps of: (i) applying generic code to the native application to produce a moditied native application which is configured to communicate with a server external to the computing device, and (ii) remotely configuring the modified native application from the server to examine and to modify the structure and behaviour of the modified native application at run time so as to he capable of altering one or more of the data, Enctions or content of a user interface generated by the modified native application.

An advantage is that the modified native application can be further modified without performing an instillation of a new version of a native application. A further advantage is that the modified native application can he ftirther modified very quickly, because it is remotely configurable fron a remote server. A further advantage is that die modified native application can be further modified in response to factors detenuined at a remote server, because the modified native application is remotely configurable from the remote server. A further advantage is that the native application is readily modified to provide the modified native application. A further advantage is that a precise measurement of the response of users to a change in an application may he provided. There is a technical effect on a process outside the computing device, because the server can remotely configure the modified native application. The computer operates in a new way, because an altered user interface is produced by the modified native application \vithout installation of a new version of the native application, the computer is a better computer because the modified native application may be further modified without its operation being interrupted in the way that is typically associated with installation of a new software versioti.

The method may be. one wherein the generic code is the only code needed to be applied to source code of the native application to enable optimization of the modified native application. An advantage is that modification of the native application is simply made.

The method may be one wherein the generic code enables many kinds of optimizatlons to be remotely configured in an optimization systems user interfaces, without changing the code within multiple versions and instances of the native application. An advantage is that multiple versions and instances of the native application do not need to be modified.

The method may he one wherein modifying the stnicture and behavior of the modified native application at mntime includes modifying one or more of the values, meta-data, properties. images and functions of the modified native application. An advantage is that the modified native application can he modified in many ways.

The method may he one wherein the computing device is a mobile device, a touch screen device, a smart phone, a video game console, a computer, a smart TV, a TV set-top-box, or a device connectable to the internet. An advantage is that the method may be applied to many types of devices, including devices which are rarely updated, if ever.

The method may be one wherein the computing device is connectable to the internet over a vired or wireless \vide area neiwork (\VAN) or over a mobile phone neiwork to a server. An adv2umtge is that the method can he used for different types of connections.

The method may he one wherein optimization includes adapting visual experiences for each user on the fly in order to increase the probability of a user completing a desired action. An advantage is that increased user interaction maybe obtained.

The method may be one wherein a desired action is a click or an order placen cut. An advantage is that increased user interaction may be obtained.

The method may be one wherein a reflection based programing language is a common reflection based programing language. An advantage is that such a language is easily used.

The method may be one wherein the. method further includes collecting one or more attributes about the user/device for use in targeting, profiling or analyzing user behavior.

An advantage is that user-customized user interface output may he provided.

The method may be one wherein the collected attributes are one of more of: Connection type to the device (e.g. 2G, 3G. 4G. Wi-Fi); Device Type; application version; Operating System type and version; Location (e.g. by coordinates, city, country, continent, postal code; Tntemet service provider; Recency and details of past visits to the app; Recency and details of past actions; Custom user defined attributes such as past interaction with a specific product category; Referral source where a user has the opportunity to launch the app from a variety of locations; time of visit by date, day, hour, and Data imported from other locations such as CRNI databases. An advantage is that user-custonaized user interface output may he provided.

The method may he one Harther including the step of including or excluding visitors/devices from a campaign using their atthbutes. An advange is that improved focus on relevant users nay be obtained.

The method may be one further including the step of: creating segmentation rules to target content to visitors/devices according to their attributes in combination with rules for other user segment information. An advantage is that inaproved focus on relevant users naay he obtained.

The method may he one wherein native application performance is optimized during a campaign as well as after the campaign. An advantage is that the benefits produced are produced over a longer time period.

Ihe method may be one wherein alterable user interface content includes one or more of: a) Simple text or image block (banners, headers of the pages) movement, resizing, hiding, reformatting; h) Buttons texts, movement. resizing, reformatting, and hiding; c) Push notification' frequency and content; d) Displayed picture (picture could he stored on external server; e) New text or image. block; f Lists & tables (ordering, how many items to display); g Navigafion paths within the applicafion (e.g. does add to cart' direct the user to the cart page or hack to a category page to make further selecions), and h) Displaying a navigation based upon the last user clii ice.

An advantage is that a wide range of user interface content may be altered.

The method may be one wherein the native application indudes a library. An advantage is that content can he ocally stored on the computing device for repeat use, or fir use during intermittent connectivity.

The method may be one. wherein the library is static for all applications.

The method may be one wherein the library is configured to send requests to, and to receive responses from, a content generator. An advantage is that the library may use the most recent configuraion data. An advartagc is that library updafing may be automated.

Ihe method may he one wherein the library interacts with a native application cache that is included in the na6ve application. An advangc is that content and a unique identifier for the user intcracthig with it can be locally stored on the compung device for repeat use, or fbr use during intermittent connectivity.

The method may he one wherein the native application cache is configured to receive content pushed from the external server.

The method may he one wherein the library makes a request to a content generator on tiadve. applicadon start An advantage is that the library's data is as up-to-date as possible, and that the. user will see only the nost up-to-date user inter&ce content.

The method may be one wherein the library is arranged to nake a request to a content generator, the request including a location and user attributes in parameters. An advantage is improved customization.

The method may be one wherein the library includes one or more of the following modules: 1) (G request module; 2) External assets module (image transaction); 3) Tracking i-nodule (actions tracking, sending results to the CG); 4) Application object identification module; 5) \fV'T module (implement alternative content visualization); 6) Personalization module; 7) Segmentation module, and 8) Administrative console module.

The method may be one wherein the library can modify only specified objects and properties in code instructions.

According to a second aspect of the invention, there is provided use of a method according to any aspect of the first aspect of the invention, to optimize a selectable button displayed on the display from among at least two alternatives for the selectable button. An advantage is that a most appropriate button may he displayed.

According to a third aspect of the invention, there is provided use of a method according to any aspect of the first aspect of the invention, to optinlize a displayed product on the display from among at least iwo alternatives for the displayed product, based on a most recently used filter option. An adv2ultmge is that a most appropriate product may he displayed.

According to a fourth aspect of the invention, there is provided use of a method according to any aspect of the first aspect of the invention, to optimize a displayed food product on the display from among at least two alternatives for the displayed food product, based on a most recently viewed food category. An advantage is that a most appropriate product may be displayed.

According to a fifth aspect of the invention, there is provided use ofa method according to any aspect of the first aspect of the invention, to optintze a displayed advertisement on the display from among at least Iwo alternatives for the displayed advertisement, based on recorded click through rates for the displayed advertisements. An advantige is that a most appropriate advertisement may be displayed.

According to a sixth aspect of the invention, there is provided a system comprising a server, a computing device including a display and a processor, and a modified native application executable on the computing device, the modified native application software being writ-ten in a reflection-based programming language, the modified native application configured to display a user interface on the display of the computing device, a native application having been modified by applying generic code to the native application t a modified native application which is configured to communicate with the server which is external to the computing device. wherein the modified native application is remotely configurable by the server to examine and to modify the structure and behaviour of the modified native application at run time so as to he capable of altering one or more of: the data, functions or content of a user interface generated by the modified native application.

An advantige is that the modified native application can be further modified \vithout performing an installation of a new version of a native application. A further advantage is that the modified native apphcation can be further modified very quickly, because it is remotely configurable from the remote server. A further advantage is that the modified native application can he further modified in response to factors determined at the remote server, because the modified native application is remotely configurable from the remote server. A further advantige is that a precise measurement of the response of users to a change in an application may he provided.

The system may he configured to perform a method according to any aspect of the first aspect of the invention.

The system may be one wherein the server is further configured to collect one or more atftibutes about the user/device for use in targeting, profiling or analyzing user behavior.

The system may he one \vherein the server is further configured to filter reports, including conversion and revenue reports, by attribute values.

The syste.m may be one wherein the server is further configured to discover segments that may require a different experience to increase conversion rates based upon the attributes.

The system may be one wherein the server is configured to automatically detect higher performing content variants and then to arrange to serve those higher performing content variants to more future visitors.

The system may be one wherein the system includes the ability te switch a concluded campaign to a mode where winners only' are served, meaning that the native application supplier does not need to make an update to the code in order to apply winning variations of content or user experience.

The system may he one wherein the system includes a self-learning, predictive algorithm which drives each nser to the best otTer for that individual, automatically.

The system may be one wherein the system includes a segment discovery process to automate the analysis process to discover the user segments \vhich require a different experience than the overall winner to increase perfonnance against measures for that segment.

The system may he one wherein the system provides multivariate testing.

The system may he one wherein the server is contigured to send segmentation rules to the native application.

The system may be one wherein the system is configured to track one or more of: a) Views of a specific screen, for example the screen shown at completion of an order; b) Button clicks or finger/stylus taps; c) tracking on the dependent website pages; d) Audio/video players playing time, and e) Action attribute and action value opportunities (e.g. to measure sales value in or other currency).

The system maybe one. wherein the. system is configured to measure a response of users to a change in the application. An advantage is that a precise measurement of the response of users to a change in an application maybe obtained.

According to a seventh aspect of the invention, there is provided use of the system of any aspect of the sixth aspect of the invention, wherein historical customer information, such as products held or a customer's propensity to buy a product, can be used to dynamic drive the best, personalized experience for each customer on native applications.

According to an eighth aspect of the invention, there is provided a computing device including a display and a processor, the computing device including a modified native application \vhose software is written in a reflection-based programming language, the modified native application executable on the device, the modified native application configured to display a user interface on the display of the computing device, the modified native application having been modified by applying generic code to a native application to produce the modified native application which is configured to communicate \vith a server external to the computing device, wherein the modified native application is remotely configurable by the server to examine and to modify the structure and behaviour of the modified native application at run tinie so as to he capable of altering one or more of: the data. functions or content of a user interface generated by the modified native application.

According to a ninth aspect of the invention, there is provided a server configured to communicate \vith a computing device, the computing device including a display and a processor, the computing device including a modified native application \vhose software is \vntten in a reflection-based programming language, the modified native application executable on the computing device, the modified native application configured to display a user interface on the display of the computing device, the modified native application having been modified by applying generic code to a native application to produce the modified native application which is configured to communicate xvith the server \vhich is external to the computing device, wherein the server is.irrnged to remotely configure the modified native application to examine and to modify the structure. and behaviour of the modified native application at run tinie so as to be capable of altering one or more of: the data, Linctions or content of a user interface generated by the modified native application.

According to a tenth aspect of the invention, there is provided a system comprising a server, a plurahty of computing devices each including a display and a processor, and a modified native application executable on the plurahty of computing devices, the modified native apphcation software being written in a reflection-based programming language, the modified native application configured to display a user interface on the display of each of the plurality of computing devices, a native application having been modified by applying generic code to the native application to produce a modified native application which is configured to communicate with the server which is external to the plurality of computing devices, wherein the modified native application is remotely configurable by the server to examine and to modify the structure and behaviour of the modified native application at run time so as to he capable of altering one or more of: the data, functions or content of a user interface generated by the modified native application, \vherein the server is arranged to configure a plurality of modified native applications on the plurality of computing devices in a plurality of different configurations and to record measurements of user actions on the plurality of computing devices in relation to the plurality of different configurations. An advantage is that a precise measurenient of the response of users to a change in an application may he provided.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects of the invention will now he described, by way of example only, with reference to the following Figures, in which: Figure 1 shows how an example of how a test works inside an application.

Figure 2 shows an example of a creation of an optimization system, in table form.

Figures 3 to 14 show examples of specifications.

Figure 15 shows an example of states which, once a campaign is setup, can be gone through starting from implementation through to live when it is published.

Figure 16 shows examples of attributes.

DETAILED DESCRIPTION

There is provided a method of optimization for native applications.

Maxymiser has created a new approach to optimize native software applications which exist on many types of Internet connected devices ranging from smart phones to video game consoles, PCs, smart TVs and TV set-top-boxes. Tn an example, optimization means adapting visual experiences for each user on the fly in order to increase the probability of a user completing a desired action, like a click or an order placement.

The invention relates to a method of optimization for native applications whose sofrtvare is written in a reflection based programing language, for example any common reflection based programing language. In compnter science, reflection is the ability of a compnter program to examine and modify the structure and behavior (e.g. the values, meta-data, properties and kinctions) of the program at runtime. The native application may run on a computer, a video game station, a smart phone or any other kind of device connected to internet, typically over a wired or wireless wide area network (WAN) or mobile phone network to a server.

The method of optimization for a native application may comprise the step of integrating a native application optnmzation system with a native application by applying generic code to the native application, that being the only code needed to be applied to the source code of the native application to enable the native application optimization system to optimize the native application by altering one or more of: the data, functions or content assets of a user interface (UT) in the native application. This is the First time that an optimization system has been able to he integrated using a generic code for integraflon which can then enable many kinds of optiniizations to be remotely configured in the optimization system's user interfaces, without changing the code within multiple versions and instances of the native application. Previous methods required new code to be applied for each optimization catnpaign, creating the requirement for a new release version to be downloaded by the user from a server before the optimization could commence. This new method removes that requirement. thereby making the optinuzation program faster and more sophisticated.

The method may be. able to collect the following attributes about the user/device for use in targeting, profiling or analyzing user behavior: * Device type, apphcation version and Operating System version * Location (e.g. Lw coordinates, ciy, country, continent, postal code) * Internet service provider and connection type * Recency and details of past visits to the app * Recency and details of past actions * Custom user deFined attributes such as past interaction with a speciFic product category * Referral source where a user has the opportunity to launch the app from a variety of locations * Time of visit by date, day, hour, day-part * Data imported From other ocations such as CRM databases The new system may include enhancements to Maxymiser's plafform to optimize a native application, these enhancements may include the following features: * Include or exclude visitors/devices from a campaign using their attributes.

* Create segmentation ruks to target content to visitors/devices according to their attributes in combination \vith rules for other user segment information.

21) * Fllter reports, induding conversion and revenue reports, by attribute va'ues.

* Discover segments that may require a different experience to increase conversion rates Ix:ised upon the attributes.

Tests can he conFigured to automatically detect higher perForming content variants and then serve those to more Future visitors. This means that overall app perFormance is optinized during the campaign as well as after it. this is known as Dynamic Optimi zati on -The system may include the ability to switch a concluded campaign to a mode where winners only' are served, meaning that the application owner does not need to make an update to the code in order to apply winning variations of content or user experience.

Making an update to app code is often a very large investment for an app developer since it is necessary to test the app, submit it for approval to the application distributor (e.g. Apple Appstore and then distribute it which in the. case of smart televisions can involve waiting a number of months until the next planned softivare update distribution, the timing of which is in the hands of the hardware manufacturer, not the application developer.

Tn addition, Maxymiser's Full Managed Services provides support for native applications, including creative design, web development, report analysis and quality control across mobile devices.

Testing with Native Applications isa development by Maxymiser. Since native application devices do not support Silverlight. the On Page Console is not provided for native application testing.

Testing for native applications may include: * the Ontilouch platform which creates an if-less environment for marketers to make changes to their native application.

* A breadth of platfom-i capability that can he applied to a native application, including recommendations based upon affinity pattems and a self-learning, predictive algorithm \vhich drives each user to the best offer for that individual, automatically.

* Segment discovery to automate the analysis process to discover the user segments which require a different experience than the overall winner to increase performance against agreed measures for that segment. This now also applies to segments that may he identified by native applications.

* Use of historical customer information, such as products held or a customer's propensity to buy a product, can he used to dynamic drive the best, personalized experience for each customer on native applications.

Specification

Examples of what it is possible to test: a) Simple text or image block (banners, headers of the pages) movement, resizing, hiding, refR)rmatting.

h) Buttons texts, movement. resizing,, reformatting, and hiding.

c) Push notification' frequency and content d) Displayed picture (picture could be stored on external senTer.

e) New text or image block.

I) Lists & tables (ordering, how many items to display).

g) Navigation paths within the application (e.g. does add to cart' direct the user to the cart page or hack to a category page to make further selections) h) Displaying a navigation based upon the last user che ice.

i) Segmentation and personalization of the above (tailoring these items to user segments rather than the whole user base).

Examples of what it is possible to track: a) Views of a specific screen, for example the screen shown at completion of an order.

h) Button clicks or finger/stylus taps.

c) Tracking on the dependentwehsite pages.

d) Audio/video players playing time.

e) Action attribute and action value opportunities (e.g. to measure sales value in or other currency).

In an example of howa test may work inside an application, a mobile device can send requests to and receive responses from a content generator, which includes a content generation service. The mobile device includes a mobile application. The mobile application includes a library which is static for all applications. The library is configured to send the requests to and to receive the responses from the content generator. The library interacts with a mobile application cache that is included in the mobile application. An external server with content assets (eg. images) can push content to the mobile application cache. Figure 1 shows how an exatnple of how a test works inside an application.

Multivariate Testing (MVT) Example 1) mobile version' library makes a request to the content generator (CC) on application stint.

2) CG creates response. with current domains' content in javaScript Object Notation (3 SON) format: * Locations list.

* Generated variants for each location.

* Content for the required variants.

* Action list with settings and location mappings.

3) U.ser going to open view with action.

* nobile. version' library finds corresponding location.

* mobile version' library reads linked actions.

* mobile version' library adds events trackers if it's necessary.

* mobile version' library calculates necessary values and attributes.

* mobile version' library makes cached request to the CG with the location and the actions in parameters.

4) User going to open view without any actions or elements.

IS * mobile version' library finds corresponding location.

* mobile version' library makes cached request to the CG with the location.

5) User going to open view with elements.

* mobile version' library Finds corresponding location.

* nobile version' library reads linked variants.

* mobile version' library shows elements' content.

* mobile version' library make cached request to the CC with the location and shown variants in parameters.

An example is shown in Figure 1.

Personalization Example 1) mobile version' library make request to the CC on application stint.

2) CC create response with current domains' content in JSON format: * Locations list.

* User attributes with settings and location mappings.

3) User going to open view with user attribute.

* mobile version' library finds corresponding location.

* mobile version' library reads linked user attributes.

* mobile version' library calculates necessary values and puts it to the corresponding user attributes.

* mobile version' library make cached request to the CG with the ocation and user attributes in parameters.

An example is shown in Figure 1.

Segmentation Example

a) mobile version' library makes a request to the (G on application start.

b) x; create response with current domains' content in JSON format: * Locations list.

* User attributes with va'ues.

* Segmentation rules with settings.

* Generated variants for each Ljcation.

* Content for the required variants.

c) User going to open view with elements.

* mobile version' library finds corresponding location.

* mobile version' library check segnaentation rules and choose the 21) segment.

* mobile version' library specify necessary variants.

* mobile version' library shows elements' content.

* mobile version' library make cached request to the CC with the location, segment, and shown variants in parameters.

An exampk is shown in Figure 1.

Examples of what the manager can do at the administrative console a For the consok running, user shouhi register device TD at the Maxymiser (Mlvi) U'.

Technical requirements / details examples.

mmu A)ile version' liE)rary mmli)dules: 1) (Xi request module.

2) External assets module (image transaction).

3) tracking module (actions tracking, sending results to the CC).

4) Application object identification module.

3) MYI module (implement alternative content visualization).

6) Personalization module.

7) Segmentation module.

8) Administrative console module.

Commercial positioning for Mobile Version Mobile smartphone adoption is increasing at a rapid rate. Significant investments are being made in mobile applications to keep up with the fast-moving market. Extending your reach to mobile devices is an opportunity, hut it requires an optimized experience to truly he effective with such a small screen.

Maxymiser's mobile version has revolutionized mobile optimization -a must for any brand investing time and budget into mobile marketing. The mobile version enables the marketer to effectively test and target content and create personalized experiences for mobile devices to increase conversion rates. Quickly optimize all of your mobile content including banner ads, data submission forms, site layout, promotions or any other content to drive the most relevant experiences for each individual user and increase revenue for your brand.

Don't you need a mobile marketing strategy to include mobile apps? The on-the-go consumer enjoys having their favorite apps at their fingertips. With the mobile version you can perform A/B and multivariate tests for mobile apps to improve the user experience without requiring end-user updates to the app. identify a new winning experience as seasonal changes occur and dynamically change the user experience without alienating the end-user with excessive updates to the app. Or, let the mobile version adapt and determine new winners automatically, driving the most relevant experience over time.

The mobile version's reporting and analytics gives you the insight into the experiences of mobile users, identifying the xvinning experience for each user segment and identifying new segments that require a different experience. You'll identify the right experience and measure the overall campaign effectiveness across web sites and mobile devices.

In an example of a creation of an optimization system, in stage zero for creating a SDK specification, a specification is created, with a project time]ine and requirements gathering. In stage one, for SDK developing and for UT adjustments, a library is created, Mi\'f UT modifications and web service developing occur, and interactions with an app store occur. Tn stage two for debugging, MM UI & SDK integration is tested with a smartphone app, internal quality assurance and bug fixing occurs, usability testing Occurs, improvements and change requests occur, and drafts of production process build-up are performed. in stage three, beta testing occurs. A test spec is created, integration within the beta app and test creation are performed. beta client testing is performed, improvements and change requests are generated based on beta feedback, and a production process build-up is finalized. Figure 2 shows an example of a creation of an optimization system, in table form.

Example with iPhone SDK, or other smartphone SDK General iPhone SDK (or another smartphone SDK functionality should alloxv easy library integration into customer's application. Integration instruction should occupy no more than 1 page. It should correspond to existing Maxymiser test campaign creation conception. It should allow production and sandbox modes. It should remember user personalization data (similar to the browser cookies). Tt should allow offline measurement.

Mobile administration console should allow: * Changing mode from sandbox to production and back; 30. Viewing and selecting interface objects and properties; * Changing property value; * Memorizing selected properties and values; * Exporting selected properties and values to Maxymiser UT; Framework. Call web-service library Example For generating web-service classes, in an example http://sudzc.com/ is used.

There may be provided MM CG web-service calls. There may be previded offline calls (calls queue and postponed calls on getting online).

Framework. Load images library Example There may be provided images loading functionality. There may be provided image caching (should load only changed images).

MMCore library Example There may he provided a set of methods and properties similarly to mmcore.js: * MMCore.CGRequestQ; * \4rvICore.setActionQ; * MMCore.render] ocationHementsQ; * \ilMCore.setU serAttributeQ; * \iMCore.setUserDefàultsQ; * i\TIVICore.settocationQ; * i\TIVICore.addl'dVTConstraintQ; * i\TMCore.domain; * MMCore.server; * MMCore.CGRequestCallhack; Encapsulate call web-service library, load images library, MYT library. MAConsole library. There may be provided different load content modes: sync, async. May allow reading, setting user personali2ation criteria

MVT library Example

There may he provided changing properties values according to the CC content. There may he provided property types and properties existence validation. There may he provided implementation of user constrarnts: * Library can modify nothing; * Library can modify everything; * Library can modify only specified objects and properties (in code instructions).

* Library can modify only specified objects and properties (in Admin UI).

Element vanant shou'd contain set of properties and their va'ues.

Tf content not ready (hasn't yet loaded from CG) defau't Content shou'd be shown.

MAConsole Library Example MAConsole (mobile administration consok) shou'd provide finding, viewing and selecting interface objects and properties. There. may be provided exporting selected properties and values to Admin UI. _\LkConsole should be. shown when user is in company IP and makes some speciFic touches, or when debug mode set in application.

MM CG web service Example CG web-service which should return data and implenent intcracon with applica6on (similar to the CG jSRenderer

Admin UI Example

New Admin Iii functionality: 1 Mobile version" c1unpaign type.

2. "Object oriented" element type.

3. Editor For variant oF "Object oriented" dement type.

4. Campaign evel property constraints.

Integration Example

Example of how to integrate Maxytniser MVT library into a sin plc view-based application. Before integrañon you need to make sure that necessary campaigns have already been created within the Maximyser UI tool.

1. (:O) MMCore classes files to the Classes folder under the "Group & files" list in your Xcode project. The classes tiles cn he downloaded 2. Include, the M_\lCore. class header to the appDe.legate class header.

#import "MMCore.h" 3. Add the following to the applicationDidFinishLaunching event handler of your appDelegate implementation tile.

[?vlTV{Core mit: self domain: cñ,"YOlJRDOMATN"]; [?vlTV{Core setServer: "cgglobal.maxynuser.con1/cg_v5/"]; [iVlTVlICore setConstraintNlode: (d"allowAll"]; [MMCore CGRequest; 4. Add the following to the viewDid Apear event handler fir all of your viewContorller implementation file.

L1\t_\tCore setLocation: self]; 3. Each time you want to set the additional location, use the function below.

IMMCore setLocation: @"\ourTocadonName"]; 13 6. You can use the following methods to report additional data.

vLMCore setAction: (ñ"YourActionName"]; 7. Content variations implemen when location is changing. You can call element rendering at any time by the following hanction.

L1\1\tCore rendertocaflonElements: c'ä"YourLocationName"]; 8. You can add properties scope which will accessible for modification.

di\iCore setConstrainflviode: (iO"allowSpecified"]; LMCore addlvlVTConstraint: ("object1/"]; [MMCore ade[tvlVTConstraint: @"object2/"]; 9. Build and run application.

Product Scenario Example Example of how to create Maxyniiser smartphone (eg. iPhone) MYl.

In Maxymiser LII 1. Choose domain.

2. Create mobile version' campaign.

3. Add own IP to the company IF.

In sniartphone (eg. iPhone) application 4. Run necessary smartphone. (e.g. iPhone) application with mobile version' libraries.

3. Find the interface element you want to include to the campaigm 6. Switch to the MAConsole and authorize. Go to the "Sdh" mode.

7. Usc marking too].

8. Choose interface element and its property Add the property to the bookmark.

9. Di 5-8 again for other properties.

\ou can use bookmarks for creating element. Do the following: In smartphone (eg. iPhone) application 1. Ge to the bookmark list. Select bookmarks from list. Press "Create element".

2. If your session is over you need to authorize. Then choose campaign, specify element name, and press "Save". When you save element system automatically creates location (if location doesn't exist) and links the element to the location.

You can use bookmarks for creating a variant for the element. Do the following: 13 In smartphone (eg. iPhone) application 1. Find the interface element you want to modify.

2. Switch to the MAConsole and authorize. Go to the "Sandbox" mode.

3. Chose element from list (system will add properties from element to the bookmark).

4. Choose property from bookmark, change property value.

3. Do 3 for other properties.

6. Press "Create variant".

7. If your session is over you need te authorize. Then specify variant name and press "Save".

You can preview the variant in application. Do the following: In smartphone (eg. iPhone) application 1. Run necessary smartphone (eg. iPhone application with mobile version'.

2. Find the interface element you want to preview.

3. Switch to the MAConsole and authorize. Go to the "Sandbox" mode.

4. Choose the element from list, choose the variant, and close MAConsole. After that you will see the variant on the screen.

Examples of Specifications

in an example of a specification, a primary action is "Order", which tracks an order placed through the application, the Value' captures the value in GBP of the order.

Attributes are "delivery" = order was placed for a pizza to he delivered, and "collection" = order was placed for a pizza to he collected from a pizza store. In a secondary action, "Clicks" tracks clicks on a customize' button An example is shown in Figure 3. Tn test variants, different button styles'td colours are tested for the customize' button. An example is shown in Figure 4.

Tn an example of a specification, a primary action is "Order", which tracks an order placed through the application. The Value' captures the value in GEP of the order.

Attributes are "delivery" = order was placed for a pizza to be delivered, and "collection" = order was placed for a pizza to be collected from a pizza store. in a secondary action, "Add to Cart" tracks whether a customized pizza has been added to a user's cart. An example is shown in Figure 5.

In an exanple of a specification, a prinary action is "Purchase", which tracks an order placed through the application. The Value' captures the value in GBP of the order. In a secondary action, "Prod Views" tracks views of a product page within the application. An example is shown in Figure 6. in test variants, when a user navigates to a category browsing page, the resultt are filtered by the last used filter option. Variants may be by title, by price or by release, fbr example. Examples are shown in Figure 7.

In an exaniple ofa specification, a primary action is "FoodViews", \vhich tracks views of one of the application food packages. In an example, both food category pages and food detail pages should he tracked. Examples are shown in Figure 8. The left hand side of Figure 8 shows an example of a food category page. The right hand side of Figure 8 shows an example of a food detail page. in an example of personalisation criteria, the last viewed food category is stored as a personalisation criteria "FoodPreference". Possible values are: cakes, breakfast, lunch, muffins, salads, sandwiches, yoghurt or blank.

Examples are shown in Figure 9.

In an example of a specification, a primary action is "RadioPlav", which tracks if the user listened to a radio station. Attributes track the nunlI)er of minutes of radio hstened to, which may grouped into the folloxving possible values: 0-30, 30-60, >60. In a secondary action, "BannerClicks" tracks clicks on an external banner at the top of the application page. An example is shown in Figure 10. In an example, three banner variants are included in a test. The three banner variants may he no advert', one advert' and two adverts'. An example is shown in Figure 11.

In an example of a specification, a primary action is "BookClicks", which tracks the clicks on a Book' button. The Value' captures the cost of the booking. Attributes tracks the agent of the journey booked, eg. Opodo. In a secondary action, "BookingCornplete" tracks completed bookings through the web-in app site. In a secondary action, "Search" tracks searches performed in the application -a view of the search results page.

Examples are shown in Figure 12.

In an example of a specification, a primary action is "Purchase", which tracks the completed purchases made through the application. The Value' captures the value of the purchase. Attributes tracks the city that the deal was purchased for.n example is shown in Figure 13.

In an example of a test page, a test page may he any of category browsing pages. In an area of a test page, a test element may be a buy colour test clement. In an area of a test page, a test clement nay be a button test element. In an area of a test page, a test den cut may he an image test elenient. In an area of a test page, a test element may he a layout test element. An example is sho'n in Figure 14.

Mobile App Testing: Instructions for Site Creation Example of How to Create a Site for Mobile App Testing (API Method) We provide a guide to setting up a Site for a client who is licensed for Mobile App lesfiug. Campaign creation is described elsewhere.

Ihe intended audience is the administrators who regularly set up Sites for clients.

Perform the following steps to prepare a site for mobile app testing: 1. Add a new site through "Adnain/Site Administration" section in _\Iaxyrniser Admin Ui.

The name of the site should he set according to this pattern: <Chent's name without domain> app For example Maxymiser.app Next the following parameters for the new Site should he set: "Visual Campaign Builder" -Disabled; "Use first-party cookies" -checked; "Mobile app testing site" -checked.

2. Select this site in the "L ampaigns" tab. Go to "Site Settings/Site Browser Rules".

3. Open a "Define custom Browser Rule" tab and add a custom inclusive browser rule with the next paraneters: Browser -Any, OS -Any, Device type -Any, Version -Empty 4. Go to "Site Settings/Personalization Criteria".

3. Add three personalization criteria with the following nanaes: OSVersion. AppVersion, ConnectionType 6. Go to "Site Settings/Site Pages".

7. Add two site pages: A page called "GenerationPage" with one included mask "Generations"; A page called "ActionsPage" with one included mask "Actions"; The field "Preview U]IL" is not used and can be populated with any URL to pass validation.

8. Add only the page "GenerationPage" to the campaign once created (see further

description below).

Maxymiser Mobile App Testing API and SDK Implementation Guide for

Developers -Example

1 Introduction

This implementation guide provides steps for integrating the Maxytniser Mobile Application Testing solution MMTApp'. It describes how an A/B testing campaign can be implemented and run within native mobile applications developed for a smartphone operating system eg. iOS. This guide assumes that you are flimiliar with creating apps for a smartphone operating system eg. iOS.

2 Getting Started Prere.quisite.s for using and integrating the APi with an App: * A development environment with XCode instlled.

* If you run your app on devices for testing purposes you will need to have an Apple Development Program account.

* A proficiency in development using Objective-C and native iOS Apps.

Note: Our SDK supports iOS 7 and onwards. For iOSó and below the SDK can be integrated into your app, but the SDK will be inactive and you will not be able to use the Maxymiser eatures.

3 Integration Guide Our integration Guide takes you through the six steps to complete the setup of an A/B test within an 105 App.

2.1 Campaign Creation \Ve have provided a number of links to our current Maxymiser UI documentation. This is where you will setup and maintain Mobile App tests. See Appendix I for our Terminology Notes and our glossary can he viewed in Appendix 3.

3.1.1 Setup a campaign for testing.

1. Create a Maiest campaign -full documentation can be found in our help guide.

2. _Map your campaign to the Page that has been created in the _Maxymiser Lii for App testing -full documentation can be found in Out I Ielp guide. Please ensure that you do not map your campaign to other Pages because the predefined Page has special properties set for App testing.

3. Create at least one Action that you would like to track inside your application -full documentation can be found in our Help guide. Please see Appendix 2 for a list of user attributes that are supported for Mobile App Testing.

4. (:t an Element which will contain variants of your user experience -full documentation can be found in our help guide. Tn an example, multivariate tests are not supported so please ensure that only one element is created per campaign.

5. Create at least one Variant for an Element of your user expenence -full documentation can be found in our Tjelp guide.

6. Specify Variant Weight and control -full documentation can be found in our TTclp guide.

lor example:

We would like to change the background color and verify which one has the higher conversion rate. Create two variants of your user experience in addition to Default as this is created automatically. The Default variant (Control variant) has awhite background.

Variant 1 -has a green background and Variant 2 has a blue background.

Now we need to set up variant weightings.

Default -50%.

Variant 1-25%.

Variant 2-25%.

This means: half of all users do not see any changes in UI, and won't participate in the campaign. A quarter of users will see the green background 2uad a quarter the blue

background.

3.2 SDK Integration Now that a campaign has been contigured in the Maxvmiser UT we can start the process of integrating the 511K.

1. Locate and extract the M_\TTApp library alp file which is distributed alongside this documentation.

2. Extract the content of the SDIC package.

3. Launch XCode and open the iOS application to be tested.

4. Add the following files from the SDIC package to your app: MMTApp.h MMTApi.h M?vlTExperiences.h MMTLoggenh libMM'f p pa 3. Link your app's target with the following libraries: CoreData.frarnework SystemConfiguration.frarnework hbMMTApp.a 6. Tnsert #import "MMTApp.h" directive tc the top of the working tiles in your project. The MMTApp is an entry point to the APT.

7. Create and configure APT: a) Tn your application delegate insert the following: / / Create. APi for myapp' site id <M_\tIApi> api = LI\trvI'1ipp apiliorSite:@"nayapp"j; // Use Sandbox configuration, not Production lapi useSandhox:YES1; 13 h) Replace mvapp' with the name of your Site in the Maxymiser LTT.

c) Later you can refer to the first created API by [MMTApp defaultApi], so there is no need to keep the reference to the created APT.

Please see API Reference for more configuration options.

3.3 Experience Fetching The code below will fetch and cache all canapaigns and their vari2uat for later use.

/ / Show interstitial page ftMMTApp defaultApi] fetchExperiences: // Hide interstitial page and display Home page H; The First argument is a block that we will invoke once experiences are fetched. It is safe to update LI inside this block as it will be invoked on the main thread.

Note: Please be aware that due to latency there may be a short delay in receiving a response from Maxymiser with details of the variant to display. therefore, we recommend fetching experiences in a place in your App where a user may naturally experience a delay, for example an interstitial page or a search term/login input page. We recommend fetching experiences at the App launch.

3.4 (:Ott Transffirmation When experiences are fetched you can use them to transform content.

There are tvo methods you can choose from to implement content transformations.

3.4.1 Variant Name Base the decision logic of your applicaflon on the variant name. In this case, Variants of the user experience for the View element usc the format shown in the following example.

\Vhen the Variant Name is Default, the Variant Content is Empty.

\Vhen the Variant Name is Variantl, the Variant Content is Empty.

\Vhen the Variant Name is Variant2, the Variant Content is Empty.

-(void)viewVVillAppear:(BOOL)animatcd { Lsuper viewWilhkppear:animatedj; UlColor color = nil; NSString variantName = 11[N1MTApp defaultApil getExperiences getVariantNameForCampaign:("Campaign" eletnent:(d"View"]; if ([(ä'Varianti' isEqualToString:variantName) { color = LilColor g-reenColor; else if ([ØJ'Variant2" isEqual'I'oSfting:variantNamej) color = LUlColorblueColori; else color = [LTIColor whiteColorl; self.view.hackgroundColor = color; 1[\ENtTApp defaultApil trackContentSeen:(tI'Carnpaign";

I

3.4.2 Variant Content You can describe content transformadon in variant content. Variants of the user experience for the View element would use the fonuat shown in the following examplc.

When the Variant Name is Default, the Variant Content is whiteColor.

\Vhen the Variant Name is Varianti. the Variant Content is green(kAor.

\Vhen the Variant Name is Variant2. the Variant Content is blueColor.

-(void)viewwillAppear:(BOOL)animate.d ( Lsupe.r viewwilhkppear:anirnatedj; NSString tvariantContent = fl\TivITApp defaultApil getExperiences getvariantcontentForcampaign:@"Campaign" elcment:(d)"View']; if(variantcontent = nil) {

self view.backgroundColor = ftJTColor

performSe]ector:NSselectorFromString(variantContent)]; else

se]fiview.backgroundColor = [UTColor \vhiteCo]or;

[T\TTTApp defau!tApi trackContentSeen:(?i)'Carnpaign"; Important: When a user is shown content from a campaign, Maxymiser platform must be notified by calling [MMTApp defaultApi] ackContentSeen: °ACarnpaignName% Failure to perform this step will mean that reporting will he corrupted since no users will be recognized as entering campaign.

3.3 Action Tracking All testing campaigns require at least one action so that you can track the inapact of altering a user experience -for example, to track a conversion to sale.

Use the following code to track an Action: [MMTApp delault/\pil trackAction:W"Purchase" value:100 attrihute:nil]; if the connecfion to _\Iaxyniiser is not available, action will be cached and sent once connecfion resumes.

3.6 QC & go-live Environments The Maxynaiser UI support Iwo environments: Sandbox and Production.

3.6.1 Sandbox Sandbox is a development test environment where campaigns are created and configured. All testing on your application should be carried out in Sandbox. Sandbox traffic is served from pre-defined Company IPs.

To preview a specific Variant in-app for Quality Control (QC) testing purposes: 1. Usc the Maxymiser UI -Campaign Settings > Campaign Content > E]ements & Variants page to set weightings of all variants to 0'.

2. Set the weight of the variant that you want te preview to 100'. Run your apphcation and make sure that the transformation is apphed correct]y and that other functionality is working as expected.

3. Repeat these steps for all variants in the user experience.

4. Complete the necessary siges to trigger all configured actions.

5. Check the Action Log report to ensure that action's value and action's attribute are being logged correctly.

Note: Data captured in Sandbox mode does not appear in the Maxymiser UT genera] reports. This mode exists for QC testing purposes. An App must not he re]eased with tests connected to Sandbox environment because Service Level Agreementt (SLAs) are not supported and reporting will not be available.

3.6.2 Production Production is the environment used to serve production (live user) traffic.

Datu gathered front this environnent is made available in the reporting area (see Mucymiser LTI -Analytics tat)) for analysis to deterntine which variant is the best performer.

The \IMTApp a]]ows the environment to he switched as fo]]ows: #ifdef DEBUG 1apiuseSandhox:\'iS1; #endif important: Never release an app to end Lisers containing a campaign which is still connected to the Sandbox environment.

3.6.3 Publish A Publish is necessary to ntove a cantpaign betxveen the Sandbox and Production states -full documentation can he found in our Help guide.

once a Campaign is setup it can go through states stirting from implementation through to live when it is published. The Stutes are described in the table shown in Figure 15.

3.6.4 Ending a campaign At the end of a campaign (when a winning user experience variant is identified) the winning variant can he weighted to 100 in the Maxymiser UI prior to republishing the campaign -this ensures that oniy the winning variant is displayed to users. It is recorntnended that the campaign is switched to the Not Collecting Data' state oncc the campaign has ended so that the performance data is fro2cn for futurc reference.

The campaign-specific code shouki he removcd from the App when the winning user experience variant has been coded to the App codebase (e.g. at the next reease.

Hease see Appendix 4 for our TrouNeshooting Guide 4 API Reference MMT App 5131K top-level class. Provides facilities to create APi for a Site and contains methods for S13K configuration. Tasks

Managing APTs + defaultApi + setDefaultApi: + apiForSite: Configuration + setLogger: + setLogLevel: (:hss i'vlethods + deflaultApi Returns defauft /\PI.

+ (id <MMTApi>)defau]tApi; Return \Tahie API object set explicitly with +setDefaultApi: or first APi object created with +apiForSite:, otherwise return nil.

+ setDefaultApi: Sets default APi.

+ (void)setDefaultApi:(id <MM'iApi>)api; Parameters api An API object to he set as default.

+ apilorSite: Cre.ate.s new APi for the given Site.

+ (id <MMTApi>)apiForSite:(NSString)site: Parameters site The name of the Site.

Return Va'ue APT object hnked with a given Site.

+ setjigger: Sets ogger to be used by SDK.

+ void)s etLogger:(id <_\ifvfflogger>)logger; Parameters logger Logger to use.

+ setLogLevel: Sets logging level.

+ (void)s etLogLcvel: (MMTLogLevel) level; Parameters 1ev ci Logging level.

MMTApi Protocol to he used to fetch experiences and track actions. Tasks

Con liguration -setRequestTimeOut: -sef\XTiFiOnly: -useSandbox: Experiences -fetchExperiences: -fetchExperiences: forPage: -getExperiences Tracking -trackAction: value: attribute: -trackAction: value: attribute: fbrPage: -trackContentSe.en: instance Methods -setRequestTime( )ut: Sets the maximum time which can he spent making a request to our servers. Default is three seconds. When the time is exceeded, the attempt to fetch experiences vvll] cease and the previous user experience will he selected.

-(void)setRequestTimeOut:(NSTimeinterval)timeout; Parameters timeout Timeout to use.

-setWifiOnly: Specifies whether new experiences can be fetched only when user is connected to a Wi-Fi network. By default any available network will he used.

-(void)setWiFiOnly:(HOOJ)wifiOnly; Parameters wifiOnly YES to use only \XTi_Fi network, NO to use any available network.

-useSandhox: Specifies whether Sandbox configuration should be used. By default Production configuration will be used.

-(void)useSandhox:(I300L)sandhox; Parameters sandbox YES to use Sandbox configuration, NO to use Production configuration.

-fetchExperiences: Fetches new experiences from Maxymiser platform for a default Page named Generaflons'. Fetching will be performed on a background thread.

-(void)fetchhxperiences:(void C) Q)handler; Parameters handler A block that will be invoked once experiences are fetched.

-fetchE.xperiences: forPage: Fetches new experiences from Maxymiser platform for a specified Page. Fetching will he

performed on a background thread.

-(void)fetchExpe.rience.s:void C) Q)handler forPage.:(NSString 1<) page; Parameters handler A block that will he invoked once experiences are fetched. page

A Page for which you want to fetch experiences.

-getExperiences Returns fetched experiences that should be used for content transformations.

-(id <MMTExperiences>)getExperiences; Return Value Experiences previously fetched from Maxymiser plafform. nil if experiences were not fetched.

-track Action: value: attribute: Tracks a user action for a default Page named Actions'.

-(void)trackAction:(Nsstring)name value:(NSTnteger)value attrihute:NSString4)attr; Parameters name Name for the action \vhich should exist in the site configuration in the Maxymiser lii. value

Integer value of the action. This can he used fbr the tracking of various parameters such as amount and sum. attr

The action attribute. This can he used for the tracking of various additional information related with the action.

-track Action: value: attribute: forPage: Iracks a user action for a specified Page.

-(void)trackAction:(NSString)name value:(NSlnteger)value attribute:(NSString)attr forPage:(NSString jpage; Parameters name Name for the action which should exist in the site configuration in the Maxymiser LII. value

Integer value of the action. Ihis can be used for the tracking of various parameters such as amount and sum. attr

The action attribute. This can he used for the tracking of various additional information related with the action. page

The Page for which you want to track the action -trackContentSeen: Tracks a special type of action for confirmation that a user has been served the content.

-(void)trackContentSeen:(NSString)campaign; Parameters campaign Campaign name which content was served.

MMTExperiences Protocol to he used to retrieve generated variants for the fetched experiences. Tasks

\Tarits -getVariant\ameForCampaign: element: -getVariantContentForCampaign: element: Instance Methods -getVariantNamePorCampaign: element: Retrieves a generated variant nanae for a campaign and the element which is related to a test configuration in the Maxymiser LTI.

-(NSString 9getVariantNameForCampaign:(NSString)campaign element:(NSstring 25)element; Parameters campaign Name of the campaign for which you want to get the variant name.

element Name of the element in the campaign for which you want to get the variant name.

Return Value The name of the generated variant for the element in the campaign.

Returns nil for the following cases: the campaign is not started, the campaign is finished, the campaign is paused or the site is disabled. In any of these situations a process to render default content should he put in place.

-getVariantcontentForcampaigm element: Retrieves a generated variant content for a campaign and the element which is related to a test configuration in the Maxymiser UT.

-(NSString jgetVariantGinte tForcampaign:(NSString 4)campaign element:(NSString >9 element; Parameters campaign Name of the campaign for which you want to get the variant content.

element Name of the element in the campaign for which you want to get the variant content.

Return Value The content of the generated variant for the element in the campaign.

Returns nil for the following cases: the campaign is not started, the campaign is finished, the campaign is paused or the site is disabled. In any of these situations a process to render default content should be put in place.

MMTLogger Protocol to he used for logging messages from the SDK. Implementation of this protocol can he provided to the MMTApp class. Tasks

Logging -debug: -info: -waniltig: -error: instance methods -debug: Logs message with log level Debug.

-void)dehug:SString >9message; Parameters mess age Message to log.

-info: Logs message with log level Info.

-(void)info:(NSString t)rnessage: Parameters message \1essage tO log.

-warning: Logs niessage with log level Warning.

-(void)warning:SString jniessage; Parameters message Message to log.

-error: Logs message with log level Error.

-(void)error:SSfting)message; Parameters message Message to log.

S Examples of possible System Limitations In an example, the Maxymiser Mobile Library has the following system limitations: * Each App can connect to one Maxymiser Site entity.

* The Site cannot contain more than 10 campaigns.

* The user interface contains some web-specific terminology, the guide above covers the settings that are supported. all others in the Maxymiser UT can he disregarded.

if in doubt, _\Iaxymiser can assist with campaign QC processes.

* A campaign must not contain more than one element.

o Multlvariate testing is not currently supported.

* An Element cannot contain more than 10 variants.

* Segmentatlon, MaxPredict, MaxRecommend and cCB are not supported.

* In an example. not all user attributes are fully supported in this version, so some attributes will show null' or even meaningless values. See Appendix 2 for a list of supported attributes.

* Filter Action by Re.ferre.r URL and Filter Action by URL pararnete.r are not supported within mobile app testing.

6 Appendices 6.1 Appendix 1 -Terminology Notes The UT for Mobile App testing has been re-purposed from web site testing so there are certain parts and terminology that still apply to web site testing.

* The Maxymiser UT uses the terminology (web) Site, this can be read as App name in mobile app testing * \Vhere the Maxymiser UT refers to a Page' this can he thought of as a container for Experiences and Actions.

6.2 Appendix 2 -Attributes The core attributes which are captured in App campaigns differ from those which are captured in web campaigns as described below.

Note: Attributes that are not listed below may still he seen in the reporting user interface and may contain either null' or meaningless values. These should be disregarded and not used for analysis.

6.2.1 New to Apps only -Examples The Attribute Connection' which is the Connection type to the device -2G, 3G, 4G, Wi-Li.

The Attribute OS' which is the Operating System type and version.

The Attribute App Version' which is the App version.

6.2.2 Present in Apps and Web Examples of these are shown in the table in Figure 16.

6.3 Appendix 3 -Glossary of Tenns A/B testing: A method of testing changes in a single area of an app. App visitors are randomly shown content "A" or "B". The response to the content is' analyed to see if the difference betiveen them is significant. A/B Testing is not limited to just Iwo variants and is commonly used as shorthand for A/B/C/D... [n] testing.

Action:A conversion event pe.rformed by the visitor, for example purchase or click-through.

Campaign: The time period a campaign needs to run in order to get statistically reliable results.

Campaign-level Settings: Core configurations that apply to individual campaigns.

Content Generator (CG): A core component of the Maxymiser system: it is the engine that executes main operations during App testing. The Content Generator serves \Tariants and tracks all selected Actions and Personalization Criteria.

Content Variant: The alternative content created in the \1axymiser UI and served to visitors during the testing process. F;ach Element has at least two variants (one of which is designated as the Control \ariant.

Conversions: Data on how many conversion actions were made with the current Content Variant Default Variant: The content displayed on the page before testing. Maxvmiser test reporting compares current @efault/control) content with the content varied in the campaign.

Element: An identifiable part of an app, such as a landing page image, the size of a banner, or the copy of a call-to-action button that is to he tested.

Experience: An experience is composed of a specific set of variants over one or more pages. For exatnple, a possible experience is a welcome page without an image, medium size banner, and call-to-action button saying View Our Best Offers'.

Implementation (status): The Campaign Status that reflects that a campaign has not et gone live.

Jive: Refers to a campaign configuration that is currently running on an App.

Maxymiser UJ:A designated workspace where business users can work collahoratively to use Maxymiser's optimization solutions.

Page: A Maxymiser Site Page that corresponds to page(s) on the web site or App that conttin Elements to be tested.

Production: The environment where visitors see campaign content.

Publish: the process that transfers a Campaign configuration from the Sandbox environment (QC test area) to the Production environment web site/app) where (:paigm content is visible to vveI) site or App visitors.

Publish Center: The area of the Maxymiser LTI where campaign content is transferred to the live \vel) site.

Ready for Publish: A feature used to inform the. Publishe.r that a campaign configuration is complete and the campaign is ready to be made public.

Sandbox: The test environment where campaigns are created and configured.

Variant (Content Variant.A Variant of an Element is a possible configuration of that particular Fiement. There arc at least two variants for every Element: a Dethu]t (or control') and at least one Challenger variant.

Variant \Veight: A value used to determine the proportion of traffic that is served a Content Vanant. Lets say we have two Content Variants "A" and "B'1 \" has a weight of 10 and "B" has a weight of 30. Tn this case "A gets 25% of the Campaign traffic, "B" gets 75%.

\isitor:A user who enters a campaign (and is tracked via a first-party cookie) and views content.

Visitor Attribute: A characteristic that identifies a web site or app visitor, such as their geographical location, activity on a site, operating system and personalization criteria, For

example gender.

6.4 Appendix 4 -Trouble Shooting 6.4.1 Debug logging mechanism The MMFApp library provides' debug logu-ing mechanism based on standard NSLog output. All requests such as query construction, sending requests and response parsing are reflected by NSLg messages. It is therefore possible to see this log of library behavior in the Output window in Xcode and monitor it for errors. Generated log messages are prefixed with "MMTApp:" 6.4.2 Emergency Campaign Stop Tn circumstances where a campaign needs to he stopped you can Pause' or Finish' the campaign by changing its state to Paused' or Finished' in the User Tnterface as referenced in the Publish section of this documentation.

For example a campaign may have been configured or programmed incorrectly leading to unexpected behavior of the application. Once the campaign is paused or finished, our APT will return nil values.

Tmportant: Ensure that the App is coded so that the nil response is processed correctly and the defimlt user experience is sho\vn.

6.4.3 Reporting if you don't see data in reports please ensure that M_\tIApp is configured to use Production configuration as Sandbox configuration does not support reporting. Also please ensure that Maxyrniser platform is notified about content is being served (seen by the user) by calling [MMTApp defaultApil trackContentSeen: %CampaignName% 7 Developnient Support if you require further assistance please raise a ZenDesk ticket or contact your account manager providing the following details where relevant: 1. Environment configuration.

a. Development machine -OS/XCodc version.

b. Deployment target -simulator or device, if this is a device, please state the type and iOS version.

2. Steps to re-produce the problem.

3. The Debug level log from the MMTApp.

4. A copy of the relevant section of the application code if applicable. Note

It is to he understood that the above-referenced arrangements are only illustrative of the application for the principles of the present invention. Numerous modifications and alternative arrangements can be devised without departing fron the spirit and scope of the present invention. While the present invention has been shown in the drawings and fully described above with particularity and detail in connection with what is presently deemed to be the most practical and preferred example(s) of the invention, it will he apparent to those of ordinary skill in the art that numerous moditications can he made without departing from the principles and concepts of the invention as set forth herein.

Claims (10)

1. CLMMS1. Method of optimization for a native application whose software is written in a reflection-based programming language, the native application executable on a computIng device including a display and a precessor, the native application contigured to display a user interface on the display ot the computing device, the method including the steps of: (i) applying generic code to the native application to prodce a modified native application which is configured to communicate with a server external to the computing device, and (ii) remotely configuring the modifie.d native application from the server to examine and to modify the structure and behaviour of the modified native application at run time so as to he capable of altering one or more oF the data, Enctions or content of a user interface generated by the modified native application.
2. 2. Method of Claim 1, wherein the generic code is the only code needed to be applied to source code of the native application to enable optimization of the modified native application.
3. 3. _Method of Claims 1 or 2, wherein the generic code enables many kinds of optimizations to he remotely configured in an optimization system's user interfaces, without changing the code within multiple versions-and instances of the native application.
4. 4. Method of any previous Claim, \vherein modifying the structure and behavior of the modified native application at runtime includes modifying one or more of the values, meta-data, properties, images and functions of the modified native application.
5. 5. _Method of any previous Claim, wherein the computing device is a mobile device, a touch screen device, a smart phone, a video game console, a computer, a smart IV, a IV set-top-box, or a device connectable to the internet.
6. 6. Method of any previous Claim, wherein the computing device is connectible to the internet over a wired or \vireless wide area netvork (WAN) or over a mobile phone network to a server.
7. 7. Method of any previous Claim, wherein optimization includes adapting visual experiences for each user on the fly in order to increase the probability of a user completing a desired action.
8. 8. Method of Claim 7, wherein a desired action is a click or an order placement
9. 9. _Method of any previous Claim, wherein a reflection based programing language is a common reflection based programing language.
10. 10. Method of any previous Claim, wherein the method further includes collecting one or more attributes about the user/device for use in targeting, protiling or analyzing user behavior.

    ii. Method of Claim 10, wherein the collected attributes are one of more of: Connection type to the device (e.g. 2G, 3G, 4G. Wi-hi); Device Iype,application version; Operating System type and version; Location (e.g. by coordinates, city, counfty, continent, postal code); Internet service provider; Recency and details of past visits to the app; Recency and details of past actions; Custom user defined attributes such as past interaction with a specific product category; Referral source \vhere a user has the opportunity to launch the app from a variety of locations; Time of visit by date, day, hour, and Data imported from other locations such as CRM databases.12. _Method of Claims 10 or ii, further including the step of: including or excluding visitors/devices from a campaign using their attributes.13. _Method of any of Claims 10 to 12, further including the step of: creating segmentation rules to target content to visitors/devices according to their atthbutes in combination with rules for other user segment infirmation.14. _\Ie.thod of any previous Claim, wherein native application pertormnance is optimized during a campaign as well as after the campaign.15. Method of any previous Claim, wherein alterable user interface content includes one or more of: i) Simple text or image Hock (banners, headers of the pages) movement, resizing, hiding, reformatting; j) Buttons texts, movement, resizing, reformatting, and hiding; k) Push notification' frequency and content; 1) Displayed picture (picture could be stored on extema] sen/er); m) New text or image block; n) Lists & tables (ordering, how many items to display); o) Navigation paths within the application (e.g. does add to cart' direct the user to the cart page or hack to a category page to make further selections), and p) Displaying a navigation based upon the last user choice.16. Method of any previous Claim, wherein the native application includes a library.17. Method of Claim 16, wherein the library is static for all applications.18. Method of any of Claims 16 to 17, wherein the library is contigured to send requests to and to receive responses from a content generator.19. Method of any of Claims 16 to 18, wherein the library interacts with a native application cache that is included in the native application.20. _Method of Claim 19, wherein the native application cache is configured to receive content pushed from the external server.21. _\Iethod of any of Claims 16 to 20, wherein the library makes a request to a content generator on native application srt.22. Method of any of Claims 16 to 18, wherein the librar is arranged to make a request to a content generator, the request including a location and user attributes in parameters.23. Method of any of Claims 16 to 22, wherein the library includes one or more of the following modules: 9) CG request module; 10) External assets module (image transaction); 11) Tracking module (actions tracking, sending results to the CG); 12) Application object identification module; 13) MYT module (implement alternative content visualization); 14) Personalization module; 15) Segmentation module, and 16) Administrative console module.24. Method of any of Claims 16 to 23, wherein the library can modify only specified objects and properties in code instructions.25. Use of the method of any of Claims 1 to 24 to optimize a selectable button displayed on the display from among at least two alternatives for the selectable button.26. LTse of the method of any of Claims 1 to 24 to optimize a displayed product on the display from among at least two alternatives fbr the displayed product, based on a most recently used filter option.27. Use of the method of any oF Claims 1 to 24 to optimize a displayed food product on the display from among at least two alternatives for the displayed food product, based on a most recently viewed food category.28. Use of the method of any of Claims 1 to 24 to optimize a displayed advertisement on the display from among at least two alternatives for the displayed advertisement, based on recorded click through rates for the displayed advertisements.29. System comprising a server, a computing device including a display and a processor, and a modified native application executable on the computing device, the modified native application software being written in a reflection-based programming language, the modified native application configured to display a user interface on the display of the computing device, a native application having been modihed by applying generic code to the native application to produce a modified native application which is configured to communicate with the server which is external to the computing device, wherein the modified native application is remotely configurable by the server to examine and tei modify the structure and behaviour of the modified native application at run time so as to be capable of altering one or more of: the data, functions or content of a user interñce generated by the modified native application.30. System configured to perform a method of any of Claims 1 to 24.31. System of Claims 29 or 30. wherein the server is further configured to collect one or more attributes about the user/device for use in targeting, profiling or analyzing user behavior.32. System of Claim 31, wherein the server is further configured to filter reports, including conversion and revenue reports, by atftibute values.fl. System of Claims 31 or 22, wherein the server is frirther conhgured to discover segments that may require a different experience to increase conversion rates based upon the attributes.34. System of any of Claims 29 to 33, wherein the server is configured to automatically detect higher performing content variants and then to arrange to serve those higher perfonning content variants to more future visitors.35. System of any of Claims 29 to 34, wherein the system includes the ability to switch a concluded campaign to a mode where winners only' are served, meaning that the native application supplier does not need to make an update to the code in order to apply winning varialions of content or user experience.36. System of any of Claims 29 to 35, wherein the syste.m includes a self-learning.predictive. algorithm which drives each user to the best offer for that individual, automatically.37. System of any of Claims 28 to 36, wherein the system includes a segment disce very process to automate the analysis process to discover the user segments which require a different experience than the overall winner to increase performance against measures for that segment.38. System of any of Claims 29 to 37, wherein the system provides mu]tivariate testrng.39. System of any of Claims 29 to 38, wherein the server is configured to send segmentation rules to the native application.40. System of any of Claims 29 to 39, wherein the system is configured to ftack one or more of: F) Views of a specific screen, for example the screen shown at completion of an order; g) Button clicks or finger/stylus taps; h) Tracking on the dependent website pages; i) Audio/video players playing time, and j) Action attribute and action value opportunities (e.g. to measure sales value in or other currency).41. System of any of Claims 29 to 40, wherein the system is configured to measnre a response of users to a change in the application.42. Use of the system of any of Claims 29 to 41, wherein historical customer information, such as product held or a customer's propensity to buy a product, can be used to dynamic drive the best, personahzed experience for each customer on native applications.43. Computing device including a display and a processor, the computing device including a modified native application whose softivare is written in a reflection-based programming language, the modified native application executable on the device, the modified native application configured to dispLiy a user interface on the display of the computIng device, the modified native application having been modified by applying generic code to a native application tc produce the modified native application which is configured to communicate with a server external to the computing device, wherein the modified native application is remotely configurable by the server to examine and to modify the structure and behaviour of the modified native application at run time so as to be capable of altering one or more of: the data, functions or content of a user interface generated by the modified native application.44. Server configured to communicate with a computing device, the computing device including a display and a processor, the computing device including a modified native application whose software is written in a reflection-based programming language, the modified native application executable on die computing device, the nodified native application configured to display a user interface on die display of the computing device, the modified native application having been modified by applying generic code to a native application to produce the modified native application \vhich is configured to con municate with die server which is external to the computing device, wherein the server is arranged to remotely configure the modified native application to examine and to modify the structure and behaviour of the modified native application at run time so as to he capable of altering one or more of: the data, functions or content of a user interface generated by the modified native application.45. System comprising a server, a plurality of computing devices each including a display and a processor, and a modified native application executable on the plurality of computing devices, the modified native application software being written in a reflection-based programming language, the modified native application configured to display a user interface on the display of each of the plurality of computing devices, a native application having been modified by applying generic code to the native application to produce a modified native application which is configured to comniunicate with the server which is external to the plurality of computing devices, xvherein the modified native application is remotely configurable by the server to examine and to modify the structure. and behaviour of the modified native application at run tinie so as to be capable of altering one or more of: the data, Linctions or content of a user interface generated by the modified native application, wherein the server is arranged to configure a plurality of modified native applications on the plurality of computing devices in a plurality of different configurations and to record measurements of user actions on the purahty of computing devices in rdatton to the plurahty of different configurations.