GB2512359A - Self adapting multi variant testing - Google Patents

Self adapting multi variant testing Download PDF

Info

Publication number
GB2512359A
GB2512359A GB1305636.1A GB201305636A GB2512359A GB 2512359 A GB2512359 A GB 2512359A GB 201305636 A GB201305636 A GB 201305636A GB 2512359 A GB2512359 A GB 2512359A
Authority
GB
United Kingdom
Prior art keywords
content
assembly
consumer
attributes
historical
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
GB1305636.1A
Other versions
GB201305636D0 (en
Inventor
Ray Gerber
Glen Manchester
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Thunderhead Ltd
Original Assignee
Thunderhead Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Thunderhead Ltd filed Critical Thunderhead Ltd
Priority to GB1305636.1A priority Critical patent/GB2512359A/en
Publication of GB201305636D0 publication Critical patent/GB201305636D0/en
Priority to PCT/GB2014/050982 priority patent/WO2014155123A1/en
Publication of GB2512359A publication Critical patent/GB2512359A/en
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0241—Advertisements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0241—Advertisements
    • G06Q30/0242—Determining effectiveness of advertisements
    • G06Q30/0243—Comparative campaigns
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0641—Shopping interfaces
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/40—Network security protocols

Abstract

A computer implemented method and associated apparatus of a content assembly generator for generating a content assembly for delivery to a content consumer, the content consumer having associated a plurality of content consumer attributes 302 from a set of possible attributes, the method comprising: receiving the content consumer attributes 302 ; generating a content assembly 304 based on the content consumer attributes and historical content configuration information; delivering the content assembly to the content consumer 306 ; recording metrics 308 associated with interaction between the content consumer and the content assembly; and updating the historical content configuration information based on the metrics 310 , wherein the historical content configuration information includes efficacy information about historical content assemblies for content consumer attributes in the set of possible attributes. The efficacy information may be for example a response rate, a sales conversion rate or progress through content. Information regarding the success (or otherwise@) of the interaction between the content consumer and the content assembly is stored as historical content configuration (figure 2). The step of generating a new content assembly includes defining a new content configuration based on an optimisation algorithm applied to the historical content information, the algorithm being a learning algorithm or an adaptive algorithm. The application also includes a computer program element comprising a computer program code to perform the method.

Description

Self Adapting Multi Variant Testing
Field of the Invention
The present invention relates to testing content for delivery to oontent consumers. In partioular, it relates to mulci variant testing.
Background of the Invention
Split Testing, or A/B Testing, is an approach to assessing the effectiveness of content, such as web page content, in order to provide improved content. The effectiveness of two versions of content are compared in order to discover which has greater efficacy. Efficacy is be measured depending on a purpose of the content. Accordingly, efficacy can be measured in cerms of, inter alia, a response rate, a sales conversion race, a progress through content etc. Multi variant testing is a process by which any number of components of content, such as components of a webpage, may be tested. Multi variant testing effectively allows numerous A/B tests to be performed for content comprising an assembly of content parts, at the same time. Multi variant testing can allow for a large number of possible combinations of content in numerous configurations. In view of the large number of possible variations, multi variant testing is constrained by the available time and population of content receivers (such as users) An example of content for which split or multi variant testing may be performed is the series of steps involved in a purchase on an electronic commerce web site. Any improvements in drop-off rates and failures to convert visitors to sales can represent additional sales for an electronic commerce provider.
Prior art approaches to multi variant testing require The selection of variatio-is of content elements, such as a component of a web page, for different users. On conclusion of a statistically significant set of tests a preferred content resource can be selected for users based on metrics arising from the multi variant testing process. Disparate content propositions can be provided for comparison, imposing a considerable burden o content providers who prepare content propositions and adjust content provision in response o repeated test results.
It would therefore be advantageous to provide multi variant testing for content without the aforementioned disadvantages.
Summary of the Invention
The present invention accordingly provides, in a first aspect, a computer implemented method of a content assembly generator for generating a content assembly for delivery to a content consumer, the content consumer having associated a plurality of content consumer attributes from a set of possible attributes, the method comprising: receiving the content consumer attributes; generating a content assembly based on the content consumer attributes and historical content configuration information; delivering the content assembly to the content consumer; recording metrics associated with interaction between the content consumer and the content assembly; and updating the historical content configuration information based on the metrics, wherein the historical content configuration information includes efficacy information about historical content assemblies for content consumer attributes i the set of possible attributes.
Thus, in accordance with the present invention, a content assembly is produced for a content consumer based on historical content coiifiguration information. An efficacy of the content assembly is measured using metrics associated with the interaction between the content consumer and the content assembly. The efficacy information of the content assembly is recorded in the historical content configuration information.
In this way, subseguent content assemblies are generated based on historical efficacy information such that a suhseguent content assembly is adapted to provide potentially more efficacious assemblies of content.
Preferably the step of updating the historical content configuration information includes the steps of: generating efficacy information about the content assembly based on the metrics; and storing a definition of the content assembly in assooiation with the generated efficacy information.
Preferably the step of generating a content assembly includes defining a new content configuration based on an optimisation algorithm applied to the historical content configuration information.
Thus the production of a content assembly based on hisuorical content configuration information includes the selection of a new content assembly nsing an optimisation algorithm based on the historical information, such as a self learning algorithm.
In this way, a content assembly is generated representing a promising new assembly having improved efficacy over those previously assembled.
Preferably a content configuration includes an identification of a template specifying one or more constituent parts of a content assembly.
Preferably the step of generating a content assembly further comprises the steps of: receiving a content assembly template specifying one or more constituent parts of the contenu assembly; accessing a content element for each constituent part of the content assembly from a repository of content elements, each content element having element attributes; configuring the element attributes of each content element in accordance with the new content configuration.
Preferably the optimisation algorithm is a maohine learning aigorithm.
Preferably the metrics inciude one or more of a conversion rate of a content consumer's interaction with the content assembly, an indication of the content consumer's progress through a predetermined interaction process, and a success indicator of a content consumer's interaction with the content assembly.
The present invention accordingly provides, in a second aspect, a content assembler system for generating a content assembly for delivery to a content consumer, the content consumer having associated a piurality of content consumer attributes from a set of possible attributes, the system comprising: a receiver for receiving the content consumer attributes; a generator for generating a content assembly based on the content consumer attributes and historical content configuration information; a delivery component for delivering the content assembly to the content consumer; a metric recorder for recording metrics associated with interaction between the content consumer and the content assembly; and an updater for updating the historical content configuration information based on the metrics, wherein the historical content configuration information includes efficacy information about historical content assemblies for content consumer attributes in the set of possible attributes.
The present invention accordingly provides, in a third aspect, an apparatus comprising: a central processing unit; a memory subsystem; an input/output subsystem; and a bus subsystem interconnecting the central processing unit, the memory subsystem, the input/output subsystem; and the apparatus as described above.
The present invention accordingly provides, in a fourth aspect, a computer program element comprising computer program code to, when loaded into a computer system and executed thereon, cause the oomputer to perform the steps of a method as desoribed above.
Brief Description of the Drawings
A preferred embodiment of the present invention is described below in more detail, by way of example only, with reference to the accompanying drawings, in which: Figure 1 is a block diagram of a computer system suitable for the operation of embodiments of the present invention; Figure 2 is a component diagram illustrating components of a content assembly process in accordance with a preferred embodiment of the present invention; Figure 3 is a flowchart of a method of a content assembler in accordance with a preferred embodiment of the present invention; and Figure 4 is a component diagram of the content assembler of Figure 2 in accordance with a preferred embodiment of The present invention.
Detailed Description of the Preferred Embodiments
Figure 1 is a block diagram of a computer system suitable for the operation of embodiments of the present invention. A central processor unit (CPU) 102 is communicatively connected to a storage 104 and an input/output (I/C) interface 106 via a data bus 108. The storage 104 can be any read/write storage device such as a random access memory (RAM) or a non-volatile storage device. An example of a non-volatile storage device includes a disk or tape storage device. The I/O interface 106 is an interface to devices for the input or output of data, or for both input and output of data. Examples of I/C devices connectable to I/C interface 106 include a keyboard, a mouse, a display (such as a monitor) and a network connection.
Figure 2 is a component diagram illustrating components of a content assembly process in accordance with a preferred embodiment of the present invention. Content assembler 200 is a software or hardware component suitable for generating and delivering content as a content assembly 208 to a content consumer 210. In an exemplary embodiment of the present invention, content assembly 208 is all or part of a web page comprised of an assembly of content elements 204, such web page components including, inter alia, textual, pictorial, interactive, media including audio and/or video, or any other conceivable content elements 204. The content assembly 208 is generated by the content assembler 200 based on a content template 202 and a plurality of content elements 204.
Content elements 204 may be stored in any suitable storage means including a data store, database, library or other suitable store. Content elements 204 are suitable for inclusion in a content assembly 208 and may have associated one or more modifiable attributes 206. Modifiable attributes 206 can include attributes affecting the presentation, rendering, position, size, content or any other configurable attribute or parameter of a content element 204. For example, for a textual content element 204, text content, formatting, size, colour, shape, font, spacing, orientation, justification, style or other textual characteristic can constitute modifiable attributes of the content element 204.
Similarly, for a media content element 204 including video and sound, modifiable attributes 206 can include video content, audio content, playback state, playback position, playback speed, audio track, size, position or any other conceivable attribute of parameter of the media content. Those skiiled in the art will appreciate that a content element 204 may have no modifiable attributes 206.
Further, it will be appreciated by those skilled in the art that the content elements 204 may not be actual content components such as web page components, rather content elements 204 may be constituted as specifications of components. For example, a content specification mechanism, such as a web page specificaticn language include HTML, XML or similar, may be employed to specify a content element 204.
The content template 202 can be a specification, definition or other indication of particular content elements 204 or types of content element 204 that are to be assembled into a content assembly 208. Thus, in use, the content assembler 200 retrieves a content template 202 for content to be delivered to a content consumer 212. The content template can indicate one or more content elements 204 for generating the content assembly 208. The content assembler additionally specifies the state of modifiable attributes 206 of content elements 204 when generating the content assembly 208. The particular content elements 204 selected by the content assembler 200 and the particular states of the modifiable attributes 206 of the selected content elements 204 constitutes a content configuration. The particular content configuration is reflected by the content assembly 208, such as by its rendering for delivery to the content consumer 210.
The content consumer 210 is an entity adapted to receive, consume and interact 214 with the content assembly 208. In the exemplary embodiment the content consumer 210 is a user accessing the content assembly 208 via a web browser served by a web server. The interaction 218 between the content consumer and the content assembly 208 can include viewing the content assembly 208, navigating the content assembly 208, interacting with the content assembly 208 and any other interaction that can be conceived between a content consumer 210 and the content assembly 208. As will be apparent to those skilled in the art, the nature of interaction 218 can vary depending on the nature of the content assembly 208 and the nature of the content consumer 210. In one embodiment, transitions by the content consumer 210 from one content assembly 208 to another content assembly 208 are included in the interaction 218. For example such transitions can include a user navigating from one web page to a particular other web page. Furthermore, interaction 218 can include, inter alia, the completion of forms, such as web page forms, the selection of options, the provision of information, the completion of a process such as an ordering or purchase process.
In the context of the content delivery system depicted in Figure 2, interactions 214 between a content consumer 210 and a ccntent assembly 208 can have a mcst desirable or preferable nature. For example, in an electronic commerce environment, a most desirable interaction 218 may be an interaction 218 that leads to a commercial transaction. Similarly, in other embodiments, other most desirable interactions 214 can be conceived. For example, where content assembly 208 relates to fault diagnosis, a most desirable interaction 218 may be an interaction that leads closer towards a diagnosis. Thus, the interactions 214 are monitored to generate content interaction metrics 214. Such metrics 214 can include, inter alia, a rate of conversion of the content consumer's 210 interaction to a particular outcome, s-job as conversion from a pre-saies state to an actual purchase state in an electronic commerce embodiment. Other metrics can include an indication of the content consumer's 210 progress through a predetermined interaction process, such as a checkout or sales process.
Further, metrics can include an indication of a state of success of a content consumer's 210 interaction with the content assembly 208.
The content consumer 210 has associated attributes 212 which can be specific to the content consumer 210 and/or can be shared with other content consumers. Attributes 212 include features, parameters, facets, gualities, behaviours, properties, states, associations or other characteristics of the content consumer 210. For example, in an exemplary embodiment where the content consumer 210 is a user accessing web content, attributes 212 can include, inter alia, the user's name, location, age, gender, preferences, details of previous interactions, or any other attributes conceivably associated with such a user.
Attributes 212 can be expiicitiy provided by the content consumer 210, such as by being provided as part of a current or previous interaction 218. Aiternatively, attributes 212 can be discerned, implied, iearned, assumed or specuiated about the content consumer 210. While attributes 212 are iilustrated as being comprised in the content consumer 210, it will be apparent to those skilled in the art that attributes 212 can alternatively and/or additionally be stored separate from the content consumer 210 such that attributes 212 are attributable to the content consumer 210 without being comprised in, with, or for the content consumer 210. For example, the content consumer 210 can be indicated as belonging to a class of content consumer, and attributes 212 can be assooiated with such a class of content consumer. Such an arrangement is equally suitable for indicating attributes 212 of a content consumer 210.
Attributes 212 can be discerned, implied, learned, assumed or speculated bout the content consumer 210 in any number of ways as will be apparent to those skilled in the art. For example, a behaviour profile of a content consumer 210 arising from the content consumer's 210 historical interactions 214 with one or more content assemblies can be used to derive attributes 212 for the content consumer. Information provided by the content consumer 210 to other, related or unrelated content providers may be associated as attributes 212 associated with the content consumer 210.
Efficacy information discerned from content interaction metrics 214 are stored along with information about attributes 212 for the content consumer 210 and a content configuration of the content assembly 208 as historical content configuration information 216. Thus, for example, information about the success (or otherwise) of an interaction 218 between a content consumer 210 and a content assembly 208 is stored as historical content configuration information 216 along with the oontent configuration of the content assembly 208 and information about the attributes 212 of the content consumer 210. The historical content configuration information 216 can be stored as one or more data structures in a data store or other suitable storage means. The historical content configuration information 216 is preferably stored in a manner that is suitable for processing by an optimisation algorithm such that possible combinations of content configuration, consumer attribute and efficacy can be optimised using an optimisation algorithm such as a machine learning algorithm.
For example, representations of combinations of content configuration, content consumer attributes 212 and efficacy, such as matrix representations, can be processed by machine learning algorithms such as hill climbing algorithms, champion-challenger algorithms, gradient descent algorithms, neural network processing algorithms or any other machine learning or general optimisation algorithm as will be apparent to those skilled in the art. Such algorithms can be adapted to be operable to identify potentially more efficacious assemblies of content based on historical information stored in the historical content configuration information 216.
Thus, in use, the content assembler 200 generates a content assembly 208 based on the historical content configuration information 216 processed by an optimisation algorithm to select a new content configuration that presents a promising adaptation from the historical configurations reflected in the historical content configuration information 216. Promising adaptations are adaptations that result in new content configurations for producing a content assembly 208 with a potentially increased efficacy as compared to historical content configurations. Preferably, the content assembier 200 undertakes to apply such optimisation technigues for every generation of a content assembly 208 such that the content assembler 200, in conjunction with the historical content configuration information 216, is continually optimising a content configuration to improve efficacy. It will, however, be apparent to those skilled in the art that, in some embodiments, it may be desirable to cease optimising when a suitably optimal content configuration has been identified, such as when a particular level of efficacy has been reached or when a local optimum has been achieved, such as a local peak or trough in a gradient descent algorithm.
Thus, in accordance with the present invention, a content assembly 208 is produced for a content consumer 210 based on historical content oonfiguration information 216. An efficacy of the content assembly 208 is measured using metrics 214 associated with the interaction 218 between the content consumer and the content assembly 208. The efficacy information of the content assembly 208 is recorded in the historical content configuration information 216. In this way, subsequent content assemblies 208 are generated based on historical efficacy information such that a subsequent content assembly is adapted to provide potentially more efficacious assemblies of content.
Further, the production of a content assembly 208 based on historical content configuration information includes the selection of a new content assembly using an optimisation algorithm based on the historical information 216, such as a self learning algorithm. In this way, a content assembly 208 is generated representing a promising new assembly having improved efficacy over those previously assembled.
While the elements of Figure 2 are illustrated in direct communication with each other, it will be appreciated by those skilled in the art that the elements of Figure 2 may be separated by intermediate elements not illustrated, such as additional elements, proxies, servers or other elements in embodiments of the present invention. Additionally or alternatively, while the elements of Figure 2 are illustrated as separate elements it will be apparent to those skilled in the art that multiple elements may be combined into single elements such that multiple elements are constituted as a single aggregate or compound element. Yet further, it will be apparent to those skilled in the art that elements of Figure 2 may be operable to communioate with other elements of Figure 2. Those skilled in the art will appreciate that such communication can take place over wired or wireless network connections or through software communication mechanisms such as interprocess, intermodule or interfunction communication facilities including message passing, function or procedure caliing, subroutine execution, library ioading, function invocation or any other suitabie communication mechanism.
Figure 3 is a flowchart of a method of a content assembler 200 in accordance with a preferred embodiment of the present invention. Initially, at step 302, attributes 212 associated with the conteut consnmer 210 are received. At step 304 a conteut assembiy 208 is generated based on the content consumer attributes 212 and historicai content configuration information 216. In an exempiary embodiment, the generation 304 inciudes defining a new content configuration based on an optimisation algorithm applied to the historical content configuration information 216, such as a machine learning algorithm. Further, in an exemplary embodiment, the generation 304 includes receiving a content assembly template 202 specifying one or more constituent parts of the contenu assembly. In the exemplary embodiment, a content element 204 for each constituent part of the content assembly 208 as defined by the template 202 is retrieved from a reposinory of content elements, each content element 204 having modifiable element attributes 206. In the exemplary embodiment the element attributes 206 of each content element are configured in accordance with the new content configuration.
At step 306 the content assembly 208 is delivered to the content consumer 210 for interaction 218. At step 308, metrics 214 associated with the interaction 218 between the content consumer and the content assembly 208 are recorded.
At step 310, the historical content configuration information 216 is updated based on the recorded metrics 214. In an exemplary embodiment, the updating 310 includes generating efficacy information about the content assembly 208 based on the metrics 214 and storing a definition of the content assembly 208 as a content configuration in association with the generated efficacy information.
Figure 4 is a component diagram of the content assembler 200 of Figure 2 in accordance with a preferred embodiment of the present invention. The assembler 200 includes a receiver 402, generator 406, delivery component 408, metric recorder 410 and an updater component 412. Each of the components 402 to 412 are hardware, software or combined hardware and software components.
The receiver 402 is suitable for receiving attributes 212 for a content consumer 210 in accordance with step 302 of The method of Figure 3. The generator 406 is suitable for generating a content assembly 208 based on consumer atuributes 212 and historical content configuration information 216 in accordance with step 304 of Figure 3. The delivery component 408 is suitable for delivering a content assembly 208 no a content consumer 210 in accordance with step 306 of Figure 3.
The metric recorder 410 is suitable for recording metrIcs 214 associated with an interaction 218 between a content consumer 210 and a content assembly 208 in accordance with step 308 of Figure 3. The updater component 412 is suitable for updating historical content configuration information 216 based on the metrics 214 in accordance with step 310 of Figure 3.
While the assembler 200 of Figure 4 is illustrated as comprising each of the elements 402 to 412 it will be apparent to those skilled in the art that any or all of the elements 402 to 412 may be provided separate to the assembler 216, such as through components external to the assembler 216 including any of the components illustrated in Figure 2. Additionally, it will be apparent to those skilled in the art that while the elements 402 to 412 of Figure 4 are illustrated as separate components, any or all of the elements 402 to 412 may be constituted in a single or combined component.
Insofar as embodiments of the invention described are implementable, at least in part, using a software-controlled programmable processing device, such as a microprocessor, digital signal processor or other processing device, data processing apparatus or system, it will be appreciated that a computer program for configuring a programmable device, apparatus or system to implement the foregoing described methods is envisaged as an aspect of the present invennion.
The computer program may be embodied as source code or undergo compilation for implementation on a processing device, apparatus or system or may be embodied as object code, for
example.
Suitably, the computer program is stored on a carrier medium in machine or device readable form, for example in solid-state memory, magnetic memory such as disk or tape, optically or magneto-optically readable memory such as compact disk or digital versatile disk etc., and the processing device utilises the program or a part thereof to configure it for operation. The computer program may be supplied from a remote source embodied in a communications medium such as an electronic signal, radio frequency carrier wave or optical carrier wave. Such carrier media are also envisaged as aspects of the present invention.
It will be understood by those skilled in the art that, although the present invention has been described in relation to the above described example embodiments, the invention is not limited thereto and that there are many possible variations and modifications which fall within the scope of the invention.
The scope of the present invention includes any novel features or combination of features disclosed herein. The applicant hereby gives notice that new claims may be formulated no such features or combination of features during prosecution of this application or of any such further applications derived therefrom. In particular, with reference to the appended claims, features from dependent claims may be combined with those of the independent claims and features from respective independent claims may be combined in any appropriate manner and not merely in the specific combinations enumerated in the claims.

Claims (16)

  1. CLAIMS1. A computer implemented method of a oontent assembly generator for generating a content assembly for delivery to a content oonsumer, the content consumer having associated a plurality of content consumer attributes from a set of possible attributes, the method comprising: receiving the content consumer attributes; generating a content assembly based on the contenu consumer attributes and historical content configuration information; delivering the content assembly to the content consumer; recording metrics associated with interaction between the content consumer and the content assembly; and updating the historical content configuration information based on the metrics, wherein the historical content configuration information includes efficacy information about historical content assemblies for content consumer attributes in the set of possible attributes.
  2. 2. The method of claim 1 wherein the step of updating the historical content configuration information includes The steps of: generating efficacy information about the content assembly based on the metrics; and storing a definition of the content assembly in association with the generated efficacy information.
  3. 3. The method of any preceding claim wherein the step of generating a content assembly includes defining a new content configuration based on an optimisation algorithm applied to the historical content configuration information.
  4. 4. The method of any preceding claim wherein a content configuration includes an identification of a template specifying one or more constituent parts of a content assembly.
  5. 5. The method of any of claims 3 or 4 wherein the step of generating a content assembly further comprises the steps of: receiving a content assembly template specifying one or more constituent parts of the ccntent assembly; accessing a content element for each constituent part of the ccntent assembly from a repository of content elements, each content element having element attributes; and configuring the element attributes of each contenc element in accordance with the new content configuration.
  6. 6. The method of any of claims 3 to 5 wherein the optimisation algorithm is a machine learning algorithm.
  7. 7. The method of any preceding claim wherein the metrics include one or more of a conversion rate of a content consumer's interaction with the content assembly, an indication of the content consumer's progress through a predetermined interaction process, and a success indicator of a content consumer's interaction with the content assembly.
  8. 8. A content assembler system for generating a contenc assembly for delivery to a content consumer, the content consumer having associated a plurality of content consumer attributes from a set of possible attributes, the system comprising: a receiver for receiving the content consumer attributes; a generator for generating a content assembly based on the content consumer attributes and historical content configuration information; a delivery component for delivering the content assembly to the content consumer; a metric recorder for recording metrics associated with interaction between the content consumer and the content assembly; and an updater for updating the historical content configuration information based on the metrics, wherein the historical content configuration information includes efficacy information about historical content assemblies for content consumer attributes in the set of possible attributes.
  9. 9. The system of claim 8 wherein the updater inciudes: means for generating efficacy information about the content assembiy based on the metrics; and means for storing a definition of the content assembly in association with the generated efficacy information.
  10. 10. The system of any of claims 8 or 9 wherein the generator inciudes means for defining a new content configuration based on an optimisation aigorithm appiied to the historical content configuration information.
  11. 11. The system of any of claims 8 to 10 wherein a content configuration includes an identification of a tempiate specifying one or more constituent parts of a content assembly.
  12. 12. The system of any of claims 10 or 11 wherein the generator further comprises: means for receiving a content assembiy tempiate specifying one or more constituent parts of the content assembly; means for accessing a content element for each constituent part of the content assembly from a repository of content elements, each content element having element attributes; and means for configiring the element attributes of each content element in accordance with the new content configuration.
  13. 13. The system of any of claims 10 to 12 wherein the optimisation algorithm is a machine learning algorithm.
  14. 14. The system of any of claims 8 to 13 wherein the metrics include one or more of a conversion rate of a content consumer's interaction with the content assembly, an indication of the content consumer's progress through a predetermined interaction process, and a success indicator of a content consumer's interaction with the content assembly.
  15. 15. An apparatus comprising: a central processing unit; a memory subsystem; an input/output subsystem; and a bus subsystem interconnecting the central processing unit, the memory subsystem, the input/output subsystem; and the apparatus as claimed in any of claims 8 to 14.
  16. 16. A computer program element comprising computer program code to, when loaded intc a computer system and executed thereon, cause the computer to perform the steps of a method as claimed in any of claims 1 to 7.
GB1305636.1A 2013-03-27 2013-03-27 Self adapting multi variant testing Withdrawn GB2512359A (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
GB1305636.1A GB2512359A (en) 2013-03-27 2013-03-27 Self adapting multi variant testing
PCT/GB2014/050982 WO2014155123A1 (en) 2013-03-27 2014-03-27 Self adapting multi variant testing

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
GB1305636.1A GB2512359A (en) 2013-03-27 2013-03-27 Self adapting multi variant testing

Publications (2)

Publication Number Publication Date
GB201305636D0 GB201305636D0 (en) 2013-05-15
GB2512359A true GB2512359A (en) 2014-10-01

Family

ID=48444889

Family Applications (1)

Application Number Title Priority Date Filing Date
GB1305636.1A Withdrawn GB2512359A (en) 2013-03-27 2013-03-27 Self adapting multi variant testing

Country Status (2)

Country Link
GB (1) GB2512359A (en)
WO (1) WO2014155123A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110245978B (en) * 2019-05-23 2023-09-05 创新先进技术有限公司 Method and device for evaluating and selecting policies in policy group

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6662215B1 (en) * 2000-07-10 2003-12-09 I Novation Inc. System and method for content optimization
WO2008052059A2 (en) * 2006-10-25 2008-05-02 Gomez, Inc. Methods and apparatus for real user monitoring
US20080275980A1 (en) * 2007-05-04 2008-11-06 Hansen Eric J Method and system for testing variations of website content
US20100145763A1 (en) * 2008-12-05 2010-06-10 Swanson International Inc. Systems, methods and apparatus for valuation and tailoring of advertising
JP2011164786A (en) * 2010-02-05 2011-08-25 Nomura Research Institute Ltd Device, method and program for verifying operation
US20120004983A1 (en) * 2010-06-30 2012-01-05 Cbs Interactive Inc. Systems and methods for determining the efficacy of advertising
WO2013109457A2 (en) * 2012-01-16 2013-07-25 3M Innovative Properties Company Systems and methods for rule-based content optimization

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6327574B1 (en) * 1998-07-07 2001-12-04 Encirq Corporation Hierarchical models of consumer attributes for targeting content in a privacy-preserving manner
US7937380B2 (en) * 2006-12-22 2011-05-03 Yahoo! Inc. System and method for recommended events
US8468051B2 (en) * 2010-06-04 2013-06-18 Microsoft Corporation Selecting and delivering personalized content

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6662215B1 (en) * 2000-07-10 2003-12-09 I Novation Inc. System and method for content optimization
WO2008052059A2 (en) * 2006-10-25 2008-05-02 Gomez, Inc. Methods and apparatus for real user monitoring
US20080275980A1 (en) * 2007-05-04 2008-11-06 Hansen Eric J Method and system for testing variations of website content
US20100145763A1 (en) * 2008-12-05 2010-06-10 Swanson International Inc. Systems, methods and apparatus for valuation and tailoring of advertising
JP2011164786A (en) * 2010-02-05 2011-08-25 Nomura Research Institute Ltd Device, method and program for verifying operation
US20120004983A1 (en) * 2010-06-30 2012-01-05 Cbs Interactive Inc. Systems and methods for determining the efficacy of advertising
WO2013109457A2 (en) * 2012-01-16 2013-07-25 3M Innovative Properties Company Systems and methods for rule-based content optimization

Also Published As

Publication number Publication date
WO2014155123A1 (en) 2014-10-02
GB201305636D0 (en) 2013-05-15

Similar Documents

Publication Publication Date Title
CN102306120B (en) Measure actual final user's performance and the availability of WEB application program
JP4536146B1 (en) RECOMMENDATION DEVICE, RECOMMENDATION METHOD, AND RECOMMENDATION PROGRAM
US8893076B2 (en) Configurable computation modules
US10672013B2 (en) Product test orchestration
US9842092B1 (en) Multi-page website optimization
US20070206584A1 (en) Systems and methods for providing a dynamic interaction router
US8166455B2 (en) Desktop application factory and application templates
US11204785B1 (en) Parameterized user interface for capturing user feedback
US9633386B1 (en) Display of items from search
US20170097749A1 (en) Integrating applications in a portal
CN102135880A (en) Widget Framework
WO2013070624A1 (en) Similarity and relatedness of content
US10152469B2 (en) Analytics report segments and controls
CN104081393A (en) Web application content mapping
US9934292B2 (en) Dynamic presentation of a results set by a form-based software application
US11126785B1 (en) Artificial intelligence system for optimizing network-accessible content
CN105164662A (en) Use of method overrides for dynamically changing visible page content
GB2553896B (en) Product test orchestration
GB2512359A (en) Self adapting multi variant testing
JP6397098B1 (en) Information processing apparatus, information processing method, and information processing program
CN107562461B (en) Feature calculation system, feature calculation method, storage medium, and electronic device
US9734529B1 (en) Method, system, and computer readable medium for selection of catalog items for inclusion on a network page
CN108780379B (en) Device controller and viewer
WO2018147954A1 (en) Automatic generation of interactive web page content
US20210042441A1 (en) Protecting user privacy in user interface data collection

Legal Events

Date Code Title Description
732E Amendments to the register in respect of changes of name or changes affecting rights (sect. 32/1977)

Free format text: REGISTERED BETWEEN 20150416 AND 20150422

732E Amendments to the register in respect of changes of name or changes affecting rights (sect. 32/1977)

Free format text: REGISTERED BETWEEN 20170105 AND 20170111

R108 Alteration of time limits (patents rules 1995)

Free format text: EXTENSION ALLOWED

Effective date: 20210415

Free format text: EXTENSION APPLICATION

Effective date: 20210326

R108 Alteration of time limits (patents rules 1995)

Free format text: EXTENSION ALLOWED

Effective date: 20210603

Free format text: EXTENSION APPLICATION

Effective date: 20210527

WAP Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1)