WO2016149347A1 - Context aware actionable behavior pattern management for evolving user behaviors - Google Patents

Abstract

Systems, devices, and methods for managing an actionable event pattern (AEP) based on feedback are provided. A management system may receive event data from a plurality of users on a real-time basis, classify multiple users into a metric group on a dynamic real-time basis based on behavior parameters and specific states of each user; conduct a controlled experiment on the multiple users of the metric group on a real-time basis based on an AEP configuration of an AEP matched to the metric group, including assigning users of the metric group to one of a control group and an experiment group; collect and analyze feedback measurements from the users of the control group and the experiment group to assess improvement between the control group and the experiment group; adjust the AEP configuration based on the assessed improvement; and store the adjusted AEP configuration.

Description

CONTEXT AWARE ACTIONABLE BEHAVIOR PATTERN

MANAGEMENT FOR EVOLVING USER BEHAVIORS

CROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. provisional application

No. 62/133,741 filed March 16, 2015, the content of which is hereby incorporated by reference herein.

BACKGROUND

[0002] Event processing refers to processing streams of information from multiple sources and reacting to meaningful combinations of events that identify opportunities or risks. Event processing and event driven architectures have been applied in various applications and services such as mobile and wireless sensor network service management systems, Internet content delivery service management systems, and Customer Relationship Management (CRM) systems. Typical event processing systems collect events from external applications and services. When one or more collected events match a set of patterns, the event processing systems may generate actions to change one or more states of the system, send messages, or invoke actions on other external systems. These actions may be used to identify business opportunities, reduce risk, or improve the efficiency of the applications and services. These actions may also be used to affect or improve user behaviors such as performance, interest, preference, and satisfaction levels to the applications and services.

SUMMARY

[0003] Systems, devices, and methods are provided for managing and adjusting an actionable event pattern (AEP) based on feedback.

[0004] For example, a method is provided for AEP management on a real-time basis, and the method includes receiving, by at least one processor, event data from a plurality of users on a real-time basis, wherein the event data corresponds to behavior parameters of the plurality of users in specific states; classifying, by the at least one processor, multiple users of the plurality of users into a metric group on a dynamic real-time basis based on the behavior parameters and specific states of each of the plurality of users; conducting, by the at least one processor, a controlled action effectiveness experiment on the multiple users of the metric group on a real-time basis based on an AEP configuration of an AEP matched to the metric group, including assigning at least one user in the metric group to a control group and assigning at least one other user of the metric group to an experiment group; collecting, by the at least one processor, feedback measurements from the multiple users during the controlled action effectiveness experiment; analyzing, by the at least one processor, the feedback measurements to assess improvement between the control group and the experiment group; adjusting, by the at least one processor, the AEP configuration based on the assessed improvement; and storing, by the at least one processor, the adjusted AEP configuration.

[0005] In another example embodiment an AEP management system is provided that includes at least one processor configured to receive event data from a plurality of users on a real-time basis, wherein the event data corresponds to behavior parameters of the plurality of users in specific states; the at least one processor is further configured to classify multiple users of the plurality of users into a metric group on a dynamic real-time basis based on the behavior parameters and specific states of each of the plurality of users; the at least one processor is further configured to conduct a controlled action effectiveness experiment on the multiple users of the metric group on a realtime basis based on an AEP configuration of an AEP matched to the metric group, including configured to assign at least one user in the metric group to a control group and at least one other user of the metric group to an experiment group; the at least one processor is further configured to collect feedback measurements from the multiple users during the controlled action effectiveness experiment; the at least one processor is further configured to analyze the feedback measurements to assess improvement between the control group and the experiment group; the at least one processor is further configured to adjust the AEP configuration based on the assessed improvement; and the at least one processor is further configured to store the adjusted AEP configuration.

[0006] Another example method is provided that includes receiving, by at least one processor, event data from a plurality of users, the event data corresponding to behavior parameters of the plurality of users in specific states; matching, by the at least one processor, the event data to an AEP having an AEP configuration; performing, by the at least one processor, an action provided by an AEP configuration based on the matching; receiving, by the at least one processor, feedback on an effectiveness of the action; adjusting, by the at least one processor, the AEP based on the feedback; and storing, by the at least one processor, the adjusted AEP configuration. The adjusting includes iteratively: profiling the plurality of users into a metric group based the behavior parameters and the specific states of each of the plurality of users; synthesizing and conducting a controlled action effectiveness experiment on the metric group, including assigning at least one user in the metric group to a control group, assigning at least one other user of the metric group to an experiment group, and providing actions under test to the at least one user in the experiment group; collecting feedback measurements from the at least one user in the control group and the at least one user in the experiment group; analyzing the feedback measurements to assess improvement between the at least one user in the control group and the at least one user in the experiment group; and adjusting the AEP configuration based on the assessed improvement.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings wherein:

[0008] FIG. 1A is a system diagram of an example communications system in which one or more disclosed embodiments may be implemented;

[0009] FIG. IB is a system diagram of an example wireless transmit/receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1A;

[0010] FIG. 1C is a system diagram of an example radio access network and an example core network that may be used within the communications system illustrated in FIG. 1A;

[0011] FIG. ID is a system diagram of an example communications system in which one or more disclosed embodiments may be implemented;

[0012] FIG. 2 is a block diagram illustrating example a system of an effectiveness management service;

[0013] FIG. 3 is a block diagram illustrating an example service architecture for Actionable Event Pattern (AEP) effectiveness management;

[0014] FIG. 4 is a block diagram illustrating an example effectiveness manager interface and function components;

[0015] FIG. 5 is a block diagram illustrating an example data model and workflow for an effectiveness management process;

[0016] FIG. 6 is a flow chart illustrating an example action accuracy adjustment process;

[0017] FIG. 7 is a diagram illustrating an example process for selecting parameter settings based on past experiments;

[0018] FIG. 8 is a flow chart illustrating an example action pattern parameter partial optimization process;

[0019] FIG. 9 is a chart illustrating an example of service usage rate improvement over cumulative usage time; and

[0020] FIG. 10 is a chart illustrating example effectiveness measurement functions. DETAILED DESCRIPTION

[0021] FIG. 1A is a diagram of an example communications system 100 in which one or more disclosed embodiments may be implemented. The communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), and the like.

[0022] As shown in FIG. 1A, the communications system 100 may include wireless transmit/receive units (WTRUs) 102a, 102b, 102c, 102d, a radio access network (RAN) 104, a core network 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and/or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and/or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d may be configured to transmit and/or receive wireless signals and may include user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, consumer electronics, and the like.

[0023] The communications systems 100 may also include a base station

114a and a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the core network 106, the Internet 110, and/or the other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and/or network elements.

[0024] The base station 114a may be part of the RAN 104, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and/or the base station 114b may be configured to transmit and/or receive wireless signals within a particular geographic region, which may be referred to as a cell (not shown). The cell may further be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In another embodiment, the base station 114a may employ multiple -input multiple-output (MIMO) technology and, therefore, may utihze multiple transceivers for each sector of the cell.

[0025] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0026] More specifically, as noted above, the communications system

100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 116 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High- Speed Downlink Packet Access (HSDPA) and/or High-Speed Uplink Packet Access (HSUPA).

[0027] In another embodiment, the base station 114a and the WTRUs

102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and/or LTE-Advanced (LTE-A).

[0028] In other embodiments, the base station 114a and the WTRUs

102a, 102b, 102c may implement radio technologies such as IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 IX, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0029] The base station 114b in FIG. 1A may be a wireless router, Home

Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, etc.) to establish a picocell or femtocell. As shown in FIG. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the core network 106. [0030] The RAN 104 may be in communication with the core network

106, which may be any type of network configured to provide voice, data, applications, and/or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. For example, the core network 106 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and/or perform high- level security functions, such as user authentication. Although not shown in FIG. 1A, it will be appreciated that the RAN 104 and/or the core network 106 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which may be utilizing an E-UTRA radio technology, the core network 106 may also be in communication with another RAN (not shown) employing a GSM radio technology.

[0031] The core network 106 may also serve as a gateway for the

WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and/or other networks 112. The PSTN 108 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and the internet protocol (IP) in the TCP/IP internet protocol suite. The networks 112 may include wired or wireless communications networks owned and/or operated by other service providers. For example, the networks 112 may include another core network connected to one or more RANs, which may employ the same RAT as the RAN 104 or a different RAT.

[0032] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities, i.e., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links. For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with the base station 114a, which may employ a cellular -based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0033] FIG. IB is a system diagram of an example WTRU 102. As shown in FIG. IB, the WTRU 102 may include a processor 118, a transceiver 120, a transmit/receive element 122, a speaker/microphone 124, a keypad 126, a display /touchp ad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and other peripherals 138. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0034] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit/receive element 122. While FIG. IB depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0035] The transmit/receive element 122 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, the transmit/receive element 122 may be an antenna configured to transmit and/or receive RF signals. In another embodiment, the transmit/receive element 122 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit/receive element 122 may be configured to transmit and receive both RF and light signals. It will be appreciated that the transmit/receive element 122 may be configured to transmit and/or receive any combination of wireless signals.

[0036] In addition, although the transmit/receive element 122 is depicted in FIG. IB as a single element, the WTRU 102 may include any number of transmit/receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit/receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0037] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 122 and to demodulate the signals that are received by the transmit/receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as UTRA and IEEE 802.11, for example.

[0038] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker/microphone 124, the keypad 126, and/or the display /touchp ad 128 (e.g., a liquid crystal display (LCD) display unit or organic hght-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker/microphone 124, the keypad 126, and/or the display /touchp ad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and/or the removable memory 132. The nonremovable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0039] The processor 118 may receive power from the power source 134, and may be configured to distribute and/or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0040] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station (e.g., base stations 114a, 114b) and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0041] The processor 118 may further be coupled to other peripherals

138, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like. [0042] FIG. 1C is a system diagram of the RAN 104 and the core network 106 according to an embodiment. As noted above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the core network 106.

[0043] The RAN 104 may include eNode-Bs 140a, 140b, 140c, though it will be appreciated that the RAN 104 may include any number of eNode-Bs while remaining consistent with an embodiment. The eNode-Bs 140a, 140b, 140c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs 140a, 140b, 140c may implement MIMO technology. Thus, the eNode-B 140a, for example, may use multiple antennas to transmit wireless signals to, and receive wireless signals from, the WTRU 102a.

[0044] Each of the eNode-Bs 140a, 140b, 140c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the uplink and/or downlink, and the like. As shown in FIG. 1C, the eNode-Bs 140a, 140b, 140c may communicate with one another over an X2 interface.

[0045] The core network 106 shown in FIG. 1C may include a mobility management entity gateway (MME) 142, a serving gateway 144, and a packet data network (PDN) gateway 146. While each of the foregoing elements are depicted as part of the core network 106, it will be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator.

[0046] The MME 142 may be connected to each of the eNode-Bs 140a,

140b, 140c in the RAN 104 via an Si interface and may serve as a control node. For example, the MME 142 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation/deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 142 may also provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM or WCDMA.

[0047] The serving gateway 144 may be connected to each of the eNode

Bs 140a, 140b, 140c in the RAN 104 via the Si interface. The serving gateway 144 may generally route and forward user data packets to/from the WTRUs 102a, 102b, 102c. The serving gateway 144 may also perform other functions, such as anchoring user planes during inter-eNode B handovers, triggering paging when downlink data is available for the WTRUs 102a, 102b, 102c, managing and storing contexts of the WTRUs 102a, 102b, 102c, and the like.

[0048] The serving gateway 144 may also be connected to the PDN gateway 146, which may provide the WTRUs 102a, 102b, 102c with access to packet -switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0049] The core network 106 may facilitate communications with other networks. For example, the core network 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the core network 106 may include, or may communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the core network 106 and the PSTN 108. In addition, the core network 106 may provide the WTRUs 102a, 102b, 102c with access to the networks 112, which may include other wired or wireless networks that are owned and/or operated by other service providers.

[0050] Other network 112 may further be connected to an IEEE 802.11 based wireless local area network (WLAN) 160. The WLAN 160 may include an access router 165. The access router may contain gateway functionality. The access router 165 may be in communication with a plurality of access points (APs) 170a, 170b. The communication between access router 165 and APs 170a, 170b may be via wired Ethernet (IEEE 802.3 standards), or any type of wireless communication protocol. AP 170a is in wireless communication over an air interface with WTRU 102d.

[0051] FIG. ID is a system diagram of an example communications system 175 in which one or more disclosed embodiments may be implemented. In some embodiments, communications system 175 may be implemented using all or a portion of system 100 as shown and described with respect to FIG. 1A.

[0052] User device 180a, server 185, and/or service server 190 may communicate over communications network 195. These communications may be wireless, wired, or any combination of wireless and wired. Communications network 195 may include the internet 110, core network 106, other networks 112, or any other suitable communications network or combination of communications networks.

[0053] User device 180a may include a WTRU (such as WTRU 102a), or any suitable user computing and/or communications device such as a desktop computer, web appliance, interactive television (ITV) device, gaming console (such as Microsoft XBOX™ or Sony Playstation™) or the like. User device 180a and/or applications executing on user device 180a may generate events such as mouse clicks, keyboard strokes, and the like. These events may be processed by user device 180a and/or may be transmitted to another device such as server 185 or service server 190.

[0054] Server 185 may include a web server, application server, data server, or any combination of these or other types of servers. Server 185 may include any suitable server device such as a server computer, personal computer, or the like. Server 185 may host applications accessible to user device 185a. For example, server 185 may include a gaming server hosting a massively multiplayer online game (MMOG), an email server, a web server hosting a website such as a social media website or blog, or other types of servers typically accessible by a user device over a computer communications network. [0055] User device 180a may access server 185 over computer communications network 175 to interact with services that it provides. For example, user device 180a may access a game server hosted on server 185 to participate in a multiplayer online game. Access of server 185 by user device 180a may be via a client application executing on user device 180a or any other suitable mechanism. In some cases, the server 185 may receive events from user device 180a, or may send events to user device 180a. For example, the server 185 may send an event to user device 180a indicating that additional in-game resources are required for continued play.

[0056] Service server 190 may include a web server, application server, data server, or any combination of these or other types of servers hosted on a server device. Service server 190 may include any suitable server device such as a server computer, personal computer, or the like. Service server 190 may be configured to communicate with server 185, for example, over network 195 or any other suitable communications medium. Service server may be co- located with, combined with, or in direct communication with server 185.

[0057] Service server 190 may communicate with server 185 to provide services, such as third party services, to users of server 185. For example, a subscriber to a game hosted on server 185 may access server 185 from user device 180A and may subscribe to third party services for the game which are hosted on service server 190.

[0058] Service server 190 may be configured to receive and/or intercept events transmitted between user device 180a and server 185. For example, in some embodiments server 185 and service server 190 may be configured such that server 185 may send an event destined for user device 180a instead or additionally to service server 190, and service server 190 may send the event or another event, signal, or message to device 180a. For instance, in a case where server 185 includes a game server, server 185 may send an event to service server 190 indicating a requirement of a user of user device 180a, and server 190 may send the event or another signal or message to device 180a indicating that a resource is available to acquire the requirement. In some embodiments, service server 190 may only forward the event to device 180a under certain conditions, such as based on a user preference and/or context information relating to the user of device 180a.

[0059] In some embodiments, the functions of service server 190 and server 185 may be implemented using the same device, or across a number of additional devices.

[0060] In some embodiments, user devices 180b and 180c may communicate with server 185 and/or service server 190 via user device 180a. For example, user device 180a may forward a notification message from service server 190 to user device 180b via a peer to peer connection and may forward a notification message from service server 190 to user device 180c via network 195. In some embodiments, user devices 180a, 180b, and 180c may form a network, such as a peer-to-peer network, and such network may have a mesh topology, a star topology using user device 180a as a coordinating node, or any other suitable topology. In such embodiments, the peer-to-peer network may operate independently of server 185 and/or service server 190, and may incorporate functionality that otherwise would be hosted by server 185 and/or service server 190, such as functionality described herein.

[0061] Everything that follows may, but is not required to be, employed and/or implemented using one or more, or part of one or more of the example systems discussed above.

[0062] Event patterns which are matched to generate meaningful actions may be referred to as "Actionable Event Patterns" (AEP). Such AEPs may include threshold parameters for detecting changes in input event parameters based on a set of conditions or states of users, devices, applications, and/or environments, for example. When one or more parameters of the event exceeds a parameter threshold value or values specified in the AEP(s), an event processing system may trigger execution of one or more actions that may impact external applications and services. [0063] Depending upon user behavior, application, and environmental contexts, actions triggered in response to events may have different levels of effectiveness on user experience and device efficiency in target applications and services. For example, user behavior may change over time and may affect the effectiveness of promotion and support actions on improving customer experience and satisfaction. Similarly, application contexts and environmental conditions such as age and preference, occupancy of a room, weather, and on-demand energy pricing, may affect the effectiveness of energy management actions. Further, location, time, and network impairment may also affect the effectiveness of service support actions for users of different types of mobile devices (e.g. smartphone, tablet, etc.) and network services. As a result of evolving user and device behaviors and changing contexts of applications and services, design and implementation of AEP and assessment of the effectiveness of deployed AEPs and actions for multiple types of applications and services may be complex, time consuming, and costly.

[0064] Effectiveness management methods, processes, and devices may be implemented which may automatically adjust an AEP configuration to affect or maximize action effectiveness. An AEP effectiveness management service architecture may integrate effectiveness management processes with an AEP engine for emerging services such as internet of things (IoTs), games, and location based services.

[0065] FIG. 2 is a block diagram illustrating example system 200 of an effectiveness management service. As shown in FIG. 2, an effectiveness management service may provide interfaces for collecting action effectiveness feedback from external users, devices, applications, and/or services 202, herein collectively referred to as event providers 202, and triggering event patterns and actions from event processing processors, engines and/or systems 204, herein collectively referred to as event processor 204. Event patterns may be realized by the event processor 204 from event data received from the event provides 202 such that a series or pattern of event data forms an event pattern, and may further include context information such as behavior parameters of an event provider 202 in a specific state. An effectiveness management service 206, including one or more processor(s), engine(s), controller(s) and/or system(s), herein collectively referred to as effectiveness manager 206, may also provide interfaces to adjust event patterns and/or control actions of the event processing systems. The effectiveness manager 206 may include one or more processors, engines, controllers and/or systems configured to perform a real-time profiling function such that evolving user and device behaviors are classified into behavior groups (e.g., metric groups) under different application contexts and environmental conditions. Based on receiving, from event providers 202, effectiveness feedback measurements of actions applied to a behavior group, the effectiveness manager 206 may adjust event patterns stored in the event processor 204 to maximize the effectiveness of the actions for each metric group under different application contexts and environmental conditions. Each event provider 202, event processor 204 and/or effectiveness manager 206 may include one or more processors coupled to memory for receiving information, storing information on the memory, reading information from the memory, processing information, and outputting information.

[0066] The effectiveness manager 206 may support multiple event processing systems for multiple applications and services. In addition to external feedback, the effectiveness manager 206 may support context aware adjustment of threshold parameters iteratively based on behavior patterns collected from groups of users with similar behavior patterns. Furthermore, combinations of effectiveness measurement methods may be used to rank the effectiveness of actions. Example systems, devices, methods, processes, data models and architectures for an effectiveness management service and an integration process flow for event processing systems are further described herein. [0067] AEP effectiveness management methods, processes, systems, and devices may perform behavior profiling, synthesis and/or analysis of events, actions, and AEP configurations, which may be collected from target applications and services. Profiling may include tracking and/or categorizing evolving user and device behaviors into multiple behavior metric groups, for example, under different application contexts and/or environmental conditions. Synthesis may include generating one or more controlled experiments on selected users and/or devices (e.g., event providers 202), applying multiple AEP configurations, and collecting effectiveness measurement feedback information (i.e., effectiveness feedback measurements) from the controlled experiments. Using effectiveness measurement feedback information, analysis may include assessing any improvements in effectiveness between a control group and an experiment group of users and/or devices. Various types of application specific functions may then be invoked to calculate correlations between the effectiveness feedback measurements and the corresponding AEPs and actions.

[0068] Profiling, synthesis, and analysis processes may iterate for each

"round" of controlled experiments. The analysis process may adjust an AEP configuration parameter, such as a threshold level or levels and/or a number of actions in each round. By observing correlations between actions and effectiveness feedback measurements, the synthesis and/or analysis processes may identify an optimized AEP configuration for each action. Where optimized AEP configurations for each action are identified for a metric group under a specific context, the iterative synthesis and analysis process may complete the round, record the AEP configuration and apply the configuration to all users and/or devices in the same metric group under the same application and environment contexts.

[0069] Devices, methods and processes for effective AEP management may include one or more of the following features: dynamic context aware behavior profiling; iterative optimization of AEP and action effectiveness; dynamic selection and grouping of users and/or devices into experiment and control groups; and programmable meta-rules-based execution flow control templates and/or experiment control objects.

[0070] Dynamic context aware behavior profiling may include classifying evolving user and/or device behaviors on a real-time basis into multi- dimensional metric groups using application defined behavior indicators such as engagement, competence, and environmental preference (e.g., hot, cold, humid, dry). It will be understood that "behavior indicators" and "behavior parameters" may be used interchangeably. Further, the metric groups may have a specific state representing the application context and a specific environmental condition, metric groups and states.

[0071] The following are examples of metric groups according to application defined behavior indicators of an event provider 202: {Beginner | Intermediate I Expert}, {Frequent Visitor | NewVisitor}, {WarmUpExercise I MildExercise I IntenseExercise}, {HighHeartBeat I MediumHeartBeat I Low heartBeat}, {VIP I HighSpendingCustomer | LowSpendingCustomer | CustomerAtRiskofAbandonment}.

[0072] The following are examples of states representing the application context and a specific environmental condition of an event provider 202: {[StatelD, ApplicationContext: {ApplicationID, TaskName, TaskDescriptor}, EnvironmentCondition: {EnvID, TypeName, ID, EnvDescriptor}].

[0073] Each state may represent a combination of a specific application context and a specific environmental condition. Examples of application context descriptors include mobile network management, integrated fitness and energy management service with IoT sensors and controllers (e.g., biometric, thermostat, humidity, heater, and air-conditioner), and content delivery services with user interaction analysis and promotion actions. Examples of environmental condition descriptors include network overload and impairments, mobile device types, locations, peak usage hour, and exterior temperature, and humidity. [0074] Iterative optimization of AEP and action effectiveness may be based on real-time feedback of effectiveness measurements from controlled experiments on event inputs, metric groups, and states.

[0075] Dynamic selection and grouping of users and devices into experiment and control groups may be based on real-time pattern matching of AEP execution traces for each metric group and state. The following example data model for an AEP execution trace may be used to facilitate triggering of actions by the same AEPs when selecting users for both control and experiment groups:

UserlD | DevicelD.

- Triggered action type, name, ID.

Triggered AEP name, type, ID.

Event parameter list for triggered AEP: [{parameter name, ID, and threshold value that triggered AEP}].

Matching the triggered AEP and action may facilitate users selected for experiment and control groups in triggering the same actions. This may be used in measuring the effectiveness improvements between users in the control group and users in the experiment group.

[0076] A programmable meta-rules-based execution flow control template and/or experiment control object may support flexible specification of execution flow for different applications and services.

[0077] An execution flow template may include one or more meta-rules specifying default and application specific functions. Such functions may include, for example, functions to create and allocate an AEP execution engine from the event processor 204 for each controlled experiment; to receive events and to dispatch events to an AEP execution engine; to collect a triggered AEP execution trace; to select and insert users into control or experiment groups and/or delete users from a control group when user behavior and state changes to a different metric group and/or state; to collect effectiveness feedback measurements by calling an external service and/or calculate effectiveness from a set of real-time behavior indicators derived from a profiler; to analyze the correlation between effectiveness measurements and actions, to adjust the AEP configurations to optimize the effectiveness measurements iteratively; and/or to report and apply desired (e.g., optimal) AEP configurations for each effective action.

[0078] An Experiment Control Object (ECO) is an object model which may include interfaces and data structures. An ECO may be dynamically created and updated to support integration and data sharing among processes implementing multiple dynamically created controlled experiments. An ECO may, for example, support one or more of the following: dynamic creation of a persistent object sharing interface for profiling, synthesis, analysis and/or AEP execution processes (or threads and/or tasks) to access metadata, metarules, AEP execution traces, metric groups, and/or states which may be assigned to specific controlled experiments; AEP engine allocation, execution, monitoring and/or action output control for an experiment group or control group; and/or metadata for controlled experiments. Such metadata for controlled experiments may include, for example, a controlled experiment identification (ID) and/or access token; a control group user list, a triggered action and/or AEP; an experiment group user list; an AEP execution trace; an AEP execution trace pattern matching a function entry point for each controlled experiment; input sources and output destinations; and/or external and/or internal specification of specific service entry points. Such external and/or internal specification of specific service entry points may include, for example, secure access control tokens; user groups, roles, names and IDs; and/or company and organization names and IDs.

[0079] An AEP management service architecture and/or function components may be implemented, for example, in order to realize various AEP effectiveness improvement devices, systems, methods, and processes, such as those described herein. An AEP management architecture may include an AEP service controller that controls profiling, synthesis and analysis processes and/or AEP execution engines.

[0080] FIG. 3 is a block diagram illustrating an example service architecture 300 for AEP effectiveness management. As shown in FIG. 3, an example AEP service controller 312 may include interfaces to interact with concurrent running AEPs serving multiple different types of applications and services. Such interfaces may require minimal modification and development efforts. The interaction interfaces may be supported by a meta-rules template and/or experiment control object, for example as described herein. The AEP service controller 312 may run as an autonomous service, may execute metarules defined in execution flow templates and may create instances of experiment control objects to support the interactions with AEPs and other components. Example processing steps are illustrated in FIG. 3. Typical AEP event processing steps are labeled using numbers from 1 to 6. Steps for synthesis and analysis processing involved in effectiveness management are labeled using letters ai, a2, b, c, d, and e. The service controller 312, behavior profiler 322, AEP engine 324, and/or effectiveness manager 326 may access a data repository 332 concurrently (e.g., 1', 2', 3', 4', 5') via a bus 334.

[0081] The proposed AEP service controller 312 may interact with a profiler, such as shown in FIG. 3, to track evolving user and device behaviors. The profiler 322 may analyze and categorize user and/or device behaviors into metric groups with respect to different system states (e.g., application contexts and environmental conditions). The service controller 312 may interact with an effectiveness experiment controller 314 to obtain metadata required to associate selected users and AEP execution engines 324 with different AEP threshold parameter configurations in each round of a controlled experiment.

[0082] Through an effectiveness experiment controller 314, the AEP service controller 312 may also interact with an AEP effectiveness manager 326 that may collect, record, and analyze effectiveness measurements of any or all of the triggered AEP execution traces from AEP execution engines 324. Each controlled experiment may include user lists for control groups (CTRUserList) and experiment groups (EXPUserList), respectively. The users may be selected dynamically, on a real-time basis, by matching AEP execution traces from users and devices with the same metric group under the same state (e.g., at the same time). As behavior and state changes over time, so may the placement of a user (i.e., event provider 202) in a new or different group. For example, a user may be removed from a current group and placed in a different group based on new situational parameters of the user.

[0083] The experiment controller 314 may also perform an AEP execution trace matching operation to ensure that only users with the same input event patterns as the experiment group are selected for the control group. Once an AEP has triggered an action, the corresponding user may be selected into a designated experiment group. The AEP of a next or later triggered action may be compared with a set of one or more previous triggered AEPs for different controlled experiments. If a triggered AEP matches any AEP specified in an experiment group, the corresponding user or users may be selected and grouped into the control group of the controlled experiment and an action may be disabled from being sent to the user. This additional dynamic AEP matching function may support tighter control of variables involved in each controlled experiment with each metric group and state.

[0084] Effectiveness measurements associated with the actions collected from controlled experiments may be calculated by application specific functions. The AEP effectiveness service controller 312 may provide an interface for an external system 340 to provide measurements for each round of actions sent to the external system 340, which includes apps, services, Software as a Service (SaaS), user devices, web services, Platform as a Service (PaaS), and the like. These measurements may be recorded in the data repository 332. Alternatively or additionally, the effectiveness measurement may be obtained from the event parameters collected by the profiler 322. The profiler 322 may analyze, classify and/or rank the event parameters to produce high level behavior indicators. These behavior indicators may be stored in the data repository 332.

[0085] The effectiveness manager 326 may classify triggered actions that produce better improvements to effectiveness measurements than a control group (which may have had the action disabled by the service controller) as effective actions. The effectiveness manager 326 may also adjust AEP threshold configurations to find a configuration that may result in maximal effectiveness measurements or which is most highly correlated with the effectiveness measurements. Such AEP configurations may be referred to as optimal AEPs.

[0086] The AEP management service 300 may provide an interface for defining application specific methods to obtain the effectiveness measurements given a set of actions. An example, effectiveness measurement and ranking methods are described as follows. Given a set of triggered actions and effectiveness measurements from multiple rounds of experiments, the analysis process may calculate the correlations between the number of the actions triggered for each action type and the effectiveness measurement. For example, Table 1 shows action and effectiveness measurement lists for multiple controlled experiments. Table 1 lists a combination of numbers of triggered actions (Al, A2, and A3) and corresponding effectiveness measurements (El, and E2) expected to be collected from a controlled experiment. It may be assumed that multiple experiments are conducted in parallel. Each experiment may control the selection of users in such a way that a particular number of actions (five (5) in this example) triggered for a control group user will have the same event pattern as in an experiment group; however, the actions will be only applied to the experiment group. Effectiveness measurements may thus be based on improvements in effectiveness measurements between the experiment group and the control group. Values of effectiveness measurement improvement for each round may be provided by applications. Values for actions may be defined based on a cost of the action normalized from 1 to 10. The correlations may be calculated incrementally for each round.

Table 1

[0087] Multiple variations of the action and effectiveness control template and threshold adjustment process may be flexibly defined based on application requirements. For example, to observe the effectiveness of one action, an action and effectiveness template table may contain one action and multiple effectiveness measurements as shown, for example, in rows 1 to 3 of Table 1. Combinations of action effectiveness measurements may be observed and evaluated as shown in rows 4 to 6 of Table 1.

[0088] Where the effectiveness measurements are collected, resulting correlations may be calculated and may be organized as in Table 2, which shows a correlation between actions and multiple effectiveness measurements. As shown in Table 2, Al has the highest ranking for effectiveness measurement E2 as well as weighted sum of E l and E2. A3 has lowest ranking for the weighted sum of E 1 and E2 but is ranked higher than A2 for E l. Accumulative Correlations

Rounds Effectiveness Al A2 A3

Measurements

1-6 E l 0.00 0.14 0.53

E2 0.96 0.26 -0.36

(E l + E2)/2

0.73 0.29 0.09

7- 13 E l

E2

(E l + E2)/2

14-20

Table 2

[0089] Using an action and effectiveness template table (e.g., Table 1), the sensitivity of threshold parameters of each AEP that triggered effective actions can be adjusted without changing other AEPs to detect the impact of the AEP on the action effectiveness. For example, a simple AEP, AEP_ID, with one threshold parameter, G, for behavior indicator, g, and action, A can be expressed as follow:

AEP_ID : If (g > G), then do (Ai).

[0090] In this example, G is a threshold parameter which may be configured with different values in different rounds of a controlled experiment. The variable G may be stored as metadata for each AEP. In each round, AEP ID, G, action Ai and a corresponding effectiveness ranking may be calculated and recorded based on application specific effectiveness measurement and ranking functions. A trace of the effectiveness measurement and correlations similar to the example entries in Table 1 and Table 2 may be recorded for rounds 7- 13 for different values of G while maintaining the same AEP configurations for the remaining actions (A2 and A3). An optimal setting G_max for behavior indicator g may be identified based on values of effectiveness measurements and/or correlation with the measurements.

[0091] Effective AEP configurations which are identified from the controlled experiments may be recorded and applied dynamically on a realtime basis to AEPs for other users and devices in the same metric group in different states. This means that where behavior evolves and state changes dynamically, the AEP may be automatically adjusted on a real-time basis without changing the AEP logic in the target event processing applications and services.

[0092] An AEP management service function architecture 300 may include various function components as shown in FIG. 3, such as an effective AEP service controller 312, behavior profiler 322, data repository 332, AEP execution engine 324, AEP effectiveness manager 326, and/or service control and effectiveness management dashboard 310.

[0093] An effective AEP service controller 312 may be configured to provide one or more of the following functions: create and assign an ECO for each controlled experiment; configure controlled experiment groups and select devices, users, and applications for conducting controlled experiments; adjust threshold levels and control triggered AEP action outputs for each control group (e.g., disable specific actions for a specific set of users in the control group); allocate AEP execution engines to process events from users, applications, and/or services with privacy isolation and security protection for concurrently executing controlled experiments and normal event processing; initiate controlled experiments and dispatch events defined in AEP-to-AEP execution engines; receive and save triggered actions and triggered AEPs in persistent data repositories; label triggered AEPs with metric group IDs; collect and save effectiveness measurements and AEP execution traces for effectiveness analysis processes (e.g., effectiveness correlation and AEP ranking, input event matching, etc.); and report and apply the effective AEPs to normal AEP engines for all users, devices, applications and services. [0094] A behavior profiler 322 may be configured to provide one or more functions.

[0095] For example, the behavior profiler 322 may collect selected subsets of events from the AEP service controller 312 for each controlled experiment. Each subset of events may include all actions and effectiveness measurements defined for the controlled experiment. Table 1 and Table 2 provide example names for actions that may be used to identify corresponding AEPs and input events for each round of controlled experiments.

[0096] The behavior profiler 322 may learn behavior pattern metrics and build behavior metric groups for different types of users, application contexts, and environmental conditions.

[0097] The behavior profiler 322 may record metric groups in persistent repository accessible by other function components.

[0098] The behavior profiler 322 may determine metric groups to which each device and user belongs based on real-time behavior events.

[0099] The behavior profiler 322 may dynamically transition users from one metric group to another on a real-time bases based on user and/or device behavior changes.

[0100] The behavior profiler 322 may dynamically transition users and/or devices to a metric group of a new state (e.g., a specific combination of application context and environment condition) on a real-time basis on a condition that the state changes.

[0101] The behavior profiler 322 may collect event parameters and calculate behavior indicators using application specific functions provided by a default weighted average function or application service interface entry points provided by external services. The behavior profiler 322 may apply the behavior indicators as real-time leading indicators for effectiveness measurements which cannot be obtained in real-time. The behavior profiler 322 may adjust and optimize AEP iteratively based on correlation with the behavior indicator. [0102] The behavior profiler 322 may adjust the weight of the event parameter in the weighted sum to optimize the correlation between the behavior indicator and the effectiveness measurement.

[0103] The behavior profiler 322 may calculate correlations between the behavior indicators and effectiveness measurements provided by external systems (where these measurements become available).

[0104] A data repository 332 may store all events, actions, effectiveness indicators, and behavior metrics. It may also store behavior metric groups, and states. The ECOs for each controlled experiment are also stored in the data repository 332. The ECOs provide access control to the rest of the data stored in the data repository 332. Examples of data models for each defined interface in FIG. 3 are listed in Table 3 which shows a high level data model for effectiveness management.

Table 3

[0105] Example behavior metrics may include engagement level (e.g., including attributes for usage frequency, time and number of user interactions per second, etc.); competence level (e.g., including attributes such as time to complete a task, number of solved tasks, number of points scored, etc.); accumulative usage time; and/or demographic data.

[0106] Example application contexts may include: application type, ID, application task, and other users; and/or game type, ID, level, tasks, and other players.

[0107] Example environmental conditions may include device or sensor types; network type (e.g., mobile, WiFi), cost, bandwidth and response time; and/or location at different application context and environment conditions, etc.

[0108] The data repository may also record configuration metadata descriptors to support dynamic binding and invocation of default and application specific services for profiling functions, AEP output actions, and/or effectiveness ranking functions Configuration metadata descriptors may include one of more of the following: metric group descriptors, AEP descriptors, and effectiveness management descriptors.

[0109] Metric group descriptors may include, for example, Group name,

ID, Parameter table, ClassifierlD (Classifier service entry points for default and application specific profiling function invocations).

[0110] AEP descriptors may include, for example, action (Name, ID,

Action service entry point); and/or threshold (Name, ID, value, type (constant or % of STD relative to average)).

[0111] Effectiveness management descriptors may include, for example, effectiveness measurement model (e.g., model name, service entry point); and/or correlation method (e.g., model name, service entry points for invoking default and application specific correlation service functions); AEP effectiveness rank (e.g., model name, ID, score (e.g., % against the model); and/or service entry points for invoking default and application specific distribution and rank scoring service functions); AEP execution trace matching (e.g., control group ID, experiment group ID; AEP execution trace; input event filter service entry point and filter configuration; AEP matching function service entry point; and default AEP matching function).

[0112] An AEP execution engine 324 may execute AEPs defined by different applications in target event processing systems. An AEP execution engine 324 may generate actions based on AEP configuration metadata specified in the AEP configuration templates for each application. The AEP execution engine 324 may be configured to perform one or more of the following functions.

[0113] The AEP execution engine 324 may be configured to receive events including user ID and application context from the AEP service controller 312 or from other sources (e.g., logs, streams, ports, etc.). For a tightly integrated AEP, the metadata (e.g., metric group IDs, state IDs, and/or controlled experiment IDs) may be part of an ECO passed to the AEP execution engine 324. The AEP execution engine 324 may access the data repository 332 through the ECO directly.

[0114] The AEP execution engine 324 may be configured to process the events against (i.e., compare the events with) AEPs and record execution traces of the AEPs to corresponding controlled experiments via the ECO. The AEP execution engine 324 may also record sources of the events and output destinations for each action.

[0115] The AEP execution engine 324 may be configured to output the actions triggered during the experiments to the AEP service controller 312. The AEP execution engine 324 may be configured to filter actions based on an effectiveness management template, which may mainly include actions for improving the performance, engagement, retention, revenue, efficiency, and/or other effectiveness indicators. The AEP execution engine 324 may be configured to send actions to external application services. If the user belongs to a control group, the action may be disabled or deferred until the end of the experiment by the AEP service controller 312. The AEP execution engine 324 may be configured to log all the actions for each user and device in the data repository 332 by the AEP service controller or directly through the ECO.

[0116] The AEP execution engine 324 may be configured record the AEP execution traces in the data repository 332 via the ECO.

[0117] An AEP effectiveness manager 326 may analyze remedial actions against effectiveness indicators defined for each application. Discussed further herein with respect to an AEP effectiveness manager 326 are interface and function components; data models and workflow for effectiveness management processes, methods, devices, and systems for identifying effective actions and AEP optimizations.

[0118] FIG. 4 illustrates an example of a system 400 having an AEP effectiveness manager 426, including interfaces to service controller 412, behavior profiler 422, actionable pattern execution engine 424, and data repositories 432, and function components, including a pattern sensitivity analysis module 427 and an action cost effectiveness analysis module 428. As shown in FIG. 4, a detailed example of pattern and action effectiveness management service process may involve interaction among one or more of various function components, which may include, for example, the action effectiveness analysis module 428; and/or an AEP sensitivity analysis and adjustment module 427.

[0119] An action effectiveness analysis module 428 may include, for example, functionality, processing and/or circuitry for selecting a pair of experiment and control groups and initiating a synthesis process to generate multiple rounds of controlled experiments and record the actions and effectiveness measurements from different sources. In this example, such different sources may include effectiveness measurements from external systems and/or effectiveness indicators from behavior patterns collected by the profile manager 412.

[0120] Effectiveness measurements may be collected from external systems, and may include for example user interest, comfort, and satisfaction levels; and/or specific business indicators such as revenues and risks. Effectiveness indicators may be collected from the behavior patterns collected by the profile manager 412, and may include, for example, revenue per week, month, and year; engagement; performance; competence; customer retention; efficiency; and/or response time or task completion time.

[0121] An action effectiveness analysis module 428 may also include, for example, functionality, processing and/or circuitry to calculate a correlation between triggered actions and effectiveness measurements. It is noted that such functionality, processing and/or circuitry may include any suitable computer, computer processor, central processing unit (CPU), nontransitory computer readable media, and/or communication devices and the like, as will be readily understood by those in the art. The specific measurements and correlation functions may be defined by business users such as application developers or service providers. For example, for customer service applications the correlation function may be defined as follows to assign cost function to different types of effectiveness measurements, such as, to calculate and determine cost benefit on percentage of application usage (e.g., percentage of users continuing to use the service or application after the first, second, and third, and fourth K-th month periods (e.g., weeks or months)). Additionally or alternatively, customer spending or revenue improvement may be calculated and determined (e.g., percentage of improvement on revenue increment or decrement trend). Additionally or alternatively, improvement in energy management efficiency of heating and air conditioning may be calculated and determined. Additionally or alternatively, improvement in comfort level, response time or task completion time may be calculated and determined.

[0122] The action effectiveness analysis module 428 may assign cost of actions (e.g., assign each action with an operation and/or delivery cost). For example, a tutorial or a bonus reward may have different costs. [0123] The action effectiveness analysis module 428 may calculate a correlation between cost effectiveness improvement and the cost of remedial actions. For example, some actions may have higher correlation with cost effectiveness improvement measurements than others. AEP optimization processes may improve the action effectiveness by adjusting a threshold value to reduce the number of actions triggered during a time interval or a frequency of occurrence of the actions. Correlation between AEP configuration and the effectiveness measurements can also be used to adjust the AEP threshold parameters.

[0124] An action effectiveness analysis module 428 may also include, for example, functionality, processing and/or circuitry to define Approximated False Positive (AFP) and Approximated False Negative (AFN) measurement functions to adjust the AEP thresholds. For example, an AFP Count may be defined as the number of ineffective actions triggered by the groups of AEPs. Ineffective actions may be those actions that have low correlations to effectiveness improvement measurement based on previous experiments (i.e., tests rounds). AFN Count may be defined as the number by which effective actions are less than an expected number of effective actions as observed from the previous experiments of same metric group.

[0125] An action effectiveness analysis module 428 may also include, for example, functionality, processing and/or circuitry to record analysis results in a data repository 432 using an ECO and to pass the results to the AEP sensitivity analysis and adjustment module 427. Such analysis results may include: ineffective actions; effective actions and associated AFP and AFN counts; effectiveness improvement measurements; threshold parameters settings; and/or metric groups for each control and experiment groups.

[0126] An AEP sensitivity analysis and adjustment module 427 may include, for example, functionality, processing and/or circuitry for receiving action effectiveness analysis outputs from the action effectiveness analysis module 428 and for executing meta-rules based on user commands or automated processes to control the adjustment of action control threshold parameters in the AEP execution engine 424. The meta-rules running in the analysis module may track, adjust, and/or update the threshold parameters that control the generation of actions (the detailed flow is described with respect to the data model and workflow herein). Meta-rules may include several types. Example types of meta-rules may include meta-rules to reduce AFP and AFN actions, and meta-rules to optimize the effectiveness measurement and/or correlation with the effectiveness measurements.

[0127] Meta-rules to reduce AFP and AFN actions may filter parameter thresholds based on the effective actions provided by the action analysis module; may adjust the threshold parameters for each round of experiments (see e.g., FIG. 7) for all effective actions using the AFP and AFN; and/or may disable actions for a preset period of time to reduce the AFP for the same user.

[0128] Meta-rules to optimize the effectiveness measurements and/or correlations with the effectiveness measurements may (e.g., to obtain finer degrees of control) be used to select most-effective threshold parameters from a set of effective threshold settings obtained from previous controlled experiments and iterative steps may be used to adjust the threshold parameters and observe (i.e., monitor) the effectiveness of each threshold parameter (see e.g., FIG. 9).

[0129] For example, such adjustment and monitoring may support non- linearly correlated parameter thresholds and action effectiveness optimization. For example, in many applications the effectiveness of actions or adjustment of AEP for an action may be influenced by many factors and may be non-linear. The optimization process may control effectiveness analysis via controlled metric groups and states and/or matching AEP execution traces between the experiments and control groups.

[0130] Additionally or alternatively, such adjustment and monitoring may provide finer degrees of action control threshold tailored for each user within each metric group. For example, the impact of an online help tutorial or a marketing advertisement action on a user may be observed from performance or engagement responses of individual users. The system may reduce a threshold for the effective actions until there are no observed cost effectiveness improvements.

[0131] FIG. 5 illustrates example logic data models and logic workflow for AEP effectiveness experiments. Modules (e.g., data repository, behavior profiler, experiment controller, AEP engine, AEP effectiveness manager and ECO meta data repository) referenced in FIG. 5 are similar to like named modules previously described herein. The effective action pattern adjustment process, as shown in FIG. 5, may receive a sequence of events 501 and trigger a set of actions 502 in real-time. AEPs may be executed 503 such that the triggered AEPs are matched and applied to corresponding users in experiment and control groups. The triggered actions over a time period may be modeled as a round of experiments which include a pair of control and experiment sets of AEP configurations predicated on two selected sets of users. The users in both the experiment group and control group may receive the same (or a similar) event pattern and may trigger the same set of rules. The actions however are only applied to users in the experiment group. The users in the control group may not receive any actions. Effectiveness measurements for one or more rounds may be collected 504 based on the triggered event patterns and/or actions applied to the experiment and control groups. The effectiveness measurement may then be output for analysis to the AEP effectiveness manager. A behavior profiler may collect event parameters and calculate behavior indicators 505. Dynamic context aware behavior profiling 505 may include classifying evolving user and/or device behaviors on a real-time basis into multi-dimensional metric groups using application defined behavior indicators. User lists for control groups (CTRUserList) and experiment groups (EXPUserList) may be defined 506. AEP execution trace matching 506 may be performed to ensure that only users with the same input event patterns as the experiment group are selected for the control group. Once an action is triggered, corresponding users may be selected into designated experiment and control groups (operation 506). The behavior indicators for mapped to the controlled experiment may be collected and provided for effective analysis and parameter adjustment 507. The example shown in FIG. 5 illustrates a method for modeling controlled experiments and adjusting the threshold based on false positive and effectiveness measurements in more detail.

[0132] An AEP set may include j number of adjustable parameters, pij, each having a threshold Pij. A threshold adjustable rule set may be modeled as, AEPi(Pii, Pij, Pij), where i = 1 to I and j = 1 to J. A set of fired remedial actions is denoted as, AEPi (Pki, Pkw, · · · , Pkw) -> At, where k = 1 to K. It is noted that {Pkw} is a subset of the set {Pij} that were evaluated as true and causing the firing of Ak. An effectiveness improvement measurement E = {Ei, E2, E_e, ...} may be application- defined. E may have multiple dimensions (as described in Table 1) each having different aspects of effectiveness (e.g., cost, efficiency, response time, satisfaction, etc .). As an example, one of the aspects of E (see e.g., Table 2) may include the weighted sum of effectiveness measurements for a set of actions, {Ak} (e.g., E =∑W_e (E_e

[0133] The action effectiveness measurement analysis module 428, or simply the effectiveness analysis module, may perform may check whether the action is effective or ineffective 508 based on the effectiveness measurements and/or a correlation with the effectiveness measurement. The action effectiveness measurement analysis module 428 may calculate effectiveness from a set of real-time behavior indicators received from the profiler. The AEP sensitivity analysis module 427 may adjust the threshold parameter {Pkw} 509 to determine whether the specific parameter threshold configuration in the AEP will affect the effectiveness measurements. If a configuration does not change the number of false positive AFP or negative AFN of actions defined over {E_p E_p2, E_p3, E_p4, ...} and { Ε_ρι', Ε_ρ2', Ε_ρ3', E_p4', ....}, then, the effectiveness measurement function is not sensitive to the parameter threshold, Pkw of an effective action Ak. If Pkw and associated Ak improves the effectiveness measurement function defined over E and E', then these parameters may be effective.

[0134] The effectiveness measurement analysis 508 performed by the action effectiveness measurement module 428 may include a sum of multiple effectiveness measurements for an action, Ak, for each threshold parameters, E(A ) I p=∑W_e (E_e ({Ak}) - E'e ({Ak}), for each parameter controllable threshold parameter, p, where e= {el, e2,..,} represents multiple effectiveness measurements, E and E' represent the effective measurements obtained from the experimental and control groups respectively, p = {p i, P2, P3, ···} for different threshold parameters, and Ak, is the triggered actions. If E(Ak) I _p < 0, Ak is ineffective actions; otherwise, calculate correlation^, Ak) to support sensitivity adjustment (e.g., Ai is effective, A2 is not). The effectiveness measurement may be implemented using rules that can be dynamically defined and updated to meet diverse application requirements.

[0135] The AEP sensitivity adjustment 509 performed by the AEP sensitivity adjustment module 427 may use E and Correlation^, Ak) to adjust p i and P2 to reduce the numbers of AFPs and AFNs for each metric group in the controlled experiments, use E_e({A k}) to adjust p i and P2 for each metric group in the controlled experiments, and set the threshold setting for all users in each metric group. The results of the AEP sensitivity analysis 509 may be provided back to the behavior profiler for optimizing the AEP iteratively based on correlation with the behavior indicator. FIG. 6 is a flow chart illustrating an example action accuracy adjustment process implemented in the AEP effectiveness manager 326 or 426 shown in FIGS. 3 and 4.

[0136] In operation 601, std and avg of user behavior event pattern parameters gij are calculated. For each round Ri, set threshold (Gij) = Sij * (std(gij )) + avg(gij), where the default coefficient, Sij, is set to Sij = 50%. Here, the initial round of experiments for multiple metric groups are conducted. [0137] In operation 602, action analysis for each round is performed by: collecting effectiveness measurement, E_g for behavior parameters g and associated threshold value, Gkw (k denotes the action label for Ak and w denotes the metric group label), to be adjusted in multiple rounds, and action, Ak for action k, to find correlation C_gk(E_g, Ak(Gkw) between E_g and Ak from multiple controlled experiment rounds for each Ak and each metric group, w.

[0138] In operation 603, pattern sensitivity analysis is performed for each round by: saving AFP and AFN, calculating new AFP and AFN, and adjust thresholds for fired actions. LC and HC are example coefficients to set the lower and higher correlation bounds as an example to determine the false positive and negative actions. Here, sensitivity is increased, if the action count of Ak is less than the average counts of previous experiments that produced good correlation with the effectiveness measurements.

[0139] In operation 604, it is determined whether (new AFP | AFN < =

AFP I AFN) or AFP < Max_AFP or AFN < Max_AFN.

[0140] In operation 605, if the conditions are met in operation 640, parameter adjustment is stopped and the correlation, AFP/AFN, and thresholds Gij are reported.

[0141] Accordingly, FIG. 6 illustrates an example method for improving the accuracy of effective actions. The example action effectiveness management analysis process, as shown in FIG. 6, performed by the AEP effectiveness manager 426 may include multiple rounds of experiments conducted in parallel for selected subsets of users/devices in multiple metric groups. The users in the control group may not receive any actions. The effectiveness measurements may be given by external systems indirectly or may be provided by a set of behavior indicators or performance indicators detected by the profiler (e.g., profiler 322 or 422). After a period of time required to collect a sufficient number of samples to determine the effectiveness of the actions, the action effectiveness analysis module 428 may generate effective action and ineffective action lists based on a correlation between the actions and effectiveness improvement. The action effectiveness analysis module 428 may also generate AFP and AFN action counts based on predefined criteria. Where there are too many AFPs, the pattern sensitivity analysis module 427 may increase the threshold for parameters that caused the firing of the effective actions.

[0142] An example of false negative action count estimation may include increasing parameter threshold sensitivity based on approximated average occurrence rate of each effective action type and/or calculate a false negative in each round.

[0143] For example, the average occurrence rate of each effective action type may be determined by calculating average occurrence rate from previous experiments. The average occurrence rate from previous experiments may be calculated by generating multiple controlled experiments, one for each different type of action on selected users in the different metric groups, and counting the number of effective actions from all actions Ai to A5 in the experiment groups that have shown improving effectiveness measurements. In an example, the actions that are correlated to the improvement of effectiveness measurements, are Ai, A3 and A5 with the average count from all the effective experiments, AvgCount(Ai) = 23, AvgCount(A3) = 30, AvgCount(A5) = 32, assuming A2, A₄ are ineffective. Using the effective action counts, the average of counts over the accumulative usage time may be calculated and the average count for each different action type may be saved (i.e., stored in memory).

[0144] Additionally or alternatively, the average occurrence rate of each effective action type may be determined by calculating the false negative using the following steps in each experiment round: generate experiments using an initial threshold setting condition; collect an occurrence count of each action in the same metric group, state and AEP execution trace; and, if the occurrence count of an action type is less than the average counts from the previous experiments (e.g., AvgCount (Ai) > Count(Ai)), reduce the sensitivity of the threshold of AEP i for the action Ai.

[0145] Another example of false negative action count estimation may include discovering new AEP pattern parameters systematically by: inserting a new AEP with new behavior indicators p derived from the profiler 322 or 422; selecting a new action Ak from an action list designed for the AEP; conducting experiments for all metric groups using the action effectiveness measurement function E_e to find a correlation between E_e and Ak; adjusting the threshold level for indicator p if it is effective; and propagating the threshold parameter to the live service AEP.

[0146] The algorithm shown in FIG. 6 may iterate for multiple rounds for all metric groups and states in parallel. When increasing sensitivity, the AFP may increase and the AFN may decrease. The adjustment may stop where both AFP and AFN are both stable (i.e., do not change from round to round) or become less than a preset maximal AFP and AFN. This automated process may be applied to different metric group sizes. In some implementations, a small percentage, δ (e.g., 5% - 20%) of standard deviation may be adjusted to control the probability of triggered AEP counts for each iteration.

[0147] Automatic AEP effectiveness optimization may be performed using metric indicators. Methods for optimizing the effectiveness of AEP may include using behavior metric indicators for effectiveness measurement directly. This direct effectiveness measurement may enable automated effectiveness analysis without delays that may be incurred in obtaining external effectiveness measurements (e.g., weekly user retention rate, customer satisfaction survey or other revenue measures). For example, behavior metric indicators such as engagement and competence may have a strong correlation with cost effectiveness of the services. In this case, if an action is effective for the engagement or competence levels, it is likely to be effective for the cost effectiveness of the service itself. Such correlation between the behavior indicators and cost effectiveness of the services may be analyzed prior to the optimization process.

[0148] Using a set of correlated (e.g., highly correlated) behavior metric indicators, the AEP service controller (e.g., AEP service controller 312 or 412) may dispatch the events to profiler 322 or 422 to generate metric indicators for each controlled experiment and may adjust the threshold parameters in realtime to find an optimal threshold value that maximizes the behavior indicators for each metric group.

[0149] FIG. 7 is a diagram illustrating an example process for selecting parameter settings based on past experiments. As shown in FIG. 7, the effectiveness cost measurement function, E(Pij) with respect to the threshold Pij of a behavior indicator g derived from the profiler 322 or 422 may be calculated directly by the effectiveness management system. Examples of behavior indicators include engagement or competence levels. The metric group for different accumulative usage times may be denoted as pij, p2j, p¾j, and p4j, and each may have multiple parameter settings, which may be modeled as threshold parameter vectors such as P21, P22, P23, P24. At each round of experiments for each behavior pattern parameter, locally optimized effectiveness measurements with corresponding behavior indicator thresholds (e.g., competence, engagement, and other parameters), {E (Pij) _max, Pij} may be recorded and used to set the parameter thresholds in similar metric groups. It is noted that threshold Pij for p^ may be non-linearly correlated with E. A curve fitting method may be used to find a value of Pij that maximizes E(Pij) using the finite number of the samples E(Pij).

[0150] FIG. 8 is a flow chart illustrating an example action pattern parameter partial optimization process implemented in the AEP effectiveness manager 326 or 426 shown in FIGS. 3 and 4.

[0151] In operation 801, a profile manager (profiler 322 or 422) calculates std and avg of gij for all users in each metric group: generate H metric groups (e.g., H > 5) each has more than N users (e.g., N > 10); select users to join the controlled experiments with same on states and AEP trace; map threshold detection AEP group for each experiment and control group; and calculate effective thresholds, Gij= {j5 std(gij) + avg(gij)} for δ = 0.1, j = 1 to 10. δ and j are adjustable to meet different application requirements.

[0152] In operation 802, a pattern threshold sensitivity analysis module conducts partial differential parameter adjustment: assign different Gij for to threshold detection rules for experimental groups; and start multiple controlled experiment in multiple rounds in parallel.

[0153] In operation 803, an action cost effectiveness analysis module conducts the experiments for selected users in all the metric groups, including: provide effectives an action list to pattern analysis module; collects {gij, Gij}, and calculates effectiveness measurement E(Gij) for all experiments; and identifies effective actions.

[0154] In operation 804, it is determined whether a threshold with dE

(Gij ) / dGij ~ 0 is found.

[0155] In operation 805, in an appropriate threshold is found, {Metric group, E (Gij )max , Gij , Action} is recorded in the data repository; and the effective threshold Gij is used for the rest of the users.

[0156] Accordingly, FIG. 8 illustrates a detailed description of an example iterative threshold adjustment process. The threshold adjustment process for each metric group may run multiple times based on the number of available users to support the controlled experiments. The process may terminate when a threshold which maximizes the effectiveness measurement is obtained from the behavior indicators. The above process may run incrementally in a moving window of time. The process monitors moving average of threshold parameter changes and tracks the effectiveness measurement changes for multiple controlled experiment groups. If a better threshold condition is discovered for a metric group, the new threshold may be applied to other users in the same metric group. [0157] By running the action analysis and AEP parameter optimization iteratively for different metric groups, the effectiveness manager (e.g., effectiveness manager 326 or 426) may adapt to evolving user and device behaviors, application contexts, and environment conditions. The optimal threshold settings and the corresponding AEP learned from early experiments on early adopters are applied to the new users.

[0158] The system may support adjustment of weight for multiple levels of parameter from a set of event parameters, to behavior indicators to effectiveness measurements for based on correlations with the effectiveness measurements from iterative execution of controlled experiments described in the proposed effectiveness management service. Correlations of each levels of behavior metric with different aspects of the effectiveness measurement dimensions can be calculated using default or application specific methods.

[0159] A service control and effectiveness management dashboard 310 may provide a user interface to any or all metric groups and statistics of AEP parameters used to control the experiments. The dashboard 310 may provide summary views of effectiveness improvements. The dashboard 310 may also allow users to provide high level user feedback on the effectiveness measurement. For example, the dashboard 310 may allow users to track the distribution of any or all of the following data, or other data: actionable event pattern name, controlled experiments, metric group, effectiveness improvement indicators, action accuracy and effectiveness, and/or threshold setting and feedback.

[0160] Data for the actionable event pattern name may include one of more of the following: a parameter name and threshold value of the action event pattern and/or action name and action description.

[0161] Data for the controlled experiments may include one or more of the following: control group name, experiment group name, user list, device list, metric group ID, action effectiveness table (e.g., as shown in Table 1), and/or action effectiveness correlation table (e.g., as shown in Table 2). [0162] Data for metric group may include one or more of the following: metric group ID, group name, metric descriptor and parameter value ranges, and/or score and distribution mode description.

[0163] Data for effectiveness improvement indicators may include one or more of the following: action cost and improvement benefit models and descriptions, cost effectiveness of control group, and/or cost effectiveness of experimental group.

[0164] Data for action accuracy and effectiveness may include one or more of the following: action ID, action description, AFP/AFN, cost effectiveness measure, current pattern threshold settings, and/or maximal effectiveness settings.

[0165] Data for threshold setting and feedback may include one or more of the following: user input for AEP threshold setting, feedback collected on the specific threshold setting (e.g., increase sensitivity or decrease sensitivity), and/or user feedback collected on comfort level or satisfaction level (e.g., level 1-10).

[0166] Applications of integrated AEP processing and effectiveness management processes for emerging IoT, game and location sensitive services are discussed further herein. Such application use cases illustrate ways in which an effectiveness management service may support diverse types of applications and services by defining AEPs with AEP configuration metadata, metric group, and state. An AEP management service may automatically conduct synthesis and analysis methods and processes to improve the effectiveness of the AEP. The complexity of coding and optimizing AEP with large combinations of application contexts and environment conditions may be significantly reduced.

[0167] Integrated AEP processing and/or AEP effectiveness management services may be implemented, for example, for context aware home or office energy management. To support the integrated effectiveness management of AEP processing in an IoT environment, one or more of the following service interfaces may be implemented.

[0168] Sensor application control parameter adjustment interface may enable the service controller to invoke remedial actions to the target thermostat devices or thermostat control applications to adjust the temperature settings. The sensor application control parameter adjustment interface may also provide an interface to energy consumption measures and occupancy sensors.

[0169] Metric group and state parameter definition template configuration interface: Table 4 shows an example metric group template including parameter and value groups for an example temperature sensor network located in a community of homes or offices buildings.

[0170] Metric group and state access interface: may support, for example, Create, Read, Update, and Delete operations on metric groups to facilitate the behavior profiler, AEP processing engine, and effectiveness management in conducting context aware controlled experiments for devices or users at different times and/or with different usage patterns.

[0171] Effectiveness measurement interface: may enable sensor applications or devices designer or administrators to define and/or configure the effectiveness measurement indicators and cost functions associated with the indicators. For example, the sensor application may provide a power consumption measurement interface to the service. The effectiveness of temperature control may be measured by the energy efficiency improvements between a set of controlled experiments.

[0172] Effectiveness management dashboard: As described herein, the dashboard may provide a summary and user feedback interface for threshold setting and/or effectiveness feedback.

[0173] As Table 4 shows sensor parameters and context variables, Table

4 may include a plurality (e.g., hundreds or thousands) of metric groups which may represent the preferences of individual users. For example, some senior citizens may prefer higher interior temperature than young students. Different utility budgets may also decide different comfort levels for different outside temperature in the winter or summer. Where a user is away for work or on vacation, desired temperature settings may be different from that of normal occupancy. Such differences may be categorized and/or recorded by the behavior profiler as metric groups and states.

[0174] Given the potentially large number of parameters for each metric group, it may be comphcated to observe and/or fine tune thermostats to achieve an optimal thermostat setting manually. The effectiveness management system service may maximize the effectiveness of energy consumption and comfort levels based on efficiency behavior analysis of the devices and energy consumption for different age groups. For example, by analyzing the time required to increase the interior temperature of the house based on the heaters' energy consumption, the effectiveness manager may obtain an energy efficiency rating of the heater and thermo-insulation rating of the house. By comparing the behavior among houses of similar size and outside temperature in the control group and experiment group, the system may provide more effective heating and cooling cycles tailored for personal preferences.

[0175] An example set of temperature control AEPs may be used to illustrate the effectiveness management service. A set of AEPs may be embedded in an intelligent thermostat or application server of the multiple thermostats. The example AEPs may include the following temperature (T) control parameters and thermostat control actions:

If T_e < T_min AND T_min < Ti < Tmax, then do Action 1: turn on heater If T_e > Tmax AND T_min < Ti < Tmax, then do Action2: turn off heater // T_e is the exterior temperature Tiis the interior temperature

If T_e > Tmax & AND Tmin < Ti < T_max, then do Action3: turn on air conditioner.

If T_e < Tmin & AND T_min < Ti <T_max, then do Action4: turn off air conditioner.

For away-state management,

//e.g., turn on air-conditioner 10 minute before arrival.

If TimeOftheDay > End-timeOfAway-State - PreSetTime, then, do

Action5: Turn on air conditioner. If Avg (vacant time) > 90% in period [9am - 5 pm] of weekday, then do Action6: Turn off heater and air-condition air-outlet. //Assume that air- outlet can be adjusted in office environment.

[0176] In traditional thermostats, temperature thresholds {T_mi_n, T_max} are set manually and statically. If the threshold is adjusted automatically using a set of AEP without effectiveness management, the AEP may need to specify a threshold for all the combinations of user preference, application and environment contexts, which may be complex and hard to change. The following use cases illustrate how the integrated AEP processing and effectiveness management service may automate adaptation to user behavior evolution and environmental changes.

[0177] As the season progresses from summer to winter, the exterior temperature distribution of a building may change. Where humidity varies, the comfort level may also be different for the same interior temperature. To minimize energy consumption, temperature threshold settings may be adjusted based on the external temperature as well as multiple parameters such as humidity, age, budget, and other parameters defined in the metric groups. For this purpose, the above example may be modeled according to parameter thresholds and metric groups.

[0178] For example, the parameter thresholds {Tmin, Tmax} may be adjusted to minimize the energy usage, e.g., using the systems, devices, methods and/or processes described herein. Tmin and Tmax are variables which may be obtained dynamically from an ECO for each controlled experiment.

[0179] A metric group may be established by the following set of parameters {Exterior Temperature, budget, age, house size, presence-time- period, vacant period}. For example, {Exterior Temperature: {[35 °F - 40 °F ], Age : [ senior | adult ], Size { small (< 1000 ft²)}, TimeOfTheDay, WeekOfTheYear, Demand-Response-Schedule, comfort level, Humidity, Start - timeOfAway-State End-timeOf Away-State}. [0180] The effectiveness measurement in this case may be the total energy consumption cost combined with comfort level feedback. The comfort level may be measured based on T_avg statistics of {Tmin, Tmax} from high budget users or from the manual feedback provided in the service dashboard. It is noted that actions for turning the air-conditioner on and off are effective actions. The continuous measurement of external and internal temperatures may be the threshold parameters for the AEP to be adjusted using meta-rules as described herein regarding effectiveness analysis.

[0181] The profile manager (i.e., the device and user "behavior profiler"

322 or 422) may use a parameter template to collect parameters from sensor applications. The profile manager 322 or 422 may also analyze and generate metric groups for different exterior temperatures, ages, sizes, and comfort levels for controlled experiments. After the deployment of a large number of smart thermostats and multiple controlled experiments are conducted for each metric group, the effectiveness of the actions may be measured and recorded. By adjusting the thresholds automatically based on previous recorded optimal thresholds for each metric group (e.g., having similar house size and/or age, comfort level and/or budget), the effectiveness management process may adapt to the changing external temperature and inside humidity to maintain a comfortable inside temperature at minimal energy consumption level.

[0182] Furthermore, by observing the time required to cool down the inside temperature for other similar size houses or offices, it may also be possible to determine a threshold for a PreSetTime to turn on the air- conditioner before arriving home or before an office opens. This PreSetTime cooling time may be adjusted based on a best effectiveness measure from the experiment groups.

[0183] The service may also estimate thermal efficiency of a house or office based on the cost measurement and temperature threshold setting and size of the house. For example, based on the average temperature change in an experiment heating cycle, ΔΤ, collected from multiple thermostats, and energy consumption cost increment collected from power meters, the effectiveness improvement may be defined as follows:

Effectiveness: AT_avg/ (AEnergyCost) for each metric group, where, Tmin *~ Tavg

[0184] Using this effectiveness measure for the controlled experiments with multiple metric groups, the effectiveness of individual house/office, AT_avg / (AEnergyCost) may be compared with other houses/office in the same metric groups to evaluate the thermal efficiency of heating or air conditioning systems.

[0185] Integrated AEP processing and/or AEP management services may also be used to improve customer service for mobile apps and games. Integrated AEP management services may be used to improve the effectiveness of player experience and retention services for games. In particular, the effective AEP service control processes discussed herein may be used to improve customer service operations for mobile apps and games using behavior indicators derived from low level input events by the profiler in realtime.

[0186] FIG. 9 is a chart illustrating an example of service usage rate improvement over cumulative usage time. An example model of the effectiveness measurement functions using player retention rate against multiple metric groups defined by accumulative usage time is described herein. Such a model may be used to establish a retention reward function for multiple control groups in the first three months of an application lifecycle. FIG. 9 shows a typical player retention rate plotted against cumulative usage time (or playing time). It may be assumed in this example that average application usage time per day is 30 minutes. In this case, the cumulative usage time for a month may be approximately 15 hours. A unit of 1 day may be used to divide the users into groups based on cumulative usage time. Retention rate improvement may then be calculated between control and experiment groups for each day. For example, by applying a set of actions to the experimental group of users, the retention rates for a group of users may improve by 30%, 32% and 15% for 1st, 2nd, and 5^th cumulative usage days, respectively. In some implementations, the more the user uses the application the longer the user may use the application.

[0187] The improvement in effectiveness of user retention rate may include multiple factors attributed to different actions that were triggered by user behavior event patterns under different application contexts (e.g., different levels of usage and behavior experiences). For example, multiple types of actions such as tutorial, bonus or better choice of difficult levels may be attributed to the improvements in cost effectiveness functions. By monitoring the effectiveness and adjusting threshold parameters for engagement and competence levels, different numbers and types of actions may be triggered which in turn may cause changes in user behavior indicators such as engagement and competence. As users become more engaged or more competent, the users may be moved to other metric groups in which the type the action and AEP threshold will be different from the previous metric group.

[0188] FIG. 10 is a chart illustrating example effectiveness measurement functions. FIG. 10 illustrates an example of a correlation between effectiveness improvement functions and behavior indicators, such as engagement and competence levels, in different metric groups which may be used to illustrate the effectiveness of AEPs for different metric groups. The effectiveness measurement model may be based on improvements on player retention rates, for example, for different users behavior metrics based on different indicators. In the example shown in FIG. 10, the lower curve represents the correlation between the customer retention percentage after 3 months and a behavior indicator (e.g., engagement and competence level). The higher the engagement level the longer the user will continue to use the application. The competence level is positively correlated with the retention percentage, however, it may have a different curvature from the engagement level and may saturate at high competence levels. By increasing the competence level and engagement level by 20%, the corresponding retention rate may be increased for both indicators, but the benefit of increased competence level will diminish. It is noted that each of the behavior pattern indicators may be derived from a selected set of application event parameters using application specific defined functions (e.g., weighted average). Example event parameters include frequency of usage, length of usage, performance, score, win rate, and failure rate.

[0189] Since each user may have different progression patterns, different users with the same usage time may acquire different levels of competence and may demonstrate different engagement levels. The profile manager 322 or 422 may classify these users into corresponding metric groups after each round. As shown in FIG. 10, where a user with low competence level in the corresponding metric group is detected, a set of actions configured to improve the competence level may be employed to conduct the controlled experiments. After a period of time, the most effective actions may be selected and applied to all users in the same metric group and state. The resulting improvements may be validated by calculating the correlation between the selected actions and AEP and the retention percentage.

[0190] Table 5 models an example metric group for a game application.

In Table 5, the groups include behavior indicators such as user engagement, competence levels, and cumulative application (i.e., game) usage (i.e., play) time. The profile manager 322 or 422 may learn the statistical patterns of these behavior parameters and use these patterns to divide the users into metric groups. The profile manager 322 or 422 may also collect and map the behavior metric into engagement levels and competence levels. Based on metric group, the profile manager 322 or 422 may select a subset of users from each group to form multiple pairs of control and experiment groups dynamically in real-time for multiple iterative rounds of controlled experiments. For each round, some of the users in the experiment group may receive actions configured to help improve their engagement or competence levels. If certain user levels exceed the ranges for their current metric group, these users may be grouped into a new metric group and participate in different controlled experiments in future rounds. Different actions such as special recognition or bonus may be more effective to more competent players. Interfaces for selecting a user from the profile database based on metric group range may be available for an administrator to control the number of users in each group and/or the number of control and experiment groups.

Table 5

[0191] The profiler 322 or 422 may track the transition of metric groups as user behavior evolves. For example, [engagement, competence] of user 1 may transition from [0.7, 0.1] [0.7, 0.5] [0.6, 0.8] ^[0.5, 0.9] while that of user 2 may transition from [0.6, 0.2] [0.8, 0.7] [0.9, 0.9] [0.5, 0.9]. As users transition from one metric group to another, the effectiveness management system may track the change and apply an optimal AEP configuration learned from the previous experiments. This optimal threshold setting may be used as an initial configuration for the user in the metric group. In some implementations the effectiveness management system may continue the optimization process and fine tuning of the control thresholds for each individual user and group of users as more effectiveness measurements, action traces, and correlations data are recorded.

[0192] Integrated AEP processing and/or AEP effectiveness management services may be implemented, for example, for context aware location sensitive ambient temperature control. Traditional home energy management for heating and air-conditioning uses thermostats to control central or local heating or cooling units. Advanced energy control systems may support occupancy detection.

[0193] An example energy management system described herein may provide personalized ambient temperature control services that may significantly reduce the energy consumption while improving comfort level using location based services for smart thermostats and/or bio-metric sensors, for example.

[0194] In an example, a location aware bio-metric enabled ambient temperature control service may include and/or utilize an integrated AEP effectiveness management service. The effective AEP service controller 312 or 412 may utilize location information from local heating, cooling and/or humidity controllers to match user locations. The effective AEP service controller 312 or 412 may also interact with wearable bio-metric monitoring devices and/or mobile phone service interfaces to track bio-metric data of users and adjust ambient temperature control settings.

[0195] Location data models are described further herein. A location service model may include multiple types of service interfaces which may provide access to location information for devices and users. The location information may include internet location, geo-location and/or indoor location. Example location sensitive bio-metric aware data models may be summarized as follows:

[Humidity controller, zone, Location] and [Humidity meter, zone, Location]

[Local heater | air-conditioner, zone, Location] [Automated ventilation control, zone, Location]

[Bio-Sensor, zone, Location]

[User, zone, Location]

[Bio-metric, Location]: Bio-metric: {heart rate, sweat, body temperature}.

[UserActivities, Location]: Activities: {exercise, sitting, walking}.

[Environment conditions, Location]: environment: {humidity, temperatures}.

Effective AEP service control model:

o Metric group model: {body temperature, heart rate, sweetness, and other bio-medical conditions}

o AEP threshold parameters: {Minimum Ambient temperature, Maximum Ambient temperature}

o Actions: control level setting of humidifier, heater, or air- conditioner.

o Apphcation context: {Walking, running, sitting}

o Environment condition: {Exterior temperature, humidity, number of occupants}

[0196] AEP effectiveness management is described further herein. An

AEP service controller 312 or 412 may minimize energy consumption while collecting comfort level feedback, for example, from a user ambient temperature feedback pooling apphcation. Users may complain that the ambient temperature is too low or too high, for example by using a dashboard to provide comfort level feedback. Such feedback may be combined with energy consumption measurements, for example using apphcation specific functions (e.g., weighted average). The effective AEP management processes may apply the combined energy consumption and comfort level to adjust a minimum threshold temperature parameter Tmin of the ambient temperature control AEP that controls heaters in different locations for users in each metric group, for example. The pseudo code for the AEP is described hereinafter. It is noted that metric group and state mapping to the threshold configuration may be stored in a data repository and accessed via an ECO automatically.

/*If in location Y, the minimal ambient temperature, Ti, is less than the minimal threshold temperature Tmin of the metric group of User X, then, set the heater state to "on" in location Y. */

If

(AND ( Te(Location-Y) < Tmin //For MetricGroup of User X. Tmin > Ti (Location-Y) )

then

Set-to-on (Heater(Location-Y)) // make sure the heater is on for location Y.

//It is also possible to set the power of the heater based on the difference of between Tmin and Ti

[0197] The effectiveness management process may invoke the profile manager 322 of 422 to collect and categorize the metric groups based on all users who visited each location as well as bio-metric data distribution. Example parameter ranges in the metric groups may include:

Body temperature, BT: {95, 96, 97, 98, 99, 100, 101, 102}

Heart beat, HB: {50-65, 65-90, 90-110, 110-140, 140-170, > 170} Activities, ACT: {Sitting, Walking, Running}

[0198] The profiler 322 or 422 may classify the users into 4 groups based on range values for {BT, HB, ACT}. For example:

Group 1: {[96-97], [65-80], Sitting} //Normal

Group2: {[98-99], [110-140], Walking} //Mild exercise Group3: {[98-99], [140-170], Running} //Intense exercise

Group4: {[99-102], [65-100], Sitting} //person with fever [0199] The profiler 322 or 422 may also consider the environmental occupancy rate:

Occupancy Environment: {Exterior temperature, humidity, occupancy}

[0200] The AEP service controller 312 or 412 may use the combined effectiveness measurements for each location to adjust the threshold parameter Tmin to a value below the average (or an default initial setting) by 0, 0.25, 0.5 and 1 standard deviations, for example, in multiple locations in parallel or in the same location in sequence for multiple selected users in multiple metric groups in different time periods. Where selected users feel cold and want to increase the temperature, these users may access an ambient temperature control mobile app dashboard to express their preferences or complaints. Such feedback may be analyzed, voted, and recorded for each individual user and collectively for a metric group for each location under different environmental conditions. If multiple users adjust the temperature up, then the average temperature may be adjusted higher to meet the minimal comfortable level. After a period of time, a set of comfortable settings that consume minimal energy may be established and applied to all users automatically. If special cases arise, such as a user who is ill and has high fever, the minimal temperature may be adjusted higher.

[0201] The ambient temperature effectiveness adjustment service may also include AEPs to turn off heaters when the temperature is greater than the maximum temperature threshold Tmax. This situation may occur for example when a combination of additional heat generation appliances, such as a large screen TV, fireplace, oven, and/or dryer are operating in conjunction with the heating system. Where heat from these appliances raises the ambient temperature above Tmax in a localized area, the local heater may already have been turned off because the Temperature is above Tmin. In that case, to lower the temperature, a cross zone temperature control AEP management service may synthesize controlled experiments to adjust Tmin to a lower level for the local zone as well as nearby zones. The analysis process may optimize energy consumption by lowering the Tmin settings in AEPs that control the heaters in local and proximate areas.

[0202] A use case for managing ineffective actions and user experience is further described herein. An effectiveness management service may manage ineffective actions to preempt damage or undesirable user experiences. An example application of an effectiveness management service to support pre-emptive protection and customer support for environment with multiple smart devices is described further herein.

[0203] For example, the following example AEPs may be configured in an event processing system to control thermostats for a home or facility with multiple rooms, where one or more rooms are far from the central air- conditioning system (e.g., a furnace).

AEP1: If Temperature < Temperture-Min, then, turn on furnace for the room.

AEP2: If Temperature > Temperature-Max, then, turn off the vent.

AEP3: If room-is-occupied, then, turn on the heater until the temperature reaches the desired temperature learned from previous minimum temperature settings by the user.

[0204] The above AEPs may encounter the following example problems.

Where the furnace is old, a window is leaking, or the temperature outside is very cold, then the furnace may continue to run to try to raise the temperature of the room. In this case, the furnace may raise the temperature of all the rest of the rooms or may overheat itself.

[0205] The AEP effectiveness manager may prevent such potential problems by ranking the effectiveness of actions and detecting effectiveness measures using furnace operation efficiency and/or change in the temperature of the room. If it is detected that the effectiveness of an action (e.g., turning on the central air-conditioner) is ranked significantly lower than for other rooms or a normal distribution of homes of similar size and user comfort level, then the threshold Temperature-min in AEP1 for the remote room may be adjusted to a higher value, for example incrementally by a small percentage of standard deviations from a similar metric group and state. The setting and effectiveness measurement and ranking may be reported to a dashboard to alert the user.

[0206] At the same time, the temperature in other rooms closer to the furnace may be higher than Temperature-max, which may trigger the action of AEP2, which in turn may turn off a vent to support more air to the remote room. For homes without venting control, AEP2 may not be feasible. In this case, it may be necessary to incorporate a smart local heater in or in the vicinity of the remote room. Such smart heater, which may have a higher effectiveness in the remote room than the central furnace, may be turned on at a lower threshold than that of the thermostat for the central air-conditioning system for the remote room.

[0207] The AEP effectiveness manager may automatically safe-guard the system from operation error. Where a user accidently sets a temperature that is much lower or higher than the average temperature settings for other users in similar metric group and state for example, the abnormal setting may be reported to the dashboard (e.g., dashboard 310) to alert the user.

[0208] To prevent discomfort of the end user, the AEP effectiveness manager 326 or 426 may automatically adjust thresholds incrementally and/or in small standard deviations relative to the norm of settings from other users in similar metrics and environmental conditions. Where the setting falls below the desired comfort level of a specific user for the first time, the user may be likely to check the temperature and may observe that it is ranked at a percentile of other users but within a common acceptable economical level. If the user prefers to increase comfort level, the user may adjust the comfort level from the dashboard. The system may classify the user into different metric groups based on desired economy and comfort levels. [0209] To prevent instability and constant changes, an iterative adjustment algorithm may be controlled by the improvements of effectiveness measurement for each iteration. Where the adjustment creates no improvement or has negative impacts, the adjustment process may halt after a certain number of attempts and may report the condition in the dashboard. The range of the adjustment may be bounded by a maximum percentage of the standard deviation of a normal distribution of users from the same metric group in the same state.

[0210] AEPs may be designed by an event processing system. All functions in the architecture may be automated and/or integrated with the event processing system (i.e., AEP processing engines 324 or 424). Such automated functions may include profiling, effectiveness measurement of action correlation, threshold adjustment algorithms, and controlled experiments. Application specific service end points may be supported by the service end-points entries in the data model.

[0211] As illustrated in the above use cases, the devices, methods, processes, architecture, detailed data model, function components and interfaces may integrate and improve effectiveness of event processing used in diverse application domains. Effective AEP management services may dynamically adjust threshold control parameters of an AEP based event processing system to support automatic synthesis and analysis of optimal setting for large numbers of possible combinations of user preferences, bio- metric vital signs, and changes in application contexts and environment conditions. This automatic adjustment process may reduce efforts required in coding, testing and deployment by multiple applications and services, and may support integration of real-time profiling and controlled experiments that may require substantial effort to develop and integrate. This automatic adjustment process may also provide additional benefits of dynamic optimization of action effectiveness to improve user experience and energy efficiency as the behavior and state changes. [0212] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer- readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted over wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD- ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

^• * *

Claims

CLAIMS What is claimed:

1. A method for actionable event pattern (AEP) management on a real-time basis, the method comprising: receiving, by at least one processor, event data from a plurality of users on a real-time basis, wherein the event data corresponds to behavior parameters of the plurality of users in specific states; classifying, by the at least one processor, multiple users of the plurality of users into a metric group on a dynamic real-time basis based on the behavior parameters and specific states of each of the plurality of users; conducting, by the at least one processor, a controlled action effectiveness experiment on the multiple users of the metric group on a realtime basis based on an AEP configuration of an AEP matched to the metric group, including assigning at least one user in the metric group to a control group and assigning at least one other user of the metric group to an experiment group; collecting, by the at least one processor, feedback measurements from the multiple users during the controlled action effectiveness experiment; analyzing, by the at least one processor, the feedback measurements to assess improvement between the control group and the experiment group; adjusting, by the at least one processor, the AEP configuration based on the assessed improvement; and storing, by the at least one processor, the adjusted AEP configuration.

2. The method as in claim 1, wherein the AEP configuration includes at least one parameter threshold and an executable action that are adjustable.

3. The method as in claim 2, further comprising: triggering, by the at least one processor, an execution of an action provided in the AEP configuration, wherein the execution of the action is triggered on the condition that an event parameter of the event data exceeds the at least one parameter threshold of the AEP.

4. The method as in claim 1, wherein the specific state represents an application context in a specific environment condition.

5. The method as in claim 1, wherein analyzing the feedback measurements includes analyzing a correlation between actions provided to the at least one user of the experiment group during the controlled action effectiveness experiment and the feedback measurements received from the at least one user of the experiment group and the at least one user of the control group.

6. The method as in claim 5, wherein conducting the controlled action effectiveness experiment on the metric group includes disabling providing actions to the at least one user of the control group during the controlled action effectiveness experiment.

7. The method as in claim 1, wherein an action provided to the at least one user of the experiment group that produces feedback measurements having an improvement compared to the feedback measurements received from the at least one user of the control group is an effective action, and the AEP configuration is adjusted based on the effective action.

8. The method as in claim 7, wherein the AEP configuration includes a parameter threshold and an executable action, and adjusting the AEP configuration includes adjusting at least one of the parameter threshold and the executable action of the AEP configuration based on parameters of the effective action.

9. The method as in claim 1, further comprising: applying, by the at least one processor, the adjusted AEP configuration to the plurality of users, wherein the adjusted AEP comprises a pattern of events for matching to a pattern of the event data received from at least one user of the plurality of users.

10. The method as in claim 1, further comprising: applying, by the at least one processor, rules for the controlled action effectiveness experiment of the metric group based on the behavior parameters and the specific states of the metric group.

11. An actionable event pattern (AEP) management system comprising: at least one processor configured to receive event data from a plurality of users on a real-time basis, wherein the event data corresponds to behavior parameters of the plurality of users in specific states; the at least one processor is configured to classify multiple users of the plurality of users into a metric group on a dynamic real-time basis based on the behavior parameters and specific states of each of the plurality of users; the at least one processor is configured to conduct a controlled action effectiveness experiment on the multiple users of the metric group on a realtime basis based on an AEP configuration of an AEP matched to the metric group, including configured to assign at least one user in the metric group to a control group and at least one other user of the metric group to an experiment group; the at least one processor is configured to collect feedback measurements from the multiple users during the controlled action effectiveness experiment; the at least one processor is configured to analyze the feedback measurements to assess improvement between the control group and the experiment group; the at least one processor is configured to adjust the AEP configuration based on the assessed improvement; and the at least one processor is configured to store the adjusted AEP configuration.

12. The method as in claim 11, wherein the AEP configuration includes at least one parameter threshold and an executable action that are adjustable.

13. The method as in claim 12, further comprising: the at least one processor is configured to trigger an execution of an action provided in the AEP configuration, wherein the execution of the action is triggered on the condition that an event parameter of the event data exceeds the at least one parameter threshold of the AEP.

14. The method as in claim 11, wherein the specific state represents an application context in a specific environment condition.

15. The method as in claim 11, wherein the at least one processor is configured to analyze the feedback measurements by analyzing a correlation between actions provided to the at least one user of the experiment group during the controlled action effectiveness experiment and the feedback measurements received from the at least one user of the experiment group and the at least one user of the control group.

16. The method as in claim 15, wherein the at least one processor is configured to conduct the controlled action effectiveness experiment on the metric group by disabling providing actions to the at least one user of the control group during the controlled action effectiveness experiment.

17. The method as in claim 11, wherein an action provided by the at least one processor to the at least one user of the experiment group that produces feedback measurements having an improvement compared to the feedback measurements received from the at least one user of the control group is an effective action, and the AEP configuration is adjusted based on the effective action.

18. The method as in claim 17, wherein the AEP configuration includes a parameter threshold and an executable action, and the at least one processor is configured to adjust the AEP configuration by adjusting at least one of the parameter threshold and the executable action of the AEP configuration based on parameters of the effective action.

19. The method as in claim 11, wherein, the at least one processor is configured to apply the adjusted AEP configuration to the plurality of users, wherein the adjusted AEP comprises a pattern of events for matching to a pattern of the event data received from at least one user of the plurality of users.

20. A method for actionable event pattern (AEP) management on a real-time basis, the method comprising: receiving, by at least one processor, event data from a plurality of users, the event data corresponding to behavior parameters of the plurality of users in specific states; matching, by the at least one processor, the event data to an AEP having an AEP configuration; performing, by the at least one processor, an action provided by an AEP configuration based on the matching; receiving, by the at least one processor, feedback on an effectiveness of the action; adjusting, by the at least one processor, the AEP based on the feedback; and storing, by the at least one processor, the adjusted AEP configuration, wherein the adjusting comprises iteratively: profiling the plurality of users into a metric group based the behavior parameters and the specific states of each of the plurality of users; synthesizing and conducting a controlled action effectiveness experiment on the metric group, including assigning at least one user in the metric group to a control group, assigning at least one other user of the metric group to an experiment group, and providing actions under test to the at least one user in the experiment group;

collecting feedback measurements from the at least one user in the control group and the at least one user in the experiment group; analyzing the feedback measurements to assess improvement between the at least one user in the control group and the at least one user in the experiment group; and

adjusting the AEP configuration based on the assessed improvement.