GB2382496A

GB2382496A - Network accounting and billing system and method

Info

Publication number: GB2382496A
Application number: GB0305270A
Authority: GB
Inventors: Eran Wagner; Limor Schweitzer; Tal Givoly
Original assignee: XACCT Technologies Inc
Current assignee: XACCT Technologies Inc
Priority date: 1997-11-20
Filing date: 1998-11-20
Publication date: 2003-05-28
Anticipated expiration: 2018-11-20
Also published as: GB0305270D0; GB2382496B

Abstract

In some embodiments, network traffic information is captured at network information sources. These sources provide detailed information about the network communications transactions at a network device. Importantly, different types of sources can provide different types of information. Gatherer devices gather the detailed information from the various information source devices and convert the information into standardized information. The gatherer devices can correlate the gathered information with account information for network transaction accounting. Manager devices manage the gatherer devices and store the gathered standardized information. The manager devices eliminate duplicate network information that may exist in the standardized information. The manager devices also consolidate the information. Importantly, the information stored by the manager devices represents the consolidated, account correlated, network transaction information that can be used for billing or network accounting. The system thereby provides a distributed network accounting and billing system. Network usage information is input to an information source module, transformed to a standard format and sent through a first program, where it is at least partially transformed in accordance with a data aggregation policy and sent to a second program, where duplicate data is merged before data records are sent to a database.

Description

NETWORK ACCOUNTING AND BILLING SYSTEM AND METHOD RELATED APPLICATIONS '" This application relates to, and incorporates by reference, the United States patent application having serial number 60/066,898, entitled"Network Accounting and Billing System, s'having inventors Limor Schweitzer and Eran Wagner, filed November 20,1997, and which has a common assignee. This application also relates to the United States patent application having serial number XXIXXX, XXX, entitled"Network Accounting and Billing System," having inventors Limor Schweitzer and Eran Wagner, filed November 19, 1998, and which has a common assignee.

COPYRIGHT NOTICE A portion of the disclosure of this patent document contains materials that are subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent disclosure, as it appears in the Patent and Trademark Office patents, files or records, but otherwise reserves all copyright rights whatsoever.

BACKGROUND OF THE INVENTION A. Field of the Invention This invention relates to the field of computer networks. In particular, the invention relates to accounting and billing for services in a computer network.

B. Description of the Related Art The low cost of Internet connectivity and a wide range of services are driving and more people onto the Internet, which is driving the deployment of TCP/IP networks. This process has led to a new market of client-server applications that enables the user to interact with other users and computer systems around the world. The use of these applications is consuming more and more Intranet and Internet bandwidth.

New applications such as"voice over IP (Internet Protocol)"and streaming audio and video require even more bandwidth and a different quality of service than email, or other less real-time applications. Also, the type quality of service can vary according to the needs of the user. For example, typically, businesses do not tolerate unavailable network services as easily as consumers.

Internet Service Providers (ISPs) therefore would like to price their available bandwidth according to a users needs. For example, flat monthly pricing may be the best billing model for consumers, but businesses may want to be billed according to their used bandwidth at particular qualities of service.

As ISPs continue to differentiate themselves by providing additional services, enterprise information technology managers will face similar problems to account for the escalating Intranet operating costs.

Therefore, ISPs and enterprise information technology managers will want to account for session logging, bandwidth usage, directory data and application session information from a variety of sources.

Due to the diversity of IP data sources (e. g. , routers, hubs etc. ), the need for effect tracking far exceeds the problems addressed by telephone companies.

Telephone companies track information such as circuit usage so it can be correlated with account information. For example, businesses may use leased lines. consumers may have"Friends and Family"plans, cellular phones have different roamer fees according to the location of the user, etc. Typically, the phone company captures all of the data and uses batch processing to aggregate the information into specific user accounts. For example, all the long distance calls made during a billing period are typically correlated with the Friends and Family list for each phone account at the end of a billing period for that account.

This requires a significant amount of computing power. However, this type of problem is significantly simpler than attempting to track and bill for every transaction in an IP network. Therefore, what is desired is a system that allows for accounting and billing of transactions on IP based networks.

The problem is even more difficult in IP network traffic because the information sources can exist and many different levels of the OSI network

model. throughout heterogeneous networks. Potential sources of information include packet use from routers, firewall authentication logging, email data, ISP session logging, and application layer use information. Therefore, what is desired is a system and method that track IP network usage information across multiple layers of the OSI network model.

SUMMARY OF THE INVENTION A network accounting and billing system and method are described. In some embodiments, the system can access any network related information sources such as traffic statistics provided by routers and switching hubs as well as application server access logs. The information can be accumulated in a central database for creating auditing, accounting and billing reports.

Alternatively, the information can be sent directly to other systems such as rating engines used in customer care and billing systems.

In one embodiment, network traffic information is captured at network information sources (examples of information sources include network devices).

These sources provide detailed information about the network communications transactions at a network device. Importantly, different types of sources can provide different types of information. Gatherer devices gather the detailed information from the various information source devices and convert the information into standardized information. The gatherer devices can correlate the gathered information with account information for network transaction accounting. Manager devices manage the gatherer devices and store the gathered standardized information. The manager devices eliminate duplicate network information that may exist in the standardized information. The manager devices also consolidate the information. Importantly, the information stored by the manager devices represents the consolidated, account correlated,

network transaction information used for billing. In addition to account information, transaction information can be correlated to other information such as geography information (e. g. , the location of an accessed server) and/or transaction routing information (as may be used in peering agreements between Internet Service Providers). The system thereby provides a distributed network accounting and billing system.

In some embodiments, the gatherer devices can access sources through proxy gateways, firewalls, and/or address translation barriers.

In some embodiments, the gatherer devices can correlate the information about a specific transaction with a particular account by accessing the transaction's source and/or destination information. The source and/or destination information is then correlated with account information from an account information database.

BRIEF DESCRIPTION OF THE FIGURES The figures illustrate the invention by way of example. The invention is not meant to be limited to only those embodiments of shown in the Figures. The same reference in different figures indicates the same element is being used in those figures.

Figure 1 illustrates a system including one embodiment of the invention.

Figure 2 illustrates an example of the data distillation used in the system of Figure 1.

Figure 3 illustrates data enhancements used in the data distillation.

Figure 4A illustrates example field enhancements that can be included in the data enhancements.

Figure 4B illustrates the creation of an enhanced record.

Figure 5 illustrates an example record merge.

Figure 6 illustrates an example of an alternative embodiment of the system.

DETAILED DESCRIPTION A. System Overview One embodiment of the system includes a multi-source, multi-layer network usage metering and mediation solution that gives Network Service Providers (NSPs), including Internet Service Providers (ISPs) and enterprise network (Intranet) operators, the information needed to set the right-price for IP (Internet Protocol) services. With the system, the providers can generate accurate usage-based billing and implement usage-based charge-back models.

The system derives IP session and transaction information, collected in real time, from a multitude of network elements. The system gathers, correlates, and transforms data from routers, switches, firewalls, authentication servers, LDAP, Web hosts, DNS, and other devices to create comprehensive usage and billing records.

The system transforms raw transaction data from network devices into useful billing records though policy-based filtering, aggregation, and merging.

The result is a set of detail records (DRs). In some embodiments, the detail records are XaCCT Detail Records (XDRsTM) available from XaCCT Technologies. DRs are somewhat similar in concept to the telephony industry's Call Detail Records (CDRs). Thus, DRs can be easily integrated with existing Customer Care and Billing (CCB) systems.

In addition to billing data, DRs enable NSPs to deploy new services based on documented usage trends, plan network resource provisioning, and audit service usage. The system provides a clear picture of user-level network service use by tracking a variety of metrics such as actual session Quality of Service (QoS), traffic routes, and end-user application transactions.

The system is based on a modular, distributed, highly scalable architecture capable of running on multiple platforms. Data collection and management is designed for efficiency to minimize impact on the network and system resources.

The system minimizes network impact by collecting and processing data close to its source. Modular architecture provides maximum configuration flexibility, and compatibility with multiple network information sources.

The system, or other embodiments, may have one or more of the following features.

Data collection can be from a wide range of network devices and services. spanning all layers of the network-from the physical layer to the application layer.

Real-time, policy-based filtering, aggregation, enhancement and merging creates accurate, detailed and comprehensive session detail records (DRs).

Real time correlation of data from various sources allows billing record enhancement.

Leverages existing investment through integration with any customer care & billing solution, reducing costs, minimizing risks and shortened time-tomarket.

Non-intrusive operation eliminates any disruption of network elements or services.

Web-based user interface allows off-the-shelf browsers to access the system, on-demand, locally or remotely.

Carrier-class scalability allows expansion to fit an NSPs needs without costly reconfiguration.

Distributed filtering and aggregation eliminates system capacity bottlenecks.

Efficient, centralized system administration allows on-the-fly system reconfigurations and field upgrades.

Customized reporting with built-in report generation or an NSPs choice of off-the-shelf graphical reporting packages.

Comprehensive network security features allow secure communication between system components and multiple levels of restricted access.

B. System Details The following describes the system 100 of Figure 1. The system 100 allows NSPs to account for and bill for IP network communications. The following paragraphs first list the elements of Figure 1, then describes those elements and then describes how the elements work together. Importantly, the

distributed data gathering, filtering and enhancements performed in the system 100 enables load distribution. Granular data can reside in the peripheries of the system 100, close to the information sources. This helps avoids reduce congestion in network bottlenecks but still allows the data to be accessible from a central location. In previous systems, all the network information flows to one location, making it very difficult to keep up with the massive record flows from the network devices and requiring huge databases.

The following lists the elements of Figure 1. Figure 1 includes a number of information source modules (ISMs) including an ISM 110, an ISM 120, an ISM 130. an ISM 136, an ISM 140, and an ISM 150. The system also includes a number of network devices, such as a proxy server 101, a DNS 102, a firewall 103, an LDAP 106, a CISCO NetFlow 104, and a RADIUS 105. The system also includes a number of gatherers, such as a gatherer 161, a gatherer 162, a gatherer 163, a gatherer 164, and a gatherer 165. The system of Figure 1 also includes a central event manager (CEM) 170 and a central database (repository) 175. The system also includes a user interface server 185 and a number terminals or clients 180.

This paragraph describes how the elements of Figure 1 are coupled. The various network devices represent devices coupled to an IP network such as the Internet. The network devices perform various functions, such as the proxy server 101 providing proxy service for a number of clients. Each network device is coupled to a corresponding ISM. For example, the proxy server 101 is

coupled to the ISM 110. The DNS 102 is coupled to the ISM 120. The firewall 103 is coupled to the ISM 130. The ISM 136 is coupled to the LDAP 106. The ISM 140 is coupled to the CISCO NetFlow 104. The ISM 150 is coupled to the RADIUS 105. Each gatherer is associated with at least one ISM. Thus, the gatherer 161 is associated with the ISM 110 and is therefore coupled to that ISM. The gatherer 162 is coupled to the ISM 120. The gatherer 163 is coupled to the ISM 130 and the ISM 136. The gatherer 164 is coupled to the ISM 140.

The gatherer 165 is coupled to the ISM 150. The various gatherers are coupled to the CEM 170. The user interface server is coupled to the terminals 180 and the CEM 170.

The following paragraphs describe each of the various elements of Figure 1.

Network Devices The network devices represent any devices that could be included in a network. (Throughout the description, a network device, unless specifically noted otherwise, also refers to an application server. ) A network device represents a subset of information sources that can be used by the system 100.

That is, the network devices are merely representative of the types of sources of information that could be accessed. Other devices such as on-line transaction processing databases can be accessed in other embodiments of the invention.

Typically, the network devices keep logging and statistical information about their activity. A network information source can be the log file of a mail server,

the logging facility of a firewall, a traffics statistics table available on a router and accessible through SNMP, a database entry accessible through the Internet, an authentication server's query interface, etc. The network devices represent the information sources accessed by the ISMs.

Each type of network device can be accessing using a different method or protocols. Some generate logs while others are accessible via SNMP, others have proprietary APIs or use other protocols.

ISMs The ISMs act as an interface between the gatherers and the network devices enabling the gatherers to collect data from the network devices. Thus, the ISMs represent modular, abstract interfaces that are designed to be platformneutral. The information source modules act as interfaces or"translators", sending IP usage data, in real time, from the network devices to the gatherers.

Each ISM is designed for a specific type of network data source. (In other embodiments, some ISM are generic in that they can extract information from multiple network devices). ISMs can be packaged separately, allowing NSPs to customize ISM configurations to meet the specific requirements of their network. For example, in the system of Figure 1, if the NSP did not have Cisco NetFlow devices, then the ISM 140 would not have to be included.

The ISMs can communicate with its corresponding network device using protocols and formats such as UDP/IP, TCP/IP, SNMP, telnet, file access, ODBC, native API, and others.

In some embodiments, the reliability of system 100 is enhanced through on-the-fly dynamic reconfiguration, allowing the NSP to add or remove modules without disrupting ongoing operations. In these embodiments, the CEM 170 can automatically update the ISMs.

The following ISMs are available in some embodiments of the invention.

* Categorizer-Classifies a session to a category according to user- defined Boolean expression.

DNS (e. g. ISM 120)-Resolves host names and IP addresses.

Generic Proxy Server (e. g. , ISM 110)-Collects data from access logs in a common log format.

'Port/Protocol Resolution-Converts protocol/port information to account names and vice versa.

* CheckPoint FireWall-l-Collects data from FireWall- ! accounting log and security log.

Cisco IOS IP Accounting-Collects accounting data from a Cisco router using IOS IP accounting.

* Cisco NetFlow Switching-Collects session data from a Cisco router via NetFlow switching.

. NETRANET-Collects information from a standard network device. o Netscape Proxy Server-Collects data from a Netscape Proxy Server.

Microsoft Proxy Server-Collects data from a Microsoft Proxy Server.

ISMs can be synchronous, asynchronous or pipe.

The data from an asynchronous ISM is dynamic so that the asynchronous ISM reacts to the information and relays it to the associated gatherer without prompting from other information sources in the system 100.

If the firewall 103 were a CheckPoint FireWall-l, then the ISM 130 would be an example of an asynchronous ISM. When a network session is initiated, the details are recorded by the FireWall-I 103. The corresponding ISM 130 receives the details and passes them on automatically to the gatherer 163.

Synchronous ISMs provide its information only when accessed by a gatherer. The ISM 120 is an example of a synchronous ISM. The DNS server 102 maintains information matching the IP addresses of host computers to their domain addresses. The ISM 120 accesses the DNS server 102 only when the ISM 120 receives a request from the gather 162. When the DNS server 102 returns a reply, the ISM 120 relays the reply information to the gatherer 162.

Pipe ISMs operate on record flows (batches of records received from information sources). Pipe ISMs process one or more enhancement flows the records as the flows arrive. The pipe ISM may initiate new record flows or may do other things such as generate alerts or provision network elements to provide or stop services. The pipe is implemented as an ISM to keep the internal coherency and logic of the architecture. (Record flows can terminate in a

database or in a pipe ISM. The pipe ISM can perform filtering and aggregation. send alarms, or act as a mediation system to provision network elements when some event occurs or some accumulated value is surpassed. Specifically, pipe ISMs can act to enable pre-payment systems to disable certain services such as a voice IP call, when the time limit is surpassed or amount of data is reached.) The gatherers can include caches and buffers for storing information from the ISMs. The buffers allow the gatherers to compensate for situations where there is a loss of connection with the rest of the system 100. The cache sizes can be remotely configured. The cache minimizes the number of accesses to the Information Source.

ISM queries can be cached and parallelized. Caching of synchronous ISM queries provides for fast responses. Parallelizing queries allows for multiple queries to be processed at the same time.

Gatherers The gatherers gather the information from the ISMs. In some embodiments, the gatherers are multi-threaded, lightweight, smart agents that run on non-dedicated hosts, as a normal user application on Windows NT or Unix, as a background process, or daemon. What is important though is that the gatherers can be any hardware and/or software that perform the functions of a gatherer.

The gatherers can be installed on the same network segment as the network device such as router and switch or on the application server itself. This placement of a gatherer minimizes the data traffic impact on the network.

The gatherers collect network session data from one or more ISMs.

Session data can be sent to another gatherer for enhancement or to the CEM 170 for merging and storing in the central database 170. The gatherers can be deployed on an as needed basis for optimal scalability and flexibility.

The gatherers perform flexible, policy-based data aggregation.

Importantly, the various types of ISMs provide different data and in different formats. The gatherers normalize the data by extracting the fields needed by the CEM 170 and filling in any fields that may be missing. Thus, the gatherers act as a distributed filtering and aggregation system. The distributed data filtering and aggregation eliminates capacity bottlenecks improving the scalability and efficiency of the system 100 by reducing the volume of data sent on the network to the CEM 170.

Aggregation can be done by accumulating groups of data record flows, generating a single data record for each group. That single record then includes the aggregated information. This reduces the flow of the data records.

Filtering means discarding any record that belongs to a group of unneeded data records. Data records are unneeded if they are known to be collected elsewhere. A policy framework enables the NSP to configure what to collect where.

Filtering and/or aggregation can be done at any point along a data enhancement (described below) so that aggregation schemes can be based on enhanced data records as they are accumulated. The filtering and/or aggregation points are treated by the system 100 as pipe ISMs which are flow termination and flow starting points (ie: like an asynchronous ISM on the starting end and like a database on the terminating end). Data enhancement paths and filtering and/or aggregation schemes can be based on accumulated parameters such as user identification information and a user's contract type.

As noted above, the PISM can be used in the context of filtering and/or aggregation. One or more record flows can terminate at the PISM and can be converted into one or more new record flows. Record flows are grouped based on matching rules that apply to some of the fields in the record flows, while others are accumulated or undergo some other operation such as "maximum" or "average". Once the groups of accumulated records have reached some threshold, new accumulated records are output. This can be used for example in order to achieve a business-hybrid filtering and aggregation data reduction by imposing the business rules or the usage-based products that are offered to the customer, onto the record flows as they are collected in real-time. This is done instead of previous system where the information is stored in a database and then database operations are performed in order to create bills or reports. The filtering and aggregation reduces the amount of data that is stored in the central

database 1 75 while not jeopardizing the granularity of data that is necessary in order to create creative usage-based products.

Typically, data collected from a single source does not contain all the information needed for billing and accounting, such as user name and organization. In such cases, the data is enhanced. By combining IP session data from multiple sources, such as authentication servers, DHCP and Domain Name servers. the gatherers create meaningful session records tailored to the NSP's specific requirements. In the example of Figure 1, the gatherer 161 can provide information to the gatherer 162 so that the source IP address for an Internet session from the proxy server 101 can be combined with the domain address from the DNS server 102.

The enhancement procedure can be triggered by an asynchronous ISM.

The information from the asynchronous ISM is associated with field enhancements in the central database 175. A field enhancement defines how a field in the central database is filled from the source data obtained from the asynchronous ISM. Through the field enhancements, the missing parameters are added to a record using the data collected from one or more synchronous ISMs.

Enhancements are described in detail below.

The gatherers can include caches and buffers for storing information from the ISMs. The buffers allow the gatherers to compensate for situations where there is a loss of connection with the rest of the system 100. The caches

can reduce the number of accesses to an information source. The buffer and/or cache sizes can be remotely configured.

Central Event Manager (CEM) The Central Event Manager (CEM) 170 acts as the central nervous system of the system 100, providing centralized, efficient management and controls of the gatherers and the ISMs.

The CEM 170 can perform one or more of the following tasks: Coordinates, controls, and manages the data collection process. The CEM 170 coordinates the operation of the gatherers and manages the flow of data through the system 100 through the collection scheme defined in the system configuration. The latter includes the configuration of the gatherers, the ISMs, the network devices, the fields in the central database 175 (described below), and the enhancement procedures. Based on the collection scheme the CEM 170 determines the system 100's computationflow (the set of operations the system 100 must perform to obtain the desired information). The CEM 170 then controls all the gatherers, instructing them to perform, in a particular sequence, the operations defined in the computation flow. The CEM 170 receives the records collected by the gatherers and stores them in the central database 175. NSPs can configure the CEM 170 to merge duplicate records

before storing them in the central database 175. Record merging is described below.

Performs clean-up and aging procedures in the database 175. The system 100 collects and stores large amounts of session information every day. The CEM 170 removes old data to free space for new data periodically. The NSP defines the expiration period for the removal of old records. The CEM 170 is responsible for coordinating the removal of records from the central database 175. The CEM 170 places a time stamp on every record when the record enters the central database 175 and deletes the record after the time period the NSP has defined elapses.

Provides centralized system-wide upgrade, licensing, and data security. The NSP can perform version upgrades of the system 100 at the CEM 170. The gatherers can be automatically upgraded once a new version is installed on the host computer of the CEM 170. ISMs are also installed via the CEM 170 and exported to the gatherers. The CEM 170 maintains a list of licenses installed in the system and verifies periodically if the system is properly licensed. This feature lets the NSP centrally install and uninstall licenses. It also prevents unlicensed use of the system 100 and any of its components.

Monitors the state of the gatherers and ISMs. The gatherers periodically communicate with the CEM 170. The CEM 170

continuously monitors the state of each gatherer and network devices in the system 100. The CEM 170 can be fault-tolerant, that is, it can recover from any system crash. It coordinates the recovery of the system 100 to its previous state.

In some embodiments, a key directory server is associated with the CEM 170. To transfer less data between the elements of the system 100. it is desirable that each piece of data to carry little descriptive data. For example, if IP address data is transferred between a gatherer and the CEM 170, a description of the IP address data is typically included. In some embodiments, data name/key, type, and length descriptions are included with the actual IP address data. In other embodiments, there the key directory server reduces the amount of descriptive information being sent. Every key in the directory server has a type and a length. Fields can be identified as variable length. Therefore, data type information need not be transmitted between elements in the system 100 if the elements use a common reference key stored in the directory server. Returning to the IP address data, by using the key directory server, elements need only send two bytes for the key id and four bytes for the actual address. Most of the data being sent in the system is relatively short in length. Therefore, the directory server helps reduce the amount of information being sent between the elements in the system 100.

Keys can be added to the directory server. The directory server can therefore support expansion of the kinds of fields being sent by allowing system

elements to update their locally stored key ids. For example, after a recipient receives a record with an "unknown" key, it contacts the directory server to get the key definition.

Central Database The central database 175 is the optional central repository of the information collected by the system 100. The central database 175 is but one example of a sink for the data generated in the system 100. Other embodiments include other configurations. The central database 175 stores and maintains the data collected by the gatherers, as well as the information on the configuration of the system 100. Thus, in configuring the system 100, the NSP defines what data will be stored in each field in the central database 175 and how that data is collected from the ISMs.

The information on network sessions is stored in the database in the form of a table. Each field in the table represents a network session parameter.

Each record describes a network session. The system 100 has a set of predefined fields that are configured by the CEM 170 on installation. The NSP can modify the central database 175 structure by adding, deleting, or modifying fields. The NSP access the data in the central database 175 by running queries and reports. The old data is removed from the central database 175 to free space for new data periodically. You can specify the time interval for which records are stored in the central database 175. The structure of the central database 175

! with some of the predefined fields is illustrated in the following figure.

As each IP session may generate multiple transaction records, during the merge process the CEM 170 identifies and discards duplications, enhancing the efficiency of the data repository. Generally, data records are passed through the merger program, in the CEM 170, into the central database 175. However, the data records are also cached so that if matching records appear at some point, the already stored records can be replaced or enhanced with the new records.

The database tables that contain the record flows can be indexed, enhancing the efficiency of the data repository. A merge is achieved by matching some of the fields in a data record and then merging the matching records from at least two record flows, transforming them into one record before updating the central database 175. In some embodiments, adaptive tolerance is used to match records. Adaptive tolerance allows for a variation in the values of fields that are compared (e. g. , the time field value may be allowed to differ by some amount, but still be considered a match). The adaptive aspect of the matching can include learning the appropriate period to allow for the tolerance. The reason that the records that do not match any previous records are sent through into the central database 175, in addition to being cached for later matching, is to avoid loss of data in case of system failure.

The following table illustrates an example of the types of records stored in the central database 175 by the CEM 170.

Source IP Desunauon Soarce Desuntton Sendee Date/Tunc Dmatl Total Counter ! HostHoston Bytes 199. 203. 13 204. 71. ! 773 pcLevj < ace yahoo. com http 1998-04-26 6464 435666 261019 2187 5 t. com 10 : 56. 55 199. 203. 13 207. 68. 137.5 prodigy.xac microsoft.co telnet 1998-04-26 747 66743 261020

2. 131 9 CLcom m 10 : 56 : 55 199.203.13 199. 203. 132. pcEitan.xac xpert.com smtp 1998-04-26 82 55667 261021 2 177 ct.com 10 56#55 199. 203 13 204.162. 80 1 pcAdi.xacc cnet.com http 1998-04-26 93 33567 261022 2.173 82 t.com 10:56:55

The system 100 supports a non-proprietary database format enabling the central database 175 to run on any of a number of commercially available databases (e. g. , MS-SQL Server, Oracle Server, DB2, etc.).

User Interface Server and Clients The User Interface Server (UIS) 185 allows multiple clients (e. g. temiinals 180) to access the system 100 through, the Microsoft Internet Explorer with JavaTM Plug-in or Netscape Navigator with JavaTM Plug-in. Other embodiments can use other applications to access the system 100. The main function of the UIS 185 is to provide remote and local platform independent control for the system 100. The UIS 185 can provide these functions through windows that correspond to the various components of the system 100. Access to the system 100 can be password protected, allowing only authorized users to log in to the system and protecting sensitive information.

The NSP can perform one or more of the following main tasks through the UIS 185: * Configure the system 100.

# Create and run queries and reports on network activity and resource consumption.

. Register and license the system 100.

C. Data Distillation Figure 2 illustrates the data distillation process performed by the system of Figure 1. The data distillation aggregates and correlate information from many different network devices to compile data useful in billing and network accounting.

First, the ISMs 210 gather data from their corresponding network device. Note that for some ISMs (e. g. pipe ISMs), real-time, policy-based filtering and aggregation 215 can also be done. This. data is then fed to the gatherers 220. The gatherers 220 perform data enhancement to complete the data from the ISMs 210. The results are provided to the CEM 170. The CEM 170 performs data merges 270 to remove redundant data. The merged data is then optionally stored in the central database 175 as a billing record 275 or is sent directly to an external system. The billing record information can be accessed from external applications, through the application interface 290, via a data record 280. Filtering and/aggregation and/or data enhancements can be done at any stage in the system 100.

D. Data Enhancement As mentioned above, the gatherers 220 provide data enhancement features to complete information received from the ISMs 210. The following

describes some example data enhancement techniques used in some embodiments of the invention.

Figure 3 illustrates an example of data enhancement. Data enhancement comprises a number of field enhancements. A field enhancement specifies how the data obtained from the trigger of the enhancement procedure is processed before it is placed in a single field in the central database 175. The data can be placed in the field directly, or new information may be added to the record by applying a Synchronous ISM function. (In the example below, the function is "resolve the IP address to a host FQDN"). Field enhancements may involve one or multiple steps. There is no limit to the number of steps in a Field Enhancement. The data record starts with fields obtained from an asynchronous ISM 300. The fields in the DR 300 are then enhanced using the field enhancements. The enhanced fields result in the DR 320.

A visual representation of an enhancement can be presented to the NSP.

The enhancement may include an itinerary of ISMs starting off with an AISM, passing through PISMs, and terminating in the CEM 170. Using this view of the system 100, the NSP need not be shown the actual flow of data since the flow may be optimized later in order to achieve better performance. This is more of a graphical logical view of how the enhancement is achieved in steps. (PISMs can terminate more than one flow and initiate more than one flow.) A visual representation of a field enhancement shows the per-field flow of data correlation. This process ends in the CEM 170 or in a PISM. The NSP

supplies information telling the system 100 how to reach each of the terminating fields (in the CEM 170 or the PISM) starting off from the initiating fields (PISM or AISM). Each step of enhancement defines cross correlation with some SISM function.

Figure 4A illustrates various field enhancements (410 through 440). A field enhancement includes applying zero or more functions to a field before storing the field in a specified field in the central database 175.

One-step Field Enhancement 410. The initial source data from the asynchronous ISM is placed directly in a field in the central database 175.

Example: the field enhancement for the Source IP field.

Two-step Field Enhancement 420. The initial source data from the asynchronous ISM is used to obtain new additional data from a synchronous network device and the new data is placed in a field in the central database 175.

Example: the field enhancement for the Source Host field.

Three-step Enhancement 430. The initial source data from the asynchronous ISM is used to obtain additional data from a synchronous ISM.

The result is used to obtain more data from another ISM and the result is placed in a field in the central database 175.

The following illustrates an example data enhancement. Suppose the data obtained from a proxy server 101 contains the source IP address of a given session, such as 199.203. 132.2, but not the complete domain address of the host computer (its Fully Qualified Domain Name), such as www. xacct. com. The

name of the host can be obtained by another network device-the Domain Name System (DNS 102) server. The DNS server 102 contains information that matches IP addresses of host computers to their Fully Qualified Domain Names (FQDNs). Through an enhancement procedure the information collected from the proxy server 101 can be supplemented by the information from the DNS 102. Therefore, the name of the host is added to the data (the data record) collected from the proxy server 101. The process of adding new data to the data record from different network devices can be repeated several times until all required data is collected and the data record is placed in the central database 175.

Figure 4B illustrates another example data enhancement where an enhanced record 490 is created from an initial netflow record 492. Fields in the enhanced record 490 are enhanced from the radius record 494, the QoS policy server record 496, the NMS DB record 498, and the LDAP record 499.

Defining Enhancement Procedures The following describes the process for defining enhancement procedures in some embodiments of the system. Typically defining an enhancement procedures for the system 100 includes (1) defining enhancement procedures for each asynchronous ISM and (2) configuring field enhancements for all fields in the central database 175 for which the NSP wants to collect data originating from an asynchronous ISM that triggers the corresponding enhancement procedure.

An enhancement procedure can be defined as follows: 1. Access the CEM 170 using the UIS 180.

2. Select the enhancement procedures list using the UIS 180.

3. Define the name of the new enhancement procedure.

4. Select a trigger for the new enhancement procedure. The trigger can correspond to any asynchronous ISM in the system 100.

Alternatively, the trigger can correspond to any asynchronous ISM in the system 100 that has not already been assigned to an enhancement procedure.

5. Optionally, a description for the enhancement procedure can be provided.

6. The new enhancement procedure can then be automatically populated with the existing fields in the central database 175.

Optionally, the NSP can define the fields (which could then be propagated to the central database 175). Alternatively, based upon the type of asynchronous ISM, a preset set of fields could be proposed to the NSP for editing. What is important is that the NSP can define field procedures to enhance the data being put into the data records of the central database 175.

7. The NSP can then define the field enhancements for every field in the new enhancement procedure for which the NSP wants to collect

data from the ISM that is the trigger of the new enhancement procedure.

Defining Field Enhancements Defining a field enhancement involves specifying the set of rules used to fill a database field from the information obtained from the trigger of the enhancement procedure. The NSP defines field enhancements for each field in which NSP wants to collect data from the trigger. If no field enhancements are defined, no data from the trigger will be collected in the fields. For example, suppose the firewall asynchronous ISM 130 that triggers an enhancement procedure. Suppose the central database 175 has the following fields: source IP, source host, destination IP, destination host, user name, total bytes, service, date/time, and URL. If the NSP wants to collect session data for each field except the URL from the firewall ISM 130, which triggers the enhancement procedure, the NSP defines a field enhancement for each field with the exception of the URL.

In some embodiments, the field enhancements are part of the enhancement procedure and the NSP can only define and modify them when the enhancement procedure is not enabled.

The field enhancements can be defined in a field enhancement configuration dialog box. The field enhancement configuration dialog box can have two panes. The first displays the name of the enhancement procedure, the name of its trigger, and the name and data type of the field for which the NSP is

defining the field enhancement. The second is dynamic and interactive. Its content changes depending on the NSP's input. When first displayed, it has two toggle buttons, End and Continue, and a list next to them. The content of the list depends on the button depressed.

When End is depressed, the list contains all output fields whose data type matches the data type of the field for which the NSP is defining the field enhancement. For example, if the field's data type is IP Address, the list contains all fields that are of the same type, such as source IP and destination IP that the AISM supplies. The fields in the list can come from two sources: (1) the source data which the gatherer receives from the trigger and (2) the result obtained by applying a synchronous ISM function as a preceding step in the field enhancement. The following notation is used for the fields:

OutputFieldName for the output of a field origination from the trigger SISName. FunctionName (InputArgument). OutputField for the output of a field that is the result of applying a function SISName... OutputField for the output of a field that is the result of applying a function as the final step of a field enhancement The following examples are presented.

Source IP is the field provided by the trigger of the enhancement procedure that contains the IP address of the source host.

DNS... Host Name and DNS. Name (Source IP). Host name are the names of a field originating from the resolved function Name of a network device

called DNS that resolves the IP address to a domain address. The input argument of the function is the field provided by the trigger of the enhancement procedure, called source IP. It contains the IP address of the source host. The function returns the output field called Host Name that contains the domain address of the source host. The notation DNS... Host Name is used when the field is the result of applying the function as the final step of a field enhancement. The notation is DNS. Name (Source IP). Host Name is used when the field is used as the input to another function.

In the user interface, if End is unavailable, none of the output fields matches the data type of the field.

When Continue is depressed, the list contains all applicable functions of the available synchronous network device configured in the system 100. If the preceding output does not match the input to a function, it cannot be applied and does not appear on the list.

The following notation is used for the functions : SISName. FunctionName (InputFie/dName : lnputFieldDataType) (OutputFieldName : OutputFieldDataT) pe) When the function has multiple input and/or output arguments, the notation reflects this. The arguments are separated by commas.

The following example shows a field enhancement.

DNS. Address (Host Name: String)- (IP Address: IP Address)

Where DNS is the name of the synchronous ISM (or network device) as it appears in the system configuration.

Address is the name of the function.

(Host Name: String) is the input to the function-host FQDN of data type String (IP Address : IP Address) is the output-IP address of data type IP Address The NSP can define the field enhancement by choosing items from the list. The list contains the option < none > when the End button is depressed.

Choosing this option has the same effect as not defining a field enhancement: no data from the trigger will be stored in the field in the central database 175.

At your request, here is one paragraph describing the algorithm (s) transforming the user representation of an Enhancement Procedure to an actual Computation Flow performed by the system. The source code accompanying this actually implements most of this. Of course, that code relies on the rest of the software (lots of other code), that may be required to completely understand

it.

Field Enhancement Computation In some embodiments, an enhancement procedure (EP) defines a flow of data records in the system 100, originating from an AISM that serves as a trigger. and terminating at the a database table, or a PISM (Pipe ISM-has the behavior of the database of being assigned input keys in much the same way,

but being installable/replacable logic in the system after it has been deployed), known as the Target of the EP. An EP is represetited to the user as a collection of Field Enhancements (FEs). Each FE identifies what is performed in order to obtain a value (or not, if it is not defined), for that Field (either database field, or input key to PISM), using values originating from the Trigger AISM, possibly, utilizing functions of SISMs available in the system. The NSP defines every FE independently of other FEs, and completely defines what value is placed in the defined field.

In contrast to the above description of the presentation to the user of an EP. the system 100 (e. g. , the CEM 170) computes what is called a Computation Flow, which is the actual operational instructions and parameters that are instructed to the gatherer's to achieve the desired NSP result (as defined in the EP). The Computation Flow is the detailed instructions required to achieve a "snowball" affect, of having data originating from a Trigger (AISM) in one gatherer, undergo specific enhancements by adding results from invocations of functions of SISMs with specific input parameters on the same gatherer or other gatherers (as required), and removing unused fields along this same path, eventually storing the user-configured results in the user-configured fields in an optimized fashion. The optimizations applied are: . Transfer minimal data between any elements in the system;

* Do not compute the same function on the same input more than once. In other embodiments, this optimization may be changed to allow for sharing of intermediary values of field enhancements ; 'Perform local enhancements before going on to a remote gatherer; * Apply a"cost"function to transferring along communication links and traverse the minimal cost in a computation flow required to achieve correct results.

This set of optimizations can be performed, in some embodiments, using the code in Appendix A. That is, the transformation (from EP to Computation Flow, or simply Flow) is achieved by coupling together eleven algorithms and data transformations one after the other. These are described in the code module attached.

Note: 1) Within the code, EP, Rule, Computation Flow are used in some cases to refer to the same objects. 2) The eleven transformations that are applied to the user representation of the EP, after being stored in a database structure, are outlined beginning at line 124 of the attached source module "FlowManager. java".

E. Record Merges Figure 5 illustrates an example record merge. Record merging removes duplicate records from the central database 175. The following example shows how merges work and illustrates the need for merging duplicate records. Suppose the system 100 is using two

asynchronous ISMs 110 and 130. All outbound network traffic going through the proxy server 101 is routed through the firewall 103. The firewall 103 records the proxy server 101 as the source of all sessions passing through the proxy server dz although they originate from different workstations on the network.

At the same time, the proxy server 101 records the destination of all sessions as the firewall 103. although their actual destinations are the different Internet sites.

Therefore, all sessions are logged twice by the system 100 and the records are skewed. The data from the firewall 103 indicates the destination of a given session, but not the source (see data record 520), while the data from the proxy server 101 records the source, but not the destination (see data record 510). Defining a merge eliminates the duplication of records.

A merge can be defined instructing the CEM 170 to store the destination data obtained from the firewall 103 and the source data from the proxy server 101 in the central database 175. The merge will also eliminate the problem of skewed data by storing the correct source and destination of the session in the central database 175. Both network devices provide information on the URL.

The latter can be used to identify the fact that the two seemingly independent records (510 and 520) are actually two logs of the same session.

Two enhancement procedures are defined for the example of Figure 5.

The trigger of the first, designated Flow One, is the Proxy Server Asynchronous Information Source Module. The trigger of the second, Flow Two, is the

Firewall Asynchronous Information Source Module. The records from Flow One and Flow Two are records of the same session. They both have the same value for the URL field. Based on this value, the CEM 170 identifies the two records are double logs of the same session. It merges the two data records taking the Source IP value from Flow One and the Destination IP from Flow Two as the values to be stored in the central database 175.

Defining Merges The following describes defining merges. A merge is a set of rules that specify how duplicate records from multiple enhancement procedures must be identified and combined before being stored in the central database 175. The NSP can merge the records from two or more enhancement procedures. To define a merge, the NSP identifies the following information.

The enhancement procedures included in the merge.

* How to identify duplicate records (which fields of the records must match).

* How to combine the records; that is, for each field, which value (from which enhancement procedure) must be stored in the central database 175. (Optional) If the NSP does not specify how records must be combined, the records are merged as follows: When the values in all but one of the fields are null, the non-null value is stored.

When the fields contain non-null values, the value of the first record received (chronologically) is stored.

F. Additional Embodiments The following describes additional embodiments of the invention.

In some embodiments, the user interface used by an NSP to configure the system 100 can be presented as a graphical representation of the data enhancement process. Every step in the enhancement can be shown as a block joined to another block (or icon or some graphical representation). The properties of a block define the operations within the block. In some embodiments, the entire data enhancement process from network devices to the central database 175 can be shown by linked graphics where the properties of a graphic are the properties of the enhancement at that stage.

In some embodiments, multiple CEMs 170 and/or central databases 175 can be used as data sources (back ends) for datamart or other databases or applications (e. g. , customer care and billing systems).

In some embodiments, the types of databases used are not necessarily relational. Object databases or other databases can be used.

In some embodiments, other platforms are used. Although the above description of the system 100 has been IP network focussed with Unix or Windows NT systems supporting the elements, other networks (non-IP networks) and computer platforms can be used. What is important is that some

sort of processing and storing capability is available at the gatherers, the CEMs. the databases, and the user interface servers.

In some embodiments, the gatherers and other elements of the system 100, can be remotely configured, while in other embodiments. some of the elements need to be configured directly. For example, a gatherer may not be remotely configurable, in which case, the NSP must interface directly with the computer running the gatherer.

In other embodiments, the general ideas described herein can be applied to other distributed data enhancement problems. For example, some embodiments of the invention could be used to perform data source extraction and data preparation for data warehousing applications. The gatherers would interface with ISMs that are designed to extract data from databases (or other data sources). The gatherers would perform filtering and aggregation depending upon the needs of the datamart (in such an embodiment, the central database and CEM could be replaced with/used with a datamart). The data enhancement would then be done before storing the information in the datamart.

Figure 6 illustrates a system 600 where multiple systems 100 are linked together. This system could be an ISPs point of presence accounting system.

The system 620 and the system 610 can store detailed network accounting information in their local detailed accounting databases. This information can then be aggregated and sent over the more expensive long distance links to the billing database in the system 630. Customer service information can still be

accessed at the detailed accounting database, but the aggregated information may be all that is needed to create the bills.

G. Conclusions A network accounting and billing system and method has been described. In some embodiments, the system can access any network related information sources such as traffic statistics provided by routers and switching hubs as well as application server access logs. These are accumulated in a central database for creating auditing, accounting and billing reports. Because of the distributed architecture, filtering and enhancements, the system efficiently and accurately collects the network usage information for storage in a form that is useful for billing and accounting.

Appendix A-FlowManager . ft. ft Example 1/*****.'. *. ***************'****... ******~.... ********..... *. *. *. ****. ***... *.

II //NAME & TITLE : Flowmanager Analyzes the various Enhancemnt Rules and creates Computation Flows II ABSTRACT : //-Analyzes the various Enhancemnt Rules and creates Computation Flows //-Respond to Gatherer Requests for Instructions // Instructions are stored in UNIRInstruction, which can either be an element of a //vector or a vector of UNIRInstructions in itself //-Allow for ReComputation of Particular Flows // - Implment Helper methods for finding interdependencies between Enhancemnt Rules //and ISs that are referenced by them. Important when attempting to delete an IS //or when an IS configuration has changed, to see which Enhancemnts mat be

affectetd.

II //this class is closely related to the Gatherer. EnhanceCompFlowStep class, which acts //on the instructions created here.

// An instruction needs to know the following: 1. what type it is (recieve, enhance, send or other) //2. If it is of type enhance: //2. 1 Where are the Input Unirs for the Enhancement Function // ("where" means position in the Arguments) //2. 2 What IS and what function to perform //2. 3 Where in the Arguments to put the function results.

3. It it is of type send, needs to know to whom to send. CEM or another

Gatherer.

II //For further understanding, see Gatherer. EnhanceCompElowStep II //CHANGE HISTORY : II Programmer Change Date Change Description //Adi Gavish 01/12/97 Initial version //--------------------------..------...---I I Adi Gavish 01/12/97 Initial version II 11*. ******'*. *. *********. *. **********************'.. ******. *..... ****. ******** import Java. util. * ;

import Until.- ; import Util. Persistence. import Util. UNIRTypes. * ; import Util. Struts. * ; import Util. Log. * ; import COM. objectspace.jgl.*;

//9.'..'. *. ~******. *. *****'"'",.... ****. 9.. ***',,,, **............ ****....... public class FlowManager f // Current Rule we are processing int m~RuleID ; //Pipe ISM variables boolean mIsTargetPipe-false ; int mTargetIS-0 ; int m~TargetNode - 1 ; int mTargetGatherer-0 ; //Holds all Enhancments for a Rule. Alll elements are of type EnhanceOpNode. public EnhanceOpSList mOpList-new EnhanceOpSList () :

public EnhanceOpSList m OpTmpList-new EnhanceOpSList f) ; //All output parameters are added to this list, with their positions in the list //used to define the actual position of UNIRs in the Data Arguments collected by the ISs of the flow.

// After Output positions are defined, the input parameters are updated with //the new positions (So the Enhancement step in the Gatherer will know where to //get the input parameters of some enhancement function) II see allocateUNIROutputData method SList UNIRAllocation ; //Temporary Storage of instructions converted from Enhancements SList mjTmpInstructions ; //HashTable For Flow Instruction Storage, access by RuleID Stores a complete set //of instructions for every Flow (-Every Rule) Hashtable m Instructions-new Hashtable () ; //Offset to fix UNIR Indexes (Take FlowID, Step into account) //See PlaeeHolders in Gatherer. EnhanceCompFlowStep final static int FLOW-OFFSET-2 ; // Globol counter used in DFS Topology Sort

r. t mTimeCtr-0 : //...........................,........................... *. *....... *.....* //Build All Flows //-Loop on all rules //-build an Instrucion vector for each Rule //-Store set of instructions per Rule in instructions //...................................................................... public void computeFlows () { Rules RuleList-CEM. instance (). getRulesO ; if (RuleList ! - null) ( //Loop on all Rules for (Enumeration El = RuleList. elements El. hasMoreElements ( //Compute Flow for specific Rule (If Enhancement is Enabled) Rule CurrentRule" < (Rule) (El. nextElement ())) ; if I CurrentRule !-null) tt (CurrentRule. getEnabledf) true) ! int RuleID-CurrentRule. getRuleIDO ; computeRuleFlows (RuleID) ; I I ) I //........................................................................

//Compute Flow for RuleID. At the end of this method, All Flowlnstructions for RuleID will be generated and saved as an SList in a Hash Table, accessed by RuleID II //parameters //RuleID-ID of rule to build the instruction set for //.........................,........................... *............,... public void computeRuleFlows tint RuleID) f Log. instance (). writeLog (new LogDebugf'Compute Flow for Rule "+RuleID)) ; mRuleID"RuleID y try ( il 1. Build Initial EnhanceOpNode set for RuleID buildGUONSByRule (RuleID) ; if (tnOpList. size () > 0) ( //2. Create Enhancement String Descriptions buildOpDescriptions () ;

//3. Convert All Opnums in Enhancement Input Params to GUONS buildTranslatedOpKeyIn () ; //4. Reduce Duplicate GUONS (with same Description) reduceSameDescriptionOps(); Sa. Generate OpKeysOut Information buildOpKeysOut(); //5b. Reduce GDONS that are NOT INPUT to any other Non End GUON.

//Dependent on Sa. reduceErrorGuons() ; //6a. Perform Topology Sorting doTopolgySort () ? //6b. Perform"Node Coupling"Optimizations doNodeCouplingt) ; //7. Allocate UNIR Data Structure Key Index allocateUNIROutputData (); //B. For every input parameter, match the appropriate Index fronm step 7 propogateIndexToInput () :

I //9. Create IntructionSet buildInstructionSet () ; //10. Save IntructionSet in Hash Table with RuleID as Key m-Instructions. put (new Integer (RuleID), m~TmpInstructions) ; //11. Build Fieldlndex Vector t send to Merger (if Target is DB, not pipe)

if ( ! aIsTargetPipe) j buildFie1dIndex () ; I Log. 1nstance (). writeLoq ("FlowManaqer. computeRuleFlows", LogEvent.LOG~DEBUG, "Done Compute Flow for Rule"+RuleID) ; ) catch (Exception E) ( Debug. writeStackTrace ("FlowManager. computeRuleFlows", E) ; I ! // ... *..,... *..................,...... *.......,....... *.............

Build an initial set of EnhanceOpNodes representing all Enhancements per some //rule. Set some additional Info on every EnhanceOpNode, such as GathererID and //mIsLast Flag.

//""IT IS ASSUMED THAT Enhancements (From PERSISTENCE) is loaded sorted by //Field/OpNum ! ! !

//.... *............... < ........... *.. *........ ****-***........ *.... protected void buildGUONSByRule int RuleID) f m IsTargetPipe-false m~TargetNode=1; m~TargetGatherer - 0 ; m~TargetIS = 0 ; m~OpList = new enhanceOpsSList (); Nodes NodesList-CEM. instance ().getNodes(); ISs ISsList = CEM. instance().getISs() ; //Loop on All Enhancements. For Every Enhancement that matches the current //Rule, create an EnhanceOpNode object and add it to the OpArray. After //loop is done, Sort the OpArray.

OpArray will sort on? Enhancements EnhancementList-CEM. instance (). getEnhancements () ;

Array OpArray"neM Array f) ; if (EnhancementList !-null) f forfEnumeration E-EnhancementList. elements () ; E. hasMoreElements(); ) ( Enhancement Enhancementltem- ( (Enhancement) (E. nextElement neo)) ;

if (Enhancernentltem. getRuleIDO-RuleID) I //printDiagnosticsPre (Enhancement) ; EnhanceOpNode EnhancementNode. new EnhanceOpNode ( Enhaneemetitltem) : //print Diagnos ticspos t (EnhancementNode) OpArray. add ( EnhaneementNode) ; J ) J Sorting. sort ( OpArray //build mOpList from OpArray if (OpArray. size () > 0) { String LastField-'" ; Rules RuleList-CEM. instanceo. getRuleso, Rule CurRule-RuleList. getRuleByRuleID (RuleID) : //If Rule Not Enabled, mOpList will be empty i. f ( (CurRule !-null) & & (CurRule. getEnabledo) I Get the AISM ID of this Flow int ISID = CurRule.getTriggerID();

set Target Pipe Boolean. Target Node and other Pipe variables mTargetIS-CurRule. getTargettDO ; if (mTargetIS > 0) I mIsTargetPipe-true ; IS TargetIS-ISsList. getISByISIDf mTargetIS) : if (TargetIS !-null) f M~TargetGatherer TargetIS. getGathererID () ; Node NNodesList. getNodeByTypeID (Util. Comm. Node. NODETYPESATHERER, mTargetGatherer) ; mTargetNode-N. getNodeIDO ; I I Get AISM Information IS I-ISsList. getISByISIDt ISID) ; if (I ! = null) I int ctr-0 //Build First entry into m-OpList from thr Rule Header int GathererID-I. getGathererID () ; int ISMID-I. getISMIDo ; Node NNodesList. getNodeByTypeID (Util. Comm. Node. NODETYPEGATHERER, GathererID) ; int NodeID-N. getNodeID () ; //Add GUON 0 with data from the Rule Header EnhanceOpNode OpZero-new EnhanceOpNode (RuleID,"", (short} O, ISID, 0, null, null) ; OpZero. setGathererID (GathererID) ; OpZero. setNodeID (NodeID) ; OpZero. setISMID (ISMID) ; setting GUON to the original index will garuentee unique value.

OpZero. setGUOHfctr) : OpZero. setLastFlagf false) ; n~OpList. add (OpZero) : ctr++ ; Loop on OpArray, build the rest of mOpList EnhanceOpNode PriorEnhanceOp-null ; for (Enumeration E-OpArray. elements f) ; E. hasMoreElements () ;) I

EnhanceOpNode Currentenhancement = ( fEnhanceOpNode) (E. nextElement ())) ; //Check if it belongs to The Rule if CurrentEnhancement null

f CurrentEnhaneement. getRuleIDf)-RuleID)) I ctr++; //set aux data, add to list I-ISsList. getISByISID ( CurrentEnhancement. getISIDo

if (I ! =null) { GathererID-I. getGathererlDO ; ISMID-I. getISMIDO ; MNodesList. getNodeByTypeID (Util. Comm. Node. NODETYPEGATHERER, GathererID) ; NodeID = N. getNodeIDO ; CurrentEnhancement. setGathererIDIGatherer ! D} ; CurrentEnhancement. setNodeID (NodeID) ; CurrentEnhancement. setISMID (ISHID) ; //set GUON to the original index will garuentee unique value.

CurrentEnhancement. setGUONtctr) ; CurrentEnhancement. setLastFlag (false) ; // Check if Field changed. If Field has changed, this means that //the previous EnhanceOpNode was the last for that field.

String CurrentField CurrentEnhancement. getFieldIDf) ;

if ! CurrentField. equals (LastField)) tt (PriorEnhanceOp'-null)) I //Field Has Changed, prior Op was last for a Field

PriorEnhanceOp. setLastFlag (true) Set Target Field of previous EnhanceOpNode

PriorEnhanceOp. addOutFields (LastField) ; )

// Save for Last Step Per Field Logic PriorEnhaneeOp- CurrentEnhancement ; LastField-CurrentField : //Add To List

mOpList. add ( CurrentEnhancement) ; Log. instance (). HriteLog ("FlowManager. buildSOONSByRule", LogEvent. LOG-DEBUG, CurrentEnhancement. toString ()) ; } else ( Log. instance().writeLog(new

LogErrorfbuildGUOHSByRule : No IS found for"+CurrentEnhancement. getISIDO)) ; } ) //End Of Loop Logic. Same as if Field has changed. if (PriorEnhanceGp != null) ( PriorEnhanceOp. setLastFlag < true) ;

PriorEnhanceOp. addOutFields (LastField) ; ) ) I t //. *........... < .,........ *......,.. *. *........ *. **...... *...... *. *.. *.. < .

//Print Diagnostics on the passed Enhancement.

//Currently Mot Used.

//... *....******.. *..*.*....,*****. **.. *... **.* < ........ < *. *..... *. ... ** protected void printDiaqnosticsPre (Enhancement Enhancementltem) / try ( byte[] buf = EnhancementItem.getOpKeyIn() ; String InputList = "[" ;

Arguments a-new Arguments (buf for (Enumeration ES-a. elements () : EK. hasMoreElementsO ;) { short value (UNIRShort) EK. nextElenent ()). getShortValueO ; InputList ±Short. toString (value) +'', * ;

value- ( (ONIRShort) EK. nextElement ()). getShortValuef) ; InputList- Short. toString (valued"," ; j InputList ± "]" ; Log. instance (). writeLog ("FlowHanager. buildGUONSByRule", LogEvent. LOG DEBUG,"PersistenceItem : "+InputList) ; I catch (Exception e) ( I I 1/*... **..... *****.. ***. **. *. *'. **. *. *. ~***'***********. ***********. ***. *.

Print Diagnostics on the passed EnhanceOpNode.

Currently Not Used.

11. -'.'.......'.... -.. -.'.'... "'.'. -................'. ****... ******. **. *. protected void printDiagnosciesPostfEnhanceOpNode EnhancementNode) i EnhancementNode} try ( byten buf-EnhancementNode. getOpKeyIn () ; String InputList-" [" ; Arguments a-new Arguments (buf) ; for (Enumeration EK-a. elements !) ; EK. hasMoreElementso ; ! short value f < UHIRShort) EK. nextElement ()). getShortValuet) : InputList ±Short. tostring (value) +", value (fUNIRShort) EK. nextElement ()). getShortValue () ; InputList ±Short. tostring (value) +", ) InputList ±"]" ; Log. instance (). writeLog CTlowManager. buildGOOMSByRule'', LogEvent. LOG~DEBUG, "Struct 1 : "+InputList) ; InputList for (Enumeration EKEnhancementNode. getParsedOpKeyIn (). elements () ; EK. hasMoreElements (J :) t int value- ( (Integer) EK. nextElement ()). lntValuef) ; InputList ± Integer. toStringlvaluel+", " ; value- ( (Integer) EK. nextElement ()). intValue () ; InputList ±Integer. toString (value) +"." ;

I InputList Log. instance().writeLog("FlowManager.buildGUONSByRule", LogEvent. LOGDEBUG,"Struct 2: "*InputList) ;

} catch (Exception e) ( I i ABSTRACT //ABSTRACT //Convert the m OpList to an Instruction Set the Gatherer Can Understand

(a set of UNIRInstructions) II Description //Translate mOpList to sets of Enhancement Instructions. Divide the Enhancements //by Different Gatherer sequences, wrap each segment with a Receive Instruction//andaSend Instruction.Instruction and a Send Instruction.

//*.. *.......... ... *.... *....... *....... **................ *.... < *.....

//. *. *.. *..... *........... *..... *..... *. **.. ***". *.

Helper method to add an instruction of type Send //*. ..................... ..... *. ***... ....... void AddSendStep (int StepCtr, int NodeID, int ISID, int ISMID, int GathererID) f UNIRInstruction u-new UNIRInstruction (m~RuleID, StepCtr, UNIRInstruction. INSTRUCTIOMSEND, NodeID, 0, ISID, ISMID, null, null, GathererID Log.instance().writeLog("FlowManager.buildInstructioniSet", LogEvent. LOG DEBUG, "Adding Instruction"+u) ;

ntT ! < ) pInstructions. pushBack (u) ; J //.......... *............... **......................

II Helper mechod to add an receive of type Receive . pe Receive void AddReceiveStepfint StepCtr, int NodeID, int ISID, int ISMID, im GathererID) ( UNIRInstruction u-new UNIRInstruction < mRuleID, StepCtr, UNIRInstruction.INSTRUCTION~RECIVE, NodeID, 0, ISID, ISMID, null, null, GathererID) ; Log. instance().writeLog("flowManager.buildInstructionsSet", LogEvent.LOG-DEBUG, "Adding Instruction "+u) ;

mTmpInstructions. pushBack (u) ; f //*.. *.. **.. *.....,,.,,,,... t., *.. ,,.... *.... *...

//Helper method to add a receive of type Enhance 11'.........................................'.'-.... void AddEnhanceStep (int StepCtr, int NodeID, int ISID, int ISMID, ini GathererID, int FunctionID, byte [] inparam, byte [] outParam) {

UNIRInstruction u-new UNIRInstruction (mRuleID, StepCtr, UNIRInstruction.INSTRUCTION~ENHANCE, NodeID, FunctionID, ISID, ISMID,

inParam, outParam, GathererID) ; Log. instance().writeLog("FlowManager,buildInstructionSet", LogEvent. LOGDEBUG, "Adding Instruction"+u) ;

mTmpInstructions. pushBac) t (u) ; J 11..... ***.. *.........'.............'...'....'.. ".. //Helper method to add an receive of type Pipe Give //....***.... *** ..,**.. t.. *.,... .. **. *..... t

void AddGive5teplint StepCtr, int NodeID, int ISID, int ISMID, int GathererID, int FunctionID, byte[] inParam, byte[] outParam)

{ UNIRInstruction u-new UNIRInstruction (m RuleID, StepCtr, UNIRInstruction. INSTRUCTIONGIVE, NodeID, FunctionID, ISID, ISMID, inParam, outParam, GathererID); Log. instance (). writeLog ("FlowManager. buildInstructionSet", LogEvent. LOG~DEBUG, "Adding Instruction"+u) ;

mnTmpInstructions. pushBack (u) ; I //............ ........ *... *... *................... protected void buildlnstructionset ( Log. instance (). writeLog ("FlowHanaqer. buildlnstruct1onSet".

LogEvent. LOG DEBUG, "Build Instruction Set for Rule "+mRuleID) ;

EnhanceOpNode PriorOp-null int PriorNodeID. -1 ; mT ! npInstruetions-new SList < ) int StepCtr - 0 ; for (SListIterator I- ! nOpList. begin () ; CI. atEndO) ; I. advance f)) z EnhanceOpNode Opl- (EnhanceOpNode) I. get if ('Opl. getLastFlagO) if (Opl. getNodeID () !-PriorNodeID) ( Add Send Instruction to close prior node sequence

if [PriorNodeID'--1) ( AddSendStep (StepCtr, Opl. getNodeIDf), Opl. getISIDO, Opl. getISMID (), PriorOp. getGathererIDO) ; StepCtr++;

} //Add Recieve Instruction to start current node sequence AddReceiveStep (StepCtr, Opl. getNodeIDO, Opl. getISID Opl. getISMIDO, Opl. getGathererID()); StepCtr++; PriorNodeID-Opl. getNodeID () ; ) PriorOp-Opl // Add Enhancement Step AddEnhanceStep (StepCtr, Opl. getNodeIDO, Opl. getISID(), Opl.getISMID(), Opl. getGathererID (), Opl. getFunctionID (),

Opl. getOpKeyInAsArray (), Opl. getOpKeyOutAsArray ()) ; StepCtr++ ; I I //The Last Step, Add Send Instruction to CEM Check globol Pipe flag. If this Flow Target is some Pipe ISM, set the Instructione Type to INSTRUCTION~GIVE and set correct Node if fPriorOp'= null) { if (mIsTargetPipe) { //if target is in a different node, add a send and recieve if (mTargetNode !-PriorOp. getNodeIDO) f AddSendStep (StepCtr, m~TargetNode, 0,0, PriorOp. getGathererIDO) ; StepCtr++ ; AddReceiveStep (StepCtr, 0,0, 0, m~targetGatherer) ; StepCtr++ ; ) //adding the final Give instruction. Dont forget Binding Information byte bindinglnfo * buildPISMBindingInformation (); AddGiveStep (StepCtr, mTargetNode, mTargetIS, 0, m~TargetGatherer, 0, binding Info, null) ;

else else //Send Data to CEM

AddSendStep (StepCtr, Util. Comm. Node. NODETYPECEH, 0, 0, PriorOp. getGathererIDO) ; } StepCtr++ ; )

} 11. ".'**...'...... **.. "'.........'.'.'.....'***... "............ +.......

//doTopolgySortf) //mOpLi. st can be seen as a Directed Acyclic Graph (DAG), and an algorithm // (Introduction To Algorithms, Chapter 23) to sort the graph is applied.

The //result of this procedure is a list where the order of operations is correct, //meaning that all enhancments whose ouput is used as input happen before the //enhancments that use them.

//*..... .. *. ...... < . *..... *......,. < **...... < ..... . ... *......... protected void visitDFS (EnhanceOpNode Op) f Op. setColor (EnhanceOpNode. GRAY) ; Op. setDFSDiscoverTime < mTimeCtr) ; mTimeCtr++ ; //For Each Adjacent Vertex (All Output GUONS) for (Enumeration AdjVert-Op. getOutputGUONs AdjVert.hasMoreElements(); ) ( int OutGuon = ((Integer)AdjVert.netElement()).inValue(); EnhanceOpNode OutOp-mOpList. flndGuon (OutGuon) ; // Explore Edges Each (Guon, OutGuon) pair is an edge

if ( (OutOp !-null) & t (OutOp. getColorOEnhanceOpNode. WHITE)) ( visitDFS (OutOp) ; OutOp. setPredecessor (Op) ; ) ) Op. setColor (EnhanceOpNode. BLACK) ; Op. setDFSFinishTime (mTimeCtr) ; m TimeCtr++ ; mOpTmpList. pushFront (Op) ; ) 1/-------

? rotected void doTopolqySort () { //Create Temprary List to Store Ordered Ops. z--OpTmpList-new EnhanceOpSList m~TimeCtr - 0 ; //For each vertex

for (int i=O ; i < rn~OpList. sizeC) ; i++1 j EnhanceOpNode Opl - (EnhanceOpNodelrn~OpList. at (i) ; //if not used in DFS yet (Color White) if (Opl. getColorO-EnhanceOpNode. WHITE) ! visitDFSi Opi) : )

I //Temporary List is now the correct one, move it to normal list mOpList new EnhanceOpSList ( ~OpTmpList) ; : //.......... **............. *. *........ ***... ****. ******. *... **.... *... *. * //doNodeCoupling () // Node Coupling attempts to move enhancments that share the same gatherer closer //in sequence, as a way to reduce traffic between gatherers. This is done without // damaging the basic order of the enhancements established earlier in the Topology Sort.

//,.....,. ............ < .. **.... < < . **.... *..... < **. **.. . .. ...

//---------------------- //Find if inserting Op at Pos will cause sort order conflict protected boolean noHarmToSort (EnhanceOpSList OpList, int pos, EnhanceOpNode Op)

! for (int i-pos ; i < OpList. size () ; i++) ! EnhanceOpNode OpToCompare- (EnhanceOpNode) OpList. at (i) ; //Check if Op has input that uses Output of OpToCompare //Loop on all inputs of Op, see if the Guon value matches OpToCompare. Guon for (Enumeration EK-Op. getInIteratorf) ; EK. hasMoreElementsO ;)

{ OpInputParam InParam- (OplnputParam) EK. nextElementC) ; if (InParam. getGUONo OpToCompare. getGUON ()) ( //No insertion Possible return false : )

) I No Conflict Found, Insertion Allowed return true ; } //-------------- //find last position of first same node sequence in OpList protected int findSameNodePos (EnhanceOpSList OpList, EnhanceOpNode Op)

{ int NodeID = Op.getNodeID(); boolean StartSeqFound-false : int ctr = 0; int ResultVal - -1 ; for (SListIterator I = OpList.begin(); (!I.atEnd()); I. advance ())

( EnhanceOpNode OpToCompare = (EnhanceOpNode) I. get () ;

if (NodeID-OpToCompare. getNodeIDf)) t StartSeqFound-true ; ) else ( if C StartSeqFound) f ResultVal=ctr; break ;//get out of loop, we found what weere looking for )

} ctr++ ; ) return ResultVal :

} //----

protected void doNodeCouplingf) f //Create Temprary List to Store Ordered Ops. mOpTmpList-= new EnhanceOpSList () ; for (SListIterator I"mOpList. begin () ; ('I. atEnd ()),-I. advance ()) ( EnhanceOpNode Opi-fEnhanceOpNodetI. gett) ; II Add Op1 to the Temporary List, either at the end (No Node Optimization Possible) //or Insert in the middle (Node Optimization, Couple Nodes) int nl-findSameNodePos (mOpTmpList, Opl) ; if (nl - -1) ( //No Node Optimizations possible, add to end mOpTmpList. pushBack (Opi) ; } else , if (noHarmToSort (mOpTmpList, nl, Opal) ( Insertion OK, insert to group same node Ops together mOpTmpList. insert (nl, Opl) ; I else ( //Insertion would hurt the input- > output order, cannot inset, add to end mOpTmpList. pushBack (Opl) ; J } } //Temporary List is now the correct one, move it to normal list mOpList-new EnhanceOpSList < m OpTmpList) ; I //..... *.. < . t. .. *.... < .. ,. *.. < , .. *t. < . *....,. ... .

//Create an image of the Data to be produced by this Rule. This allocates ouput //positions for every Enhancement. These positions are used later to update the //input params.

11-.'.. "~...'. ". *'.'. -'. -.. *... ~-*--.'''.'..'''...''''. ~'. w.. *. ********.....

..... ~. *.. * protected void allocateUNIROutputDataO (

UNtRAllocation-new SList () ; //For Each OP for (int i-O ; i < mOpList. size () ; i++) ( EnhanceOpNode Opl-fEnhanceOpNode) mOpList. atfi) ; if (Opl null) i //For Each Output Parameter for (Enumeration EK - Op1. qetOutputKeys () ; EK. hasMoreElementsf) ;) ( Integer Key- (Integer) EK. nextElement first is key, 2nd is quon GuonKeyParam NewKey-new GuonKeyParam < Key, new integer (Opl-getGUONo if INewKey ! -nu11) ( UNIRAllocation. pushBackINewKey) ; ! ) ) 1 ) /} Based on the Unir Allocation, match every input parameter with the index in the //UNIRAllocation.

II propogateIndexTolnputl) Based on the Unir Allocation, match every input parameter with the index in the UNIRAllocation.

// < .,. .... *..,... ... *. *,. *. ,.......... * < *.......... ,. . *... protected int findIndexinAllocation < opinputparam Keyin) < GuonKeyParam CurInKey-KeyIn. getGuonKeyO ; for (int i-O ; i < UNIRAllocation. sizel) : i++1 ( GuonKeyParam KeyInAllocation * (GuonKeyParam) ONIRAllocation. at (i) ; if (CurInKey. equa1sIKeyInAllocaton)) I return i ; I return-1 I 11---

protected void propogatelndexToInputj) \ For Each OP for (SListIterator I"mOpList. begin () ; ( ! I. atEnd ()) ; I. advance ()) f EnbanceOpNode Opl- (EnhanceOpNode) I. get () ; //For Each Input Parameter for (Enumeration EK-Opl. getlnlterator () ; EK. hasMoreElementsO ; ) j OpInputParam InParam = (OpInputParam) EK. nextElement () ; //Get index of pair in UNIRAllocation int n =* findIndexinAllocation (InParam) ; if (n-1) { TBD, Error } InParam. setIndex (n + FLOW-OFFSET ) Log. 2nstance (). writeLoq ("FlowHanaqer. propogatelndexToInput", LogEvent. LOG-DESUG, "Propagated Input Op :"+Opl) ; I I //.............. .....,.,..... *..... *.......... ...... < ...... < ...., //For every Enhancement //Build List of Enhancments that need the output of some enhancement as Input.

/1 This information is needed later to weed out Non productive enhancements and to perform the Topological sort on mOpList protected void buildOpKeysOuto protected void buildOpKeysOutO ( Opl Loop : Loop on All Enhancements in m-OpList for (SListIterator I-mOpList. begin () ; CI. atEndO) ; 1. advance ()) { EnhanceOpNode Opl- (EnhanceOpNode) I. get () ; //Op2 Loop : Loop on All Enhancements in mOpList Compare all Enhancments to Opl (Except itself) for (SListlterator J. m~OpList. beqin () ; ( ! J. atEnd ()) ; J. advance ()) ( EnhanceOpNode Op2 - (EnhanceOpNode) J. get () ; //No need to compare Op to itself if (Opl !-Op2)

f //If Opl participates as input in Op2, add Op2 to list of //Enhancements that need Opl output. This list is managed in Opl. for (Enumeration EK-Op2. getinlterator EK. hasMoreElementsO ;)

f OpInputParam InParam" (OpInputParam)EK.nextElement() ; if (InParam. getGUONt Opl. getGUONO) (

Opl. addOutputParam (InParam. getKeyO, Op2. getGnON ()) ; I I ) ) //..... < ....... ..... ....... *... *...... < ............... *............

Reduce Duplicates. No need for duplication of process if another enhancement //already creates the needed Output. Keep one of the duplicates, delete the other.

* // Pass thru all input keys of all enhancments, replacing any reference to the

//Deleted GUON with the Surviving GUON.

//...... ., *. *,.... *...., .... < .... . < ...**... *.... .... ... *........ protected void redueeSameDescriptionOps () ( for (int i=0 ; i < mOpList. size () ; i++) ( EnhanceOpN04e Op1 = (EnhanceOpNode)m~OpList.at(i); //Move from end to Current Pos because we might delete an object for (int j-m~OpList. size()-1;j > i;j--) (

EnhanceOpNode Op2 - (EnhanceOpNode) m~OpList. atlj) ; if [Opi. getDescriptionf). equals) Op2. getDeseription ())) { Log. instanceo writeLogl"FlowManager. reduceSameDescriptionops", LogEvent. LOG DEBUG, ''Reducing Description : "+Op2.getFieldID()+":"+ Op2.getGUON()+":"+

Op2. getDescription ()) ; //Merge outfields of deleted quon with surviving guon Op1. addOutFields (Op2. getOutFields < )) ; int newGUON = Op1.getGUON(); int oldGUON = Op2.getGUON(); mOpList. remove (j) ;

//GUON Replacement Pass for (int k-0 ; k < mOpList. sizef) ;) c++) ( EnhanceOpNode Os3 = (EnhanceOpNode) OpList. ask), Op3. SwapInputKeyGUON (oldGUON, neMGUON) ; } //Mark for Garbage Collection Op2 - null ; ) ) ) ) 1/...... "..'. "". ""'**.'**'.. *. *****'.'.'***""'.... **.... ****. *. *. * //Reduce Error Guons, that do not produce an output needed as input by any other //guon. They can be identified if they are not "Last" and the output list is empty.

1/**.'****************'*. ****'.. *****************"". **. ****. ********** -. ****** protected void reduceErrorGuonsf) ( if (mOpList. size () > 0) ( for (int i-mOpList. size < )-l ; i > -0 ; i-) EnhanceOpNode Opl- (EnhanceOpNode) mOpList. at (i) ; //If Not Last Guon per some field... if ( ! Opl. getLastFlag ()) ( // Check if any Outputs are defined Enumeration ChkOutput =

Opl. getOutputGUONKeys () ; if ( ! ChkOutput. hasMoreElements ()) f Log. instance (). writeLogfFlowManager. reduceErrorGuons", LogEvent. LOG-DEBUG, "Reducing Bad Enhancment :"Opl. getGuon()+":'+Op1.getDescription());

//No Output Elements for this Guon, delete it mOpList. removed) ; //Mark for Garbage Collection Ope-nul I I I 1 ) 11.'*.'.. ****.'. ". ***.... ********'****......'.. ******.... *. *. *..... *....

//Build Translated OpKeyIn for every Enhancment II ! ! IT IS ASSUMED THAT Enhancements (From PERSISTENCE) are loaded sorted by //Field/Opnum !' ! //.*.....*.*.*............. *.... *..... *..........................*.

****.. *. protected void buildTranslatedOpKeyInf) ( for (SListIterator I-mOpList. begin [) ; ( ! I. atEnd () ; I. advance [)) I EnhanceOpNode Op" (EnhaneeOpNode) I. getO ? if (Op ! - null) l Op. translateOpKeyIn (mOpList) ; Log. instance (l. writeLog ("FlowManager. buildTranslatedOpKeyIn", LogEvent. LOGDEBUG, Op. toStringO) ; ) } 1 //....,............................. *....................................

//Build a Description for every Enhancment //' !" IT IS ASSUMED THAT Enhancements (From PERSISTENCE) is loaded sorted by //Field/Opnum !" Field/Opn= ******** protected void buildOpPeseriptionsf) l for (SListIterator I - m~OpList. begin () ; ( ! I. atEnd ()) ; I. advanced) ( EnhanceOpNode Op - (EnhanceOpNode) I. get () ; if (Op !-null) ( String S - getDescriptionl Op) ;

Log. instance (). wr1teLoq (-FlowManaqer. buildOpDescriptions", LogEvent. LOG DEBUG, "Op Description ., Op. get FieldID +" :"+Op. getGUON () + +S) ; ) ) ) //............................................... ***...... *.............. get Textual representation of the first EnhanceOpNode Input Key //......................*..**.......................................*..*.

**...... protected String getKeyTexttEnhanceOpNode Op) I String ResultVal-"' ; try z if (Op ! - null) I Arguments args-new Arguments (Op. getOpKeyInf)) ; int n- ( (UNIRShort) args. gettArgKeys. KEYKEYID)). getShortValueO ; ResultVal-Integer. toString < n} ; } ) catch (Exception e) () return ResultVal ; return ResultVal ; ) 11. "". "'.. -'-.. "''''.''...'.'.'.'''...''.''.. -'. -''-'. **********. ******. * Build a Textual description of a regular EnhanceOpNode Op II //the methods"parseArgs'* and"getDescription"are used in a recursive manner //.**..**..** *...*.**......*.............*..*.*. *.......................

**.. ±** protected String parseArgs (EnhanceOpNode Op) I String s try I //Loop On All Op Key In Arguments a-new Arguments (Op. getOpKeyln for (Enumeration EK-a. elements () ; EK. hasMoreElementsO ;) { //Add Description of Input Param OpNum

short OpValue- ( (UNIRShort) EK. nextElement()).getShortValue(); EnhanceOpNode CurrentOp = m~OpList.findOp(OpValue, Op. getFieldIDO) ; if ( !Op.equals(CurrentOp) ) { s += getDescription(CurrentOp)+"."; //Add Description of Input Param Key short KeyValue- ( (UNIRShort) EK. nextelement(j).getshortValue(); s ±Integer. toString (KeyValue) ;

//If More Param, delimit by comma if (EK. hasMoreElementsO) ( s-',' ? ) } else ( Log. instance (). writeLog ("FlowHanager. parseArgs", LogEvent. LOGERROR, 'Corruption Detected in Data (Illegal Recursion) *) ; throw (new Exception 0) : ) I catch (Exception EA) s s. "It ; } return s ; )

II....'.. **..'***'. **. *. ****.... *'* ;..'. *.... *. ****..... *............ *. *.

1/Build a Textual description of the EnhanceOpNode Op /I //the methods'parseArgs* and'getDescription"are used in a recursive manner 1/***... *. ;. *. *...'. ***'***'*. *.. *. **********..'******. *.. *.......,.. protected String getDescriptiontEnhanceOpNode Op) / String s = Op.getDescription(); //If No description was built yet, build it and save for next time if (s-null) ( if last Step for some field, just parse if (Op. getLastFlag ())

I s-parseArgs (Op) ; ) else if (Op. getOpNumO-0) { //First Op, has no function s-Op. getISID () +".'+ getKeyText (Op) ; ) else I //Function, Must show Func ID + Input Parameters. s-Op. getISID {) -'."+ Op. getFunetionIDf) " (' ; s parseArgs (Op) ; 5 ± ") " ; ) 11---- Save for next time Op. setDeseriptionfs) ; ) l return s ; //Scan A11 Rules, Loop On All Instructions, send matching instructions return s ; ) //***-****.... *. ******... ******. ********************.'**. *..... ***. ******* //Scan Ml Rules, Loop On Ml Instructions, send matching instructions to Gatherer 1/**...'*******.. *... *. ***~. *. ****. *****'. ******. ***-'... +. *. ** < .. * public void set ! nstructionsByGatherer (int GathererID, GathererController GC ) ( Rules Ru1eList - CEH. instance (). qetRules () ; if (RuleList !-null) ( For Each Rule for (Enumeration El"RuleList. elements !) ; El. hasMoreElements () ; ( //Build Flow for specific Rule RuleCurrentRule-f < Rule) (El. nextElementf))) ; if ( (CurrentRule !-null) & : (CurrentRule. getEnabledf) - true)) z int RuleID-CurrentRule. getRuleIDO : setInstructionsByRuleByGatherer < GathererID, RuleID, SC) ; ) )

i I 1/***.'*. *'..'.........'....... *********"***......'........... *~.... +...

Send All Instructions per Rule/per Gatherer to the Gatherer //Thru the GathererController 11...'... *.. ". -'.... "'....'. *..... ".'. "'. ".. ". "......... -. *. *. *... synchronized public void setInstructionsByRuleByGatherer int GathererID, synchronized public void setInstructionsByRuleByGatherer (int GathererID, int RuleID, GathererController GC) SList Tmplnstructions- (SList) mInstructions. get (new Integer (RuleID)) ; Vector v-new VectorO ;//H (tld ONIRInstruction belonging to some Gatherer //Loop on All instructions of a Rule, Add Gatherer Matches to v, Send v to GC for (Enumeration EOps-Tntplnstructions. elements () ; EOps. hasMoreElementst) ;) ! UNIRlnstruction Op- OMIRInstruction) EOps. nextElement () ; if (Op. getGathererIDo-GathererID) { v. addElement (Op) ; } I try ( Log. instance 0. writeLogC'FlowManager. setInstructionsByRuleByGatherer", LogEvent. LOGDEBUG,'Disable Flow"+RuleID*'on * + GathererID) ; GC. disableFlow (RuleID) ; if (v. size () > 0) I Log. instance f). MriteLog ("FlowManager. setlnstructionsByRuleByGatherer", LogEvent. LOGDEBUG,'Upload Flow"+RuleID+"on"+ GathererID) ; GC. sendFlowData (RuleID, new UNIRInstruction (v) Log. instance () writeLog ("FlowManager. setlnstructionsByRuleByGatherer", LogEvent. LOGDEBUG,"Enable Flow *+RuleID+"on *'+ GathererID) ; GC. enableFlow (RuleID) ; I J catch (Exception e) { !

//***************************************************************** < ***** //Pack the Binding Information between current Flow to the input //parameters of the target Pipe ISM.

This information is added to the Give instruction and sent to the //Gatherer where it is used to pass data from the output UNIR of //the current flow to the input parameters of the target flow.

//RETURNS // a byte array containing Binding information in the following format: pairs of shorts made of the index of the data in the output UNIR, followed

//by the key value of the target input parameter.

11... -.... *........ **'**..... -*'... ****..... ~..'***'*-... **************..

* t try ! try Arguments args-new Arguments Vector v-new Vector () ; //Loop on All Enhancements

for (SListIterator I-mOpList. begu) [) : f'l. atEndO) ; I. advance ()) { EnhanceOpNode Op- (EnhanceOpNode) 1. get if (Op null) t int Index-0 ;//Position of data in Output Unir short Key-0 ; Key ID of data in Output Unir int inKey-0 ;//Key ID of data in Input Parameters //the binding process will //"take the UNIR in the Index position from the flow data and put // in the PISM input parameter in inKey" One to many relation : One input to Many output //DEFINE INPUT: Get Index & key of first input key

Enumeration EK-Op. getInIteratorO ; if (EK. hasMoreElementsf)) z OpInputParam Inparam = (OpInputParam)EK.nextElement(); Index-InParam. mKeylndex ; Key * (short) InParam. mnKeyID :

}

//DEFINE OUTPUT (S) Loop on all Out Values for (Enumeration EFlds -

Op. getOutFieldsf). elements 0 ; EFlds. hasMoreElementsO ;) I String FieldID- (String) EFlds. nextElement () : Log. instance().writeLog("FlowManager.buildPISMBindingInformation", LogEvent. LOGINFO, "Got Pipe Binding :"+ FieldID Index +"Key) ; //in PISH, FieldID is a String Representation of the KeyID // defined as string for compatabiliy with Fields (see buildFieldIndex) inKey-Integer. valueoft FieldID ). intValueC) ; //Add to Arguments ;

args. put (ArgKeys. ARGKEYE ! ! HANCEMENT TOPNUM, new UNIRShort ( (short) Index)) ; args. put (ArgKeys. KEYKEYID, new UNIRShort((short)inKey)); )

} return args. toByteArrayO ; ) catch (Exception e) i Log. instance (). writeLog ("FlowHanager. buildPISMBindingInformation", LogEvent. LOG-ERROR, "Error in Binding Info, "+e); Debug.writeStackTrace("FlowManager.buildPISMBindingInformation", e) ; ) return null ; )

1/9~...... "..'.'... *...'....'. ". ""'. "***. """............ +..... *..

* //Builds a vector of all DB fields that participate in a specific Flow.

(The current flow, in m~OpList) This list is built of FieldIndexStruct elements and sent to the Merger.

//The Merger uses this information in processing incoming UNIRs from this //flow.

//The vector is a cross reference between the DB FieldID and the position the data to be put in the field from the incoming UNIR for this flow.

//.. *........ * .....**. **.. *..... * < *............. ... .. t...........

* void buildFieldIndex () { Vector v-new Vector () ; //Loop on All Enhancements

for [SListIterator I-mOpList. begin () ; CI. atEndO) ; I. advance ()) ; EnhanceOpNode Op- (EnhanceOpNode) I. get () : if (Op null) ( int Index-0 ; short Key - 0 ;

Enumeration EK"Op. getInIteratorO ; //Get Index ; key of first input key if (EK. hasMoreElementsf)) ( OpInputParam InParam (OpInputParamIEK. nextElementC) ; Index-InParam. mKeyIndex ; Key- (short) InParam. mKeyID ; ) //Loop on all Fields Out, add all fields in list to vector with // Index α ket stored earlier. for (Enumeration EFlds- Op. getOutFields (). elements () ; EFlds. hasMoreElements () ; ) ( String FieldID- (String)EFlds.nextElement(); v. addElement(new FieldIndexStruct(FieldID, Index, Key)) ;

} ) )

if (v. sizes > 0) f //CallMerger Merger m = CEM.instance9).getMerger(); Log. instance (). writeLog(new LogDebug("Vector:=n"+v)); m. addRule(m~RuleID, v); ) else

//Call Merger Merger m-CEM. instance (). getMerger () ; Loq. instance (). writeLog (new LogDebugfDeleting Rule"+mRuleID)) ; m. deleteRule (mRuleID) ; I // ...... < ....... ....,... *....... *.... < . ............... < .............

Get All Enhancments with matching FieldID Return Vector of affected Rules //.......... *.......,........... ...,.....,.......... *.............. public Veecor getRulesWithField (String FieldIB) ( Log. instance (). wri teLog ("FlowHanager. getRul esWi thRul eFi eld", LoqEvent. LOG~DEBUG,"Get Enhancements with Field"+FieldID) ; Vector ResultVal = new Vector !) ; OrderedSet AftectedRules - new OrderedSetlfalse) ; Vector ToDel-new Vector f) ; Enhancements EnhancemenfLisc"CEM. mstaneeO. getEnhaneementsO ; fortEnumeration E-EnhaneementList. elementsO ; E. hasMoreElementsf) :) l Enhancement CurrentEnhancement - ( (Enhancement) CE. nextElement ()) I ; if ( < FieldID. equals ; CurrentEnhancement. getFieldIDO))) I AffectedRules. add ( new Integer ( CurrentEnhancement. getRuleIDf])) ; //Set Result Val for (Enumeration AR =* AffectedRules. elements () ; AR. hasMoreElements () ;) C Integer RuleIDObj- (Integer) AR. nextElement ResultVal. addElement (RuleIDObjt ;//add Rule to Result Set i return Result Val ; J .......................................... w.......................

//Delete All Enhancments with matching FieldID, RuleID from List 1/"""... "".'........... *.. 9.........................................

public'.'oiQ deleteRulesHithField (Vector RuleList. S : : ino FieldID) ! for fine i=0 ; i < RuleList. size () ; i±) ( int Ruled < (Integer) (RuleList. elementAt (i))). intValueO ; deleteRulesWithRuleField f RuleID, FieldID) : f .

Delete All Enhancments with matching RuleID, FieldID //Return Vector of affected Rules 1/"""""". "". """""""""'.. ""'. ""'"................... public vector deleteRulesMithRuleFieldfinc RuleID, String FieldID ! 1 Log. instance (). writeLog ("FlowManager. deleteRulesWithRuleField", LogEvent. LOG DEBUG,"Delete Enhancements with Fj. eld *+FieldID) ; vector ResultVal-new Vector () ; OrderedSet AffectedRules"new OrderedSet (false) ; Vector ToDel-new Vector U ; Enhancements EnhancementList-CEM. instance (). getenhancements forfEnumeration E-EnhancementList. elements () ; E. hasMoreElementsO ;) ! Enhancement CurrencEnhancemenc"f f Enhancement) tE. nextElemenn))) ; if : i FieldID. equalst CurrentEnhancement. getFiedID () n ( (RuleID"-CurrentEnhancement. getRuleIDf ; t i // ( (RuleID'* CurrentEnhancement. gecRuleID ()) I fRuleID .. -1))) ToDel. addelement (Current Enhancement) ; J for ! in : i"0 : i < ToDel. sizef) : i) Enhancement CurrentEnhancement t (Enhancement) TcBel. eiementAte AffectedRules. addl new Integer (CurrentEnhancement. getRuleIDO) ,.

Log. instance)). writeLog (*FlowManager. deleteRulesWithRuleField", LogEvent. LCGDEBUG,''Deleteing"+CurrentEnhancement) ; try (CurrentEnhancement. deleteO :) catch (Except :. on e) fl i

//Set Result Val for (Enumeration AR-AffectedRules. elements ) : AR. hasMoreElements () ;) f Integer RulelDObj-i Integer) AR. nextElement ResultVal. addElemencfRuleIDOb]) ://add Rule to Result Set J return ResultVal ; //........... *...... *,............. < ... *,.. *.. *... *. **.... < .... *..... *...

//deleteRuleEnhancements .............. public void deleteRuleEnhaneementsfint RuleID) Vector ToDel - new Vector !) ; Enhancements EnhancementList-CEM. instance ). getEnhancements () ; for (Enumeration E-EnhancementList. elements () : E. hasMoreElementsO ;) ( Enhancement CurrentEnhancement- ( (Enhancement)fE. nextElementO)) ; if (CurrentEnhancement. getRuleID !)"* RuleID) (, ToDel. addElement (CurrencEnhancement) : t t : or (-n- : i=O ; i < ToDel. sizef) : l++) Enhancement CurrentEnhancement'* (Enhancement !"cDel. elementAt -) ; try ( CurrentEnhancement. delete ! catch (Exception e) ( Log. instance (). writeLog (new LogDebug ("Deiete Rule Enhancement Failed...'*+)) ; Debug. writeStackTrace ("FlowManager. deleteRuleEnhancements", e., ; I I II'. """"""'.'. """""""'. """""""""...................

Delete All Erhancments with matching ISID :/ If All enhancements per rule are deleted, disable the Rule //If Trigger is Deleted, Delete Rule // Return Vector of affected Rules

/**. *... **......... *..,............... *... *.. *.................... < ...,. public Vector deleteRulesWithIs(int SIID) Log. instance().writeLog(new LogDebug("Delete Enhancements for "+ISID)); Vector ResultVal-new Vector f) OrderedSet AffectedRules = new OrderedSet(false); OrderedSet AffectedRuleFields = new OrderedSet(false); Vector ToDel-new Vector () ; //Loop on all Rules

Rules RuleList = CEM. instanceo. getRuleso ; forlEnumerat10n El - RuleList. elements () ; El. hasMoreElementst) ;) I /1 Compare Trigqer 10 to the ISID being deleted Rule CurrentRule- ( (Rule)(El.nextElement(1)),

int RuleID-CurrentRule. getRuleIDO ; if (CurrentRule. getTriggerIDo-ISID) ( //yes. found a match. Add to Affected List //and delete the whole Flow

AffectedRules. adds new Integer f RuleID ToDel. addElement (CurrentRule) : deleteRuleEnhancements (RuleID) ; I I Delete from Collection for tint iO : i < ToDel. size < ) ; i < -) Log. lnstance)). writeLogfnew LogOebuq ("Delete Rule vith Trigger "+IS : : ; Rule CurrentRule (Rule) ToDel. elementAt (i) : try f CurrentRule. deletef) ;) catch (Exception e) t) J //Loop on All Enhancements, find Rule/Feild pairs affected by IS removal Enhancements EnhancementList = CEM.instance9).getEnhancements();

forfEnumeration E2 < = EnhancementList. elements f) ; E2. hasKoreElements () ;) [ Enhancement CurrentEnhancement - ( (Enhar. cement) (E2. nextElement < ))) ; if (CurrentEnhancement. getISIDO ** ISID) f Log. instanee (t. writeLog (newLogDebug ("Adding Rule Field to Affected List"+

CurrentEnhancement.getRuleID()+" : "+ CurrentEnhancement. getFieldIDo AffectedRuleFields. add(new Pair(new Integer

(CurrentEnhaneetnent. getRuleIDo), Current--nhancement. getFieldrDt) t ) forfEnumerat on E3 * AffectedRuleFields. elements ( : : E3. hasMoreElements < ) ;) ( Pair CurrentPair = { (Pair)(E3.nextElement())); //Delete All Enhancemnts for this Rule t FieldID

deleteRulesWithRuleField f f Integer) CurrentPair. first). intValuef), tScnr : ?) CurrentPair. seconQ) AffeccedRules. add ( (IntegeriCurrentPair. first) : ) // Set Result Val for (Enumeration AR = AffectedRules.elements() ; AR. hasMoreflements(); ) Integer RuleIDObj = (Integer)AR.nextElement(); ResultVal. addElement(RuleIDObj); // add Rule to Result Set

} /1 Return Affected Rules return ResultVal ; //. ....... ..,.... ..... *.. ... ...... *,.. ................ Delete from the given Rule all the Fields that have Enhancements using at least 1 from the given set of Trigger Input Keys. //***********************************************************************

public OrderedSet deleteRuleFieldsWithTriggerKeys (int F.'-leId, Vector TriggerKeys) ( Log. instance f). writeLog (new LogDebug :"deleceRuleFieldsWithTriggerKeys"+"Ruleld-"+ Ruleld+ ", TriggerKeys-"+ TriggerKeys)) ; int OpNum-0 ;

OrderedSet AffectedRule - new OrderedSetCfalse) ; OrderedSet AffectedFields-new OrderedSet (false) ;

. : (TnggerKeys-null !) TriggerKeys. si : e ()'--0) return null ; Log. instance !). wrceLog (new LogDebug ("Add Rule to Affected List"+ Ruled)) ; AffectedRule. add (new Integer (RuleId) //Get all Enhancements Enhancements EnhancementList = CEM. instance O. getEnhancementsf) ; for (Enumeration E-EnhancementList. elements () ; E. hasMoreElementsf) ; í Enhancement CurrentEnhancement" (Enhancement) (E. nextElemento) if [CurrentEnhancement. getRuleIDO-RuleId) //Search InputKeys for Trigger ones (OpNmm-O ! ( EnhanceStruct CurrentEnhanceStruct-new EnhanceStructCCurrentEnhancementJ ; Vector ParsedOpKeyln-CurrentEnhanceStruct. getParsedOpKeyInO : III : nt CurrentOpNum ; int CurrentOpKey : for tine i*'0 ; KParsedOpKeyIn. sizef) ; i+='2) r CurrentOpNum- (fInteger) ParsedOpKeyln. elementAt (i)). intValuef) : CurrentOpKey- (f Integer) ParsedOpKeyIn. elementAt (i+l)). intValue () ; if (CurrentOpNum-OpNum & t TriggerKeys. contains (new Integer (CurreniOpKey))) f Log. instance writeLogfnew LogDebug 'Adcing Rule Field to Affectea List"- : ; rrentEnhancement. qetRuleIDo+" Curren tenhancement. getFieldrD () AffectedFields. addf CurrentEnhanceStruc t. getFieldIDf)) ; break ;//pass to next Enhancement I )//for over ParsedOpKeyIn }//if (CurrentEnhancement. getRuleID (--Ruleld) )//loao over Enhancements for (Enuneration E-AffectedFields. elements () ; E. hasMoreElementsf) :) ! String Fieldid- (String) E. nextElement [) ; //Delete All Enhancemnts for this Rule : 21dID deleteRulesWithRuleField (RuleId, Fieldid ! ; 1 returs AffectedRule ; )//deleteRuleFieldsWithTriggerKeys

//*********************************************************************** //Delete in all Rules all Fields including an Enhancement from the given // ISID with Functionld equal to one from the given set.

/......,.............................................................. public CrderedSet deleteRulesFieldsHithISFunetions (int ISm, Vector ChangeaFunctions) I Log. instancef). writeLogfnew LogDebug ("deleteRulesFieldsWithISFunctions''+''ISID-"+ISID+ ", ChangedFunctions="+ChangedFunctions !); Vector ToDel-new Vector () ;

OrderedSet AffectedRules = new OrderedSet (false) ; OrderedSet AffectedFields = new OrderedSet(false); if ( ChangedFunctions==null !! ChangedFunctioins.size()==0) return null ;

Enhancements EnhancementList-CEM. instancel). getEnhancementsfi ; : or (Enumeration E-EnhancementList. elementsf) : E. hasMoreElementsf ; ; Enhancement CurrentEnhancement-( (Enhancement) (E. nextElementIJI I ; if (CurrentEnhancement. getISIDO = ISID tt ChangecFunctior. s. contains (new integer (CurrentEnhancement. getFunctlonIDf)))) I //toDel.addElement(currentenhancement); Log. instance().writeLog(new LogDebug("Adding Rule Field to AffectedList"+

CurrentEnhancement. gecRuleID ()-'' : *+ CurrentEnhancement. getFieldID 0)) : AffectedFields.add(new Pair(new Integer ! Currentenhancement.getRuleID()),

CurrentEnhancement. getFieldIDf))) : I J forlEnumeration E2 - AffectedFields. elements () ; E2. hasMoreElements () ;) t Pair CurrentPair = ( (Pair) (E2.nextElement()) ); //Delete All Enhancemnts for this Rule ; : leld1D deleteRulesWithruleField( (f Integer) CurrentPair. first). intValue(), (Str ;) CurrentPair. second) ; Log. isntance().writeLo(new LogDebug("Add Rule to Affected List"-CurrentPair. first)) : AffectedRules. add( (Integer)CurrentPair.first );

I return AffectedRules :

//deleteRulesFieldsWithISFunccions .................. *............... I........ *.......... !..............

In all Rules find all Enhancements which met the fllowlng conditions : ! SID- < given ISID > , FunctionID- < one from the given lisez For all these Enhancements the Functionld is replaced with new one.

: I The "new" Id 1S placed immidiately after the matching"eld"Id in the TI, h //2-nd input parameter.

//**..,.. *.... < . *. ..... < . *.... < .. < ......... **... *****... *. **.. *. *...... public OrderedSet editEnhancementsFunctlOnIDWithFunct1onIDIlnt ISID.

Vector nffectedFunctionIds) throws Exception { Log. instance (). writeLog (new LogDebug feditEnhancementsFunctionIDWithFunctionID"+"ISID-"+ISID+ ", AffectedFunctionIds-"+AffectedFunctionIds)) ; OrderedSet AffectedRuleSet-new OrderedSet (false) ; if f AffectedFunctionIds-null I AffectedFunctionIds. sizeO-O) { Log. instancef). MriteLogfnew LogDebug ("editEnhaneementsFunctionIDWithFunctionID :","There are no changed function IDs'). ; return null ; } Enhancements EnhancementList-CEM. instance getenhancements forfEnumeracion E-EnhancementList. elementsO ; E. hasMoreElements t ;) Enhancement Curren tenhancement (Enhancement) (E. nextelemenr for tint 1=0 ; i < AffectedFunctlonlds. sizes ; i±2) if (CurrentEnhancement. gecFunctionIDO-- (I Integer) AffectedFunctionlds. elementAt (i)). intValueO) ! int OldFuneId'* CurrentEnhancement. getFunctionIDf) ; int NewFuncId'* < i Integer) AffectedFunctionIds. elementAtli)). intValueO ; Log. instance C). writeLog ! new LogDebug ("Replace Functionid "+OldF1 ; ncld + "with"+ NewFuncId+"in Enhancement =''+CurrentEnhancement)) ; Current Enhancement. edit CurrentEnhancement. setFunctionID (NewFuncId) ; CurrentEnhancement. update () ; Log. instance (). writeLog (new LogDebuq ("Add Rule to Affected List"-urrentEnhancement. getRuleIDO)) ; AffectedRuleSet. add (new Integer (CurrentEnhancement. getRuleIDO)) ;

! t return AffectedRuleSet : ) )//elass Illclass //Internal Helper Classes to support Flow computations //...... . *.. ... .................. < ......... *.. *....................... ..................................................

// //........................... *.......................................

Class EnhanceOpSList extends SList ! //.....................................................................

.

// public EnhanceOpSList () I super () ; i /..,................................................,...,.............

II .............................. public EnhanceOpSList (int n) ( super In) ; I ............................................................

.

II

public EnhanceOpSList (EnhanceOpSLst s' ( super (s) : I ............................... *......

II 1/""""""""""". *'. "".'... """""""'"............. *.... public EnhanceOpNode flndOp lint OpValue, String OpField) { for tint i-O ; i < sizei) ; i++) I EnhanceOpNode Op- [EnhaneeOpNode i at < i) ; if fOp. getOpNumo-OpValuel I //Field does not matter if we are searcning for OpNum C if ( (OpField. equals Op. getFieldlOO)'H ssopvalue == f return Op ; ) t } return null ) ......................................... public EnhanceOpNode findGuon fine GuonValue) j for (int i-O ; i < size () ; i++) EnhanceOpNode Op =* (EnhanceOpNodelat (i : ; if (Op. getGUONt)-GuonValuei f return Op : ) , return null }

I ......... v...................... //.... .................... t......,........ *.".... *. *"...... **. *......

//Object that represents a single Input Parameter of a single Enhancement Operation.

1/Every Enhancment should maintain a set of OpInputParam .......................................... class OpInputParam f //Original Opnum of Enhanememt owning the Input Param public int m~OpNum - 0 ; 1/ArqKey Value of the Output Parameter public int mKeyID = 0 ; 1/GUON value of Enhanonent owning the Input Param public int m OpGUON-O : //Index of Input Parameter. Defined from the'UNIR Allocation" public int m~KeyIndex - 0 ; //////////// //Constructors ////////11// public OpInputParam int OpNum, int KeyID, int GUON) f mopNum-OpNum ; mKeyID = KeyID ; : ~OpGUON-GUON : mKeyIndex-0 : I /11/111//1/1 //Get Set /1/111111111 public int getGOONO (return mOpGUOK ;) public void setGOON (int Value) (mOpGUON-Value :I public int getkeyo (return mKeyIC ;) public void setkeytint Value) (mKeyID-Value :) public GuonKeyParam getGuonKey () (return new GuonKeyParaa (new Integer < m~KeyID), new Integer (m~OpGUON)) public mnt getIndexo) return mKeyIndex :) public void setlndexiint Value) ( < mKeyIndex-Value :I

...........

.

/ 'I : lass GuonKeyParam extends Pair public int m GUON-0 ; public int m-Key-0 ; public GuonKeyParam () i super 0 : l public GuonKeyParam (Integer Key, Integer GUON) I super (Key, GUON) ; } public int hashCode (1 ( Integer nl- (Integer) first : Integer n2 = (Integer) second ; return ( (nl. intValueO 16) + n2. intValel)) ; public boolean equals (Pair pair) { Integer n1 - (Inteqerlfirst ; Integer n2- (Integer) pair. first ; Integer n3- (Integer) second ; Integer n4- (Integer) pair. second, return ( (nl. intValue () -- n2. intValue zizi n3. 1nalue () -- n4. intValuel) f) ; r 1 Object that represents a single output Parameter of a single Enhancement Operation.

// Every Enhancment should maintain a set of OpOutpucPara //***********************************************************************

class opOutputParam I //Set of Key/GUON pairs, elements are JGL Pairs public OrderedSet mKeyGUONList-new OrderedSet (false, ;//No Duplicates //Set of Keys. elements are Integers public OrderedSet mKeyList = new Ordered (false ); // No Duplicates //Set of GUONs. elements are Integers public OrderedSet mGUONList-new orderedSet (false) ;//No Duplicates //*********************************************************************** Get Set //*********************************************************************** public Enumeration getGUONs() (return m~GUONList.elements();) public Enumeration getKeys() (return m~KeyList.elements();) public Enumeration getGUONKeys() (return m~KeyGUONList.elements();) //*********************************************************************** FFU : //public Keys getGUONKeys (GUON) //public GUONs getKeyGUONs (Key) :

................. *.....................................................

II II""""""""'. ".'..'. """"'. """'. """".................... public void addKey (int Key, int GUON) ! Pair KeyGuon = new GuonKeyParam (new Inceger (Key), new Integer (GUON) mKeyGUONList. add KeyGuon) : mKeyList. add (new Integer (Key)) ;

mGUONList. addfnew Integer (GUOM)) : I //

//........................................................................ public byte ] getOpKeyOutAsArrayO ( try I Arguments args * new Arguments f) ; for (Enumeration EK-getKeysO ; EK. hasMoreElements () ;) I Integer outKey" (IntegeDEK. nextElemenco ; args. put (ArgKeys. KEY~KEY~ ! D, new UNIRShort ( (short) outKey. intValue ())) ; I return argas. toByteArray () ; z catch (Exception e) (I return null } rexurn null ; , //Object that representes a single Enhancement Operation.

Flow Manager should maintain a set of EnhanceOpNode //....................................................................... ass EnhanceOpNode extenas EnhanceStruct //Unique Identifier of the Op, Created at Runtime."Global Unique Cp Num" int mGUON-0 ; //String Description of the Operation, Created"Recursively"at Runtime.

String m~Description-null ; //OrderedSet version of EnhanceStruct. OpKeysIn. Each element is of type OpInputParam OrdereaSet:n~TranslatedOpKeysln-new OrderedSe* : ; false ; ;//r) o Duplicates Vector m TranslatedOpKeysInVector-new Vector () ; //List of Fields that are the"End Product"of this GUON. null in most objects, has value only for //"End"Objects. Becuase of Optlmizations, one GUON can feed more than one field. Elements are Strings //that were the FieldID of Original Enhancement.

; : dereaSet CutFields-new OrderedSett false) ; //Object that manages the lists of output param keys of this Enhancment OpOutputParam mOpKeysOut-new opOutputparamo ; ISM, Gatherer t Node ID of this (derivea from EnhanceStruct. IS : D) me mISHID-0 ; m nt mGathererID-0 ; int m~NodeID. 0 ; //Flag if this GUON was originally the"lase* (Last OpRum for a FIELDID) boolean mIsLast = false ; //The list of EnhanceOpNodes can be viewed as a Directed Acyclic Graph (DAG), with the GUONS as vertices //and the pairs of (GUON n, GUON m using output of n as Input) as edges.

//The Following are Variables Used in a TOPLOGICAL SORT using

Depth F. : st Search (DFS). final static int WHITE-0 : final static int GRAY-1 ; final static int BLACK-2 ;

public int aColorStatus-WHITE ; public EnhanceOpNode m-Pi-null ; Predecessor Node. if null,- : hen root (Several roots possible) public int m~DiscoverTime = 0 ; // Start Time of DFS on this GUON public int m~Finshtime = 0; // End Time of DFS on this GUON (and adjaceny list) //////////// Constructors //////////// public EnhanceOpNode Enhancement E) ( super (Ei : ) public EnhanceOpNode(int RuleID, String FieldID, short OpNum, int ISID, int FunctionID, byte [] OpKeyIn, Vector ParsedOpKeyIn

i super (RuleID, FieldID, OpNum, ISID, FunctionID, CpKeyIn.

FarsesOpKeyIn' ; , 11/1//1/111/ Operations ////////////

//,..................... *.................. ........... *. *...............

Create a list of OplnputParamin m~TranslatedOpKeysIn from the pair //arguments in mOpKeyIn. Add the corresponding GUON for every OpNum in the OpKeyln. opium //in she OpKeyin. public void translateOpKeyIn (EnhanceOpSLlst OpList) ( try f Arguments a-new Arguments (getOpKeyIn ()) ; for (Enumeration EX-a. elements () ; EK. hasMoreElementst) ; ) i short OpValue-' (UNIRShort) EK. nexElement ()). getShortValuef) ; short KeyValue- (UNIRShort) EK. nextElemenc ()). getshortValue () ; OpInputParam tmp-null ; EnhanceOpNode CurrentOp OpList-findOp (OpValue, FieldID) ; emp-new OpInputParam (Opvalue, KeyValue, Curren top. getGUON m TranslatedOpKeysIn. addftmp) ; m~TranslatedOpKeyslnVector. addEleent (trnpl ; ) ) catch (Exception EA) ) i //....................... *.......... ...................................

//pass thru mTranslatedOpKeysln, replacing occurences of oldGOC : : with newGUOR //. *....................... < .............................................

. public v01. d $waplnputKeyGUON (int oldGUOH, int newGUON) ( for (Enumeration EK * mTranslatedOpKeysln. elements () ; EK. hasKoreElementsO :) f OpInputParam InParam- (OpInputParam) EK. nextElementt) ; if (InParam. getGUONt)"=-oldGUON)

f InParam. seiGUON (newGUON) ;

..

Add Op to Lists of output of current EnhanceOpNode //....................................................................... public void addOutputParam fine Key, int GUON) I mOpKeysOut. addKey (Key, GUON) ; I 1/"". """"""""""""""""""""""'"...................

Get Set ................. < ........................................... .... t.. public int getGUON () freturn mGUON :) public void setGUON (int Value) (m~GUON = Value;} public int getGathererIDO (return mtGathererID ;) public void setGathererID(int Value) (m~GathererID = Value;) public int getNodeID () ireturn m~NodeID;) public void setNodeID(int Value) (m~Description = value ;) public coolean getLastFlaç() (return m~r sLast ;) public void setLastFlag(boolean value) (m~IsLast = value ;) public String getDescription() (return m~Description ;} public void setDescription (String value) (m~Descritpion = value ;) public int getISMIDO (return mISMID ;)

public void setISMIDfint Value) Im ISMIDValue ; i //.............. ........................................................

Topology Sort .............

public mi oeiColorO : return m~ColorStatus ; t publicvoidsetColor(intValue0(m~ColorsTatus=

value ; I public void seiDFSDiscoverTimeflnt Value) mDiseoverTimeValue ; I public void secDFSFinishTimednc Value) (mFinishTimeValue ;) public void setPredecessor (EnhanceOpNode Value) (mPi-Value ;) //... *...... < ........... ..... ... < ... *.. *........................... public void addOutFieldsf String FieldID) (mOutFields. add < FieldID) :} public void addOutFieldsfOrderedSet FieldList) < mOutFields - mOutFields. union (FieldList) ;) public OrderedSet getOutFields() freturn : nOutFields ;) public Enumeration getlnIteratort)) return : nTranslatedOpKeysInVector. elements () ;) public Enumeration getOutputGUONs() (return m~OpkKeysOut.getGUONs();) public Enumeration getOutputGUONKeys() (return m~OpKeysOut.getGUONKeys();)

public Enumeration getOutputKeys ( ! return mOpKeysOut. getKeys () ;) //*********************************************************************** Convert OpKeyln structure to 3 oyte array for te UNIRInstruction

........................ *.. *............ *............................ public byte [] getOpKeyInAsArray ) f try ! Arguments args = new Arguments(); for (Enumeration EK = getInIterator(); EK. hasMoreElements(); ) ( OpInputParam InParam- (OpInputParam;EK.nextElement(); args. put(ArgKeys.ARKEY~ENHANCEMENT~OP~NUM, new UNIRShort ((short)InParam.m~KeyIndex)) ;

args. put (ArgKeys. KEYKEY : :. new BNIRShort f tshort) InParam. mnKeyID)) ; } return args. toByteArray (i ; t catch (Exception e) (t return null ; } ...

//Convert OpKeyOut structure to a byte array for the UNIRInstruct10n .......................... *. *.......... I................... public byte I] getOpKeyOucAsArrayO ( return mOpKeysOut. getOpKeyOutAsArray () ; } 11'... "'. """'. ". ".. "..'.'.. *. ".'. "'. """'.....................

String representation of this Enhancement /....................................................................... punic String tostringo f String s-m GUON +"," ; for (Enumeration EK-getlnlteratorf) : EK. hasMoreElementsO ;) { OpInputParam InParam - (OpInputParam) EK. nextElement () s ±" ("+ InParam. m-OPGUON +" :"+ InParam. mOpNum-t'* :'* + InParam. mKeyIndex +" :" + InParam. mKeyID < -")" : ! s *-"j. r : for (Enumeration EK2. m~OutFields. elements () ; EK2. hasMoreElementsl) ;) { String OutFld- (StringtEK2. nextElement ( : ;

s OutFld+"I" ; s± "1, en ; for [Enumeration EK3-getOutputGUONKeysO ; EK3. hasMoreElements < ) ;) ( Pair GuonKey-'Pair) EK3. nextElement () ; s + ' f"*GuonKey+")" ; s ±"],"+ mISMID t" :"-'-mGathererID +" : -'- mModeID ; return s : return s ; I II'*"W""""'"'*"""'*'*"'+"""""'. """'9""'"""""" II to support special operation of slist public boolean equals (Object obj) return (m~GUON a. (CEnhanceOpNode) obj). getGUON ()) ; I to support special operation of slist //.......................................................................

//to support special operation of slost public int hashCode () t //byte vl- (byte) (mGUON & OxOOOOOOff ; byte vl-0 ; int v2-0 ; if CmFieldID. equalsf'"')) v2-mFieldID. hashCode < ) t OxOOOCffff ; return (vl 24) + (v2 < < B) + mOpMun) : ) !

./.... I...................... I........................ v..........

//.................................................. .....................

This application is a divisional application of UK patent application number 0002516.3. The aspects and features that were set out in the claims of the parent application as filed are repeated below as numbered clauses.

Clauses 1. A system for tracking network session information, the system comprising : an information source module having a source information input and a standardized information output, a source information corresponds to network usage information, a standardized information corresponds to the network usage information transformed into a standard format ; a first program having at least a first standardized information input and an enhanced data output, a first standardized information input corresponding to the standardized information, an enhanced data corresponding to the standardized data after at least a partial transformation, the at least partial transformation being defined according to a data record format; a second program having at least a first enhanced data input and a data record output, the first enhanced data corresponding to the enhanced data, a data record corresponding to the first enhanced data, the data record being formatted according to the data record format ; a database storing the data record; and wherein the second program merges duplicate data records that represent the same network usage information.

2. The system of clause I wherein the at least partial transformation is defined from a data enhancement procedure, and wherein the data record fonnat includes a plurality of fields and wherein the data enhancement procedure defines how the standardized information is to be transformed into the plurality of fields of the data record format.

3. The system of clause 2 wherein the data enhancement procedure includes at least a field enhancement wherein the field enhancement defines a source for a predetermined field in the plurality of fields.

4. The system of clause 2 wherein the data enhancement procedure includes at least a field enhancement wherein the field enhancement defines a function to be applied to at least a portion of the standardized data.

5. The system of clause 2 wherein the data enhancement procedure defines a plurality of field enhancements, wherein each field enhancement defines network usage information to be stored in the plurality of fields.

6. The system of clause 5 wherein at least a first field enhancement corresponds to an information source module that performs aggregation on at least a portion of the network usage information to be stored in at least a first field in the plurality of fields.

7. The system of clause 6 wherein the aggregation occurs by aggregating packet flow information. the packet flow information corresponding

to a plurality of packets. where each packet in the aggregated packets has the same IP source address, destination address and port information : 8. The system of clause 5 wherein at least a first field enhancement corresponds to a set of information source modules that performs filtering and aggregation on at least a portion of the network usage information to be stored in at least a first field in the plurality of fields.

9. The system of clause 5 wherein at least a first field enhancement corresponds to a set of information source modules that performs merging, filtering, and aggregation on at least a portion of the network usage information to be stored in at least a first field in the plurality of fields.

10. The system of clause 5 wherein at least a first field enhancement corresponds to a set of information source modules that performs event notification and provisioning activation.

11. The system of clause 2 further comprising a second information source module, the second information source module having a second source information input and a second standardized information output. a second source information corresponds to a second network information, a second standardized information corresponds to the second network information transformed into a standard format, and wherein the data enhancement procedure includes a first definition of at least a first field in the plurality of fields being from the standardized information, and at least a second definition

of a second field in the plurality of fields being from the second standardized 9 information.

12. The system of 11 further comprising a first information source and a quality of service information source, and wherein the information source module receives the network usage information from the first information source. and wherein the second information source module receives the second network information from the quality of service information source, and wherein the first definition defines that a source IP address supplied by the first information source should be put into the first field, and wherein the second definition defines a supplied by the DNS server should be put into the second field.

13. The system of clause I wherein the second program manages the first program and the information source module.

14. The system of clause I wherein the second progi-f. causes the data record to be stored in the database.

15. The system of clause I wherein the information source module is configured to receive the network usage information from a predetermined network device.

16. The system of clause 1 wherein the at least partial transformation

includes policy-based data aggregation which defines how network usage data 9 should be aggregated.

17. The system of clause I wherein the network usage information includes IP session data.

18. The system of clause I wherein the data format includes a plurality of fields including a source IP field. a destination IP field, a source host field. a destination host field. a service type field, a date and time field, a duration field, a total number of bytes field, and a counter field.

19. The system of clause I further comprising a customer care a billing system coupled to the database, the customer care and billing system for accessing the database to generate a bill from the data record.

20. A network usage accounting system comprising : an information source module coupled to receive network information from a network device ; a gatherer coupled to receive the network information source module. the gatherer for performing data enhancements on the network information to create a plurality of data records; a central database storing the plurality of data records: and a central event manager coupled to receive the plurality of data records, the central event manager merging duplicate records in the plurality of data records, the duplicate records representing the same network usage information.

21. The system of clause 20 wherein the information source module is configured to receive network information from a network device chosen

from the group of network devices consisting of a proxy server. a domain name service server, a firewall, a RADIUS server, and a router.

22. The system of clause 20 wherein the gatherer performs filtering and aggregation on the network information.

23. The system of clause 20 wherein the plurality of data records have a predefined data format comprising a plurality of fields, and wherein the data enhancements includes at least a first data field enhancement to enhance the network information to fill in the first data field.

24. The system of clause 23 wherein the first data field corresponds to a source IP address field and wherein the data enhancement includes extracting a source IP address value from the network information.

25. The system of clause 23 wherein the first data field corresponds to a domain name field and wherein the data enhancement includes requesting a domain name from a domain name service server.

26. A method of gathering and aggregating network usage information from a set of network devices. the system using at least a first program and a second program coupled in communications, the method comprising: accessing network communications usage information ; filtering and aggregating the network communications usage information using the first program;

completing a plurality of data records from the filtered and aggregated network communications usage information, the plurality of data records corresponding to network usage by a plurality of users; storing the plurality of data records : and merging duplicate records in the plurality of data records.

27. The method of clause 26 wherein completing the plurality of records includes accessing user account information.

28. The method of clause 26 wherein completing the plurality of records includes for each data record determining a corresponding source IP address. a corresponding domain name, a corresponding type of service used. and a corresponding amount of time used.

29. The method of clause 26 wherein the system includes a third program coupled in communications with at least the second program and wherein completing the plurality of records includes accessing the third program to determine network account information and including the network account information in at least a first record in the plurality of records.

30. The method of clause26 wherein merging the duplicate records includes comparing a plurality of fields in the data records to identify data records corresponding to the same network session and merging the corresponding records.

31. The method of clause 26 wherein merging the duplicate records includes automatically deleting a duplicate record.

32. The method of clausel 26 further comprising using the second program to automatically update the filtering and aggregation performed by the first program.

33. A network usage tracking system comprising: means for accessing network communications usage information : means for filtering and aggregating the network communications usage information using the first program; means for completing a plurality of data records from the filtered and aggregated network communications usage information. the plurality of data records corresponding to network usage by a plurality of users; means for storing the plurality of data records; and means for merging duplicate records in the plurality of data records.

34. The network usage tracking system of clause 34 wherein the means for completing the plurality of data records includes one or more : networked computers running one or more programs.

35. The network usage tracking system of clause 34 wherein the means for storing the plurality of data records includes a relational database.

36. The network usage tracking system of clause 34 wherein the means for storing the plurality of data records includes an object database.

37. A system for accounting for network usage comprising :

a plurality of gatherers coupled to receive network usage information. each gatherer for performing data enhancements on the network information to create a plurality of data records; and a central event manager coupled to receive the plurality of data records, the central event manager merging duplicate records in the plurality

of data records, the duplicate records representing the same network usage information.

38. The system of clause 37 further comprising a customer care and billing application for receiving the plurality of data records and generating bills.

Claims

1. A system for tracking network session information, the system comprising : an information source module having a source information input and a standardized information output, a source information corresponds to network usage information, a standardized information corresponds to the network usage information transformed into a standard format; a first program having at least a first standardized information input and an enhanced data output, a first standardized information input corresponding to the standardized information, an enhanced data corresponding to the standardized data after at least a partial transformation, the at least partial transformation being defined according to a data record format ; a second program having at least a first enhanced data input and a data record output, the first enhanced data corresponding to the enhanced data, a data record corresponding to the first enhanced data, the data record being formatted according to the data record format ; a database storing the data record; wherein the second program merges duplicate data records that represent the same network usage information; and wherein the at least partial transformation includes policy-based data aggregation which defines how network usage data should be aggregated.

2. The system of claim 1 wherein the at least partial transformation is defined from a data enhancement procedure, wherein the data record format includes a plurality of fields, wherein the data enhancement procedure defines how the standardized information is to be transformed into the plurality of fields of the data record format and wherein the data enhancement procedure includes at least a field enhancement wherein the field enhancement defines a source for a predetermined field in the plurality of fields.

3. The system of claim I wherein the at least partial transformation is defined from a data enhancement procedure, wherein the data record format includes a plurality of fields, wherein the data enhancement procedure defines how the standardized information is to be transformed into the plurality of fields of the data record format and wherein the data enhancement procedure includes at least a field enhancement wherein the field enhancement defines a function to be applied to at least a portion of the standardized data.

4. The system of claim I wherein the at least partial transformation is defined from a data enhancement procedure, wherein the data record format includes a plurality of fields, wherein the data enhancement procedure defines how the standardized information is to be transformed into the plurality of fields of the data record format and wherein the data enhancement procedure defines a plurality of field enhancements, wherein each field enhancement defines network usage information to be stored in the plurality of fields.

5. The system of claim 4 wherein at least a first field enhancement corresponds to an information source module that performs aggregation on at least a portion of the network usage information to be stored in at least a first field in the plurality of fields.

6. The system of claim 5 wherein the aggregation occurs by aggregating packet flow information, the packet flow information corresponding to a plurality of packets, where each packet in the aggregated packets has the same IP source address, destination address and port information.

7. The system of claim 4 wherein at least a first field enhancement corresponds to a set of information source modules that performs filtering and aggregation'on at least a portion of the network usage information to be stored in at least a first field in the plurality of fields.

8. The system of claim 4 wherein at least a first field enhancement corresponds to a set of information source modules that performs merging, filtering, and aggregation on at least a portion of the network usage information to be stored in at least a first field in the plurality of fields.

9. The system of claim 4 wherein at least a first field enhancement corresponds to a set of information source modules that performs event notification and provisioning activation.

10. The system of claim 1 wherein the at least partial transformation is defined from a data enhancement procedure, wherein the data record format includes a plurality of fields, wherein the data enhancement procedure defines how the standardized information is to be transformed into the plurality of fields of the data record format, further comprising a second information source module, the second information source module having a second source information input and a second standardized information output, a second source information corresponds to a second network information, a second standardized information corresponds to the second network information transformed into a standard format, and wherein the data enhancement procedure includes a first definition of at least a first field in the plurality of fields being from the standardized information, and at least a second definition of a second field in the plurality of fields being from the second standardized information.

11. The system of 10 further comprising a first information source and a quality of service information source, and wherein the information source module receives the network usage information from the first information source, and wherein the second information source module receives the second network information from the quality of service information source, and wherein the first definition defines that a source IP address supplied by the first information source should be put into the first field, and wherein the second definition defines a supplied by the DNS server should be put into the second field.

12. The system of claim 1 wherein the second program manages the first program and the information source module.

13. The system of claim 1 wherein the second program causes the data record to be stored in the database.

14. The system of claim 1 wherein the information source module is configured to receive the network usage information from a predetermined network device.

15. The system of claim 1 wherein the network usage information includes IP session data.

16. The system of claim 1 further comprising a customer care and billing system coupled to the database, the customer care and billing system for accessing the database to generate a bill from the data record.