WO2007008590A2 - Distributed capture and aggregation of dynamic application usage information - Google Patents

Distributed capture and aggregation of dynamic application usage information Download PDF

Info

Publication number
WO2007008590A2
WO2007008590A2 PCT/US2006/026337 US2006026337W WO2007008590A2 WO 2007008590 A2 WO2007008590 A2 WO 2007008590A2 US 2006026337 W US2006026337 W US 2006026337W WO 2007008590 A2 WO2007008590 A2 WO 2007008590A2
Authority
WO
WIPO (PCT)
Prior art keywords
application
web
server
feature
log file
Prior art date
Application number
PCT/US2006/026337
Other languages
French (fr)
Other versions
WO2007008590A3 (en
Inventor
Andrew S. Wilson
Brian J. Shin
David J. Angel
Original Assignee
Visible Measures, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Visible Measures, Inc. filed Critical Visible Measures, Inc.
Priority to JP2008521441A priority Critical patent/JP5354776B2/en
Priority to EP06786477A priority patent/EP1907941A4/en
Priority to CA002657487A priority patent/CA2657487A1/en
Priority to CN200680033282.9A priority patent/CN101371245B/en
Priority to KR1020087003403A priority patent/KR101300360B1/en
Publication of WO2007008590A2 publication Critical patent/WO2007008590A2/en
Publication of WO2007008590A3 publication Critical patent/WO2007008590A3/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/04—Processing captured monitoring data, e.g. for logfile generation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/542—Event management; Broadcasting; Multicasting; Notifications
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G06F11/3495—Performance evaluation by tracing or monitoring for systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/535—Tracking the activity of the user

Definitions

  • the present invention relates generally to software monitoring, tracking, and return on investment (ROI) analysis.
  • ROI return on investment
  • the usage information or, more generally, the log file is transmitted in a highly compact manner over the Internet (preferably using conventional HTTP transport) to allow for a high performance, distributed system with lightweight processing requirements.
  • the second component is the server, which takes the results of the remote node and combines it with data collected from other remote nodes. These two components make it possible for a user of the present invention to collect feature tracking and configuration information from many simultaneous nodes. There are also three important goals that the remote node should have. The first is that it must be easy to integrate. The second is that it must run fast. The third is that the transfer of information must be fast and appear incidental to the application's normal execution. Integration is made simple by preferably requiring only one line of code to be added per feature. The second goal is met by the application running fast because this one line of code, and its resulting calls, only introduce several milliseconds of overhead to an application per feature call.
  • the third goal is met because the log file format ensures that the file will always be very small even under the worst of conditions, leading to very fast transfer times and low CPU utilization at those times, thus ensuring that the user's application does not perform poorly while a transfer is in progress.
  • the remote node collects feature tracking and configuration information for one instance of a given application. At some user defined point, the remote node attempts to connect to the server component and dispatch a series of log files. The remote component relies on several sub-components that collect and transfer this information. Some of these components require the user to perform certain implementation tasks. Table 1 below describes each component.
  • Component Description Messenger User calls the messenger to store the execution of a feature. This component performs some processing of events, invokes the Message
  • Handler to determine if the message should be logged and if it is appropriate to dispatch the log file.
  • Event An object that contains interesting information concerning the state of a running application.
  • Message Handler A user defined component that determines if a message should be stored, where to store it, when to dispatch the various event repositories, and where to dispatch it to.
  • Log Dispatcher A component that allows the remote node to dispatch log files to a server.
  • the server side simply serves an HTML form for the file transfer to work as shown in the code snippet illustrated in Figure 8.
  • the remote node parses the HTML page and finds the upload form, sets the file field, and then submits the information.
  • the step of finding the form ensures that the system does not try to arbitrarily perform a file upload on a form that cannot accept the incoming file.
  • the server Upon receipt, the server writes the incoming log file to an import directory.
  • the server's log import component preferably scans this directory at regular intervals. The import component opens the new log file and adds the data to the database.
  • the second configuration (type B) is typically implemented with the logging server located in the DMZ of an enterprise web site.
  • an arbitrary HTTP server 613 serves the file transfer form and waits for the remote node connections at step 614.
  • the server 613 processes the upload form request at step 616, writes the log file to a local drive 618 and, at step 620, copies the files to log import directory of the logging server 615 associated with the hosted service.
  • the logging server 615 runs concurrently. In particular, logging server 615 scans the import directory at step 622.
  • the configuration B is more secure and reliable (as compared to configuration A), as the HTTP server can be an existing corporate HTTP server, such as the one used to serve the company's web pages. Configuration B also is more secure in the sense that no critical log data is accessible from the outside world. Further, the second configuration is more reliable in that the corporate web server is implemented to handle a large number of simultaneous users, and it is constantly monitored so that it can be brought back up quickly in the event of a failure.
  • the 32 character unique identifier may be provided by the server to a user working through a user interface, or through any other convenient means.
  • the identifier is created and displayed when the user creates a new build for a particular project.
  • the user then copies that 32 character unique identifier into his MessageHandler class implementation.
  • the 32 character identifier is used to match the log file data to a particular project and build on the service provider's server.
  • the read data from the log file is then put into the server's data repository.
  • Project and build names preferably are also used to match data to a project and build in the event that the 32 character identifier fails, e.g., due to a typographic error on the part of the developer.
  • the MessageHandler's dispatching mechanism preferably also requires a destination URL and may include other optional items such as a login name, password, proxy server configurations, and additional code to script the process of transmitting a log file back to an import server.
  • the MessageHandler may dispatch a log file at the time the application starts, stops, during configuration data collection or after a feature has been executed. The developer's implementation of these methods simply returns a "true" response, and the MessageHandler then automatically dispatches all (or some subset of the) log files in the system.
  • the dispatcher Upon getting a "true" response, the dispatcher typically uses the proxy information, if present or necessary, to pass through a first intervening firewall and make a connection to the server specified in the URL parameter.
  • the URL may require special login information or expose additional scripting. At this point, the dispatcher executes the additional login information and code to get a file upload form. The dispatcher then sends the log files through this form.
  • HandleConfigMessage Called by the Messenger prior to the first HandleMessage call. Allows the user to define custom configuration information. HandleMessage Called by the Messenger whenever a feature is executed. The user code may return ti'ue, which will cause the Messenger to dispatch the log files. Regardless of the return value, the messenger will then examine the event object to determine if the event has been processed. The event will not be stored if the event has been processed.
  • the Constructor method provides a significant amount of information concerning the initialization of the application.
  • the code block in Figure 10 is the MessageHandler abstract class.
  • the various getXXX and setXXX methods allow the user to set critical configuration information.
  • the illustrated framework allows the user to define many elements to further enhance the ability of the application to transfer data.
  • Table 3 describes various methods and what operation they perform (with the asterisk methods being optional).
  • LogDispatcher with a custom implementation may support any communication mechanism the user wishes.
  • getLoginName The form on the server may be behind a basic HTTP authentication scheme.
  • a custom LogDispatcher calls this method to get the name of an authorized user when using a multi-page authentication scheme.
  • getLoginPassword A custom LogDispatcher calls this method to get the password of an authorized user when using a multi-page authentication scheme.
  • the getLoginUserQ method must return a user name for this method to work.
  • getLoginUrl A custom LogDispatcher calls this method to get the URL for the
  • HTML page containing the file upload form when using a multi-page authentication scheme HTML page containing the file upload form when using a multi-page authentication scheme.
  • getProjectName The name of the application 's specific project in the form of a string.
  • getProxyServerHost The default LogDispatcher calls this method to get the name of the proxy server in the same domain as that of the application running the remote node.
  • getProxyServerPort The default LogDispatcher calls this method to get the port of the proxy server in the same domain as that of the application running the remote node.
  • getProxyUserName The default LogDispatcher calls this method to get the name of the user to authenticate on proxy server in the same domain as that of the application running the remote node.
  • getProxyUserPassword The default LogDispatcher calls this method to get the password of the user to authenticate on proxy server in the same domain as that of the application running the remote node.
  • getWorkDirectory Returns the location where the Messenger should write log files prior to dispatching. setBuildName Sets the name of the build. Set during initialization of the customized
  • MessageHandler setProxyUserName* Sets the user name for proxy authentication. Set during initialization of the customized MessageHandler. setProxyUserPassword* Sets the password for the user to be authenticated by the proxy server.
  • Set during initialization of the customized MessageHandler setUrl Sets the URL containing the file upload form. Set during initialization of the customized MessageHandler. setWorkDirectory Sets the directory where the Messenger will write log files. Set during initialization of the customized MessageHandler. setDomain A set method for unique name representing the domain of the company producing the application that is using the logging code.
  • the value is used to match a log file to a project.
  • getDomain A get method to obtain the current value of the domain name.
  • setKey A set function for the unique 32 character string identifying the project and build to associate log file data.
  • getKey A get function for the unique 32 character string identifying the project and build to associate log file data.
  • the Messenger and LogDispatcher read these methods to persist and transfer log files.
  • Step 2 The Messenger.storefeature
  • the Messenger.storefeature method tracks feature data. It takes two parameters. The first is an instance of the customized message handler. The second is a string containing the name of the feature. The resulting information about the number of executions of the feature indicates the value of that feature. Of course, a highly used feature is often more valuable than a rarely or never used feature.
  • the Messenger.storefeature method is added to each feature entry point within the application. Step 3
  • the feature tracking code is executed.
  • a test is then ran at step 704 to determine whether this is the first feature execution for the application ran. If so, the MessageHandler startup method is called at step 706.
  • a test is then ran at step 708 to determine whether the log files should be dispatched. If not, the MessageHandler' s handle configuration message method is called at step 710.
  • the routine then calls the MessageHandler' s handle message method at step 712, which step is also reached when the outcome of the test at step 714 is positive.
  • Step 714 tests whether a dispatch is due to a startup event.
  • the feature statistics are updated at step 716.
  • a test is then performed at step 718 to determine whether the feature should be written to the log file.
  • a log file preferably is a binary file containing a series of specially formatted events.
  • the file comprises aggregated feature information rather than one entry per feature execution (as is common in traditional log files) to ensure that the file is smaller than traditional log files.
  • aggregated feature information is not a limitation of the invention, however.
  • the format can be read and integrated into a database very efficiently because each field is well defined. Traditional log files have to be read line by line and then the data must be parsed out of the text. This is often difficult to implement and consequently error prone and performs poorly. The formatting is described in terms of number of unsigned bytes.
  • Each event construct preferably is derived from a based event format such as described in Table 4 below.
  • Event_Data_Length The length of the data portion of the event.
  • Event Data The event data itself. Table 4
  • the events are derived from this one format. Consequently, the events preferably begin with Event_Type and Event_Data_Length but may not include the Event_Data section (as the events typically override this field with their specific implementations), hi a representative embodiment, the file format is a collection of Base_Events.
  • Feature_Name_Length The length of the name field.
  • Configuration J3ntry_Name_Length The length of the configuration entry name.
  • Configuration_Entry_Name The configuration entry name.
  • Configuration_Entry_Value_Length The length of the configuration's value.
  • Configuration_Entry_Value The configuration entry's value.
  • the code logs a first feature extraction.
  • the feature logging code preferably opens a new log file and the logging code writes the following messages, preferably in order: project, configuration, node type and feature.
  • the code logs follow on feature executions.
  • the logging code preferably adds just one message, indicating feature execution
  • hi a third scenario the logging code logs a feature failure.
  • the logging code preferably adds just one message, indicating feature failure.
  • the logging code logs a feature fault. Just one message is created again, indicating a feature fault.
  • Sample logging code with the associated logging statements is illustrated in Figure 9.
  • each log file is made up of a series of messages. Each message typically has some common formatting and then become message type specific. The following are illustrative message types:
  • ⁇ UserConfig Message placeholder for additional configuration information desired by the user of the API (e.g., gather more info about a server upon which an application is installed)
  • Feature Failure Message information about a feature failing (feature fails to complete because for example, user entered improperly formatted data), the time it failed, and some info about why it failed
  • Subnode Feature Execution - features executed on the end user's configuration, versus features executed on a server (needed to track client processing in situation such as with Rich Internet Applications or DHTML executing on a end user client versus on the web application server)
  • Subnode Fault Message faults occurring on end user's machine - e.g., Javascript runtime errors across browsers ⁇ Subnode Reset Message - as with Feature Reset Message but for end-user features
  • Configuration - preferably only one message can be present per log file 3.
  • User Configuration - 0 or more can be present
  • Node Type preferably only one message can be present per log file
  • Data is stored in the log file in the manner of a database record, such as indicated below:
  • the log files preferably use a binary file format, which is now described in more detail.
  • Figure 10 illustrates a representative file format (with bytes represented as hexadecimal).
  • the following describes an encoding method for generating the binary file format and, in particular, that takes in a number and compresses it down to a smallest number of bytes possible.
  • the method begins by initializing a buffer to a base value that represents zero, the smallest non-negative whole number.
  • the most significant bit (MSB) for each byte in the buffer is reserved to indicate whether the byte is a part of a number or the end of a number. This means that only 7 bits of a byte actually contain a numeric value while the most significant bit is an indicator.
  • the encoding process then goes on to break the number from 8 to 7 bits across the buffer, shifting the bits 7 places to the right.
  • the MSB portions of the number are encoded within each byte of the buffer with the MSB set to 1.
  • a data type represents a whole number that is compressed to fit into a smallest number of necessary bits.
  • the scheme preferably works by breaking numbers across a series of bytes. The first N bytes have the most significant bit (MSB) set to 10xx xxxx where x are the bits that represent the actual number. The last byte has its MSB set to Oxxx xxxx. How the scheme operates is best seen by example. For example, the number 1 is stored according to the present scheme as 0000 0001. Normal encoding would be 0000 0000 0000 0000 0000 0001, taking up 4 bytes; as can be seen, the present scheme uses just 1 byte. As another example, the number 1,000 is stored as 1000 0111 0110 1000.
  • Normal encoding would be 0000 0000 0000 0000 0000 1110 1000, taking up 4 bytes, whereas the present scheme uses just 2 bytes. The number 100,000 is stored aslOOO 0110 1000 1101 0010 0000. Normal encoding would be 0000 0000 0000 0001 1000 0110 1010 0000, once again taking up 4 bytes, wherein the present scheme uses just 3 bytes. This technique substantially reduces the log file size, especially as compared to human readable (e.g., XML) or even conventional binary number (4 byte) encoding..
  • a log file comprises a series of messages. Each message represents some interesting event, such as a feature execution or failure.
  • the messages are preferably formatted with specific formats and field lengths.
  • Variable length fields preferably are prefixed with a field indicating the length of the following field.
  • Numeric fields preferably are variable length but use the above-described data type. As described above, the encoding scheme enables a number to be stored within a binary field.
  • a machine typically comprises commodity hardware and software, storage (e.g., disks, disk arrays, and the like) and memory (RAM, ROM, and the like).
  • storage e.g., disks, disk arrays, and the like
  • RAM random access memory
  • ROM read-only memory
  • a given machine includes network interfaces and software to connect the machine to a network in the usual manner.
  • the present invention may be implemented as a managed service (e.g., in an ASP model) using the illustrated set of machines, which are connected or connectable to one or more networks.
  • the service is provided by an operator using a set of one or more computing-related entities (systems, machines, processes, programs, libraries, functions, or the like) that together facilitate or provide the inventive functionality described above.
  • the service comprises a set of one or more computers.
  • a representative machine is a network-based server running commodity (e.g. Pentium-class) hardware, an operating system (e.g., Linux, Windows, OS-X, or the like), an application runtime environment (e.g., Java, .ASP), and a set of applications or processes (e.g., Java applets or servlets, linkable libraries, native code, or the like, depending on platform), that provide the functionality of a given system or subsystem.
  • the service may be implemented in a standalone server, or across a distributed set of machines.
  • a server connects to the publicly-routable Internet, a corporate intranet, a private network, or any combination thereof, depending on the desired implementation environment.
  • the hosted service may be implemented in a multi-server cluster environment that is designed to scale efficiently.
  • Each server is designated with a primary and secondary series of tasks.
  • one server is dynamically set to be a master server, which server determines the secondary tasks to be performed by all servers. All servers update their existence within a database, and the servers cooperate to determine which server will be the master.
  • the servers in the cluster are assigned tasks (such as log import and event processing) by the master server.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Quality & Reliability (AREA)
  • Data Mining & Analysis (AREA)
  • Debugging And Monitoring (AREA)

Abstract

A web-based hosted solution through which application developers create, manage and monitor application usage analytics in an online manner. Preferably, an application under test is one of : application software, a script-enabled web application, or a rich Internet application (RIA). During the development process, a usage monitoring API is integrated into the application and the application is deployed. As users interact with the application, a log file is generated, typically in one of two ways. If the application is able to write to a local file system (in the user's machine), usage information is gathered in a log file local to the deployed application and then dispatched to an upload server for processing in a batch manner. If the application is not able to write to the user machine's local file system, the usage information is sent to a remote logging server, preferably on a just-in-time basis, and then the log file is generated on the logging server. In either case, preferably the usage information that is tracked comprises 'features', 'faults' and 'failures' of the application, independent of platform, location, and number of deployed application instances.

Description

DISTRIBUTED CAPTURE AND AGGREGATION OF DYNAMIC APPLICATION USAGE INFORMATION
This application includes subject matter that is protected by copyright. This application is based on and claims priority from Serial No. 60/698,567, filed
My 12, 2005.
BACKGROUND OF THE INVENTION Technical Field
The present invention relates generally to software monitoring, tracking, and return on investment (ROI) analysis. Description of the Related Art
Powerful trends are reshaping the way the software industry operates and the way software is being developed. The biggest trend is arguably the one toward running Information Technology (IT) as a business. The United States Department of Commerce estimates that 50% of all capital spending is spent on IT and software. According to published studies, a not-insignificant portion of this spending is wasted because of the lack of metrics regarding productivity and ROI, and because during the software development process developers lack the ability to gather feedback easily from users and customers. Without such feedback, developers and product managers cannot determine which features of an application are most popular, which features cause the most problems when the software is used, and the like. Decisions about where to optimally allocate and leverage development resources thus are uninformed. The problem is exacerbated as software development and testing activities become distributed around the world. In the past, developers and IT managers have attempted to ascertain and evaluate application usage information through various techniques including guess-estimation, user surveys and focus groups, but such techniques typically represent only a small sampling of a customer base, and they often contain inaccurate and untimely data.
It is known in the prior art to provide methods and systems for centrally coordinating, collecting and storing error, trace, audit and other information in a computer network. A representative example is U.S. Patent No. 6,470,388, issued to Niemi et al. In this patent, selected applications or processes running at various entities or hosts within the network instantiate one or more "debug" objects that collect given information. Each entity also includes at least one logging service layer that communicates with the application or process, and includes a communications resource and one or more state machine engines. In response to collecting error, trace, audit or other information, each debug object passes that data to the respective logging service layer, which decides whether to forward the data to a centralized logging facility disposed within the network. The forwarding of collected information depends on the state of the debug object. At the centralized logging facility, the information is time-stamped and appended to a log file along with the application's name and the name of the entity or host at which the application is running.
Another representative patent is U.S. Patent No. 6,591,228, issued to Hall et al. In this patent, a logging service logs to a centralized log diagnostic messages from applications executing in a computing environment in which a plurality of mainframes are coupled to data storage area. Each mainframe has one or more subsystems that execute applications. A logging service API writes diagnostic messages to the data storage and, depending on the type of problem, contacts an alert facility.
While the above-described application logging techniques are generally useful, they work across homogeneous computer environments, using a relatively modest number of tracked systems, and they collect a relatively constrained set of information. There remains a need in the art to provide methods and systems that track application usage information independent of platform, location and number of deployed application instances, especially in the context of the software development process. The present invention addresses the need in the art. BRIEF SUMMARY OF THE INVENTION
The present invention is a software platform that delivers business value analytics for technology optimization, especially during the software development process. According to the invention generally, a tracking platform is run as a hosted (or managed) service to monitor, collect and aggregate application usage information. It is assumed that an application under test is being executed on a set of distributed machines, such as end user client machines. The application is instrumented to collect the usage information, which in one embodiment is then transported reliably to a central location, where it is aggregated and exported for viewing. By collecting and measuring detailed usage information about an application, the service helps software developers build high quality software applications more efficiently. The system preferably exports a web interface to enable users (e.g., IT managers, application developers and the like) to transact with the system over the Internet using conventional technologies (a computer having a web browser and network connectivity) and using standard communication techniques (HTTP, secure HTTP, SOAP -based web service, or the like). Alternatively, the system is implemented as an extranet, over a private network, or the like. Preferably, entities access the hosted solution on a subscription basis, although a given entity may also elect to access the service on a transaction-based basis.
According to more specific aspects of the present invention, the application under test is one of application software, a web application, or a rich internet application (RIA). During the development process, a usage monitoring API is integrated into the application and the application is deployed. As users interact with the application, a log file is generated, typically in one of two ways. If the application is able to write to a local file system (in the user's machine), usage information is gathered in a log file local to the deployed application and then dispatched to an upload server for processing in a batch manner. If the application is not able to write to the user machine's local file system (because, for example, it is a web application or RIA), the usage information is sent to a remote logging server, preferably on a just-in-time basis, and then the log file is generated on the logging server. In either case, preferably the usage information that is tracked comprises "features," "faults" and "failures" of the application, independent of platform, location, and number of deployed application instances. As used herein, "feature" data refers generally to a collection of information such as which features were used, when, in what order, by whom, on what platform, and with what end-user environment. The "fault" data refers generally to which features caused programmatic errors (e.g., exceptions). The "failures" data identifies which features failed to complete successfully, for example, if data was entered into a field in an incorrect format.
According to a further feature of the present invention, the usage information or, more generally, the log file, is transmitted in a highly compact manner over the Internet (preferably using conventional HTTP transport) to allow for a high performance, distributed system with lightweight processing requirements.
The foregoing has outlined some of the more pertinent features of the invention. These features should be construed to be merely illustrative. Many other beneficial results can be attained by applying the disclosed invention in a different manner or by modifying the invention as will be described.
BRIEF DESCRIPTION OF THE DRAWINGS
Figure 1 is a block diagram of a service provider infrastructure for implementing a brand integration technology platform according to the present invention;
Figure 2 illustrates the basic logging service of the present invention wherein application instances are instrumented to provide usage data sets that are then transported to a remote logging server;
Figure 3 illustrates how the hosted service can be used to provide feedback to a developer during a software development process;
Figure 4 provides a more detailed implementation of the hosted service; Figure 5 is a simplified process flow diagram illustrating how a remote node interacts with a logging server;
Figure 6 illustrates representative logging server configurations according to the present invention;
Figure 7 shows a representative process flow that occurs when a user executes a tracked feature;
Figure 8 illustrates a representative HTML code snippet for the HTML form for a file transfer;
Figure 9 is representative logging code for use in creating a log file; and
Figure 10 is a representative log file illustrating the binary file format. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
Figure 1 illustrates a representative service provider or system architecture, which in the preferred embodiment is implemented in or across one or more data centers. A data center typically has connectivity to the Internet. In one embodiment, the system provides a web-based hosted solution through which application developers (or others such as IT personnel) create, manage and monitor application usage analytics in an online manner. Participants preferably interact with the platform as a hosted service. In an alternative embodiment, the system may be implemented over a private network, or as a product (as opposed to a hosted or managed service). A user of the service has an Internet accessible machine such as a workstation or notebook computer. Typically, the user accesses the service provider architecture by opening a web browser on the machine to a URL associated with a service provider domain or sub-domain. The user then authenticates to the managed service in the usual manner, e.g., by entry of a username and password. The connection between the machine and the service provider infrastructure may be encrypted or otherwise secure, e.g., via SSL, or the like. Although connectivity via the publicly-routed Internet is typical, the user may connect to the service provider infrastructure over any local area, wide area, wireless, wired, private or other dedicated network. As seen in Figure 1, the service provider architecture 100 comprises an IP switch 102, a set of one or more web server machines 104, a set of one more application server machines 106, a database management system 108, and a set of one or more administration server machines 110. A representative web server machine 104 comprises commodity hardware (e.g., Intel- based), an operating system such as Linux, and a web server such as Apache 2.x. A representative application server machine 106 comprises commodity hardware, Linux, and an application server. The database management system 108 may be implemented as an Oracle database management package. In a high volume use environment, there may be several web server machines, several application server machines, and a number of administrative server machines. Although not shown in detail, the infrastructure may include a name service, other load balancing appliances, other switches, network attached storage, and the like. The system typically will also include connectivity to external data sources, such as third party databases. Each machine in the system typically comprises sufficient disk and memory, as well as input and output devices. Generally, the web servers 104 handle incoming business entity provisioning requests, and they export a display interface that is described and illustrated in more detail below. The application servers 106 manage the data and facilitate the functions of the platform. The administrator servers 110 handle all back-end accounting and reporting functions. The particular hardware and software implementation details described herein are merely for illustrative purposes are not meant to limit the scope of the present invention.
Figure 2 illustrates the basic operation of the logging service. In this example, application instances 200a and 200b have been instrumented with a usage monitoring API according to the invention. The usage monitoring API is sometimes referred to as a remote node. Thus, application instance 200a has associated therewith the remote node 202a and application instance 200b has associated therewith the remote node 202b. Of course, the use of two instances is merely illustrative, as the present invention is designed to provide a highly scalable distributed logging service wherein a large number of instances of the deployed application are instrumented and tracked. In operation, the remote node 202a generates usage data set 204a, and remote node 202b generates usage data set 204b. This usage data is transported in a highly efficient manner (as will be described below) to a central server (or to a set of servers), where the data sets are aggregated (reference numeral 206) and processed within an analytics and reporting engine 208.
The present invention is useful in the context of software development and testing, although one of ordinary skill in the art will appreciate that the invention is not limited to such use. Figure 3 illustrates a representative business case. In this example, a web application developer 300 adds the usage monitoring API into the web application or rich Internet application under development. The application so instrumented is then made available from a web site or other publishing server 302. An end user 304 navigates to the site and downloads and interacts with the application to generate usage data. This data is sent to the logging server 306, which then uploads such data to the hosted service database 308. A manager 310 (or the developer 300) can then log into the hosted service web site 312 and access the logged data.
Figure 4 illustrates an operation of the hosted service in more detail. In this example, a large number of end users 400a-400n use the application and generate usage data sets that are provided to the logging server 402. The logging server 402 may comprise one or more servers. The logging server 402 periodically uploads the data sets through firewall 410 to an application server 404, which stores the processed data in a database 406. The user 405 of the hosted service logs into the service through server 412 and views the usage reports through server 414, which accesses the usage data through the firewall 410.
Preferably, the application under test is one of: application software (such as a program written in Java, .Net, C++, C#, or the like), a script-enabled web application (such as a web page including Javascript, ActionScript or the like), or a rich Internet application (RIA) (e.g., Flash, AJAX-enabled, or the like). During the development process, a usage monitoring API is integrated into the application and the application is deployed. As users interact with the application, a log file is generated, typically in one of two ways. If the application is able to write to a local file system (in the user's machine), usage information is gathered in a log file local to the deployed application and then dispatched to an upload server for processing in a batch manner. If the application is not able to write to the user machine's local file system (because, for example, it is a web application or RIA), the usage information is sent to a remote logging server, preferably on a just-in-time basis, and then the log file is generated on the logging server. This is the technique used for Web browser-based logging. Preferably, such logging is accomplished by transmitting the data through http URL parameters to the logging server, which then translates the data into a log file. In either case, preferably the usage infoπnation that is tracked comprises "features," "faults" and "failures" of the application, independent of platform, location, and number of deployed application instances. As noted above, "feature" data refers generally to a collection of information such as which features were used, when, in what order, by whom, on what platform, and with what end-user environment. Typically, features are exposed to end users. The "fault" data refers generally to which features caused programmatic errors (e.g., exceptions). The "failures" data identifies which features failed to complete successfully, for example, if data was entered into a field in an incorrect format.
Thus, according to one illustrative use of the invention, a user of the hosted service instruments an application under development with tracking modules to better understand its beta customers as well as the progress of their beta testing. As noted above, preferably the "tracking modules" are embedded in (or otherwise associated with) the software application to be tracked. Usage data is sent back in a lightweight proprietary manner to an aggregation and reporting server. The following description provides additional details of an embodiment where an application has the ability to write a log file to an end user machine local file system. In this embodiment, the hosted service platform collects data from the end users through a number of components that are now described. The first component is the remote node, which is responsible for collecting feature tracking and configuration information. The second component is the server, which takes the results of the remote node and combines it with data collected from other remote nodes. These two components make it possible for a user of the present invention to collect feature tracking and configuration information from many simultaneous nodes. There are also three important goals that the remote node should have. The first is that it must be easy to integrate. The second is that it must run fast. The third is that the transfer of information must be fast and appear incidental to the application's normal execution. Integration is made simple by preferably requiring only one line of code to be added per feature. The second goal is met by the application running fast because this one line of code, and its resulting calls, only introduce several milliseconds of overhead to an application per feature call. The third goal is met because the log file format ensures that the file will always be very small even under the worst of conditions, leading to very fast transfer times and low CPU utilization at those times, thus ensuring that the user's application does not perform poorly while a transfer is in progress. The following describes the remote node and all of its parts and then moves on the relevant server components. The remote node collects feature tracking and configuration information for one instance of a given application. At some user defined point, the remote node attempts to connect to the server component and dispatch a series of log files. The remote component relies on several sub-components that collect and transfer this information. Some of these components require the user to perform certain implementation tasks. Table 1 below describes each component.
Component Description Messenger User calls the messenger to store the execution of a feature. This component performs some processing of events, invokes the Message
Handler to determine if the message should be logged and if it is appropriate to dispatch the log file.
Event An object that contains interesting information concerning the state of a running application.
Message Handler A user defined component that determines if a message should be stored, where to store it, when to dispatch the various event repositories, and where to dispatch it to.
Log Dispatcher A component that allows the remote node to dispatch log files to a server.
Log File A repository of feature and configuration events.
Log Writer A component that stores event objects to a log file.
Server An arbitrary HTTP server that provides access to a file upload form
File Upload Form A specially formatted and named for used only for the transfer of log files from a remote node to a sei-ver.
Import A component on the server that processes incoming log files and incorporates the log file 's data into the main Visible Measures ' database.
Table 1
These components will be described below. First, the following describes the flow of data between the remote node and the server in a representative embodiment. Then, the process of integrating these components into an application so that a user can collect feature tracking information is described. Preferably, data flows in a single direction from the remote node to the server and the server does not communicate with the remote node. Figure 5 illustrates a representative process flow. The process for transmitting data is straightforward. The remote node waits for a dispatching event 500, which causes the remote node to connect to a user defined URL. At step 502, the remote attempts to connect to the logging server. A test at step 504 determines whether the remote node can connect to the server. If so, the remote node then transfers all log files, preferably by submitting them to an HTML form on an HTML page specified by the URL. Preferably, the remote node then deletes the successfully dispatched log files. Moreover, preferably the remote node stores the log files for a user-defined number of days if a connection cannot be made to the server. The remote node also preferably checks the date of each log file after a failed dispatch event. The remote node then deletes the files that are older than the maximum number of days.
In this embodiment, the server side simply serves an HTML form for the file transfer to work as shown in the code snippet illustrated in Figure 8. The remote node parses the HTML page and finds the upload form, sets the file field, and then submits the information. The step of finding the form ensures that the system does not try to arbitrarily perform a file upload on a form that cannot accept the incoming file. Upon receipt, the server writes the incoming log file to an import directory. The server's log import component preferably scans this directory at regular intervals. The import component opens the new log file and adds the data to the database.
Preferably, there are two basic configurations for the logging server to make available the file upload form. Figure 6 illustrates the process flows for these configurations. The first configuration (type A) is shown in the top portion of the drawing. In this configuration, the server makes the form available and waits for remote nodes to connect. This is step 600. At step 602, a test is run to determine whether a remote node is attempting to comiect. Upon receiving a connection, the server accepts the log file at step 604 and writes the log file to the import directory at step 606. At step 608, the new log file is imported into the service database. At step 610 the database is updated, after which the log file is deleted at step 612. The server then returns to a state to await another upload. The second configuration (type B) is typically implemented with the logging server located in the DMZ of an enterprise web site. In this implementation, an arbitrary HTTP server 613 serves the file transfer form and waits for the remote node connections at step 614. When a remote node connects, the server 613 processes the upload form request at step 616, writes the log file to a local drive 618 and, at step 620, copies the files to log import directory of the logging server 615 associated with the hosted service. The logging server 615 runs concurrently. In particular, logging server 615 scans the import directory at step 622. If a new log file is present as indicated by the test at step 624, the log file is imported at step 626, the database is updated at step 628, and the log file is deleted at step 630. The configuration B is more secure and reliable (as compared to configuration A), as the HTTP server can be an existing corporate HTTP server, such as the one used to serve the company's web pages. Configuration B also is more secure in the sense that no critical log data is accessible from the outside world. Further, the second configuration is more reliable in that the corporate web server is implemented to handle a large number of simultaneous users, and it is constantly monitored so that it can be brought back up quickly in the event of a failure.
In this embodiment, a software developer typically performs a series of steps to facilitate the collection of the feature and configuration information. In particular, the following steps describe how the developer instruments the application to generate the log data (familiarity with Java is presumed in the following):
1. Implement a derivation of a MessageHandler class.
2. Add a series of Messenger.store(...) methods to the application's existing code. Each addition should be at the point in code that represents a given feature's entry point. 3. Update the application installation tool such that it collects HTTP proxy information and generates a configuration file readable by the developer's application and that can forward this information to the MessageHandler.
Each of the above steps will now be described in more detail. Step 1 The MessageHandler is a Java abstract class. Consequently, according to the invention, an integrator derives a concrete implementation of this class that is specific to his or her application. This message handler performs a series of tasks, which are now described. In a representative embodiment, the MessageHandler class is responsible for filtering and dispatching log files. An instance of the class typically begins with initialization code that sets a series of variables necessary for creating and dispatching a log file. Creating a log file typically requires the name of the integrator's domain name, such as mycompany.com, a unique 32 character identifier provided by the service provider's server, and project and build names. The 32 character unique identifier may be provided by the server to a user working through a user interface, or through any other convenient means. The identifier is created and displayed when the user creates a new build for a particular project. The user then copies that 32 character unique identifier into his MessageHandler class implementation. The 32 character identifier is used to match the log file data to a particular project and build on the service provider's server. The read data from the log file is then put into the server's data repository. Project and build names preferably are also used to match data to a project and build in the event that the 32 character identifier fails, e.g., due to a typographic error on the part of the developer. The MessageHandler's dispatching mechanism preferably also requires a destination URL and may include other optional items such as a login name, password, proxy server configurations, and additional code to script the process of transmitting a log file back to an import server. The MessageHandler may dispatch a log file at the time the application starts, stops, during configuration data collection or after a feature has been executed. The developer's implementation of these methods simply returns a "true" response, and the MessageHandler then automatically dispatches all (or some subset of the) log files in the system. Upon getting a "true" response, the dispatcher typically uses the proxy information, if present or necessary, to pass through a first intervening firewall and make a connection to the server specified in the URL parameter. The URL may require special login information or expose additional scripting. At this point, the dispatcher executes the additional login information and code to get a file upload form. The dispatcher then sends the log files through this form.
MessageHandler instances describe a set of methods that are explained below in Table 2.
Method Description Constructor Called when the MessageHanlder instance is first created by the application.
HandleConfigMessage Called by the Messenger prior to the first HandleMessage call. Allows the user to define custom configuration information. HandleMessage Called by the Messenger whenever a feature is executed. The user code may return ti'ue, which will cause the Messenger to dispatch the log files. Regardless of the return value, the messenger will then examine the event object to determine if the event has been processed. The event will not be stored if the event has been processed.
Shutdown Called as the application is shutting down via the Java shutdown hook mechanism via the Messenger. The user code may return true, which will cause the Messenger to dispatch the log files.
Startup Called during the processing of the first event by the Messenger prior to invoking the HandleConfigMessageQ and HandleMessageQ methods.
The user code may return true, which will cause the Messenger to dispatch the log files.
Table 2
Typically, the Constructor method provides a significant amount of information concerning the initialization of the application. The code block in Figure 10 is the MessageHandler abstract class. The various getXXX and setXXX methods allow the user to set critical configuration information.
The illustrated framework allows the user to define many elements to further enhance the ability of the application to transfer data. Table 3 below describes various methods and what operation they perform (with the asterisk methods being optional).
Method Description addFormElement* Allows the user to set additional form element and values. Useful for building more complex file upload mechanisms that may require additional user name and password fields. areMessagesEnabled Returns true if message processing is allowed. enableMessages* User may turn on and off message processing. Turning off message processing means that messages will not be logged and log files cannot be transferred. getBuildName The name of the application's specific build in the form of a string. getFormElements A HashMap of all the custom form elements specified by calls to the addFormElement method. Called by the default LogDispatcher. getLogDispatcher An instance of the LogDispatcher that will send log files back to the server. The user has the option of replacing the default
LogDispatcher with a custom implementation. This custom implementation may support any communication mechanism the user wishes. getLoginName The form on the server may be behind a basic HTTP authentication scheme. A custom LogDispatcher calls this method to get the name of an authorized user when using a multi-page authentication scheme. getLoginPassword A custom LogDispatcher calls this method to get the password of an authorized user when using a multi-page authentication scheme. The getLoginUserQ method must return a user name for this method to work. getLoginUrl A custom LogDispatcher calls this method to get the URL for the
HTML page containing the file upload form when using a multi-page authentication scheme. getMaximumFileAge Called by the Messenger to determine how long files may be stored on the system running the application containing the remote node component. getProjectName The name of the application 's specific project in the form of a string. getProxyServerHost The default LogDispatcher calls this method to get the name of the proxy server in the same domain as that of the application running the remote node. getProxyServerPort The default LogDispatcher calls this method to get the port of the proxy server in the same domain as that of the application running the remote node. getProxyUserName The default LogDispatcher calls this method to get the name of the user to authenticate on proxy server in the same domain as that of the application running the remote node. getProxyUserPassword The default LogDispatcher calls this method to get the password of the user to authenticate on proxy server in the same domain as that of the application running the remote node. getϋrl Called by the default LogDispatcher to get the URL of the html page containing the file upload form on the server. getWorkDirectory Returns the location where the Messenger should write log files prior to dispatching. setBuildName Sets the name of the build. Set during initialization of the customized
MessageHandler. setLogDispatcher* Sets a customized LogDispatcher. Set during initialization of the customized MessageHandler. setLoginName* Sets the login name used by a customized LogDispatcher. Set during initialization of the customized MessageHandler. setLoginPassword* Sets the login password used by a customized LogDispatcher Set during initialization of the customized MessageHandler. setLoginUrl* Sets the login URL used by a customized LogDispatcher Set during initialization of the customized MessageHandler. setMaximumFileAge* Sets the maximum number of days to retain a given log file. Set during initialization of the customized MessageHandler. setProjectName Sets the project name. Set during initialization of the customized
MessageHandler. setProxyServerHost* Sets the proxy server host. Set during initialization of the customized
MessageHandler. setProxyServerPort* Sets the proxy server port. Set during initialization of the customized
MessageHandler. setProxyUserName* Sets the user name for proxy authentication. Set during initialization of the customized MessageHandler. setProxyUserPassword* Sets the password for the user to be authenticated by the proxy server.
Set during initialization of the customized MessageHandler. setUrl Sets the URL containing the file upload form. Set during initialization of the customized MessageHandler. setWorkDirectory Sets the directory where the Messenger will write log files. Set during initialization of the customized MessageHandler. setDomain A set method for unique name representing the domain of the company producing the application that is using the logging code.
The value is used to match a log file to a project. getDomain A get method to obtain the current value of the domain name. setKey A set function for the unique 32 character string identifying the project and build to associate log file data. getKey A get function for the unique 32 character string identifying the project and build to associate log file data.
Table 3
The Messenger and LogDispatcher read these methods to persist and transfer log files.
Step 2 The Messenger.storefeature (...) method tracks feature data. It takes two parameters. The first is an instance of the customized message handler. The second is a string containing the name of the feature. The resulting information about the number of executions of the feature indicates the value of that feature. Of course, a highly used feature is often more valuable than a rarely or never used feature. In a representative embodiment, the Messenger.storefeature (...) method is added to each feature entry point within the application. Step 3
The remotely running application may be behind a transparent or authenticating proxy server. The user's application installation component should find this out and collect the proxy server's address, port, user name and password if necessary and then write this information to a configuration file that the custom MessageHandler instance can read. The MessageHandler should then call the setProxyServerHostO, setProxyServerPort(...), setProxyUserName(...), setProxyUserPassword methods with the configured information. These integration steps make it possible for the user's application to now collect and transfer feature-tracking information. Figure 7 shows a representative process flow that occurs when a user executes a tracked feature. As seen in Figure 7, the routine begins when a user executes a feature at step 700. At step 702, the feature tracking code is executed. A test is then ran at step 704 to determine whether this is the first feature execution for the application ran. If so, the MessageHandler startup method is called at step 706. A test is then ran at step 708 to determine whether the log files should be dispatched. If not, the MessageHandler' s handle configuration message method is called at step 710. The routine then calls the MessageHandler' s handle message method at step 712, which step is also reached when the outcome of the test at step 714 is positive. Step 714 tests whether a dispatch is due to a startup event. After step 712, the feature statistics are updated at step 716. A test is then performed at step 718 to determine whether the feature should be written to the log file. If so, the log file is written at step 720. Following step 720, or if the outcome of the test at step 718 is negative, a test is performed at step 722 to determine whether the log files should be dispatched. If so, the remote node collects all log files at step 724, connects to the remote server at step 726, and then tests at step 728 to determine whether it can connect to the server. If so, a test is performed at step 730 to determine whether the remote server has a file upload form. If not, or if the outcome of the test at step 728 is negative, a test is performed at step 734 to determine if the file is older than a user-defined maximum number of days. After a positive outcome of the test at step 730, the file is uploaded at step 732. After step 732 or a positive outcome to step 734, the file is deleted at step 736 and control returns to the test at step 714 to complete the processing. A log file preferably is a binary file containing a series of specially formatted events. Preferably, the file comprises aggregated feature information rather than one entry per feature execution (as is common in traditional log files) to ensure that the file is smaller than traditional log files. The use of aggregated feature information is not a limitation of the invention, however. The format can be read and integrated into a database very efficiently because each field is well defined. Traditional log files have to be read line by line and then the data must be parsed out of the text. This is often difficult to implement and consequently error prone and performs poorly. The formatting is described in terms of number of unsigned bytes. The following key describes the nomenclature as defined by the Java Virtual Machine (JVM) Specification. In particular, U followed by a number is an unsigned 8-bit byte of the given length. Ul is a single unsigned while U4 represents 4 unsigned bytes. U followed by [] indicates that it is an array of bytes. A [] enclosing another field name indicates that that field name specifies the length of the array.
Each event construct preferably is derived from a based event format such as described in Table 4 below.
Field Description EventJType Each event has a numeric type associated with it.
This type field makes it possible for the reader to know the format for the given event.
Event_Data_Length The length of the data portion of the event. Event Data The event data itself. Table 4
Preferably, all events are derived from this one format. Consequently, the events preferably begin with Event_Type and Event_Data_Length but may not include the Event_Data section (as the events typically override this field with their specific implementations), hi a representative embodiment, the file format is a collection of Base_Events.
A feature event type represents a series of feature invocations. Each feature may have one or more events in the log file. There are often many feature events in the log file for various features. Table 5 below describes the fields.
Field Description
Feature_Name_Length The length of the name field.
Feature_Name The name of the feature.
Executions The number of times the feature have been executed. Table 5
A user configuration event type is generated by the integrator and allows storage of an arbitrary amount of additional configuration information. Table 6 below describes the fields.
Field Description
Configuration J3ntry_Name_Length The length of the configuration entry name. Configuration_Entry_Name The configuration entry name. Configuration_Entry_Value_Length The length of the configuration's value. Configuration_Entry_Value The configuration entry's value. Table 6
The following provides additional details regarding a preferred technique by which log files are processed and transported according to the invention. As noted above, preferably log files track features, faults and/or failures of a software application, and they are transported in a highly compact footprint to allow for high performance, lightweight processing in a scalable manner. To this end, a preferred log file format according to the invention is highly efficient, sequential, relative, and referential. Efficiency is achieved by using small messages, preferably in a binary file (as opposed to, say, XML or other human readable text), and by compressing whole numbers to save log file space. Preferably, all messages are written in order of their real world occurrence. Timestamps for messages preferably are relative to a first time-stamped message. Thus for example, assume a first message indicates that an event occurred at 12:00 and that a second message occurred at 12:01. The first message stores 12:00 in milliseconds from the UTC epoch date, and the second message is stored with the number of milliseconds between the first event and the second event. Once again, this encoding technique saves log file space. The reporting mechanism uses the time data to provide detailed information about when a user took a given action with respect to the application or when the application responded in a certain way. The reporting mechanism can segment the data so that feature usage information can be examined per hour, per day, per week, per month, or any other time scale. The log file format preferably is also referential, as some messages (e.g., like feature execution messages), reference standard feature messages. Moreover, preferably a feature message creates an index for a feature name and uses this index to count the number of feature invocations.
In a representative embodiment, there are several different scenarios that are tracked. In a first scenario, the code logs a first feature extraction. In such case, the feature logging code preferably opens a new log file and the logging code writes the following messages, preferably in order: project, configuration, node type and feature. In a second scenario, the code logs follow on feature executions. In this scenario, the logging code preferably adds just one message, indicating feature execution, hi a third scenario, the logging code logs a feature failure. In this scenario, the logging code preferably adds just one message, indicating feature failure. Finally, in a last scenario, the logging code logs a feature fault. Just one message is created again, indicating a feature fault. Sample logging code with the associated logging statements is illustrated in Figure 9. Preferably, each log file is made up of a series of messages. Each message typically has some common formatting and then become message type specific. The following are illustrative message types:
■ NodeType Message - the type of deployed instance of software ■ Project Message - the project name, e.g., the name of the deployed instance of software that is being tracked
■ Feature Name - the feature being tracked
■ Startup Message - the time the application started up
■ Feature Execution Message - the feature executed and when it executed ■ Config Message - information about the environment and configuration in which the application is deployed
■ UserConfig Message — placeholder for additional configuration information desired by the user of the API (e.g., gather more info about a server upon which an application is installed) ■ Feature Failure Message - information about a feature failing (feature fails to complete because for example, user entered improperly formatted data), the time it failed, and some info about why it failed
■ Feature Fault Message - information about a fault occurring (exception), the time it occurred, and some info about why it occurred ■ Feature Reset Message - mechanism to internally identify features taking up minimum data size - resets the feature count in case of very high number of features
■ Subnode User Message - information about the configuration and environment of the end user's machine (e.g., what web browser used, etc.) ■ Subnode User Configuration - placeholder for customizable information about an end-user's configuration
■ Subnode Feature Execution - features executed on the end user's configuration, versus features executed on a server (needed to track client processing in situation such as with Rich Internet Applications or DHTML executing on a end user client versus on the web application server)
■ Subnode Failure Message - failures encountered on end user's machine
■ Subnode Fault Message — faults occurring on end user's machine - e.g., Javascript runtime errors across browsers ■ Subnode Reset Message - as with Feature Reset Message but for end-user features
■ Sub Type Message - space to grow to track additional information
■ Null Message - for testing purposes to align columns within log file to make them more easily discernable
Preferably, the order of messages in a log file is as follows:
1. Project - preferably only one message is present per log file
2. Configuration - preferably only one message can be present per log file 3. User Configuration - 0 or more can be present
4. Node Type — preferably only one message can be present per log file
5. {Startup | Feature | Sub Node User} — 1 or more can be present per log file
6. {Feature | Feature Execution | Feature Failure | Feature Fault | Reset | Sub Node User I Sub Node User Configuration | Sub Node Feature Execution | Sub Node Failure | Sub Node Fault | Sub Node Reset } - 0 or more can be present
Data is stored in the log file in the manner of a database record, such as indicated below:
• Message Format is "name" : size - Type : 1 byte - Time Stamp: 3 bytes
- Message Type Specific Data: variable length depending on the type of Message, e.g., Feature Execution Message or Subnode User Configuration or other To capture the desired information while still maintaining high performance goals for the feature tracking system, the log files preferably use a binary file format, which is now described in more detail. Figure 10 illustrates a representative file format (with bytes represented as hexadecimal).
The following describes an encoding method for generating the binary file format and, in particular, that takes in a number and compresses it down to a smallest number of bytes possible. The method begins by initializing a buffer to a base value that represents zero, the smallest non-negative whole number. The most significant bit (MSB) for each byte in the buffer is reserved to indicate whether the byte is a part of a number or the end of a number. This means that only 7 bits of a byte actually contain a numeric value while the most significant bit is an indicator. The encoding process then goes on to break the number from 8 to 7 bits across the buffer, shifting the bits 7 places to the right. The MSB portions of the number are encoded within each byte of the buffer with the MSB set to 1. The final bits of the number are encoded with a 0 at the MSB. Decoding the number is a process of checking if the buffer's MSB bit is set to 1. If so, the routine reads 7 bits and then shifts the values left for seven bits into a target data type, such as a 32 or 64 bit data type. When the MSB is set to 0, the routine reads the remaining 7 bits, performs the same shift operation and terminates the read process. The result is the number used as the input for the encoder.
Thus, according to an embodiment of the invention, a data type represents a whole number that is compressed to fit into a smallest number of necessary bits. The scheme preferably works by breaking numbers across a series of bytes. The first N bytes have the most significant bit (MSB) set to 10xx xxxx where x are the bits that represent the actual number. The last byte has its MSB set to Oxxx xxxx. How the scheme operates is best seen by example. For example, the number 1 is stored according to the present scheme as 0000 0001. Normal encoding would be 0000 0000 0000 0000 0000 0000 0000 0001, taking up 4 bytes; as can be seen, the present scheme uses just 1 byte. As another example, the number 1,000 is stored as 1000 0111 0110 1000. Normal encoding would be 0000 0000 0000 0000 0000 0000 1110 1000, taking up 4 bytes, whereas the present scheme uses just 2 bytes. The number 100,000 is stored aslOOO 0110 1000 1101 0010 0000. Normal encoding would be 0000 0000 0000 0001 1000 0110 1010 0000, once again taking up 4 bytes, wherein the present scheme uses just 3 bytes. This technique substantially reduces the log file size, especially as compared to human readable (e.g., XML) or even conventional binary number (4 byte) encoding..
The above-described log file format and the compressed data type enable the system to track very advanced software application usage data about features, failures and faults in a high performance manner that is transparent for the end user. Most of the log data, which are mostly numbers, are directly mapped from a disk representative to an in-memory representation. As a result, the data is very compact and can be loaded and transported very efficiently. Moreover, preferably, the location of the data is very specific and ordered, which facilitates reducing the overall size of the log file. Thus, in an illustrative embodiment, a log file comprises a series of messages. Each message represents some interesting event, such as a feature execution or failure. The messages are preferably formatted with specific formats and field lengths. Variable length fields preferably are prefixed with a field indicating the length of the following field. Numeric fields preferably are variable length but use the above-described data type. As described above, the encoding scheme enables a number to be stored within a binary field.
As previously noted, the hardware and software systems in which the invention is illustrated are merely representative. The invention may be practiced, typically in software, on one or more machines. Generalizing, a machine typically comprises commodity hardware and software, storage (e.g., disks, disk arrays, and the like) and memory (RAM, ROM, and the like). The particular machines used in the network are not a limitation of the present invention. A given machine includes network interfaces and software to connect the machine to a network in the usual manner. As illustrated in Figure 1, the present invention may be implemented as a managed service (e.g., in an ASP model) using the illustrated set of machines, which are connected or connectable to one or more networks. More generally, the service is provided by an operator using a set of one or more computing-related entities (systems, machines, processes, programs, libraries, functions, or the like) that together facilitate or provide the inventive functionality described above. In a typical implementation, the service comprises a set of one or more computers. A representative machine is a network-based server running commodity (e.g. Pentium-class) hardware, an operating system (e.g., Linux, Windows, OS-X, or the like), an application runtime environment (e.g., Java, .ASP), and a set of applications or processes (e.g., Java applets or servlets, linkable libraries, native code, or the like, depending on platform), that provide the functionality of a given system or subsystem. As described, the service may be implemented in a standalone server, or across a distributed set of machines. Typically, a server connects to the publicly-routable Internet, a corporate intranet, a private network, or any combination thereof, depending on the desired implementation environment.
The hosted service may be implemented in a multi-server cluster environment that is designed to scale efficiently. Each server is designated with a primary and secondary series of tasks. Preferably, one server is dynamically set to be a master server, which server determines the secondary tasks to be performed by all servers. All servers update their existence within a database, and the servers cooperate to determine which server will be the master. The servers in the cluster are assigned tasks (such as log import and event processing) by the master server.
Having described our invention, what we now claim is set forth below.

Claims

1. A method of tracking an application that is deployed as a set of application instances, each of the application instances being instrumented to cause generation of a log file, comprising: providing a web-based hosted service at which the log files from the set of application instances are received, wherein a given log file comprises data associated with a set of application feature executions, failures and faults; aggregating the log files; and enabling a permitted entity to access and view the aggregated log files.
2. The method as described in claim 1 wherein the application is one of: a server-based application, a web-based application, and a rich Internet application.
3. The method as described in claim 1 wherein the log files are received at the web-based hosted service via http transport.
4. The method as described in claim 1 wherein given information in the log file is encoded in a compact data structure to reduce a size of the log file.
5. The method as described in claim 1 wherein the log file data comprises a set of messages associated with the application feature executions, failures and faults, wherein the set of messages are written in a sequential order of the application feature executions, failures and faults.
6. The method as described in claim 5 wherein at least first and second messages of the set of messages each have timestamps, and wherein the timestamp of the second message is a function of the timestamp in the first message.
7. The method as described in claim 1 wherein the permitted entity is one of: a provider of the application, and a third party.
8. The method as described in claim 1 wherein the application instances are deployed across a public network and the log files are received from a set of end users that are located throughout the public network.
9. The method as described in claim 1 wherein the log file is generated at an end user machine.
10. The method as described in claim 1 wherein the log file is generated at a gateway server associated with the web-based hosted service.
11. A method of tracking a web application deployed across a wide area network as the web application is used within local end user web browser environments, comprising: providing a web-based hosted service at which given feature information generated from attempted use of the web application within each local end user web browser environment is received; aggregating the given feature information; and enabling a permitted entity to access and view the aggregated feature information.
12. The method as described in claim 11 wherein the web application is one of: a web page having a browser-enabled script, and a rich Internet application.
13. The method as described in claim 11 wherein the given feature information is received at the web-based hosted service via http transport.
14. The method as described in claim 11 wherein the given feature information is encoded in a compact data structure to reduce a size of the given feature information.
15. The method as described in claim 11 wherein the given feature information is one of: a feature execution, a feature failure, and a feature fault.
16. The method as described in claim 11 wherein the permitted entity is one of: a provider of the application, and a third party.
17. A computer-readable medium having computer-executable instructions for performing the method steps of claim 1.
18. A server comprising a processor, together with a computer-readable medium having processor-executable instructions for performing the method steps of claim 1.
PCT/US2006/026337 2005-07-12 2006-07-07 Distributed capture and aggregation of dynamic application usage information WO2007008590A2 (en)

Priority Applications (5)

Application Number Priority Date Filing Date Title
JP2008521441A JP5354776B2 (en) 2005-07-12 2006-07-07 Distributed acquisition and aggregation of dynamic application usage information
EP06786477A EP1907941A4 (en) 2005-07-12 2006-07-07 Distributed capture and aggregation of dynamic application usage information
CA002657487A CA2657487A1 (en) 2005-07-12 2006-07-07 Distributed capture and aggregation of dynamic application usage information
CN200680033282.9A CN101371245B (en) 2005-07-12 2006-07-07 Distributed capture and aggregation of dynamic application usage information
KR1020087003403A KR101300360B1 (en) 2005-07-12 2006-07-07 Distributed capture and aggregation of danamic application usage information

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US69856705P 2005-07-12 2005-07-12
US60/698,567 2005-07-12
US11/481,530 US7356590B2 (en) 2005-07-12 2006-07-06 Distributed capture and aggregation of dynamic application usage information
US11/481,530 2006-07-06

Publications (2)

Publication Number Publication Date
WO2007008590A2 true WO2007008590A2 (en) 2007-01-18
WO2007008590A3 WO2007008590A3 (en) 2007-07-19

Family

ID=37637743

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2006/026337 WO2007008590A2 (en) 2005-07-12 2006-07-07 Distributed capture and aggregation of dynamic application usage information

Country Status (7)

Country Link
US (3) US7356590B2 (en)
EP (1) EP1907941A4 (en)
JP (1) JP5354776B2 (en)
KR (1) KR101300360B1 (en)
CN (1) CN101371245B (en)
CA (1) CA2657487A1 (en)
WO (1) WO2007008590A2 (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2010025274A1 (en) * 2008-08-27 2010-03-04 Eric Sven-Johan Swildens Method and system for testing interactions between web clients and networked servers
WO2012032019A1 (en) 2010-09-07 2012-03-15 Nestec S.A. Ergonomic handle with user-interface
CN105224445A (en) * 2015-10-28 2016-01-06 北京汇商融通信息技术有限公司 Distributed tracking system
EP2696297B1 (en) * 2011-03-30 2020-06-03 China Unionpay Co., Ltd. System and method for generating information file based on parallel processing
US20210026811A1 (en) * 2007-11-30 2021-01-28 Red Hat, Inc. Generating file usage information

Families Citing this family (114)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7849085B2 (en) * 2004-05-18 2010-12-07 Oracle International Corporation System and method for implementing MBSTRING in weblogic tuxedo connector
US7356590B2 (en) * 2005-07-12 2008-04-08 Visible Measures Corp. Distributed capture and aggregation of dynamic application usage information
US20070055602A1 (en) * 2005-09-02 2007-03-08 Mohn Anne M Methods and systems for financial account management
US7930363B2 (en) * 2005-10-12 2011-04-19 Powerreviews, Inc. Application service provider delivery system
WO2007056344A2 (en) 2005-11-07 2007-05-18 Scanscout, Inc. Techiques for model optimization for statistical pattern recognition
US20070192161A1 (en) * 2005-12-28 2007-08-16 International Business Machines Corporation On-demand customer satisfaction measurement
US8856276B2 (en) * 2006-05-11 2014-10-07 International Business Machines Corporation Method, system and program product for collecting web metric data
US20090271324A1 (en) * 2006-08-11 2009-10-29 Evapt, Inc. Systems and methods for metered software as a service
US8255489B2 (en) * 2006-08-18 2012-08-28 Akamai Technologies, Inc. Method of data collection among participating content providers in a distributed network
US8014726B1 (en) 2006-10-02 2011-09-06 The Nielsen Company (U.S.), Llc Method and system for collecting wireless information transparently and non-intrusively
US8260252B2 (en) 2006-10-02 2012-09-04 The Nielsen Company (Us), Llc Method and apparatus for collecting information about portable device usage
US7996817B1 (en) * 2006-10-27 2011-08-09 Avaya Inc. Automatic validation of test results using componentized software object reflection
US8745647B1 (en) 2006-12-26 2014-06-03 Visible Measures Corp. Method and system for internet video and rich media behavioral measurement
US20080228581A1 (en) * 2007-03-13 2008-09-18 Tadashi Yonezaki Method and System for a Natural Transition Between Advertisements Associated with Rich Media Content
US20080288498A1 (en) * 2007-05-14 2008-11-20 Hinshaw Foster D Network-attached storage devices
US7747988B2 (en) * 2007-06-15 2010-06-29 Microsoft Corporation Software feature usage analysis and reporting
US7870114B2 (en) 2007-06-15 2011-01-11 Microsoft Corporation Efficient data infrastructure for high dimensional data analysis
US7739666B2 (en) 2007-06-15 2010-06-15 Microsoft Corporation Analyzing software users with instrumentation data and user group modeling and analysis
US8176476B2 (en) * 2007-06-15 2012-05-08 Microsoft Corporation Analyzing software usage with instrumentation data
US8990378B2 (en) * 2007-07-05 2015-03-24 Interwise Ltd. System and method for collection and analysis of server log files
US8321556B1 (en) 2007-07-09 2012-11-27 The Nielsen Company (Us), Llc Method and system for collecting data on a wireless device
US7975262B2 (en) * 2007-08-16 2011-07-05 Microsoft Corporation Error tracing with context history
US8577996B2 (en) * 2007-09-18 2013-11-05 Tremor Video, Inc. Method and apparatus for tracing users of online video web sites
US8549550B2 (en) 2008-09-17 2013-10-01 Tubemogul, Inc. Method and apparatus for passively monitoring online video viewing and viewer behavior
US20090083363A1 (en) * 2007-09-26 2009-03-26 Microsoft Corporation Remote monitoring of local behavior of network applications
US8108513B2 (en) * 2007-09-26 2012-01-31 Microsoft Corporation Remote monitoring of local behavior of network applications
US8543683B2 (en) * 2007-09-26 2013-09-24 Microsoft Corporation Remote monitoring of local behavior of network applications
US8271836B2 (en) * 2007-09-27 2012-09-18 Microsoft Corporation Capturing diagnostics in web browser applications
US20090089751A1 (en) * 2007-09-28 2009-04-02 Microsoft Corporation Exposing features of software products
US20090113397A1 (en) * 2007-10-25 2009-04-30 Signetag, Inc. Dynamic, secure software tagging for software asset management with respect to deployment, configuration, and usage
US20090276856A1 (en) * 2007-11-26 2009-11-05 International Business Machines Corporation License management facility
US20090138292A1 (en) * 2007-11-26 2009-05-28 International Business Machines Corporation Driving software product changes based on usage patterns gathered from users of previous product releases
US8307101B1 (en) 2007-12-13 2012-11-06 Google Inc. Generic format for storage and query of web analytics data
US8429243B1 (en) * 2007-12-13 2013-04-23 Google Inc. Web analytics event tracking system
US20090248729A1 (en) * 2008-03-25 2009-10-01 Microsoft Corporation Online application platform and user communities
US20090248855A1 (en) * 2008-03-31 2009-10-01 International Business Machines Corporation Method for monitoring web page statistics
US20090313628A1 (en) * 2008-06-13 2009-12-17 Microsoft Corporation Dynamically batching remote object model commands
US8612751B1 (en) * 2008-08-20 2013-12-17 Cisco Technology, Inc. Method and apparatus for entitled data transfer over the public internet
US9612995B2 (en) 2008-09-17 2017-04-04 Adobe Systems Incorporated Video viewer targeting based on preference similarity
US8560675B2 (en) * 2009-04-01 2013-10-15 Comscore, Inc. Determining projection weights based on a census data
AU2009345651B2 (en) 2009-05-08 2016-05-12 Arbitron Mobile Oy System and method for behavioural and contextual data analytics
US20100306363A1 (en) * 2009-05-26 2010-12-02 Erwien Saputra Determining completion of a web server download session at a database server
US8868601B2 (en) 2009-08-17 2014-10-21 International Business Machines Corporation Distributed file system logging
CN101645905B (en) * 2009-08-31 2012-10-24 深圳市茁壮网络股份有限公司 Data downloading method and data uploading method and browser
US20110093783A1 (en) * 2009-10-16 2011-04-21 Charles Parra Method and system for linking media components
US9298569B1 (en) * 2009-11-13 2016-03-29 Arris Enterprises, Inc. Log storage and retrieval for broadband communications devices
CA2781299A1 (en) * 2009-11-20 2012-05-03 Tadashi Yonezaki Methods and apparatus for optimizing advertisement allocation
US20110138307A1 (en) * 2009-12-09 2011-06-09 Microsoft Corporation Publishing client device usage data
US8332500B1 (en) * 2010-01-29 2012-12-11 Google Inc. Transmitting data requests based on usage characteristics of applications
CN102195795B (en) * 2010-03-19 2014-03-12 Tcl集团股份有限公司 Intelligent district log system and log recording method thereof
JP5896382B2 (en) 2010-06-24 2016-03-30 ザ ニールセン カンパニー （ユーエス） エルエルシー Network server equipment configuration and related methods for handling non-parametric, multi-dimensional, spatial and temporal human behavior or a wide range of technical observations
US9280391B2 (en) 2010-08-23 2016-03-08 AVG Netherlands B.V. Systems and methods for improving performance of computer systems
KR101021812B1 (en) * 2010-07-05 2011-03-17 (주)지란지교소프트 Method for organizational network analysis using monitoring endpoint devices and recording-medium for recorded program thereof
CN101895431B (en) * 2010-08-20 2012-02-29 烽火通信科技股份有限公司 RIA-based high-performance Web transmission network management system and implementation method thereof
US8340685B2 (en) 2010-08-25 2012-12-25 The Nielsen Company (Us), Llc Methods, systems and apparatus to generate market segmentation data with anonymous location data
WO2012031165A2 (en) * 2010-09-02 2012-03-08 Zaretsky, Howard System and method of cost oriented software profiling
US8498213B2 (en) * 2010-09-14 2013-07-30 Brocade Communications Systems, Inc. Manageability tools for lossless networks
US8542583B2 (en) 2010-09-14 2013-09-24 Brocade Communications Systems, Inc. Manageability tools for lossless networks
US20120117227A1 (en) * 2010-11-10 2012-05-10 Sony Corporation Method and apparatus for obtaining feedback from a device
US8996689B2 (en) * 2010-11-11 2015-03-31 Telefonaktiebolaget L M Ericsson (Publ) Methods of operating networks, application servers, and wireless devices supporting machine-to-machine applications
US10102354B2 (en) 2010-11-19 2018-10-16 Microsoft Technology Licensing, Llc Integrated application feature store
US20120140255A1 (en) * 2010-12-02 2012-06-07 Ricoh Company, Ltd. Application launcher apparatus
US20120159341A1 (en) 2010-12-21 2012-06-21 Microsoft Corporation Interactions with contextual and task-based computing environments
US20120166522A1 (en) * 2010-12-27 2012-06-28 Microsoft Corporation Supporting intelligent user interface interactions
JP5648911B2 (en) * 2010-12-27 2015-01-07 インターナショナル・ビジネス・マシーンズ・コーポレーションＩｎｔｅｒｎａｔｉｏｎａｌ Ｂｕｓｉｎｅｓｓ Ｍａｃｈｉｎｅｓ Ｃｏｒｐｏｒａｔｉｏｎ System, program and method
US20120246619A1 (en) * 2011-03-23 2012-09-27 Oracle International Corporation Automated testing of browser based rich internet applications (ria) delivered through proprietary frameworks
US9223632B2 (en) 2011-05-20 2015-12-29 Microsoft Technology Licensing, Llc Cross-cloud management and troubleshooting
US8789020B2 (en) 2011-09-01 2014-07-22 International Business Machines Corporation Interactive debugging environments and methods of providing the same
US9344335B2 (en) 2011-09-09 2016-05-17 Microsoft Technology Licensing, Llc Network communication and cost awareness
CN103019918B (en) * 2011-09-27 2015-07-15 腾讯科技（深圳）有限公司 Method and device for acquiring operating frequency of application program
US9075914B2 (en) 2011-09-29 2015-07-07 Sauce Labs, Inc. Analytics driven development
CN108256092B (en) * 2011-10-04 2020-04-17 谷歌有限责任公司 Combined activity history on a device
US8359389B1 (en) * 2011-10-11 2013-01-22 Google Inc. Monitoring application program resource consumption
EP2793463A4 (en) 2011-12-12 2015-06-03 Sony Corp Client terminal, client terminal operating method, transmission server, transmission server operating method, and program
KR101691245B1 (en) * 2012-05-11 2017-01-09 삼성에스디에스 주식회사 System and method for web service monitoring
US9704171B2 (en) * 2012-06-05 2017-07-11 Applause App Quality, Inc. Methods and systems for quantifying and tracking software application quality
US20130346917A1 (en) * 2012-06-22 2013-12-26 Microsoft Corporation Client application analytics
US20140025441A1 (en) * 2012-07-19 2014-01-23 Sap Ag Peer support gamification by application knowledge scoring in social networks
US9183110B2 (en) 2012-11-26 2015-11-10 Google Inc. Centralized dispatching of application analytics
US10187481B2 (en) * 2012-12-12 2019-01-22 Facebook, Inc. Organizing application-reported information
CN102981934A (en) * 2012-12-21 2013-03-20 曙光信息产业(北京)有限公司 Log transition method and log transition device
US9628414B1 (en) * 2013-03-15 2017-04-18 Google Inc. User state based engagement
US20140282618A1 (en) * 2013-03-15 2014-09-18 Telemetry Limited Digital media metrics data management apparatus and method
US9998536B2 (en) 2013-05-29 2018-06-12 Microsoft Technology Licensing, Llc Metered network synchronization
US9405531B2 (en) * 2013-07-16 2016-08-02 Software Ag Methods for building application intelligence into event driven applications through usage learning, and systems supporting such applications
RU2598988C2 (en) * 2013-08-07 2016-10-10 Фиизер Инк. Methods and systems for searching for application software
US9349142B2 (en) 2013-10-09 2016-05-24 Ebay Inc. Reflow of data presentation using tracking data
US20160048850A1 (en) * 2014-08-15 2016-02-18 Clarence Wooten Collecting, aggregating, analyzing and reporting traction data on internet-connected applications
US11695657B2 (en) 2014-09-29 2023-07-04 Cisco Technology, Inc. Network embedded framework for distributed network analytics
GB2546205B (en) 2014-10-24 2021-07-21 Google Llc Methods and systems for automated tagging based on software execution traces
GB2594372B (en) * 2014-10-24 2022-04-13 Google Llc Methods and systems for automated tagging based on software execution traces
US9928233B2 (en) 2014-11-12 2018-03-27 Applause App Quality, Inc. Computer-implemented methods and systems for clustering user reviews and ranking clusters
US10706073B1 (en) * 2014-12-03 2020-07-07 Amazon Technologies, Inc. Partitioned batch processing for a usage analysis system
US10362086B2 (en) 2014-12-12 2019-07-23 Medidata Solutions, Inc. Method and system for automating submission of issue reports
US10305759B2 (en) * 2015-01-05 2019-05-28 Cisco Technology, Inc. Distributed and adaptive computer network analytics
CN104598372B (en) * 2015-02-16 2017-11-03 联想(北京)有限公司 A kind of information processing method and device
US10467000B2 (en) 2015-03-18 2019-11-05 International Business Machines Corporation Extending the usage of integrated portals for a better user experience
US10003669B2 (en) * 2015-07-28 2018-06-19 DISH Technologies L.L.C. Methods and apparatus to create and transmit a condensed logging data file
US10311500B2 (en) * 2016-09-01 2019-06-04 Facebook, Inc Methods and systems for developer onboarding for software-development products
CN107517128B (en) * 2017-08-24 2020-06-19 北京小米移动软件有限公司 Data transmission method, device and equipment
US10810099B2 (en) * 2017-09-11 2020-10-20 Internatinal Business Machines Corporation Cognitive in-memory API logging
US10691082B2 (en) * 2017-12-05 2020-06-23 Cisco Technology, Inc. Dynamically adjusting sample rates based on performance of a machine-learning based model for performing a network assurance function in a network assurance system
US11263328B2 (en) * 2018-09-13 2022-03-01 Vmware, Inc. Encrypted log aggregation
CN110727567B (en) * 2019-09-09 2024-02-02 平安证券股份有限公司 Method, device, computer equipment and storage medium for detecting software quality
CN111597097B (en) * 2020-04-21 2023-12-12 山西宇钶技术发展有限公司 Big data processing method and system
US20220114092A1 (en) * 2020-10-13 2022-04-14 Dynatrace Llc Method And System For Estimating Garbage Collection Suspension Contributions Of Individual Allocation Sites
US11347625B1 (en) 2020-10-19 2022-05-31 Splunk Inc. Rendering a service graph illustrate page provider dependencies at query time using exemplars
US11693850B1 (en) 2020-10-19 2023-07-04 Splunk Inc. Rendering a service graph to illustrate page provider dependencies at an aggregate level
WO2022086610A1 (en) * 2020-10-19 2022-04-28 Splunk Inc. End-to-end visibility of a user session
JP7072625B1 (en) 2020-11-26 2022-05-20 任天堂株式会社 Game development system
CN112612664B (en) * 2020-12-24 2024-04-02 百度在线网络技术（北京）有限公司 Electronic equipment testing method and device, electronic equipment and storage medium
US20220374457A1 (en) * 2021-05-21 2022-11-24 Databricks Inc. Feature store with integrated tracking
CN113986270B (en) * 2021-11-16 2022-09-23 前海飞算云智软件科技(深圳)有限公司 Distributed application deployment method and device, storage medium and electronic equipment
US20230247034A1 (en) * 2022-02-01 2023-08-03 Sap Se Log entry buffer extension network

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6591228B1 (en) 2001-02-13 2003-07-08 Charles Schwab & Co., Inc. Centralized diagnostic logging service
US20050028171A1 (en) 1999-11-12 2005-02-03 Panagiotis Kougiouris System and method enabling multiple processes to efficiently log events

Family Cites Families (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6073241A (en) * 1996-08-29 2000-06-06 C/Net, Inc. Apparatus and method for tracking world wide web browser requests across distinct domains using persistent client-side state
US6647548B1 (en) * 1996-09-06 2003-11-11 Nielsen Media Research, Inc. Coded/non-coded program audience measurement system
US6353929B1 (en) * 1997-06-23 2002-03-05 One River Worldtrek, Inc. Cooperative system for measuring electronic media
US7302402B2 (en) * 1998-03-30 2007-11-27 International Business Machines Corporation Method, system and program products for sharing state information across domains
JPH11288381A (en) * 1998-04-02 1999-10-19 Tokyo Electric Power Co Inc:The Remote monitoring system for software
US6470388B1 (en) * 1999-06-10 2002-10-22 Cisco Technology, Inc. Coordinated extendable system for logging information from distributed applications
JP2001356940A (en) * 2000-06-13 2001-12-26 Canon Inc Processor, its method and computer readable memory
JP2002297258A (en) * 2001-03-30 2002-10-11 Canon Inc User supporting device, system and method for application, information processor, recording medium and program products
JP4752125B2 (en) * 2001-04-06 2011-08-17 大日本印刷株式会社 Computer system
JP2003050719A (en) * 2001-08-03 2003-02-21 Katsuyoshi Nagashima Device for monitoring utilization state of application software, and its software program
US7415437B2 (en) * 2001-10-31 2008-08-19 The United States Of America As Represented By The Secretary Of The Navy Business development process
JP4050497B2 (en) * 2001-11-06 2008-02-20 インフォサイエンス株式会社 Log information management apparatus and log information management program
JP2003186702A (en) * 2001-12-18 2003-07-04 Tsubasa System Co Ltd Terminal operation monitoring system and terminal operation monitoring method
WO2003069428A2 (en) * 2002-01-18 2003-08-21 Sales Automation Support, Inc. Mobile marketing system
US7546365B2 (en) * 2002-04-30 2009-06-09 Canon Kabushiki Kaisha Network device management system and method of controlling same
JP2004021549A (en) * 2002-06-14 2004-01-22 Hitachi Information Systems Ltd Network monitoring system and program
JP4490029B2 (en) * 2002-06-28 2010-06-23 キヤノン電子株式会社 Information analysis apparatus, control method therefor, information analysis system, and program
US20040186738A1 (en) * 2002-10-24 2004-09-23 Richard Reisman Method and apparatus for an idea adoption marketplace
JP2004185463A (en) * 2002-12-05 2004-07-02 Hitachi Electronics Service Co Ltd Web application monitoring system, computer for monitoring center and computer software
US20050216844A1 (en) * 2004-03-03 2005-09-29 Error Brett M Delayed transmission of website usage data
JP2004280398A (en) 2003-03-14 2004-10-07 Toshiba Corp Service provision side program, user side program and method
JP2007524144A (en) * 2003-06-18 2007-08-23 フジツー シーメンス コンピュータース ゲゼルシャフト ミット ベシュレンクテル ハフツング Cluster device
CA2433750A1 (en) * 2003-06-27 2004-12-27 Ibm Canada Limited - Ibm Canada Limitee Automatic collection of trace detail and history data
US7581224B2 (en) * 2003-07-10 2009-08-25 Hewlett-Packard Development Company, L.P. Systems and methods for monitoring resource utilization and application performance
US7457872B2 (en) * 2003-10-15 2008-11-25 Microsoft Corporation On-line service/application monitoring and reporting system
US20050125271A1 (en) * 2003-12-09 2005-06-09 Peltz Christopher J. Web services project planning
US7614064B2 (en) * 2004-08-23 2009-11-03 Microsoft Corporation Determining program boundaries through viewing behavior
US8078596B2 (en) * 2004-09-16 2011-12-13 International Business Machines Corporation Method and system for a non-hierarchical trace system using a hierarchical control interface
US7707189B2 (en) * 2004-10-05 2010-04-27 Microsoft Corporation Log management system and method
US7661030B2 (en) * 2004-11-30 2010-02-09 Sap Ag Propagating debug information in a web services environment
US8141111B2 (en) * 2005-05-23 2012-03-20 Open Text S.A. Movie advertising playback techniques
US7610266B2 (en) * 2005-05-25 2009-10-27 International Business Machines Corporation Method for vertical integrated performance and environment monitoring
US7356590B2 (en) * 2005-07-12 2008-04-08 Visible Measures Corp. Distributed capture and aggregation of dynamic application usage information
JP4638370B2 (en) * 2006-03-29 2011-02-23 富士重工業株式会社 Lane departure prevention device
US8992304B2 (en) * 2006-04-13 2015-03-31 Igt Methods and systems for tracking an event of an externally controlled interface
US8495204B2 (en) * 2006-07-06 2013-07-23 Visible Measures Corp. Remote invocation mechanism for logging
US7689682B1 (en) * 2006-08-16 2010-03-30 Resource Consortium Limited Obtaining lists of nodes of a multi-dimensional network
US20080066107A1 (en) * 2006-09-12 2008-03-13 Google Inc. Using Viewing Signals in Targeted Video Advertising
US8113844B2 (en) * 2006-12-15 2012-02-14 Atellis, Inc. Method, system, and computer-readable recording medium for synchronous multi-media recording and playback with end user control of time, data, and event visualization for playback control over a network
US7747988B2 (en) * 2007-06-15 2010-06-29 Microsoft Corporation Software feature usage analysis and reporting
US20090094584A1 (en) * 2007-10-04 2009-04-09 International Business Machines Corporation Non-intrusive asset monitoring framework for runtime configuration of deployable software assets
US8621491B2 (en) * 2008-04-25 2013-12-31 Microsoft Corporation Physical object visualization framework for computing device with interactive display

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050028171A1 (en) 1999-11-12 2005-02-03 Panagiotis Kougiouris System and method enabling multiple processes to efficiently log events
US6591228B1 (en) 2001-02-13 2003-07-08 Charles Schwab & Co., Inc. Centralized diagnostic logging service

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
GRAHAM HAMILTON, 18 October 2000 (2000-10-18), pages 1 - 81
See also references of EP1907941A4

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210026811A1 (en) * 2007-11-30 2021-01-28 Red Hat, Inc. Generating file usage information
US11669493B2 (en) * 2007-11-30 2023-06-06 Red Hat, Inc. Generating file usage information
WO2010025274A1 (en) * 2008-08-27 2010-03-04 Eric Sven-Johan Swildens Method and system for testing interactions between web clients and networked servers
US8185910B2 (en) 2008-08-27 2012-05-22 Eric Sven-Johan Swildens Method and system for testing interactions between web clients and networked servers
US8732722B2 (en) 2008-08-27 2014-05-20 Eric Sven-Johan Swildens Method and system for testing interactions between web clients and networked servers
USRE46055E1 (en) 2008-08-27 2016-07-05 Eric Sven-Johan Swildens Method and system for testing interactions between web clients and networked servers
WO2012032019A1 (en) 2010-09-07 2012-03-15 Nestec S.A. Ergonomic handle with user-interface
EP2696297B1 (en) * 2011-03-30 2020-06-03 China Unionpay Co., Ltd. System and method for generating information file based on parallel processing
CN105224445A (en) * 2015-10-28 2016-01-06 北京汇商融通信息技术有限公司 Distributed tracking system
CN105224445B (en) * 2015-10-28 2017-02-15 北京汇商融通信息技术有限公司 Distributed tracking system

Also Published As

Publication number Publication date
JP2009501386A (en) 2009-01-15
EP1907941A4 (en) 2009-10-21
JP5354776B2 (en) 2013-11-27
CN101371245B (en) 2012-09-05
US8135827B2 (en) 2012-03-13
CN101371245A (en) 2009-02-18
EP1907941A2 (en) 2008-04-09
US8626908B2 (en) 2014-01-07
US20100030785A1 (en) 2010-02-04
US20070016672A1 (en) 2007-01-18
WO2007008590A3 (en) 2007-07-19
US20130111016A1 (en) 2013-05-02
CA2657487A1 (en) 2007-01-18
KR20080052560A (en) 2008-06-11
KR101300360B1 (en) 2013-08-28
US7356590B2 (en) 2008-04-08

Similar Documents

Publication Publication Date Title
US8135827B2 (en) Distributed capture and aggregation of dynamic application usage information
US7966398B2 (en) Synthetic transaction monitor with replay capability
US7792948B2 (en) Method and system for collecting, aggregating and viewing performance data on a site-wide basis
EP1490775B1 (en) Java application response time analyzer
US9965758B2 (en) Troubleshooting transactions in a network environment
US8495204B2 (en) Remote invocation mechanism for logging
US8438276B1 (en) Method of monitoring network and application performance by analyzing web clients and web servers
US8533279B2 (en) Method and system for reconstructing transactions in a communication network
CN112039701B (en) Interface call monitoring method, device, equipment and storage medium
US20080162690A1 (en) Application Management System
US9760874B2 (en) Transaction tracing in a network environment
US10049403B2 (en) Transaction identification in a network environment
US11531612B2 (en) Methods for providing an enterprise synthetic monitoring framework
US20170012814A1 (en) System Resiliency Tracing
CN113411206B (en) Log auditing method, device, equipment and computer storage medium
US20170012839A1 (en) System Architecture Mapping

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 200680033282.9

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application
ENP Entry into the national phase

Ref document number: 2008521441

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

REEP Request for entry into the european phase

Ref document number: 2006786477

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2006786477

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 1020087003403

Country of ref document: KR

ENP Entry into the national phase

Ref document number: 2657487

Country of ref document: CA