Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
  • G06F11/25—Testing of logic operation, e.g. by logic analysers
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
  • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
  • G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
  • G06F11/0709—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
  • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
  • G06F11/079—Root cause analysis, i.e. error or fault diagnosis
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
  • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F15/00—Digital computers in general; Data processing equipment in general
  • G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F15/00—Digital computers in general; Data processing equipment in general
  • G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
  • G06F15/163—Interprocessor communication
  • G06F15/173—Interprocessor communication using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake

Definitions

• the present application is related to the concurrently-filed U.S. patent applications respectively identified as: attorney docket no. YOR920020097US1 entitled: "Methods And Apparatus For Managing Dependencies in Distributed Systems;” attorney docket no: SOM920020003US1 entitled: “Methods And Apparatus For Topology Discovery and Representation of Distributed Applications and Services;” attorney docket no. SOM920020004US1 entitled: “Methods And Apparatus For Impact Analysis and Problem Determination; " and attorney docket no. SOM920020005US1 entitled: “Methods And Apparatus For Dependency-based Impact Simulation and Vulnerability Analysis;” the disclosures of which are incorporated by reference herein.
• the present invention relates to distributed computing systems and, more particularly, to methods and apparatus for analyzing and determining the root cause of a service outage based on dependencies between the various components of such distributed computing systems.
• runtime generally refers to the time period when a piece of software is being executed and active in a computer system's memory, as opposed to being dormant and merely sitting in storage on a computer's hard drive.
• being able to compose e-business applications at runtime means having the capability to do so without the need to bring down and restart the system/application and without the need to recompile the application.
• the lifecycle of a computer program is: write program code -> compile (translate into machine code) -> run.
• a service often plays both roles (e.g., a name service is required by many applications and services but depends, itself, on the proper functioning of other services, such as the operating system and the network protocols and infrastructure) .
• dependency relationships are transitive, i.e., the dependent of a given component requires, in addition to the component itself, the components ' antecedent (s) .
• IEEE 1995 addresses software distribution/deployment/installation.
• the IEEE standard defines a mechanism for ensuring that new software components
• the IEEE standard identifies three kinds of relationships: prerequisite, exrequisite, corequisite, that facilitate such compatibility checks . This is done individually or every system on which new software needs to be installed. With the IEEE standard, the software inventories present on other systems are not taken into account. Furthermore, the IEEE standard does not deal with instantiated applications and services and therefore does not represent any means of determining the dependencies between components at runtime.
• Open Group Systems Management: Distributed Software Administration, CAE Specification C701, The Open Group, January 1998) extends IEEE 1387.2 by defining several commands (swinstall, swlist, swmodify, etc.) that are invoked by software installation tools on a specific system. Open Group also defines a software definition file format to make sure that the information required by the aforementioned commands is available from the system on which the commands are invoked.
• the shortcomings of IEEE 1387.2 i.e., confined to a single isolated system, no means for determining software dependencies at runtime) also apply to the Open Group specification .
• Patent No. 5,721,824 or updates/corrections/ fixes/patches
• the present invention provides techniques for identifying the root cause of a component failure and performing appropriate problem determination procedures in accordance with a computing environment.
• the inventive techniques may be applied to a distributed computing environment.
• the computing environment may also be an autonomic computing environment .
• a computer-based technique for determining a root cause of a condition (e.g., service outage) of at least one subject component in a computing environment comprises the following steps/operations .
• Identification comprises traversing at least a portion of a model representative of an existence of one or more relationships associated with at least a portion of components of the computing environment and which is capable of accounting for a full lifecycle (e.g., including deployment, installation and runtime) associated with at least one component of the computing environment .
• one or more procedures are performed in accordance with the one or more identified components to determine a condition status associated with each of the one or more identified components.
• the procedures may be carried out either stepwise or in combination, and may include, by way of example, process inspection, exercising, heartbeats, and status indicators.
• a component may be a service, an application, middleware, hardware, a device driver, an operating system or a system associated with the computing environment.
• the term "component” is not limited to these examples .
• FIG. 1 is a block diagram illustrating an example of a client-server application architecture with which features of the present invention can interact to produce information
• FIG. 2A is a block diagram illustrating a system for providing dependency management according to an embodiment of the present invention
• FIG. 2B is a block diagram illustrating a generalized hardware architecture of a computer system suitable for implementing a system for providing dependency management according to an embodiment of the present invention
• FIG. 3 is a block diagram illustrating a functional dependency model of services according to an embodiment of the present invention.
• FIG. 4 is a block diagram illustrating a structural dependency model of services according to an embodiment of the present invention.
• FIG. 5 is a block diagram illustrating a service lifecycle addressed by functional, structural and operational dependency models according to an embodiment of the present invention
• FIG. 6 is a block diagram illustrating relationships between functional, structural and operational dependency models according to an embodiment of the present invention.
• FIG. 7 is a block diagram illustrating components involved in analyzing and computing a root cause of a service outage according to an embodiment of the present invention
• FIG. 8 is a block diagram illustrating components of a root cause analyzer according to an embodiment of the present invention
• FIG. 9 is a flow diagram illustrating steps for invoking a dependency service and performing root cause analysis on an operational model according to an embodiment of the present invention
• FIG. 10 is a flow diagram illustrating tasks of an administrator for creating and updating a functional dependency model according to an embodiment of the present invention.
• FIG. 11 is a flow diagram illustrating steps for updating a structural dependency model by installing or removing hardware/software components on a computer system according to an embodiment of the present invention
• FIG. 12 is a flow diagram illustrating performance of root cause analysis on an operational model according to an embodiment of the present invention
• FIG. 13 is a flow diagram illustrating performance of root cause analysis on antecedents of a service according to an embodiment of the present invention
• FIG. 14 is a flow diagram illustrating steps for determining a status of a service according to an embodiment of the present invention.
• FIG. 15 depicts examples of root cause analyzer application programming interfaces according to an embodiment of the present invention.
• system may be used to refer to a computer system, a software system and/or some combination thereof.
• system may also be used to refer to an application and/or a service.
• multi systems refers to a collection of several systems.
• component may refer to a system, itself, or one or more portions of a system.
• the first scenario deals with managing outsourced services, typically offered by Internet or Application Service Providers (ISP/ASP) .
• Outsourcing services leads to layered service hierarchies where, e.g., the services of an ASP depend on the IP-connectivity (Internet Protocol-connectivity) offered by an ISP, which, in turn, relies on the wide area network of a telecom carrier.
• IP-connectivity Internet Protocol-connectivity
• a SAP delimits the boundary between the different organizational domains and is the place where Service Level Agreements (SLAs) are defined and observed. Usually, this is done at every layer by monitoring a set of specific parameters that are exposed by the provider.
• the second scenario deals with the regular maintenance tasks that can not be done "on-the-fly" and therefore affect services and their customers: e.g., e-mail servers get updated with a new release of their operating system, network devices are exchanged or upgraded with a new firmware version, etc.
• e-mail servers get updated with a new release of their operating system
• network devices are exchanged or upgraded with a new firmware version, etc.
• impact analysis it is important for the network and server administrators to determine in advance how many and, more specifically, which services and users are affected by the maintenance. We refer to this task as impact analysis.
• Servlets gets instantiated by the servlet engine of the application server, performs its tasks and is then removed by the servlet engine. Consequently, a system for computing dependencies among these dynamic entities should address the trade-off between the accuracy of the data and the workload generated for retrieving this data.
• Heterogeneity comes in three different flavors. First, the services offered to customers differ to a large degree. Second, there may be various providers involved in offering a service to a customer. Finally, the products implementing a service may stem from a variety of vendors. A system for computing dependencies should provide a language that is independent of specific operating systems, network protocols, software products and the services offered to a customer.
• Taxonomy for Dependencies The notion of dependencies is very coarse and needs to be refined in order to be useful. Examples for this are the strength of a dependency (indicating the likelihood and the degree to which a component is affected if its antecedent fails) , the criticality (how important this dependency is with respect to the goals and policies of an enterprise), the degree of formalization (i.e., how difficult it is to obtain the dependency) and many more. There is a need to add attributes to dependencies that allow them to be qualified more appropriately; and accordingly, a need to reflect these attributes in the dependency representation.
• an (abstract) service being offered to customers e.g., "Web Hosting,” “Managed Storage,” “IP Connectivity,” “Managed Database,” etc.
• the implementation of a service i.e., the product (s) being used for providing the service, e.g., "IBM Universal Database version 7.1,” “WebSphere Application Server version 3.2;” and
• the invention offers a generic and uniform approach for retrieving dependency information from computer systems that provide a mechanism for retrieving configuration information of individual computer systems or that provide such data in a machine-readable format.
• One benefit of the aforementioned system is that a large amount of application/service management information can be obtained from these computer systems without having the need to instrument individual applications/services. However, if such application/service instrumentation is available, it can be used by the present invention.
• the execution of the system described by the present invention may be triggered either by a specific (management) application (such as: impact analyzer, root cause analyzer), a network management platform (such as IBM/Tivoli NetView, HP OpenView or Aprisma Spectrum) or a management application based on traditional network management systems and platform .
• a specific (management) application such as: impact analyzer, root cause analyzer
• a network management platform such as IBM/Tivoli NetView, HP OpenView or Aprisma Spectrum
• the present invention provides, inter alia, features for: (a) observing performance degradations and outages of subscribed services; (b) tracking down the root cause of the problem by traversing the different layers of the dependency model from the top to the bottom (since various services may be outsourced to other service providers, this (recursive) traversal of a dependency model crosses domain boundaries) ; and (c) analyzing the impact of a service outage by navigating the dependency model from the bottom to the top.
• the invention combines dependency information that is available during the lifecycle of an application or service
• the functional model defines dependencies between different generic services (database service, name service, web application service, connectivity service, etc.).
• the functional model does not describe client/server relationships within a specific service.
• the functional model neither takes into account which concrete products have been chosen to implement the services nor their actual configuration.
• the functional model establishes the principal constraints to which the other models (described below) are bound, i.e., further models may refine the dependencies defined in the functional model with respect to a concrete system infrastructure but should not introduce new dependencies between service categories .
• the model is very compact and generic and is preferably stored on the managing system.
• the structural model contains the detailed descriptions of software components that realize the services defined in the functional model.
• the structural model provides details captured during the installation/deployment phase and complements the functional model by taking the software inventory of concrete systems into account.
• the structural model provides information about which services are installed and configured on a specific system and, for every service, whether the system operates in a client or a server role . The potentially high numbers of systems and services make it difficult to track these dependencies from a remote location.
• Operational Model In a preferred implementation, when software packages get instantiated and bindings between services and applications are established, the operational model of dependencies is created. The high degree of dynamics of this model and large number of involved systems places a limitation on the extent to which the complete model can be instantiated and stored. It is not practical to define and store such a model, rather, the model has to be computed dynamically and stepwise. The operational model is therefore computed "on demand" and relies on the functional and structural models.
• the amount of dependencies and their dynamics are extremely high in large-scale distributed systems.
• the features of the present invention keep their impact on the distributed system (in terms of resource and bandwidth usage) as small as possible and leave as many configuration options that might affect the performance up to the user. Examples for this are: the time intervals for retrieving an updated dependency model, the range of the systems whose dependencies should be tracked, the depth of the dependency model (only services immediately affected versus transitive closure for a given service versus whole service hierarchy) .
• inter-system are perceived as dependencies between the client and server parts of the same service. It is not possible that a client for service A issues requests to a server which provides a different service B.
• One goal of the present invention is to retrieve information mainly from a few well-known/well-defined places (e.g., system repositories) in order to achieve a maximum degree of independence from concrete service/application instrumentation.
• the invention defines a minimal and sufficient amount of commonly available dependency information.
• the invention comprises facilities for storing dependency models persistently or leaves this to the discretion of a management application or another service that uses the invention .
• the invention is enabled to have a notion of history in order to detect and determine changes in the dependency model .
• the invention provides a publish/subscribe interface for notifying software components that have previously registered for changes within the dependency model. Another possible use of the invention is to leave the detection of changes in the dependency model up to the discretion of a management application (or a change management service) to issue periodic calls to the invention in order to determine whether changes in the dependency model have occurred.
• the present invention provides techniques for identifying the root cause of a service outage and performing appropriate problem determination procedures.
• Finding the root cause of a service outage involves the traversal of the service dependency graph from the top to the bottom to identify the candidate services that may have experienced a problem, which is then propagated to the service in question. This traversal from the service where the outage or degradation has been observed towards its antecedents retrieves the entities that a specific service directly depends on, or recursively selects the complete set of nodes, including sub-antecedents, that this service requires to function properly.
• the list of nodes received by a root cause analyzer enables it to perform, in a second step, specific problem determination routines to check whether these services are operational .
• These problem determination procedures involve determining whether a service is functioning properly or not.
• the problem determination procedures may be carried out either:
• stepwise i.e., the function test is carried out for every single service returned, or
• problem determination procedures are carried out for all of them simultaneously.
• the order in which these tests are carried out may be determined by precedence values computed from the strength of dependencies between services, which, in turn, are expressed as weights associated with such dependencies .
• problem determination procedures may be used. Examples of problem determination procedures for distributed services and applications may include, but are not limited to:
• Process Inspection For software applications that implement a service, one way to determine whether they are functioning properly is to verify if their processes (daemons) are running. This can be done in a non-intrusive way by inspecting the process table (or task list) of an operating system and does not require any instrumentation of the application. If the process (es) of an application are running, the application is considered to be in a healthy state. (ii) Exercising an application is an intrusive (and more precise) way of determining whether the application is fully operational, i.e., running, and performing its business function.
• An “exerciser” is a transaction or command invoked from outside of the application that .exercises the application in some fairly complete way to determine if it is really alive and able to deliver its functionality in a timely way. What is exercised is a function of the application itself.
• An analogy in networking is to test the connectivity via the ICMP (Internet Control Message Protocol) "ping" command, which sends time-stamped IP (Internet Protocol) packets to a resource. These packets are returned by the resource and thus allow a system to determine whether the resource is alive and to measure the round-trip delay.
• ICMP Internet Control Message Protocol
• Heartbeats allow the application to demonstrate that it is alive and well.
• the application regularly announces that it is in a healthy operational state by generating events automatically and repeatedly.
• the system listening for heartbeats must understand that if the timeout period expires between the heartbeat events then the application may not be functioning correctly.
• Status indicators are specific management variables of an application that reflect the current status of an application implementing a service. Querying one or more status variables indicates the overall health of a service. However, this approach requires the application to be instrumented appropriately by exposing this information to the outside.
• the root cause determination methodology of the invention may be performed for one or more subject components substantially concurrently (e.g., for one component at a time, or for multiple components in parallel) . At least a portion of results of the root cause determination methodology of the invention may be persistently stored, however, such results may not be persistently stored. Still further, a history of results associated with the root cause determination methodology of the invention may be maintained. Such history may be used to derive heuristics for subsequently determining the sequence of steps for determining the root cause. For example, a ranking may be established, from the history, of the most common outages and the next time a root cause analysis is invoked, the component that has been failing the most in the past is checked first.
• FIG. 1 a block diagram illustrates an example of an electronic commerce system in the form of a client-server application architecture with which the features of the present invention can interact to produce information.
• the architecture of FIG. 1 will be described below to illustrate how such an architecture may handle a transaction in the absence of the techniques of the invention.
• a client system 105 is used to initiate a request, for example via keyboard. Requests, however, could be initiated by any conventional means such as by mouse click, voice command, bar code swipe, etc.
• Examples of the client system 105 are personal computers, kiosks, data entry terminals, scanners, telephones, pagers, handheld or wearable devices, wireless devices, personal digital assistants, network-enabled watches, etc.
• the request is acted upon locally where the request is formulated and forwarded to a web application server 120 over a network 110 and by passing through one or many network access 115 devices.
• An example of the network 110 and communication protocol are socket-based communications riding on a TCP/IP (Transmission Control Protocol/Internet Protocol) transport across a local area network (LAN) that is connected by network access 115 devices such as routers and switches to a wide area network (WAN) containing many switching locations that create a virtual circuit to a service provider and eventually to a web application server 120.
• TCP/IP Transmission Control Protocol/Internet Protocol
• Examples of a web application server 120 are high-end personal computers, RISC-based PowerPC's, UNIX-based workstations, minicomputers or mainframe computers running software fielding requests from clients and distributing the requests to appropriate back-end database servers when appropriate.
• a web application server includes, but are not limited to, those available from IBM Corporation under the trademark WEBSPHERE, from BEA Systems, Inc. under the trademark WEBLOGIC, or from Lotus under the trademark LOTUS DOMINO SERVER.
• the business logic of the web application server 120 processes the incoming request and provides authentication and/or identification of the client system 105. Once the business logic implemented by the web application server 120 determines that the client may proceed with the purchase, it communicates another request via a network 123 to a database server 125 to decrement the inventory.
• the database server 125 processes the request, accesses its database 130 and prepares a response to the web application server 120. Examples of database servers include, but are not limited to those sold by Microsoft under the trademark SQL/SERVER or TRANSACTION SERVER and by IBM Corporation under the trademark DB2 UNIVERSAL DATABASE SERVER.
• the web application server 120 receives the response from the database server 125 and returns it via the network 110 to the client system 105.
• the client system 105 then processes the response to format it for display and presents the response for the transaction initiator to review.
• An administrator 100 observes the various software and hardware components, through which the business transaction is processed, located at the site of the service provider to determine if they function properly. In case of an outage 135 occurring at the database 130, such as a corrupted tablespace or a failure of the database runtime system, the task of the administrator 100 is to locate the cause of the outage, correct the problem and verify if the overall system is functioning properly again. It is to be understood that the invention is intended to operate with any form of outage or performance degradation.
• the administrator 100 interacts with the software and hardware components either directly or through a management system that processes the administrative information (such as status and health data) exposed by software and hardware components at a well-defined management interface.
• the hardware and software components are perceived by the administrator as isolated resources and not as being part of an overall system serving a specific business purpose.
• errors occurring in one component may not be noticed because the administrator does not become aware of them due to lack of continuous monitoring.
• no explicit information regarding the interdependencies between the various components is available to an administrator in a straightforward way. Thus, it is possible that an error within a component, which is not continuously monitored may go unnoticed until the failure propagates to a monitored component .
• an administrator may eventually only become aware of the outage if the web application server 120 is not functioning properly anymore (for example, the load on the web application server increases drastically because it continuously retries to connect to the database server 125, and is unable to complete the requests sent by the client system 105) .
• the administrator 100 would first examine the web application server 120, then determine if there is a network 123 connectivity problem and finally verify if the database server 125 is experiencing difficulties that might stem from an internal error in the database 130.
• the client-server application architecture described above can be regarded as a precursor to an emerging computing environment referred to by IBM Corporation as an "autonomic" computing environment.
• the term derives from the body's autonomic nervous system, which controls key functions without conscious awareness or involvement. More specifically, one of the goals of autonomic computing is to automate some or all of the tasks an administrator 100 would typically carry out. Motivation for doing so is as follows.
• FIG. 2A a block diagram illustrates a system for providing dependency management according to an embodiment of the present invention. More specifically, FIG. 2A depicts a dependency management system that addresses the aforementioned issues.
• the system comprises four layers (application layer 200, service layer 205, middleware layer 210 and resource layer 215) and an administrator graphical user interface 285 by which an administrator 100 interacts with the system.
• the resource layer 215. comprises managed resources 220, a resource dependency repository 225 and a repository agent 230.
• managed resources 220 include, but are not limited to, physical and logical hardware components (examples of the former are hard disks, random access memory, central processing units, network adapters, channel controllers, etc.; examples of the latter are disk partitions, file systems, etc.) and software components (such as operating system, system services like print spoolers or name services, and end-user applications) .
• the resource dependency repository 225 contains the inventory of the hardware and software components of every managed resource 220 and the dependency information on a per-resource basis (i.e., the dependencies between the components within a managed resource 220).
• the resource dependency repository 225 can be either co-located with every individual managed resource 220 or reside at a centralized location.
• the resource dependency repository 225 can be queried, updated and modified through a repository agent 230, which makes the information of the resource dependency repository 225 available to other components of the system.
• the middleware layer 210 comprises a management communication infrastructure 235 such as the protocols and object request brokers by which the different components of the system exchange (management) information.
• the service layer 205 comprises various generic management services 250 such as policy, event and directory, which can be used by a variety of management applications .
• a specifically important service is the dependency service 245, which retrieves information, both from the managed resources 220 and from the repository agent 230, and processes this information to establish an end-to-end dependency model of the overall resource environment.
• This model (or parts of it) is stored in the end-to-end dependency repository 240, according to the needs (e.g., caching for faster retrieval) of the dependency service 245.
• the dependency service 245 is the only component in the described system that directly interacts with the end-to-end dependency repository 240.
• the application layer 200 comprises various management applications that use the generic management services 250 and/or the dependency service 245. Examples of such management applications include, but are not limited to, fault manager 260, topology generator 265, impact analyzer 270, impact simulator 275 and root cause analyzer 280.
• the root cause analyzer 280 determines the root cause of an outage (i.e., the component that initially caused the outage) based on traversing the dependency model (provided by the dependency service 245) from a component affected by an outage towards its antecedents .
• the impact analyzer 270 determines the impact of an outage (i.e., the components that are likely to be affected by the outage) based on traversing the dependency model (provided by the dependency service 245) from a component experiencing an outage towards its dependents .
• the impact analyzer may employ the techniques disclosed in the above-referenced and concurrently-filed U.S. patent application identified by attorney docket no.
• the impact simulator 275 based on the impact analyzer 270, allows an administrator 100 to carry out a "what-if” analysis by simulating the effects of an outage of a specific component on the overall system. This enables the provisioning of appropriate failover solutions.
• the impact simulator may employ the techniques disclosed in the above-referenced and concurrently-filed U.S. patent application identified by attorney docket no. SOM920020005US1 entitled: “Methods And Apparatus For Dependency-based Impact Simulation and Vulnerability Analysis.”
• the fault manager 260 performs appropriate "sanity checks" or tests on components that have been identified either by the root cause analyzer 280 or the impact analyzer
• the fault manager can perform such tests upon the direction of the root cause analyzer 280 or the impact analyzer 270 (i.e., serve as an interface for these modules), and report back results thereto.
• root cause analyzer 280 or the impact analyzer
• the fault manager is preferably comprised of a collection of application-specific or resource-specific tools that allow a determination to be made whether the component that is being tested is working properly or not.
• the fault manager may return a message indicating whether the component "works” or “does not work.”
• These tools may be automated and/or manual.
• a so-called "ping" program checks network connectivity. If the subject remote system answers to a ping, it is online and its network protocol stack (and all the underlying hardware, e.g., network adapter, cables, intermediate network components, etc.) works.
• the fault manager may employ the ping program, as well as any number and type of other tools needed to test components of the distributed computing environment (e.g., heartbeat detection, status indication, etc . ) .
• the topology generator 265 establishes (a subset of) the overall topology of a distributed system, comprising a vast amount of highly dynamic components, such as web applications, database instances and transactions.
• An example for using the topology generator 265 is to display the components of a distributed system that are involved in fulfilling the requests of a specific client system 105.
• the dependency model (or parts of it) is stored in the topology database 255, according to the needs (e.g., caching for faster retrieval) of the topology generator 265.
• the topology generator 265 is the only component in the described system that directly interacts with the topology database 255.
• the topology generator may employ the techniques disclosed in the above-referenced and concurrently-filed U.S. patent application identified by attorney docket no. SOM920020003US1 entitled: "Methods And Apparatus For Topology Discovery and
• FIG. 2B a block diagram is shown illustrating a generalized hardware architecture of a computer system suitable for implementing the various functional components/modules of a system for providing dependency management as depicted in the figures and explained in detail herein.
• the individual components of the dependency management system namely, components associated with the graphical user interface 285, the application layer 200, the service layer 205 and the middleware layer 210 (FIG. 2A)
• the dependency management system may be implemented on one or more computer systems having an architecture as shown in FIG. 2B.
• the other components shown in FIG. 2A e.g., the components associated with the resource layer 215, may also be implemented on similar computer systems.
• the computer system may be implemented in accordance with a processor 290, a memory 292 and I/O devices 294.
• processor as used herein is intended to include any processing device, such as, for example, one that includes a CPU (central processing unit) and/or other processing circuitry.
• memory as used herein is intended to include memory associated with a processor or CPU, such as, for example, RAM, ROM, a fixed memory device (e.g., hard drive), a removable memory device (e.g., diskette), flash memory, etc.
• RAM random access memory
• ROM read only memory
• a fixed memory device e.g., hard drive
• removable memory device e.g., diskette
• flash memory etc.
• input/output devices or “I/O devices” as used herein is intended to include, for example, one or more input devices
• processing unit e.g., keyboard
• output devices e.g., CRT display and/or printer
• processor may refer to more than one processing device and that various elements associated with a processing device may be shared by other processing devices.
• software components including instructions or code for performing the methodologies of the invention, as described herein, may be stored in one or more of the associated memory devices (e.g., ROM, fixed or removable memory) and, when ready to be utilized, loaded in part or in whole (e.g., into RAM) and executed by a CPU.
• ROM read-only memory
• RAM random access memory
• FIG. 3 a block diagram illustrates a functional dependency model of services according to an embodiment of the present invention. More specifically, FIG. 3 depicts a functional application dependency graph between various components in an electronic commerce system such as the one depicted in FIG. 1.
• This functional dependency model represents both the functional components of a distributed system and their dependencies.
• the model defines dependencies between generic services, which are considered atomic from a business perspective. This implies that the functional model is not concerned with dependencies occurring within a business service.
• Such a decomposition makes sense in the scope of a specific product being used to implement the service and will be discussed in more detail with reference to FIG. 4.
• Functional components are the (sub-) services that a service provider needs to deploy for providing an end-to-end service to a customer, the latter being defined in a service level agreement.
• the functional model focuses on the design of an end-to-end service and abstracts from the details of an end-to-end service's technical realization, such as the products being used for the service offering, their location
• the provider domain i.e., whether the provider itself outsources some of its services to another service provider, transparent to the customer), etc.
• an e-business application 300 service depends on a web application service 305 for hosting the business logic.
• the web application service 305 requires two further services.
• the static content of the electronic commerce website is provided by a web service 310, while a back-end database service 330 stores the dynamic content (such as product descriptions, user and manufacturer data, shopping carts, user profiles and preferences, payment information, etc.) of the e-business application 300 being offered to a customer.
• the web service 310 depends, itself, on two services, namely, the name service 315 for mapping host names to IP addresses, and the IP service 320 for network connectivity.
• FIG. 4 a block diagram illustrates a structural dependency model of services according to an embodiment of the present invention. More specifically, FIG. 4 depicts a structural application dependency graph between various components in an electronic commerce system such as the one depicted in FIG. 1.
• This structural dependency model extends the functional model (FIG. 3) in the following ways.
• the structural dependency model deals with the implementation of a business service and is focused on concrete products and their logical (modules, components) and physical (files, shared libraries) architecture.
• the structural dependency model captures the detailed descriptions of software components, i.e., the system inventory, which is usually recorded in the various system repositories or in well-defined places, e.g., the configuration files of a managed resource 220.
• system repositories include, but are not limited to, the IBM AIX Object Data Manager (ODM) , the Linux Red Hat Package Manager (RPM) or the Microsoft Windows Registry.
• ODM IBM AIX Object Data Manager
• RPM Linux Red Hat Package Manager
• Microsoft Windows Registry Information relating to software components is typically captured during the installation and deployment of a software package.
• the structural model contains the dependencies between the various system components, depicted as arrows. For the sake of clarity, the names of the business services are written without quotes, while the names of the elements of the structural model are written with quotes in FIG. 4.
• the system with the fully qualified domain name wslab8.watson.ibm.com 400 hosts the following components: the e-business application (a business service defined in the functional model) , which is implemented as storefront servlets 410, the latter encapsulate the business logic of the application.
• the web application service is implemented by IBM WebSphere version 3.5 415, while the web service is realized by the IBM HTTP Server version 1.3.6 420.
• the IP service is implemented by the default IP protocol stack 430, the operating system (OS) is Win(dows) NT version 4 425.
• the system with the fully qualified domain name rslab2.watson.ibm.com 405 hosts the following components: a database service implemented by (IBM) DB2 Universal Database
• UDB version 5.2 435
• AIX Advanced Interactive Executive
• FIG. 5 a block diagram illustrates a service lifecycle addressed by functional, structural and operational dependency models according to an embodiment of the present invention. More specifically, FIG. 5 depicts the relationships between a functional model 500 and a structural model 510, described above, and introduces a third dependency model, an operational model 520. These three models enable the invention to track the services during their whole lifecycle, i.e., from the design stage to the installation and deployment stage, to the operational or runtime stage.
• the functional model 500 relates to the design of the business services and is thus captured at the design time of a business system.
• the structural model 510 is established.
• the operational model 520 is created when the various components of the structural model 510 become instantiated (step 515) and when runtime bindings between them are established.
• the operational model represents the characteristics of the previously described models at runtime.
• the web application service 305 is implemented by IBM WebSphere 415; one or more instances of the latter are referred to as websphere-daemon 545.
• the web (or WWW) service 310 is implemented by two products, namely, Apache 1.3.4 525 and Lotus Domino 530.
• the running instances of these products can be identified as http daemons "httpd" 550.
• the database service 330 is implemented by two products, namely, Oracle v7 535 and DB2 UDB 435; however, no instances of Oracle v7 535 are active because no server processes are visible in the operational model 520.
• DB2 UDB 435 In contrast, four instances of DB2 UDB 435 are running, as can be seen from the presence of four DB2 daemons "db2d" 555 in the operational model 520.
• the name service 315 is implemented by BIND version 5.6 540; the running instance of BIND can be observed as "named” 560 in the operational model 520.
• the operational model 520 is highly dynamic and potentially very large. In contrast to the functional and structural dependency models, the operational model 520 is not stored in a repository or a database, but computed on demand and to the extent needed.
• FIG. 6 a block diagram illustrates relationships between functional, structural and operational dependency models according to an embodiment of the present invention. More specifically, FIG. 6 depicts the details of the data template used for the three dependency models and the means for tying these models together by an example. The example details the template and its associated values for describing the name service during its lifecycle.
• the functional template 605, used for the functional model 500 contains the "hostName” (the unique name of the computer system that hosts the service) , the "serviceName”
• the structural template 610 used for the structural model 510, contains all the fields of the functional template 605, which allows to link the functional template 605 with the structural template 610 in order to navigate from the functional model 500 to the structural model 510, and vice-versa.
• the structural template 610 contains the "componentName” (the name of the product component) , an "identifier” (a globally unique name for identifying the component), the "version,” “release” and “modification” (e.g., maintenance or patch/fix level) numbers, the "installState"
• the operational template 615 used for the operational model 520, contains the fields "hostName” (the unique name of the computer system that hosts the service) and "processName"
• the operational template 615 contains the fields "operState” (the operational state of the process, i.e., running, interrupted, zombie, etc.), "portNumber” (the number of the TCP/UDP port by which an application can connect to the process) , and, "instancelD” (to distinguish the various application instances within the scope of a computer system) .
• the three dependency models are stored and computed at different places to achieve a maximum degree of efficiency.
• the functional model 500 is gathered and stored at the management system 620, i.e., the central point of control by which the administrator 100 interacts with the distributed environment. Some of the reasons for this choice are as follows. As seen in the descriptions of FIG. 3 and FIG. 5, the functional model 500 is fairly compact because the amount of possible business services is limited. In addition, the functional model is not subject to overly frequent changes.
• the functional model is defined at the time when a business service is provided to a customer and remains unchanged until the service offering period ends. Since the administrator 100 is responsible for setting up and updating the functional model 500, it is a natural choice to keep it close to the management system 620.
• the structural model 510 captures the detailed descriptions of software components, i.e., the system inventory, which is usually recorded in the various system repositories or in well-defined places, e.g., the configuration files of managed resource 220. Consequently, it is both large sized (the content of a system repository tends to be between several hundred kilobytes and up to a few Megabytes) and also subject to frequent changes. Therefore, keeping the structural model 510 of a system at the managed resource 220, itself, eliminates both the communication overhead for updating the model and the need for vast amounts of storage that would occur if the structural models 510 of all managed resources (220) is stored at a centralized place.
• the operational model 520 has been described in FIG. 5 as very dynamic and also extremely large because it covers potentially multiple instances of every application present on the computer systems of the distributed environment and the dependency relationships between them. Given the fact that current data centers of Internet/Application/Storage Service Providers and outsourcers consist of several thousands of computer systems with each hosting close to 100 applications and system services, an operational model comprising all the currently instantiated applications and their dependencies may be impractical. Thus, a practical approach is to compute the relevant parts of the operational model on demand (step 625) .
• FIG. 7 a block diagram illustrates components involved in analyzing and computing the root cause of a service outage based on dynamic information technology (IT) service dependencies according to an embodiment of the present invention. More specifically, FIG. 7 depicts the data flows between the various components used for analyzing and computing such a root cause. It is assumed that the managed resources 220 are able to provide XML (Extensible Markup Language) descriptions of their system inventory, configuration files and their various dependencies. However, it should be noted that any data description format may be used in accordance with the invention. The details on how this information can be acquired are as follows.
• XML Extensible Markup Language
• the dependency service 245 makes use of information stored in system repositories 745 for generating appropriate service dependency information. This information is made available to the other components of the system through a web server 730.
• the managed resources 220 expose their information by means of an instrumentation agent, called CIM (Common Information Model, which is a standardized management framework) provider 750, which interacts with a CIM Object Manager (CIMOM) 735, as proposed by the Distributed Management Task Force (DMTF) .
• CIM Common Information Model
• DMTF Distributed Management Task Force
• various management services being part of the service layer 205, are depicted. These are: a name service 700, a trader service 710, an event service 715 and the dependency service 245.
• the main tasks of the dependency service 245 are as follows:
• the management system issues queries to the application programming interface (API) of the dependency service (245) .
• API application programming interface
• the invention aims at keeping the load on every involved system as low as possible.
• the invention decouples the management system from the managed resources 220 and encapsulates the time consuming filter and join operations in the dependency service 245, which can be replicated on various systems. It is therefore possible to achieve a maximum level of parallelism for query operations, since the selection of an instance of the dependency service 245 can be done flexibly by the management system.
• Another important advantage is that the (very large and highly dynamic) operational model 520 is not stored at a specific place but computed on demand in a stepwise manner.
• the different parts of the structural model 510 are stored at the managed resources 220.
• the management system therefore always receives the most recent information but is still free to store it according to elaborate caching policies.
• a block diagram illustrates components of a root cause analyzer according to an embodiment of the present invention.
• a root cause correlator 870 which functions as a flow coordinator for the entire root cause analysis process, receives service problem reports 880 from the administrator 100, providing the name of the service and the host name where the problem has been observed or reported.
• the root cause correlator 870 interacts with the dependency service 245 in order to obtain a list of the basic services on which the problematic service depends . Examples of basic services would be: domain name service, IP connectivity service, etc.
• the task of the dependency service 245 is to find the antecedents of the service in question, even if the e-commerce environment spans different managed domains 800. In order to deal with multiple domains, various (cascaded) instances of the dependency service 245 may work jointly together. In FIG. 8, the e-commerce environment is shown by the dotted rectangle. Typically, such environments contain one or more managed domains, eventually each with its own dependency database 810 and dependency service 245. The dependency service 245 returns the names and identifiers of the antecedents back to the root cause correlator 870, which then initiates further problem determination procedures to verify whether one or more antecedents experience difficulties. This can be accomplished via a number of approaches, some of which are illustrated below. A first approach assumes the presence of an event monitoring and distribution function, provided by an event monitor 820. Examples of such event monitors include, but are not limited to HP OpenView Event Services and Tivoli
• the event monitor 820 receives events/alerts related to resources within the e-commerce environment and forwards the ones , which are relevant to an event database 830. Some events are informational in nature or are related to errors that are fixed automatically by the domain management services . These are typically filtered out and not forwarded to the event database 830. In practice, the event monitor 820 may comprise of a hierarchical arrangement of multiple event monitors, one for each managed domain.
• an event-service correlator 840 The main function of an event-service correlator 840 is to provide, for a given service or resource, a list of outstanding alerts associated with that resource or service. It does so by interacting with event monitors 820 and stores the events in the event database 830 for further retrieval.
• the root cause correlator 870 uses the event-service correlator 840 to estimate the likelihood of this being the case by getting a list of events that may have been reported about the suspected service or resource. If there is an undesirable event, for example buffer overflow, the likelihood of this resource being a root cause is high.
• the root cause correlator 870 also interacts with the dependency services 245 for each of the managed domains in order get a set of possible root causes that lie within that domain.
• the dependency service 245 interacts with domain dependency data 810, which is constructed during deployment and periodically updated during the operation phase of the e-commerce setup.
• a second approach is to obtain the current status of a service or resource from a status monitor 860, which directly interacts with the services and verifies whether they are functioning properly or not (e.g., failed, not failed, degraded, etc.).
• the status monitor may serve as the interface between the root cause analyzer and the fault manager so as to, among other advantages, make the root cause analyzer independent of any particular test procedure.
• the status monitor 860 may therefore serve as a single point of contact for the root cause correlator 870 to determine the health of the service in question.
• no functionality at all it is either not present or does not offer its functionality to other components through a programmatic interface. So while a human user may interact with the fault manager through a GUI (graphical user interface), no program (such as the correlator 870) can use it. Also note that a fault manager is meant to refer to a loose collection of system administration tools having completely different interfaces. The status monitor thus serves to integrate these various tools under a uniform interface. That is, the status monitor preferably operates to account for any mismatch between a root cause correlator which expects everything to be testable for status and the fault manager which may provide between 0 percent and 100 percent of this required functionality.
• the root cause correlator 870 thus has two ways to determine whether a service is functioning properly or experiences a problem, i.e., the correlator can query the event-service correlator 840 for problem event reports relating to a service or directly look up the status of a service from the status monitor 860.
• the root cause correlator 870 is free to choose by which of these at least two means it determines the health of a service. Further, the root cause correlator 870 may choose to obtain information both from the event-service correlator 840 and from the status monitor 860.
• FIG. 9 a flow diagram illustrates steps for invoking a dependency service and performing root cause analysis on an operational model according to an embodiment of the present invention. More specifically, FIG. 9 depicts a methodology of invoking a dependency service (e.g., dependency service 245), collecting its results, and applying root cause analysis to them. This methodology is initiated either by an administrator 100 or a management application being part of the application layer 200, as described in FIG. 2A.
• a dependency service e.g., dependency service 245
• the methodology begins at block 900 and proceeds as follows.
• a business service is selected (step 905), typically from the functional model, since an administrator is interested in the business services provided by the distributed system.
• the structural model is queried to provide a selection of hosts that are involved in the provisioning of the business service. This can be either done by locating the structural models present on every host of the distributed system, or (for efficiency purposes) by querying a (periodically updated) service/host lookup table stored at the management system, which contains a mapping between the services and hosts present in the distributed system.
• the administrator selects a host at his discretion (step 910) .
• query parameters include, but are not limited to, the direction of the traversal (towards the service dependents, or towards its antecedents) , the depth of the traversal (e.g., only the immediate antecedents/dependents; the overall possible antecedents/dependents, i.e., the complete transitive closure of the operational model; only between the m-th and __-th layers of the operational model) , filtering criteria related either to the presence of attributes or to their values.
• step 905 The fact that the order of the steps for selecting service (step 905), host (step 910) and the options for composing the query is prescribed here, emphasizes the "service-centric” approach (versus the "host-centric” approach of existing art) of the present invention.
• steps 905, 910 and 915) may be made without departing from the spirit and scope of the present invention.
• the dependency service is invoked (step 920) with these parameters.
• the mode of invocation can be either synchronous (i.e., blocking the invoker until the results are returned by the dependency service) or asynchronous (thus allowing the invoker to perform additional tasks during the computation) .
• the dependency service computes the appropriate part of the operational model and, depending on the mode of invocation, either sends the results back to the invoker or notifies the invoker that the results are available.
• the invoker then collects the results and applies root cause analysis and appropriate problem determination procedures to them (step 925).
• the methodology ends at block 930.
• FIG. 10 a flow diagram illustrates tasks of an administrator for creating and updating a functional dependency model according to an embodiment of the present invention. This is necessary either if new (business) services are deployed and offered, or changes are applied to an existing model, or existing (business) services are withdrawn from an offering.
• the methodology begins at block 1000 and proceeds as follows. An administrator or a management application evaluates whether a new business service should be added or an existing service is to be deleted (step 1005) . If this is not necessary, the methodology proceeds directly to block 1025. Otherwise, in step 1010, the service and its description are entered in (or removed from) the template 605 of the functional model, which has been described in FIG. 6.
• step 1015 the service dependencies, i.e., its relationships regarding its antecedents, need to be added to (or removed from) the template 605 of the functional model.
• the dependencies from the service dependents need to be adjusted to point to the antecedents of the service that is to be removed. This may involve checking for eventual duplicate descriptions within the dependencies of the antecedents.
• the updated functional model is stored in the repository of the management system (step 1020). The methodology ends at block 1025.
• FIG. 11 a flow diagram illustrates steps of updating a structural dependency model by installing or removing hardware/software components on a computer system according to an embodiment of the present invention. This is necessary either if new components are deployed and installed on a host, or existing components are removed from the host.
• the methodology begins at block 1100 and proceeds as follows. If new hardware components are installed/removed, the verification and adjustment of their dependencies is usually performed by the operation system and is thus not described further here. Instead, the following description focuses on the task of adding/removing software components.
• An administrator or a management application performing software distribution and installation evaluates whether a new software component should be added or an existing software component is to be deleted (step 1105) . If this is not necessary, the methodology proceeds directly to block 1125. Otherwise, in step 1110, the software components' descriptions are entered in (or removed from) the template 610 of the structural model, which has been described in FIG. 6. Then, in step 1115, the software components' dependencies, i.e., its relationships regarding its antecedents, needs to be added to (or removed from) the template 610 of the structural model.
• the updated structural model is stored in the resource dependency repository of the host (step 1120) .
• the methodology ends at block 1125.
• FIG. 12 a flow diagram illustrates performance of root cause analysis on an operational model according to an embodiment of the present invention.
• the methodology begins at block 1200 and proceeds as follows.
• the system performing the root cause analysis on the operational dependency model is continuously listening for requests at a specific port of the host on which the system is executed, which is illustrated by the loop that connects step 1205 with itself. This is the standard behavior for server processes (“daemons”) that implement services, which can be invoked by applications at any time.
• the system On reception of a request, the system extracts the input parameters from the request (step 1210) .
• input parameters include, but are not limited to, the names of the service and the host in question, the direction of the traversal, the depth of the traversal, filtering criteria related either to the presence of attributes or to their values . These input parameters are then used to invoke the computation of the operational model, which is invoked in step 1215.
• the results of the computation i.e., the operational model
• the root cause analysis is then performed on the operational model (step 1220).
• the results of the root cause analysis are returned to the invoking application, in accordance with the mode of invocation specified at the time of the invocation (step 1225).
• any allocated resources of the host on which the system is running are released (step 1230) . Examples of host resources include, but are not limited to, memory, disk space or CPU registers.
• the system returns back to its initial stage and listens for subsequent incoming requests (return to step 1205) .
• a flow diagram illustrates performance of root cause analysis on antecedents of a service according to an embodiment of the present invention.
• the methodology begins at block 1300 and proceeds as follows. First, the names of the targeted service and the host are obtained (step 1305) . These parameters are provided by the invoking management application, which obtains these parameters either directly from the administrator or from an event message that arrives at the management console. In addition, the search criteria according to the status of the antecedents are specified. They indicate whether the system should return the services that have encountered a problem
• defective or the services that perform well.
• a management application is interested in the former, i.e., the defective services.
• search path is specified.
• step 1310 the computation of the operational model is performed by the dependency service, according to the parameters "Service Name,” “Host Name,” “Antecedent status,” “search path” (step 1310).
• step 1315 the results, i.e., the list of antecedent service elements ("Candidate List"), are obtained from the dependency service.
• the first service element of the candidate list is selected (step 1325) and checked to determine whether it is functioning properly (step 1330) .
• step 1350 the methodology then proceeds directly to step 1350, where either the list of working service elements or the list of defective service elements is returned to the invoker.
• the results content depends on whether the invoker has been asking in step
• FIG. 14 a flow diagram illustrates steps for determining a status of a service according to an embodiment of the present invention. More specifically, FIG.
• FIG. 14 illustrates interactions of the root cause correlator 870 with the event-service correlator 840 and the status monitor 860, depicted in FIG. 8.
• the methodology begins at block 1400 and proceeds as follows.
• the name of the targeted service element is obtained (step 1405) .
• This problem determination problem is described from the perspective of checking the status of a single service element, as may be done by the root cause correlator 870. It is to be understood that this procedure is repeated for every service in question. It is to be further understood that this procedure is carried out in step 1330 of FIG. 13.
• the names of the targeted service elements are provided by the invoking management application, which obtains these parameters either directly from the administrator or from an event message that arrives at the management console.
• the root cause correlator 870 queries the event-service correlator 840 for the presence of events relating to the service in question (step 1410) . These events indicate whether any problems with this service have been observed.
• step 1415 the service element is flagged as "defective" (step 1415) , and the result is returned to the invoker (step 1430) . If step 1410 yields that no fault events have occurred for this service, it is necessary to perform further work to determine the status of a service. This is done by querying the status monitor 860 for the status of the service (step 1420) .
• the task of the status monitor 860 comprises providing a uniform interface to status information for every service in the distributed system, regardless of how the status of the service is determined.
• the table includes base APIs that can generate, send and request receipt of appropriate operational models for a given service and host name.
• APIs can use one or more parameters (not shown) to identify characteristics (specified in the functional description) used by the APIs.
• a “getDirectAntecedents (parameters)” API retrieves direct antecedents, regardless of their current status, of a service located on a specific host.
• the “getAntecedentsRecursive (parameters)” API performs a recursive "drill-down,” i.e., it retrieves all the antecedents of a given service (regardless of their current status) , which is located on a specific host. It therefore returns all the services residing "below” a given service in the dependency hierarchy.
• the “getLowestAntecedents (parameters)” retrieves the lowest antecedents of a service located on a specific host. This method yields the services that do not have any antecedents themselves (i.e., they reside in the lowest layer of the dependency hierarchy of a given service) .
• the "getWorkingDirectAntecedents (parameters) " API retrieves direct antecedents with status "up" of a service located on a specific host.
• “getWorkingAntecedentsRecursive (parameters)” API performs a recursive "drill-down,” i.e., it retrieves all the antecedents with status "up” of a given service, which is located on a specific host. It therefore returns all the working services residing "below” a given service in the dependency hierarchy.
• the “getWorkingLowestAntecedents (parameters) " API retrieves the lowest antecedents with status "up” of a service located on a specific host. This method yields the services that do not have any antecedents themselves (i.e., they reside in the lowest layer of the dependency hierarchy of a given service) .
• the “getDefectiveDirectAntecedents (parameters) " API retrieves direct antecedents with status "down” of a service located on a specific host.
• the “getDefectiveAntecedentsRecursive (parameters) " API performs a recursive "drill-down,” i.e., it retrieves all the antecedents with status "down” of a given service, which is located on a specific host. It therefore returns all the failed services residing "below” a given service in the dependency hierarchy.
• the “getDefectiveLowestAntecedents (parameters)” retrieves the lowest antecedents with status "down” of a service located on a specific host.
• This method yields the services that do not have any antecedents themselves (i.e., they reside in the lowest layer of the dependency hierarchy of a given service) .
• This API is particularly useful if a user or management application wants to determine quickly whether any "fundamental" service (e.g., network connectivity service) has failed without having to verify the status of "intermediate” services (such as name services) .

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Computer Hardware Design (AREA)
• Quality & Reliability (AREA)
• Health & Medical Sciences (AREA)
• Biomedical Technology (AREA)
• Software Systems (AREA)
• Mathematical Physics (AREA)
• Debugging And Monitoring (AREA)
• Test And Diagnosis Of Digital Computers (AREA)
• Multi Processors (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (2)

Family

ID=31991140

Family Applications (1)

Country Status (8)

Cited By (4)

Families Citing this family (264)

Family Cites Families (33)

• 2002
  • 2002-09-11 US US10/241,213 patent/US7096459B2/en not_active Expired - Fee Related
• 2003
  • 2003-07-21 TW TW092119800A patent/TWI238329B/zh not_active IP Right Cessation
  • 2003-08-13 WO PCT/EP2003/010080 patent/WO2004025471A2/en not_active Ceased
  • 2003-08-13 CA CA 2498065 patent/CA2498065C/en not_active Expired - Fee Related
  • 2003-08-13 AU AU2003264290A patent/AU2003264290A1/en not_active Abandoned
  • 2003-08-13 CN CNB038214431A patent/CN100350391C/zh not_active Expired - Lifetime
  • 2003-08-13 KR KR1020057004305A patent/KR100763326B1/ko not_active Expired - Fee Related
  • 2003-08-13 JP JP2004535485A patent/JP2005538459A/ja active Pending

Also Published As

Similar Documents

Legal Events