WO2001001221A9 - Architecture a geometrie variable amelioree et procedes pour des applications de commerce electronique dans un systeme informatique en grappe - Google Patents

Architecture a geometrie variable amelioree et procedes pour des applications de commerce electronique dans un systeme informatique en grappe

Info

Publication number
WO2001001221A9
WO2001001221A9 PCT/US2000/017857 US0017857W WO0101221A9 WO 2001001221 A9 WO2001001221 A9 WO 2001001221A9 US 0017857 W US0017857 W US 0017857W WO 0101221 A9 WO0101221 A9 WO 0101221A9
Authority
WO
WIPO (PCT)
Prior art keywords
computer
computers
software
software program
module
Prior art date
Application number
PCT/US2000/017857
Other languages
English (en)
Other versions
WO2001001221A3 (fr
WO2001001221A2 (fr
Inventor
Souza Roy P D
Original Assignee
Biztro Inc
Souza Roy P D
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US09/346,000 external-priority patent/US6446218B1/en
Priority claimed from US09/346,074 external-priority patent/US6453468B1/en
Application filed by Biztro Inc, Souza Roy P D filed Critical Biztro Inc
Priority to AU57769/00A priority Critical patent/AU5776900A/en
Publication of WO2001001221A2 publication Critical patent/WO2001001221A2/fr
Publication of WO2001001221A3 publication Critical patent/WO2001001221A3/fr
Publication of WO2001001221A9 publication Critical patent/WO2001001221A9/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions

Definitions

  • the present invention relates to an improved computer architecture. More particularly, the present invention relates to techniques for improving the reliability and response time of a scalable computer system ofthe type employed in e-commerce applications through the Internet.
  • E-commerce or electronic commerce through the Internet, places stringent requirements on the planning and implementation ofthe computer infrastructure that supports the service.
  • the e-commerce service is in its infancy, it is important for economic reasons to minimize the cost ofthe computing infrastructure employed to service the few initial users or early adopters.
  • the use ofthe service becomes wide-spread among many users, which in the e-commerce age could be in a matter of days or weeks, the initial computing infrastructure must grow correspondingly to offer reliable and fast service to users or risk losing users to competing services.
  • the processing load is borne by a single centrally located computer and as the processing load increases, that computer may be upgraded to have a more powerful processor or, in the case with parallel processors, be endowed with additional processors to handle a higher processing load.
  • Clustering represents another computer architecture that readily scales to adapt to changing processing loads.
  • clustering multiple inexpensive and/or low power computers are clustered together to service the processing load.
  • the individual computers are interconnected using some type of network connection, such as Ethernet.
  • network connection such as Ethernet.
  • the number of computers in the cluster may be correspondingly increased or decreased to meet the need ofthe changing processing load.
  • FIG. 1 illustrates a prior art computer architecture wherein the computers are clustered in various stages to service the processing needs ofthe stages.
  • a computer system 102 representing a typical prior art clustered computer system employed to service Internet-based transaction requests.
  • Computer system 102 which is typically connected to a larger network such as the Internet or a portion thereof, includes a webserver stage 104, application server stage 106, and a data repository stage 108.
  • each stage is implemented by a group or cluster of servers.
  • a user may access computer system 102 by typing in a URL (Uniform Resource Locator) and obtaining a page from a webserver of webserver stage 104.
  • the first few pages returned may include general introductory information as well as an authentication facility to allow the user to sign in.
  • a menu of contents and/or applications may then be served up to the user. If the user chooses an application, the request is serviced by one ofthe application servers in application server stage 106, which acts in concert with one or more databases in the data repository stage 108, to respond to the user's request.
  • webserver router 112 which arbitrates among the webservers 114(a)- 114(e), to decide which of these webserver should service this user's request. As a threshold determination, webserver router 112 may ascertain whether the user had recently accessed the service through a particular webserver of webserver stage 104. If he did, there is usually data pertaining to this user that is cached at the webserver that last serviced him, and it may be more efficient to continue assigning this user to the webserver that serviced him earlier.
  • webserver router 112 may assign the user to one of webservers 114(a)-l 14(e). The decision of which webserver to assign is typically made based on the current load levels on the respective webservers, the information pertaining to which is periodically received by webserver router 112 from the webservers through path 116. Once the user is assigned one ofthe webservers, subsequent traffic may be directly transmitted between the user's te ⁇ ninal and the assigned webserver without going through the router.
  • application server router 118 picks among application servers 120(a)- 120(d) of application server stage 106 based on the current load levels on the application servers. The information pertaining to the current load levels on the application servers are periodically received by application server router 118 through path 122 as shown. At any rate, one of application servers 120(a)- 120(d) will be assigned to the user to service the user's request. As in the case with the webservers, once the user is assigned one ofthe application servers, subsequent traffic may be directly transmitted between the web server that services the user and the assigned application server without going through the router that performed the assignment.
  • the application server may consult yet another router (shown in Fig. 1 as database router 130), which may pick the most suitable database server 132(a)-l 32(c) for serving up the data.
  • data base router 130 has information pertaining to the level of load on each database server since it periodically receives feedback from the database servers (via path 134).
  • fault tolerance is achieved at the server level, i.e., by maintaining a sufficiently large number of servers per cluster to ensure that if there is a failure in one ofthe servers, there still remains sufficient processing capability in the surviving servers to allow the computer system as a whole to continue handling the transaction requests.
  • prior art fault tolerance solutions are typically offered on homogeneous clusters and are specifically tied to specific computers from specific vendors. With reference to Fig. 1, for example, the prior art technique of fault tolerance typically requires that all servers in a cluster (i.e., all servers serviced by a router such as servers 112a -112d of Fig. 1) be homogeneous. There are, however, disadvantages to the prior art approach to implementing fault tolerance.
  • Another area that system engineers always strive to improve relates to reducing transaction request processing time. Because of scaling and the desire to implement fault tolerance, it is typically the case that there exist multiple copies of any given application program per cluster. With reference to Fig. 1, for example, there typically exist multiple copies of an application program, distributed among two or more of servers 112a- 112d. Because there are multiple copies present in the cluster to service incoming transaction requests, it is important to appropriately distribute the processing requirements ofthe multiple users across the servers so that transaction requests may be more efficiently serviced, with no single server being overtaxed while others are idle.
  • the decision regarding which server in the cluster should service a new user can be made by simply examining the relative load levels among the servers that have the appropriate software to handle the incoming transaction request of that user, and by assigning the new user to the server that is least heavily loaded.
  • the average processing time for transaction requests is, in theory, minimized.
  • most modern routers have the capability to receive relative load level data for the servers they service, and can make decisions pertaining to user routing based on the relative load level data.
  • the lightly loaded server is being stress-tested and not yet certified to handle a full load, or due to the fact that the lightly loaded server also implements another application program, which is ofthe type that is subject to sudden, rapidly fluctuating processing demands and therefore needs a large reserve processing capacity).
  • the prior art method of routing incoming transaction requests leaves a lot to be desired.
  • the relative load level information among the cluster triggers an alert.
  • the response is typically to add additional servers to the cluster to increase the number of copies ofthe application program that is in heavy demand, thereby increasing the ability ofthe computer system as a whole to handle transaction requests that require the attention of that application program.
  • the addition of a server to a cluster is typically an expensive option and usually involves a substantial delay and investment in time since it requires the acquisition, installation, and configuration of new hardware in the existing cluster.
  • the new server is acquired and/or installed, user responsiveness suffers as the overloaded servers struggle to keep up with incoming transaction requests.
  • the servers that implement the software program in demand may be overloaded one-by-one and that overload may lead to a situation wherein none ofthe users' transaction requests are serviced in a timely manner.
  • proactive approaches to load balancing that can ready the cluster for handling the increased processing load before it occurs.
  • outside influences such as natural and manmade disasters, may pose a serious threat to the reliability ofthe e-commerce service.
  • some regions ofthe United States are exposed to seasonal storms or to earthquakes.
  • 1 may be implemented by two clusters of servers, with one being located in San Francisco while the other is located in New York.
  • the presence ofthe redundant servers further complicates the earlier mentioned challenges regarding maintaining reliability during and after software upgrades, efficient routing of transaction requests, maintaining an acceptable fault tolerance level in a heterogeneous cluster, and handling increases in the number of transaction requests both reactively and prospectively.
  • the invention relates, in one embodiment, to a clustered computer system having at least one cluster of computers operatively coupled to service transaction requests for a plurality of software programs implemented as software modules on the cluster of computers.
  • the clustered computer system includes a first computer, and a second computer operatively coupled with the first computer in a cluster configuration.
  • an intelligent director agent operatively coupled with both the first computer and the second computer, the intelligent director agent receiving both computer-specific information and software module- specific information from the first computer and the second computer. Further, the intelligent director agent performs at least one of a routing ofthe transaction requests to respective ones of the first computer and the second computer and a reconfiguration ofthe software modules implemented on the cluster of computers.
  • the invention relates to an intelligent director agent configured for use in a clustered computer system having at least a first computer and a second computer connected in a cluster configuration, wherein both the first computer and the second computer are configured to run software modules pertaining to a software program.
  • the intelligent director agent is configured to be operatively coupled with both the first computer and the second computer.
  • the intelligent director agent includes a first input module for receiving a transaction request that requires the software modules on the first computer and the second computer.
  • the intelligent director agent further includes a second input module for receiving both computer-specific information and software module-specific information from the first computer and the second computer.
  • a software module selector configured for selecting one ofthe first computer and the second computer for servicing the transaction request.
  • the software module selector performs the selecting responsive to both the computer-specific information and the software module-specific information from the first computer and the second computer.
  • the invention relates to a method for routing a transaction request that requires a software program implemented as software modules on a plurality of computers of a clustered computer system, the plurality of computers being connected in a cluster configuration, wherein at least two ofthe plurality of computers being configured to run software modules pertaining to a software program.
  • the plurality of computers being coupled to an intelligent director agent.
  • the method includes receiving at the intelligent director agent computer-specific information pertaining to the plurality of computers.
  • the computer-specific information includes load level information on respective ones ofthe plurality of computers.
  • the method further includes receiving at the intelligent director agent software module-specific information pertaining to the software modules.
  • the method additionally includes selecting a given one of the plurality of computers to route the transaction request to, the selecting being performed responsive to both the computer-specific information and the software module- specific information. There is also included issuing a command to route the transaction request to the given one ofthe plurality of computers responsive to the selecting.
  • the invention in another embodiment, relates to a method for upgrading a software program from a first version to a second version.
  • the software program is implemented as software modules running on a plurality of computers coupled in a cluster configuration in a clustered computer system.
  • the method includes replacing a subset ofthe software modules with the second version ofthe software program, the method also includes assigning the subset of software modules with a first certification level. There is further included monitoring performance ofthe subset of software modules to ascertain whether the subset of software modules meet a predefined reliability criteria after the replacing.
  • the method includes designating the subset of software modules with a second certification level, wherein the subset of software modules receive transaction requests that require the software program at a first rate when assigned the first certification level.
  • the subset of software modules receives the transaction requests that require the software program at a second rate when assigned the second certification level, the second certification level being higher than the first certification level.
  • the invention in another embodiment, relates to a method for enhancing reliability while upgrading a software program implemented in a clustered computer system from a first version to a second version.
  • the software program is implemented as software modules running on a plurality of computers coupled in a cluster configuration in a clustered computer system.
  • the method includes ascertaining a certification level associated with each ofthe software modules. If a certification level of a given software module ofthe plurality of software modules has a first certification level, the method includes limiting a load level on the given software module to a first load level. If a certification level of a given software module of the plurality of software modules has a second certification level, the method includes allowing the load level on the second routing transaction requests to reach a second load level higher than the first load level.
  • the invention in another embodiment, relates to techniques for maintaining an adequate level of fault tolerance for a software program implemented on computers of a cluster in a clustered computer system.
  • the invention includes the use of intelligent director agents that are coupled to the computers ofthe cluster and the computer-specific as well as the software module-specific information stored therein. These pieces of information permit the intelligent detection of a deficiency in fault tolerance, the intelligent selection of a computer capable of running another copy ofthe software program, the loading of another copy ofthe software program on the identified computer, and the registration ofthe identified computer for servicing transaction requests pertaining to the software program after another copy ofthe software program is loaded thereon. Techniques are described to permit both computers in a local cluster and computers in a remote cluster to serve the fault tolerance relief role.
  • the invention in another embodiment, relates to a method for maintaining a predefined acceptable fault tolerance level for a plurality of software modules implementing a software program running on a first plurality of computers coupled together in a cluster configuration in a first cluster in a clustered computer system.
  • the first plurality of computers being coupled to a first intelligent director agent.
  • the method includes tracking, using the first intelligent director agent, status ofthe software modules running on the first plurality of computers.
  • the method also includes ascertaining a fault tolerance level associated with the software program, with the ascertaining being ascertained by examining the status ofthe software modules running on the first plurality of computers.
  • the method also includes searching for a first suitable computer among the first plurality of computers to load another module ofthe software program thereon.
  • the first suitable computer represents a computer of the first plurality of computers that does not have a module ofthe software program running thereon.
  • the first suitable computer is compatible to execute the another copy ofthe computer program. If the first suitable computer is available, the method further includes loading the another module ofthe software program on the first suitable computer, registering the first suitable computer as a computer capable of servicing transaction requests pertaining to the software program after the another module of the software program is loaded onto the first suitable computer, and routing the transaction requests pertaining to the software program to the first suitable computer after the registering.
  • the invention in yet another embodiment, relates to a method for balancing load levels among a plurality of computers coupled together in a cluster configuration in a clustered computer system.
  • the plurality of computers are coupled to a first intelligent director agent.
  • the method includes ascertaining a first set of load levels associated with the plurality of computers.
  • the ascertaining is made responsive to data received from the plurality of computers at the intelligent director agent.
  • the data represents load level information experienced substantially currently by the plurality of computers. If one ofthe first set of load levels associated with a first computer of the plurality of computers exceeds a predefined threshold, the method includes ascertaining a software program that causes stress on the first computer ofthe plurality of computers.
  • the second computer represents a computer that is compatible to run the another module ofthe software program.
  • the second computer further represents a computer ofthe plurality of computer that did not run the software program prior to the loading.
  • the mvention in another embodiment, relates to a method for balancing load levels among a first plurality of computers coupled together in a cluster configuration in a first cluster of a clustered computer system.
  • the first plurality of computers are coupled to a first intelligent director agent and being located at a first geographic location.
  • the clustered computer system further includes a second cluster that includes a second plurality of computers also coupled together in the cluster configuration.
  • the second cluster being located at a second geographic location that is remote from the first geographic location.
  • the method includes ascertaining a first set of load levels associated with the first plurality of computers.
  • the method also includes identifying, using the intelligent director agent, a given local computer among the first plurality of computers.
  • the given local computer represents a computer that is capable of running another module ofthe software program and did not run the software program prior to the identifying. If the given local computer among the first plurality of computers is not available, the method includes identifying, using the intelligent director agent, a given remote computer among the second plurality of computers.
  • the given remote computer represents a computer that is capable of running the another module ofthe software program and did not run the software program prior to the identifying.
  • the invention in another embodiment, relates to a method for predictively preventing computer stress by reconfiguring a plurality of computers coupled together in a cluster configuration in a clustered computer system.
  • the reconfiguring is performed prior to the stress occurring.
  • the plurality of computers is coupled to a first intelligent director agent.
  • the method includes predicting a first computer ofthe plurality of computers that would experience the computer stress at a future point in time.
  • the computer stress is related to a number of transaction requests being serviced by the first computer at the future point in time. If the first computer is predicted to experience the computer stress at the future point in time, the method includes ascertaining a software program that causes stress on the first computer ofthe plurality of computers.
  • the method also includes loading another module ofthe software program on a second computer ofthe plurality of computers.
  • the second computer represents a computer that is compatible to run the another module ofthe software program.
  • the second computer further represents a computer ofthe plurality of computer that did not run the software program prior to the loading.
  • the invention in another embodiment, relates to a method for predictively preventing computer stress on a plurality of computers coupled together in a cluster configuration in a clustered computer system.
  • the reconfiguring is performed prior to the stress occurring.
  • the plurality of computers is coupled to a first intelligent director agent and being located at a first geographic location.
  • the clustered computer system further includes a second cluster that includes a second plurality of computers also coupled together in the cluster configuration.
  • the second cluster is located at a second geographic location that is remote from the first geographic location.
  • the method includes predicting a first computer ofthe first plurality of computers that would experience the computer stress at a future point in time.
  • the computer stress is related to a number of transaction requests being serviced by the first computer at the future point in time.
  • the method includes ascertaining a software program that causes stress on the first computer ofthe plurality of computers.
  • the method also includes identifying, using the intelligent director agent, a given local computer among the first plurality of computers.
  • the given local computer represents a computer that is capable of running another module of the software program and does not run the software program prior to the stress occurring.
  • the method also includes identifying, using the intelligent director agent, a given remote computer among the second plurality of computers.
  • the given remote computer represents a computer that is capable of running the another module ofthe software program and did not run the software program prior to the stress occurring, and loading the another module ofthe software program on the given remote computer ofthe second plurality of computers to permit the given remote computer to receive and service a subset of transaction requests that require attention of the software program to reduce load on other computers ofthe first plurality of computers that also service the transaction requests.
  • Fig. 1 illustrates a prior art computer architecture wherein the computers are clustered in various stages to service the processing needs ofthe stages.
  • Fig. 2 illustrates, in accordance with one aspect ofthe present invention, a clustered computer system architecture wherein an intelligent director agent (IDA) is included with each ofthe clusters that implement the webserver stage, the business logic stage, and the data repository stage.
  • IDA intelligent director agent
  • Fig. 3 illustrates, in accordance with one embodiment ofthe present invention, a simplified logic block diagram of an exemplary business logic intelligent director agent (IDA).
  • Fig. 4 illustrates, in accordance with one embodiment ofthe present invention, a flowchart illustrating the steps employed to perform the software upgrade in a manner so as to improve the reliability ofthe clustered computer system.
  • IDA business logic intelligent director agent
  • Fig. 5 illustrates in detail, in accordance with one embodiment ofthe present invention, the step of routing the transaction request to the uncertified business logic module to handle.
  • Fig. 6 illustrates, in accordance with one embodiment ofthe present invention, a clustered computer system architecture that includes both a remote site and a local site, and the IDA's therefor.
  • Fig. 7 illustrates, in accordance with one embodiment ofthe present invention, a clustered computer system having a business logic stage which comprises a cluster of heterogeneous computers
  • Fig. 8 illustrates, in accordance with one embodiment ofthe present invention, a flowchart illustrating the steps for maintaining a proper level of fault tolerance for a business logic software.
  • Fig. 9 is a flowchart illustrating, in accordance with one embodiment ofthe present invention, a method for increasing the fault tolerance level pertaining to a particular business logic software which may also include the use of remote servers.
  • Fig. 10 illustrates, in accordance with one embodiment ofthe present invention, the steps involved in performing load balancing by shuffling the business logic modules among the business logic servers of a cluster if it is ascertained that the load level on any ofthe business logic servers is unacceptably high.
  • Fig. 11 illustrates in detail, in accordance with one embodiment ofthe present invention, the step of shuffling business logic modules among servers ofthe cluster to increase the processing capability ofthe business logic software identified to be the cause of server stress.
  • Fig. 12 is a flowchart illustrating, in accordance with one embodiment ofthe present invention, a method for performing load balancing by shuffling the business logic modules among the remote and local business logic servers.
  • Fig. 13 illustrates, in accordance with one embodiment ofthe present invention, the steps involved in performing load balancing prospectively by shuffling the business logic modules among the business logic servers of a cluster if it is ascertained prospectively from data available to ID As, such as the historical profile, that the load level on any ofthe business logic servers may become unacceptably high at some point in time.
  • FIG. 2 illustrates, in accordance with one aspect ofthe present invention, a clustered computer system architecture wherein an intelligent director agent (IDA) is included with each ofthe clusters that implement the webserver stage, the business logic stage, and the data repository stage.
  • IDA intelligent director agent
  • IDA there is an IDA for each cluster, although more than one cluster may be provided per stage, in which case multiple ID As may be provided.
  • the clusters may be disposed at one local site or may be dispersed among geographically remote locations.
  • Fig. 2 shows an intelligent director agent for each of these stages, it is contemplated that in some clustered computer systems, not every stage needs to be provided with an intelligent director agent and that significant benefits may be achieved by endowing even only one ofthe stages with one or more intelligent director agents.
  • a stage may comprise multiple clusters, in which case multiple ID As may be provided.
  • Clustered computer system 202 which is typically connected to a larger network such as the Internet or a portion thereof.
  • Clustered computer system 202 includes a webserver stage 204, and business logic stage 206, and a data repository stage 208.
  • Data repository stage 208 represents the stage wherein data for use by the business logic software modules are kept and includes the data stores as well as the database logic employed to access the data stores.
  • Business logic stage 206 represents the stage wherein the computer cluster(s) employed to execute the business logic software modules is implemented. For simplicity, only one cluster comprising four business logic servers is shown in Fig. 2.
  • Webserver stage 204 represents the stage wherein the computer cluster(s) employed to execute the webserver logic is implemented.
  • Webserver stage 204 generally facilitates the users' interaction with the rest of clustered computer system 202 using the web-based paradigm or a suitable paradigm for interacting with the Internet. Again, only one cluster comprising five webservers is shown in Fig. 2 to simplify the illustration.
  • the servers within each stage and within each cluster may be heterogeneous (i.e., implemented on different platforms and having different capability) and each may operate a different set of business logic modules, i.e., application software modules.
  • servers 216, 218, 220 and 222 within business logic stage 206 may be implemented using different hardware/software platforms and configurations that are adapted for operating the business logic software modules implemented therein.
  • the servers associated with a given stage or cluster or even those running copies of a particular software module be homogeneous (although such can be readily accommodated by the instant clustered computer system architecture without any major modification, as can be appreciated by those skilled in the art after reading this disclosure).
  • the servers in a cluster can communicate with the IDA that is . associated with that cluster and can be adapted to operate cooperatively with one another within a cluster, they can be implemented in the cluster architecture ofthe present invention.
  • IDA webserver logic intelligent director agent
  • webserver logic IDA 212 may ascertain whether the user had recently accessed the service through a particular webserver of webserver stage 204. If he did, there may be data pertaining to this user that is cached at the webserver that last serviced him, and it may be more efficient to continue assigning this user to that webserver to take advantage ofthe cached data.
  • webserver logic IDA 212 may assign the user to one of webservers 214a-214e. As in the prior art, the decision of which webserver to assign may be made based on the current relative load levels on the respective webservers, the information pertaining to which is periodically received by webserver logic IDA 212 from the webservers through path 232. Additionally, however, webserver logic IDA 212 also receives additional information pertaining to the webservers and the webserver logic software modules implemented on the webservers to facilitate improved access speed and reliability.
  • the webserver logic IDA 212 arbitrates among the webserver computers based not only on the relative load level information associated with the individual webservers but also based on information pertaining to the individual webserver logic software modules. For brevity sake, this aspect ofthe invention will be discussed in greater detail in the analogous discussion made in connection with the business logic IDA later herein.
  • the assigned webserver may then authenticate the user to ascertain whether the user is registered and/or properly authorized to use the service offered through clustered computer system 202. After successful authentication, if the user subsequently indicates that he wishes to employ a particular business logic software (by, for example, inputting data or taking an action that requires the attention of a particular business logic module), the webserver assigned to him then accesses a business logic IDA 240 to ascertain the appropriate business logic server (i.e., the appropriate server in the business logic stage such as one of business logic servers 2216, 218, 220 or 222) to which the user's transaction request may be sent.
  • a business logic IDA 240 to ascertain the appropriate business logic server (i.e., the appropriate server in the business logic stage such as one of business logic servers 2216, 218, 220 or 222) to which the user's transaction request may be sent.
  • the decision pertaining to which business logic server to assign may be made based on the current relative load levels on the respective business logic servers, the information pertaining to which is periodically received by business logic IDA 240 from the business logic servers through path 242. Additionally, however, business logic IDA 240 also receives additional information pertaining to the business logic servers and more importantly the business logic software modules implemented on the business logic servers to facilitate improved access speed and reliability. Accordingly, the routing decision taken by the business logic IDA is based not only on information pertaining to the individual business logic servers but also based on information pertaining to the individual business logic software modules implemented thereon.
  • the availability ofthe additional business logic server- specific information and the business logic module-specific information also facilitates inventive techniques to improve access speed and reliability during software upgrades, to maintain a desired level of fault tolerance for the business logic software and/or the business logic servers, to reactively and/or prospective load balance among the business logic servers, and to efficiently employ remote business logic servers to accomplish improving access speed and reliability.
  • a business logic software refers to a business logic program.
  • a business logic module refers to a copy ofthe business logic software.
  • the servers of a cluster may implement many different business logic software programs. Each of these business logic software programs has many copies distributed among the servers ofthe cluster to facilitate redundancy and scalability.
  • the business logic software module may consult yet another IDA (shown in Fig. 2 as database logic IDA 250), which picks the most suitable database server 252, 254, and/or 256 for serving up the data.
  • database logic IDA 250 the decision regarding which database server to assign may be made based on the current relative load level on the respective database servers that have the necessary data, the information pertaining to which is periodically received by database logic intelligent director agent 250 from the database servers through path 260.
  • the database logic IDA 250 also receives additional information pertaining to the database servers as well as the database server logic modules implemented on the database servers to facilitate improved access speed and reliability.
  • an intelligent directory agent receives more than just load status data from the servers it services.
  • the business logic IDA tracks one or more of the additional information such as server processing capability, server geographic identification (e.g., remote or local to the site that implements the webserver stage and/or the data repository stage), the average latency for servicing a transaction request (e.g., due to the server's geographic remoteness or the speed ofthe network connection), the list of business logic modules that are compatible with each server, the list ofthe business logic modules actually implemented on each server, the version ofthe business logic modules implemented, and/or the load experienced by the business logic modules on the servers.
  • server geographic identification e.g., remote or local to the site that implements the webserver stage and/or the data repository stage
  • the average latency for servicing a transaction request e.g., due to the server's geographic remoteness or the speed ofthe network connection
  • the list of business logic modules that are compatible with each server, the list ofthe business logic modules actually implemented on each server, the version ofthe business logic modules implemented,
  • the business logic IDA also receives information pertaining to external historical profiles (268) of transaction requests and processing loads on the business logic modules and/or the business logic servers in order to predict usage demands placed on the business logic modules and to prospectively balance the loads among the business logic servers if needed so that an anticipated surge in usage does not overwhelm any particular business logic module.
  • Fig. 3 illustrates, in accordance with one embodiment ofthe present invention, a simplified logic block diagram of an exemplary business logic intelligent director agent (IDA) 240.
  • IDA business logic intelligent director agent
  • the webserver logic IDA and the database logic IDA may be similarly formed. However, their similar construction will not be discussed in details for brevity sake.
  • business logic requests from the webservers are received by business logic IDA 240 via path 270.
  • business logic intelligent director agent 240 both server-specific and software-specific information is received and maintained in addition to the relative load status on individual business logic servers.
  • Some ofthe additional pieces of information are received from the business logic servers via path 242 and stored in exemplary blocks 304, 306, 308, 310, 312, 314, and 316 respectively.
  • exemplary blocks 304, 306, 308, 310, 312, 314, and 316 For ease of illustration, not every piece of information is shown in Fig. 3.
  • static information includes server processing capability and business logic module version number.
  • Other information may be dynamically received by the IDA from the servers (such as the list of business logic modules implemented on each server) and other network monitoring tools (such as conventional software tools that track network congestion at specific locations).
  • other information may be derived from the information received dynamically and/or statically (such as the average latency time for servers, which may be calculated periodically based on average network latency between the webserver and the business logic server, the average network latency between the business logic server and the available database cluster, the processing capability ofthe servers, and the like).
  • Business server directory 304 may track information pertaining to the list of business logic servers available to the clustered computer system, their remote/local status, their certified/uncertified status (which may be expressed as Boolean values or may be a numerical value that reflects their preference in receiving and servicing transaction requests), the list of business logic servers capable of being loaded with a particular business logic software, the list of business logic servers capable of being used for running a particular business logic module, their relative weight which reflects the relative preference with which traffic should be directed to the individual servers (e.g., due to network conditions or other factors), and the like.
  • Business logic module version block 306 may track information pertaining to the software versions ofthe business logic modules implemented on the various business logic servers. Further, business logic version block 306 may track information pertaining to the certified/uncertified status of each copy ofthe business logic modules, the relative weight of each copy of business logic module which reflects the relative preference with which traffic should be directed to it, and the like.
  • Business logic module load status block 308 may track information pertaining to the level of load currently experienced by the individual business logic modules (in number of transactions per second or the number of users currently using a business logic module, for example). This information may be tracked for business logic modules currently in operation, individually and/or as a group average.
  • Server processing capacity block 310 may track the processing capability (again in number of transactions per second or the number users that can be supported concurrently) of the individual business logic servers in order to ascertain how much bandwidth a particular server may have, how much has been used, and how much is available.
  • Business logic server load status block 312 may track a similar type of data as business logic module load status, albeit at the server level instead ofthe business logic module level.
  • Business logic server average latency block 314 may track the average latency to be expected if a particular business logic server is employed to service the transaction request. The average latency may be calculated based on the processing capability ofthe server, how remote it is from the webserver that issues the transaction request (which is impacted by network latency), how remote it is from the database that may be needed to service the transaction request (which is also impacted by network latency).
  • Business logic server log file block 316 may track the operational status ofthe business logic server and/or the business logic modules implemented thereon to determine, for example, the length of time that the server and/or the business logic module has been in operation without failure and other types of log file data.
  • Business logic intelligent director agent 240 also includes a data mining module 330, which receives the external historical, profiles (268 of Fig. 2) of past usage trends on the various business logic modules and/or business logic servers, and ascertains prospectively the load demand on the various business logic modules and/or business logic servers.
  • Data mining module 330 may be implemented using a variety of available data mining methodologies. One implementation of data mining is further discussed in the aforementioned data-mining related applications, which are incorporated herein by reference.
  • a business logic selector module 334 selects one ofthe business logic servers to service the pending business logic request and transmits the selection to the requesting webserver via path 272.
  • a configurator module 340 representing the module that either reactively or prospectively reconfigures and/or reshuffles the business logic modules among the business logic servers to permit the clustered computer system to better handle the processing load and to achieve the desired level of fault tolerance.
  • the reliability and service quality risks associated with software upgrade operations are vastly reduced by allowing the copies ofthe new business logic module to be gradually phased in on only a percentage ofthe servers that eventually need to be upgraded.
  • software version upgrade at least a number of copies ofthe older version ofthe business logic module being upgraded are preferably kept unchanged to continue to service transaction requests in the interim.
  • copies ofthe new business logic module are loaded on a percentage ofthe servers that need to be upgraded, their load levels are increased gradually, either incrementally in stages or smoothly over time, until their operation log files indicate that they or the servers on which they are implemented have passed some predefined reliability criteria (which may be set in term ofthe number of hours of continuous operation, the number of users supported concurrently, a combination thereof, or the like).
  • some predefined reliability criteria which may be set in term ofthe number of hours of continuous operation, the number of users supported concurrently, a combination thereof, or the like.
  • the number of servers to be loaded at any given in time is preferably chosen so that if they, as a group, fail, the remaining servers are adequate to handle the existing load without undue impact to the users (i.e., without undue degradation in performance).
  • the use of remote business logic servers may allow a greater number of servers of a particular cluster to be loaded at once since the remote business logic servers may be able to serve as redundant servers to handle the stream of transaction requests should the servers currently undergoing certification fail.
  • the number of servers to be loaded with copies of a new business logic module may be determined based on the expected level of usage of the business logic module and the processing capability ofthe servers ofthe clusters, among other factors. If the load is expected to be high, more servers may be loaded with copies ofthe new business logic module. On the other hand, if the processing capability ofthe servers to be loaded is fairly high, the number of servers required may be reduced since each copy may support a greater number of concurrent users.
  • Fig. 4 illustrates, in accordance with one embodiment ofthe present invention, a flowchart illustrating the steps employed to perform the software upgrade in a manner so as to improve the reliability ofthe clustered computer system.
  • the invention preferably upgrades only a percentage ofthe number of servers that need upgrading in the cluster at any given time.
  • the new business logic modules are initially loaded, they are deemed uncertified until they pass some reliability criteria.
  • user transaction requests are routed to both the certified business logic modules (the old but proven copies ofthe business logic software) and the new business logic modules.
  • a routing function ensures that the traffic load on the uncertified business logic modules are brought up gradually.
  • the uncertified business logic modules pass some reliability criteria, they become certified and another set of old business logic modules can be replaced (or another set of servers can be loaded) in a similar manner.
  • step 402 as user transaction requests are received at the cluster, the intelligent director agent is consulted to determine whether there exists an uncertified business logic module in the cluster.
  • a business logic module may be flagged as uncertified if it is a new, unproven copy of a business logic software on a server and/or an existing business logic module that is implemented on a server which happens to also be loaded with another business logic module undergoing certification. The latter situation may also present a reliability risk since the entire server may crash (due to e.g., conflict with the newly loaded business logic module).
  • IDA 240 is consulted to determine whether there exists any uncertified business logic module on any of servers 216, 218, 220, and 222. If no uncertified business logic module is found in the servers ofthe cluster (step 404), the method proceeds to route the incoming transaction request (or the user) to one ofthe certified business logic modules using a conventional routing methodology (such as round robin, based on the relative load levels, and the like). This routing step is shown in step 406 of Fig. 4.
  • a conventional routing methodology such as round robin, based on the relative load levels, and the like.
  • step 408 a routing function is ascertained to determine whether one ofthe uncertified business logic modules should service the incoming transaction request.
  • the routing function is configured to increase the load level ofthe uncertified business logic module in a gradual manner.
  • the uncertified b ⁇ siness logic module may be brought up to capacity gradually over time, or to some predefined threshold initially, allowed to level off for a period of time to assure that the uncertified business logic module can function satisfactorily at that threshold level before that uncertified business logic module is permitted to receive additional loads.
  • multiple threshold levels may be involved.
  • the routing function allows the uncertified business logic module to be brought on line gradually, the specific mathematical construct ofthe routing function may vary widely depending on need and preference.
  • the incoming transaction request is then routed to one ofthe uncertified business logic modules to handle in step 410.
  • the IDA since the IDA also has available to it performance data pertaining to the individual servers, the IDA may intelligently route the incoming transaction request to a specific uncertified business logic module that can service the transaction request in the most timely manner among all uncertified business logic modules. Additionally and/or alternatively, all uncertified business logic modules may be loaded with transaction requests gradually and equally so as to minimize the impact on the whole clustered computer system if any random module fails.
  • the adverse impact of one or more server crashing during certification may be reduced even further by staging the number of servers simultaneously undergoing certification such that initially, only a very small number (e.g., 1 or only a few) is first allowed to undergo certification. For subsequent groups of servers undergoing certification, their number may be allowed to increase such that a greater number of servers concurrently undergoing certification may be allowed. With this technique, initial crashes may be experienced by only a small percentage ofthe servers.
  • the servers to be loaded with the uncertified business logic modules in this case may be new servers that are installed locally with the cluster or servers that are remote to the cluster but are registered with the local IDA for receiving and servicing transaction requests for the purpose of providing redundancy during software upgrade.
  • the remote servers may run the old, certified modules to provide redundancy while the uncertified modules are loaded onto the existing local servers to leave the capacity attributable to the certified business logic module substantially unchanged.
  • the transaction requests routed to the uncertified copies may be executed concurrently on a certified copy or cached to allow seamless recovery in the event of a crash.
  • the method ascertains whether the uncertified business logic module has passed some predefined reliability criteria.
  • the reliability criteria may be ascertained from reviewing the log file associated with the uncertified business logic module and/or the server undergoing certification (e.g., by consulting business logic server log file block 316 of Fig. 3, for example). If the reliability criteria is satisfied, the uncertified business logic module may have its status changed to certified in step 414. Thereafter, the steps of Fig. 4 end at step 416.
  • Fig. 5 illustrates, in accordance with one embodiment ofthe present invention, step 410 of Fig. 4 (routing the transaction request to the uncertified business logic module to handle) in greater detail.
  • the transaction request is forwarded to the uncertified business logic module.
  • the transaction being performed is optionally safeguarded by additionally caching the transaction request data or by running the request concurrently on one ofthe certified business logic modules, which may be local or remote. If the uncertified business logic module or the server on which it is implemented crashes (step 506), the transaction request currently underway may be completed by the certified business logic that runs the transaction concurrently or the transaction may be executed again using the cached data pertaining to the transaction using another certified business logic module. This is shown in step 508.
  • the uncertified business logic module that failed is removed from the cluster (step 510) and its status may be updated accordingly with the business logic IDA.
  • its reliability indicator is upgraded (by, for example, upgrading the operation log file ofthe uncertified business logic module (step 512)).
  • the uncertified business logic module is able to complete the transaction, there may be no need to complete the transaction request by the redundant certified business logic module since only one business logic module should complete servicing the transaction request by the user.
  • a preference rule may be set such that the transaction is always completed by the uncertified business logic module if no crashing occurs to simulate as closely as possible the conditions that the uncertified business logic module will experience once it becomes certified.
  • another preference rule may dictate that the certified business logic module always complete the transaction during the software upgrade period so as to minimize any impact on customer service and system reliability if the uncertified business logic module fails, since the uncertified business logic modules are not relied on to complete the transactions anyway.
  • the business logic modules may be dynamically reshuffled among the servers ofthe cluster, may be upgraded by the e-commerce service and/or its partners, and may be implemented on a variety of heterogeneous servers all having different capabilities and mix of business logic modules. Accordingly, at any given time, some ofthe business logic modules may be in the process of being upgraded or some of the resources they may need (such as the local database) may be temporarily unavailable due to maintenance/upgrade. Further, some business logic modules may be implemented on servers that are more powerful than others, and some may be off-site at a remote location. All these activities impact the response time for transaction requests and need to be dynamically accounted for in order to minimize the wait time for customers ofthe e-commerce site.
  • the routing of traffic is made more efficient utilizing the additional information pertaining to the business logic modules and business logic servers that are tracked by the ID As.
  • the IDA ofthe present invention further employs, among others, information pertaining to the processing capacity of the servers, the certified/uncertified status ofthe business logic modules, and the average latency ofthe servers on which the requisite business logic modules are implemented, in order to make its routing decision.
  • Information pertaining to the processing capacity ofthe servers may powerfully impact the routing decision since a more powerful server may be able to process a transaction request faster than a less powerful server even if the more powerful server may appear to be more heavily loaded.
  • the server processing capability is tracked by the business logic IDA (as well as other ID As for their clusters) in block 310.
  • the server processing capability may be obtained when the server is first installed and registered with the IDA.
  • the certified/uncertified status ofthe business logic modules may impact the ability of a business logic module to accept transaction requests since, as mentioned earlier, a routing function may limit the speed at which the load on an uncertified business logic module is ramped up after software upgrade.
  • the certified/uncertified status may be registered by the business logic module undergoing certification or by the server for all the business logic modules implemented on that server if one ofthe business logic modules currently undergoes certification and poses a reliability risk to the server. This is because, as mentioned, even if the business logic module being requested has not been upgraded recently, another business logic module on its server may have been upgraded or loaded recently, which affects the reliability risk of that server as well as all business logic modules implemented thereon. When a business logic module is labeled as uncertified, it may be deemed less preferred for servicing transaction requests by a routing function.
  • the geographic distribution ofthe clusters and servers may also impact routing decisions.
  • servers of a given e-commerce service widely dispersed over a large geographic area, both to improve service to its worldwide customer base and also to provide some measure of protection against natural or man-made catastrophes that may impact a given geographic location.
  • transaction requests originated from a given locality be serviced by business logic servers that are closest to the place of origination.
  • remote servers need to be employed in order to reduce the transaction request processing times.
  • the available resources of a particular local cluster for servicing transaction requests may be reduced.
  • the delay may be less if remote servers are called upon to service the transaction requests originated locally, particularly if the remote servers have ready and quick access to the needed database resource at the remote site.
  • the business logic server average latency is kept by block 314 of Fig. 3, for example.
  • the servers are interconnected with one another and with other components ofthe clustered computer system using networking technologies, network congestion at specific locations may unduly increase the time required to process a transaction request. Due to network latency, it may be possible to reduce the transaction request service time by routing the transaction request to a remote server for servicing.
  • Fig. 6 illustrates, in accordance with one embodiment ofthe present invention, a clustered computer system architecture wherein a business logic IDA 602 of a local site 604 receives feedback data from both the business logic servers/business logic modules of a remote site 606 (via a connection 608) and business logic servers/business logic modules of local site 604 so that it can, through connection 610, direct traffic to the business logic servers ofthe remote site.
  • Network traffic data pertaining to specific connections within and between the various sites may be obtained through the use of appropriate network sniffers or other software tools and furnished to the business logic IDA 602 of local site 604 so that the appropriate calculation pertaining to average latency can be made.
  • a connection 612 is also shown, indicating that business logic IDA 602 is also capable of directing the business logic servers of remote site 606 to reconfigure themselves to achieve load balancing and fault tolerance.
  • business logic IDA 602 is also capable of directing the business logic servers of remote site 606 to reconfigure themselves to achieve load balancing and fault tolerance.
  • one or both ofthe routing and the reconfiguration connections from one site to another may also be made between IDA's. Reconfiguration ofthe business logic modules to achieve load balancing and fault tolerance is discussed in detail later herein.
  • connections that facilitate routing and reconfiguration ofthe business logic servers/business logic modules of local site 604 by business logic IDA 614 of remote site 606 are also not shown.
  • the reverse connections that allow business logic IDA 614 of remote site 606 to track information pertaining to the business logic servers/business logic modules of local site 604 are not shown in Fig. 6.
  • similar connections between the servers and ID As ofthe web server stage and the data repository stages ofthe various sites are also not shown in order to simplify the illustration. Additionally, more than one remote site may be present. However, the details pertaining to these connections should be readily apparent to the artisan given the disclosure above.
  • FIG. 7 illustrates, in accordance with one embodiment ofthe present invention, a clustered computer system 702 having a business logic stage which comprises a cluster of heterogeneous computers, as indicated by their diverse shapes to symbolically indicate that servers 704, 706, 708, and 710 may be implemented by computers running on different hardware and operating systems.
  • Fig. 8 illustrates, in accordance with one embodiment ofthe present invention, a flowchart illustrating the steps for maintaining a proper level of fault tolerance for a business logic software. In contrast to the prior art, fault tolerance may be implemented in the present invention for a business logic software instead of at the server level.
  • the fault tolerance level for a particular business logic module is ascertained. Typically, this is ascertained by determining the number of servers that have thereon the business logic module in question and compare this number to some predefined fault tolerance threshold number. This information may be obtained by reviewing the list of business logic modules implemented on the various business logic servers ofthe cluster. Since failure typically occurs at the server level, i.e., a business logic module failure typically affects the entire server or at least all copies of that business logic module on that server, it is generally the number of servers having thereon copies ofthe business logic software at issue that is material in step 802. In one embodiment, uncertified business logic modules pertaining to a particular business logic software (or servers undergoing maintenance/software upgrade) are not considered sufficiently reliable and may not be counted (or only partially counted) toward the number of business logic modules available to provide fault tolerance.
  • step 806 If the fault tolerance level for the business logic module in question is below a predefined acceptable level (as determined in step 804), the method proceeds to step 806 to warn the system operator and give the operator an opportunity to add additional servers.
  • additional business logic modules pertaining to the business logic software at issue may be loaded onto existing business logic servers ofthe cluster (particularly those that did not already have a copy of that business logic module running).
  • the addition of a software module no matter how well tested and proven, always involves reliability risks (due to, for example, software conflicts) and it is typically less risky to employ a server that is new to the cluster so as not to interfere with the other servers already running in the cluster.
  • the method proceeds to step 808 to search for the least utilized server in the cluster (or a powerful server in the local cluster that is relatively lightly loaded) that does not already have a copy ofthe business logic module at issue loaded.
  • the selected server is also one that is known to the IDA to have the ability or compatibility to accept another copy ofthe business logic software having the inadequate fault tolerance level. Again, this information may be ascertained by reviewing the IDA, e.g., the business logic server directory 304 of Fig. 3.
  • the utilization level ofthe new server is of course about zero in step 808 and if that new server is compatible to receive another copy ofthe business logic module in question, that new server may be selected. At any rate, one ofthe existing servers in the cluster that is both least utilized and compatible/able to accept another copy ofthe business logic module at issue will be selected.
  • step 810 another copy ofthe business logic module pertaining to the business logic software that has the inadequate fault tolerance level is loaded onto the server selected in step 810.
  • the business logic IDA may accomplish this by issuing an appropriate command to the selected business logic server through its reconfiguration facility. In the Internet case, this may include instructing the business logic server to access another computer on the net to retrieve a copy ofthe business logic module to be loaded and load it. Thereafter, the server that has just been loaded with the business logic module that previously has the inadequate fault tolerance level is registered with the IDA (step 812) to begin accepting transaction requests.
  • the server that has just been loaded with a copy ofthe business logic module may be (but not required to be) registered as uncertified and the addition of another copy of this business logic module may be treated as a software upgrade operation to this server to allow the load to be increased gradually in accordance with the software upgrade method described earlier herein.
  • Fig. 9 is a flowchart illustrating, in accordance with one embodiment ofthe present invention, a method for increasing the fault tolerance level pertaining to a particular business logic software which may also include the use of remote servers.
  • the clusters ofthe clustered computer system may be scattered among different geographically dispersed sites to improve service to geographically dispersed customers and/or to increase survivability in case of a natural/manmade disaster that affects one site.
  • the business logic servers of a remote site e.g., remote site 606 of Fig. 6
  • a local site e.g., local site 604 of Fig. 6
  • the fault tolerance level for a particular business logic software of a given local site is ascertained. In the context of a multiple-site clustered computer system, this may be ascertained by determining the number of servers at the local site (e.g., local site 604 of Fig. 6) that have thereon copies ofthe business logic module in question and compare this number to some predefined fault tolerance threshold number. In one embodiment, uncertified business logic modules (or modules implemented on servers undergoing maintenance/software upgrade) are not considered sufficiently reliable and may not be counted (or only partially counted) toward the number of business logic modules available to provide fault tolerance.
  • step 906 If the fault tolerance level for the business logic module in question is below a predefined acceptable level (as determined in step 904), the method proceeds to step 906 to warn the system operator and give the operator an opportunity to add additional servers to the local cluster. Additionally or alternatively, additional copies ofthe business logic software having the inadequate level of fault tolerance can be loaded onto existing business logic servers ofthe local cluster (particularly those that did not already have a copy of that business logic software running). If the operator does not respond after a predefined period of time or if the operator affirmatively indicates that no additional server will be added, the method proceeds to step 908 to search for the least utilized server in the local cluster (or a powerful server in the local cluster that is relatively lightly loaded) that does not already have the business logic module in question loaded.
  • this determination may be made by reviewing information collected at the IDA, such as the list of business logic servers, the list of business logic modules on each server, the load level on the servers, and the like.
  • the selected local server is also one that is known to the local IDA to have the ability or compatibility to accept another copy ofthe business logic having the inadequate fault tolerance level. If there is one or more local server that has the capability (defined as, for example, a minimum processing capability threshold) or compatibility to accept another copy ofthe business logic software having the inadequate fault tolerance level (as determined in step 910), the method proceeds to step 916 to load another copy ofthe business logic module onto selected business logic server at the local site.
  • step 912 select a business logic server in the remote cluster (e.g., the cluster in the business logic stage of remote site 606 of Fig. 6) to provide fault tolerance for the local cluster.
  • a business logic server that already has a copy ofthe business logic module in question loaded to serve as the redundant business logic server for the local cluster for the purpose of increasing fault tolerance therein.
  • one or more ofthe remote servers are now selected to contribute their processing capability to the local cluster to increase fault tolerance.
  • the least utilized server in the remote cluster (or a powerful server in the remote cluster that is relatively lightly loaded) that does not already have the business logic module in question loaded and that is known to the local IDA to have the ability or compatibility to accept another copy of he business logic software having the inadequate fault tolerance level is selected to be loaded with another copy ofthe business logic software needing the increased level of fault tolerance.
  • the loading may be accomplished via the local IDA through its reconfiguration facility or through the remote IDA (under instruction from the local IDA).
  • the selected server (either remote or local) having thereon another copy ofthe business logic software that requires the increased fault tolerance level is registered with the local IDA (step 914) to begin accepting transaction requests.
  • the newly registered server may be (but not required to be) registered as uncertified to allow the load to be increased gradually in accordance with the software upgrade method described earlier herein.
  • the newly registered server is a remote server, its status may be noted by the IDA so that it is less preferred in the routing of incoming transaction requests at the local site in order to avoid creating network congestion unnecessarily or to avoid the long latency typically associated with using remote servers.
  • a business logic server to provide additional fault tolerance protection may also be made by reviewing the load level data, the latency data and/or the processing capability data kept at the business logic ID As without regard to whether the additional server is "local” or "remote.” This may very well be the case if the clusters ofthe clustered computer system network are connected through reliable, high speed network connections and the geographical distinction may therefore be less important. In this case, the business logic server that is mostly lightly loaded may well be selected to be loaded with another copy ofthe business logic software needing increased fault tolerance.
  • a rule may be stated wherein it is more preferable to employ a remote server that already has thereon a copy ofthe business logic software for the purpose of increasing fault tolerance at the local cluster (provided that the load level and latency are acceptable) than to load another copy ofthe business logic software onto another local server (since such software loading operation may be deemed in some systems to take too long and/or unduly increase the reliability risk).
  • a remote server that already has thereon a copy ofthe business logic software for the purpose of increasing fault tolerance at the local cluster (provided that the load level and latency are acceptable) than to load another copy ofthe business logic software onto another local server (since such software loading operation may be deemed in some systems to take too long and/or unduly increase the reliability risk).
  • the fault tolerance level for a business logic software may be increased prospectively to account for activities or events that may increase the reliability risk.
  • software upgrade or software loading operations may heighten the risk of one or more server crashing (and may therefore potentially affect the reliability of the copy being upgraded/loaded/modified and/or any business logic module that is implemented on a server undergoing the reliability risk-affecting activities). This is particularly true if software upgrade and/or maintenance activities are performed on a group of business logic servers and their simultaneous crashing would lead to a catastrophic shortage in the processing capability for one or more ofthe business logic software even when a "normal" level of fault tolerance exists prior to failure.
  • fault tolerance level at the local site may be increased just in case fault tolerance relief is needed and the extra capacity is not available in the remote servers.
  • the fault tolerance level may be increased prospectively over the level normally existing in the absence of such reliability risk-affecting activities.
  • fault tolerance may be raised by either increasing the predefined acceptable fault tolerance level for the business logic software that experiences the heightened reliability risk or by not taking into account (or taking into account only partially) the contribution ofthe copies ofthe business logic module at risk in the calculation of available fault tolerance.
  • the copies ofthe business logic modules implemented thereon may be downgraded (or discounted altogether) in terms of their ability to provide redundancy for fault tolerance purposes. Since different business logic servers ofthe cluster may have thereon different sets of business logic modules, there may be times when there is more demand placed on a particular business logic software than others. Thus, even with correct routing, the set of business logic servers having thereon copies ofthe business logic software in demand will be more heavily loaded than other business logic servers which do not have thereon a copy ofthe business logic software in demand. In extreme cases, some business logic servers ofthe cluster may be stressed while other business logic servers may sit idle.
  • Adding additional servers to the cluster to handle the spikes in demand on a particular business logic software has its disadvantages.
  • the addition of a server to a cluster is typically an expensive option and usually involves a substantial delay (during which time transaction request response suffers) and investment in time (since it requires the acquisition, installation, and configuration of new hardware in the existing cluster).
  • the number of servers required to handle peak demand for every business logic software implemented in the cluster may be disproportionately large relative to the average processing requirement placed on the cluster since the demands on different business logic modules may fluctuate at different times, and a business logic module that may be idle at one point in time may be heavily used at other times, and vice versa.
  • a load balancing technique which involves reconfiguring the business logic servers using business logic module-specific load information collected by the ID As.
  • the present invention preferably obtains the load information on the business logic modules themselves. With this information, it is possible to ascertain the specific business logic module(s) that contribute to server stress, and to identify the business logic module(s) that are relatively idle at any given point in time. Once identified, the business logic modules may be shuffled among the business logic servers ofthe cluster to allow the load to be better balanced among the business logic servers.
  • Fig. 10 illustrates, in accordance with one embodiment ofthe present invention, the steps involved in performing load balancing by shuffling the business logic modules among the business logic servers of a cluster if it is ascertained that the load level on any ofthe business logic servers is unacceptably high, e.g., greater than some predefined load level for some predefined time period.
  • the load levels for the business logic servers ofthe cluster are ascertained.
  • the load level information is transmitted periodically or on demand from the business logic servers to the IDA. If the load level on any particular business logic server is greater than a predefined acceptable load level (as determined in step 1004), the method proceeds to step 1006 to ascertain the business logic module(s) that are causing the high load level ofthe stressed servers.
  • the identification ofthe business logic modules that are causing server stress may be made by reviewing the business logic module-specific load information received periodically or on demand by the IDA (e.g., by reviewing the processing demand placed on individual business logic modules that exist on the stressed server).
  • the method may proceed to an optional step 1008 to warn the system operator and give the operator an opportunity to take action to increase the processing capability ofthe business logic software that causes the server stress condition (since additional processing capability may relieve stress on the stressed servers) and/or reduce the demand on the business logic servers experiencing the server stress condition. If no action is taken or if the default is automatic load balancing, the method proceeds to step 1010 to perform load balancing among the existing business logic servers ofthe business logic stage.
  • Load balancing may be performed only among local servers, as is discussed in connection with Fig. 11 in one embodiment, or may be performed in a manner so as to also include the remote servers, as is discussed in connection with Fig. 12 in one embodiment.
  • the method After load balancing is performed, the method returns to step 1002 to continue to monitor the load level information on the servers to ascertain whether load balancing has addressed the server stress problem. Preferably, some time should be allowed to permit the routing mechanism to distribute the load among the reconfigured servers ofthe cluster before load balancing is performed again (to ensure system stability and prevent wild gyrations in the distributed loads among the servers).
  • load balancing involves identifying servers ofthe cluster that can be loaded with copies ofthe business logic software identified to be causing server stress so that the demand on that business logic software may be spread among a greater number of servers ofthe cluster.
  • Fig. 11 illustrates, in accordance with one embodiment ofthe present invention, the steps for performing step 1010 of Fig. 10, i.e., for shuffling business logic modules among servers ofthe cluster to increase the processing capability ofthe business logic software identified to be the cause of server stress.
  • the method searches for the least utilized server in the cluster (or a powerful server in the cluster that is relatively lightly loaded) that does not already have a copy ofthe business logic module identified to be the cause of server stress already implemented thereon.
  • the selected server is also one that is known to the IDA to have the ability or compatibility to accept another copy ofthe business logic module identified to be the cause of server stress.
  • the server identified as a candidate to relieve the server stress condition is evaluated to ascertain whether it has sufficient processing capability to receive a copy ofthe business logic software identified to be the cause of server stress.
  • step 1102 If there is sufficient processing capability in the server identified in step 1102 (as determined in step 1104), the method proceeds to step 1106 wherein another copy ofthe business logic software that was identified to be the cause of server stress is implemented on that server in order to increase the processing capability ofthe business logic module identified earlier to be the cause of server stress.
  • step 1108 to attempt to move one or more ofthe business logic modules currently implemented on that server to another server to create the needed processing capability.
  • one or more existing business logic modules on the server identified in step 1102 may be moved onto another server ofthe cluster that is also relatively lightly loaded to make room for a copy ofthe business logic module ascertained to be the cause of server stress to be loaded onto the server identified in step 1102. It is preferable, of course that due attention is paid (by checking with the IDA beforehand) to compatibility issues during business logic module shuffling.
  • the business logic IDA may accomplish this by issuing appropriate commands to the selected business logic server(s) through its reconfiguration facility.
  • step 1106 represents the step wherein another copy ofthe business logic software that was identified to be the cause of server stress is implemented on that server in order to increase the processing capability of the business logic module identified earlier to be the cause of server stress.
  • step 1110 the selected server having thereon another copy ofthe business logic software that requires the increased fault tolerance level is registered with the local IDA (step 914) to begin accepting transaction requests.
  • the newly registered server may be (but not required to be) registered as uncertified to allow the load to be increased gradually in accordance with the software upgrade method described earlier herein.
  • load balancing may be performed by increasing by one at a time the number of servers having thereon the business logic software that has the high demand.
  • load balancing may be performed by increasing by one at a time the number of servers having thereon the business logic software that has the high demand.
  • the traffic spike on a given business logic software is fairly severe (as ascertained by reviewing the historical profile of transaction requests)
  • a greater number of servers may be simultaneously loaded with copies ofthe business logic software that causes server stress in order to more quickly relieve the stress condition.
  • the additional number of servers required may be moderated if one ofthe more powerful servers is employed to increase the processing capability ofthe business logic software causing the original server stress condition.
  • the number of business logic servers that are loaded with copies ofthe business logic software that causes the stress condition may stay the same after shuffling, albeit with the more powerful servers ofthe cluster being substituted in to increase the processing capability of that business logic software.
  • Fig. 12 is a flowchart illustrating, in accordance with one embodiment of he present invention, a method for performing load balancing by shuffling the business logic modules among the remote and local business logic servers.
  • the clusters ofthe clustered computer system may be scattered among different geographically dispersed sites to improve service to geographically dispersed customers and/or to increase survivability in case of a natural/manmade disaster that affects one site.
  • the business logic servers of a remote site e.g., remote site 606 of Fig. 6
  • the method searches for the least utilized local server in the local cluster (or a local, powerful server in the local cluster that is relatively lightly loaded) that does not already have a copy ofthe business logic software identified to be the cause of server stress already implemented thereon.
  • the selected local server is also one that is known to the IDA to have the ability or compatibility to accept another copy ofthe business logic software identified to be the cause of server stress.
  • the server identified as a candidate to relieve the stress condition is evaluated to ascertain whether it has sufficient processing capability to receive a copy ofthe business logic software identified to be the cause of server stress.
  • step 1204 If there is sufficient processing capability in the server identified in step 1202 (as determined in step 1204), the method proceeds to step 1206 wherein another copy ofthe business logic software that was identified to be the cause of server stress is implemented on the identified server in order to increase the processing capability ofthe business logic software identified earlier to be the cause of server stress.
  • step 1208 the method proceeds to step 1208 to ascertain whether it is possible to move one or more ofthe business logic modules currently implemented on that local server to another server to create the needed processing capability.
  • one or more existing business logic modules on the server identified in step 1202 may be moved onto another server ofthe cluster that is also relatively lightly loaded to make room for a copy ofthe business logic module ascertained to be the cause of server stress to be loaded onto the server identified in step 1202. This is performed in step 1209. It is preferable, of course that the new local server(s) that receive these lightly loaded copies are not the ones that also need relief through load balancing themselves.
  • the business logic IDA may accomplish this by issuing appropriate commands to the selected business logic server(s) through its reconfiguration facility.
  • the business logic modules to be moved are lightly used anyway, it may be possible to simply delete or disable the copy ofthe lightly loaded business logic modules from the local server that is identified for relieving the server stress condition. This approach may be acceptable if there is sufficient fault tolerance and/or processing capability in the remaining copies ofthe lightly loaded business logic module after deletion.
  • a copy ofthe business logic module that is to be moved to create additional processing bandwidth on the server that is identified for relieving the server stress condition may be loaded on a remote server to still leave the processing capacity of that lightly loaded business logic module unchanged, albeit through the use of a remote server.
  • step 1202 If reshuffling the business logic modules existing on the local server identified in step 1202 would result in sufficient processing capacity to allow another copy ofthe business logic software identified to be the cause of server stress to be implemented thereon (as determined in step 1208), the method proceeds to step 1206, which, as mentioned earlier, represents the step wherein another copy ofthe business logic that was identified to be the cause of server stress is implemented on the identified server in order to increase the processing capability ofthe business logic module identified earlier to be the cause of server stress.
  • step 1206 represents the step wherein another copy ofthe business logic that was identified to be the cause of server stress is implemented on the identified server in order to increase the processing capability ofthe business logic module identified earlier to be the cause of server stress.
  • step 1208 determines whether reshuffling the business logic modules existing on the local server identified in step 1202 would not result in sufficient processing capacity to allow another copy ofthe business logic software identified to be the cause of server stress to be implemented thereon.
  • the method proceeds to step 1212 to search a suitable remote server to relieve the server stress condition on the local cluster.
  • the method may try to ascertain with a few local servers to determine whether shuffling locally would result in local capacity to receive another copy of the business logic software that causes the server stress.
  • the suitable remote server may be a lightly loaded remote server that already has a copy ofthe business logic software identified to be the cause of server stress already implemented thereon or a lightly loaded remote server in the remote cluster (or a powerful remote server in the remote cluster that is relatively lightly loaded) that does not already have a copy ofthe business logic software identified to be the cause of server stress already implemented thereon but can also accept, or be arranged via shuffling at the remote site to accept, a copy ofthe business logic software identified to be the cause of server stress.
  • the remote cluster is first searched for the presence of a lightly loaded remote server that already has a copy ofthe business logic software identified to be the cause of server stress already implemented thereon.
  • step 1214 If such a server exists (as determined in step 1214), it is registered with the local IDA and the local IDA may subsequently employ it to relieve the server stress condition locally.
  • the method proceeds to step 1216 to search for the least utilized server in the remote cluster (or a powerful server in the remote cluster that is relatively lightly loaded) that does not already have a copy ofthe business logic software identified to be the cause of server stress implemented thereon.
  • the remote server identified as a candidate to relieve the stress condition is loaded with another copy ofthe business logic software that was identified to be the cause of server stress in order to increase the processing capability ofthe business logic software identified earlier to be the cause of server stress.
  • the remote server is then registered with the local IDA to begin receiving transaction requests to relieve the server stress condition.
  • load balancing has revolved around reactive load balancing, i.e., balancing the load after the stress condition is detected on one ofthe business logic servers.
  • load balancing is insufficient to address the stress condition.
  • certain business logic modules may experience an increase in usage so rapidly that there may be no time to perform load balancing reactively (i.e., after detection ofthe stress condition) without suffering poor transaction request processing performance or increased reliability risks due to dangerously high stress conditions.
  • a potential stress condition may be averted by performing the load balancing among the local servers and/or the remote servers prospectively. Since the ID As receive the historical profiles of transaction requests, data mining techniques may be applied to ascertain the trends of demand placed on various business logic software programs. By way of example, if the business logic software services bank withdrawals, an examination ofthe historical profiles of transaction requests may reveal that bank withdrawals tend to happen prior to a major holiday and may be the heaviest at the close ofthe business day immediately preceding the holiday.
  • This information may be employed to determine whether a stress condition is likely to occur on one or more servers ofthe local cluster and whether load balancing should be performed prior to the expected peak demand.
  • Fig. 13 illustrates, in accordance with one embodiment ofthe present invention, the steps involved in performing load balancing prospectively by shuffling the business logic modules among the business logic servers of a cluster if it is ascertained prospectively from data available to ID As (such as the historical profile) that the load level on any ofthe business logic servers may become unacceptably high at some point in time, e.g., greater than some predefined load level.
  • step 1302 the load levels for the business logic servers ofthe cluster are forecasted from data available to ID As (such as the historical profiles of transaction requests).
  • ID As such as the historical profiles of transaction requests.
  • the load level information is forecasted using a data mining technique. Implementations of data mining for this purpose may be found in the aforementioned data-mining applications, which are incorporated by reference herein.
  • step 1304 the method to an optional step 1308 to warn the system operator and give the operator an opportunity to take action to increase the processing capability ofthe business logic software that is forecasted to cause the server stress (since additional processing capability may relieve the potential server stress from the anticipated increase in traffic) and/or reduce the forecasted demand on the business logic software (e.g., by diverting traffic away from this cluster). If no action is taken or if the default is automatic load balancing, the method proceeds to step 1310 to perform load balancing among the existing business logic servers ofthe business logic stage.
  • the load balancing is performed only a short time before the expected stress condition so that interference with the normal distribution of processing capacity among the business logic servers is kept minimal.
  • Exemplary techniques of load balancing among local servers and among both local and remote servers are discussed in details in connection with Figs. 11 and 12 herein and is not repeated here for brevity's sake.

Landscapes

  • Business, Economics & Management (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Strategic Management (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Development Economics (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Stored Programmes (AREA)

Abstract

La présente invention concerne un agent directeur intelligent conçu pour être utilisé dans un système informatique en grappe comportant au moins un premier ordinateur et un second ordinateur, à la fois les premier et second ordinateurs étant configurés pour exécuter des modules de logiciels relatifs à un programme de logiciel. Par ailleurs, un sélecteur de module de logiciel est configuré pour sélectionner le premier ordinateur et/ou le second ordinateur afin de traiter la demande de transaction. De plus, cette invention concerne un procédé permettant d'améliorer la fiabilité d'un programme de logiciel mis en oeuvre dans un système informatique en grappe, tout en le mettant à niveau d'une première version à une seconde version. Le programme de logiciel est mis en oeuvre sous forme de modules de logiciel, ce procédé consistant alors à vérifier un niveau de certification associé à chacun des modules de logiciel. En outre, cette invention concerne un procédé permettant de maintenir une marge acceptable de tolérance aux défaillances prédéfinie pour un ensemble de modules de logiciel mettant en oeuvre un programme de logiciel s'exécutant dans un premier ensemble d'ordinateurs couplés ensemble selon une configuration en grappe, et couplés à un premier agent directeur intelligent. Ce procédé consiste également à utiliser l'agent directeur intelligent pour effectuer un suivi de l'état des modules. Par ailleurs, cette invention concerne un procédé d'équilibrage des niveaux de charge dans un groupe d'ordinateurs couplés ensemble selon une configuration en grappe, dans un système informatique en grappe. On vérifie que les niveaux de charge sont compatibles avec les données reçues du groupe d'ordinateurs au niveau de l'agent directeur intelligent. Un autre procédé permet d'anticiper sur les contraintes de l'ordinateur en configurant un ensemble d'ordinateur couplés ensemble selon une configuration en grappe. Cette configuration est exécutée avant la survenue de contraintes. Ce procédé consiste à déterminer à l'avance, à partir du nombre de demande de transactions en cours de traitement par l'ordinateur, un premier ordinateur qui risquerait de subir des contraintes à un moment donné. Ce procédé consiste également à charger un autre module du programme de logiciel dans un second ordinateur parmi l'ensemble d'ordinateurs.
PCT/US2000/017857 1999-06-30 2000-06-28 Architecture a geometrie variable amelioree et procedes pour des applications de commerce electronique dans un systeme informatique en grappe WO2001001221A2 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU57769/00A AU5776900A (en) 1999-06-30 2000-06-28 Improved scalable architecture and methods for e-commerce applications in a clustered computer system

Applications Claiming Priority (10)

Application Number Priority Date Filing Date Title
US34525099A 1999-06-30 1999-06-30
US34426699A 1999-06-30 1999-06-30
US34615599A 1999-06-30 1999-06-30
US09/346,000 US6446218B1 (en) 1999-06-30 1999-06-30 Techniques for maintaining fault tolerance for software programs in a clustered computer system
US09/346,074 US6453468B1 (en) 1999-06-30 1999-06-30 Methods for improving reliability while upgrading software programs in a clustered computer system
US09/346,074 1999-06-30
US09/346,000 1999-06-30
US09/344,266 1999-06-30
US09/346,155 1999-06-30
US09/345,250 1999-06-30

Publications (3)

Publication Number Publication Date
WO2001001221A2 WO2001001221A2 (fr) 2001-01-04
WO2001001221A3 WO2001001221A3 (fr) 2001-10-04
WO2001001221A9 true WO2001001221A9 (fr) 2002-07-25

Family

ID=27541186

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2000/017857 WO2001001221A2 (fr) 1999-06-30 2000-06-28 Architecture a geometrie variable amelioree et procedes pour des applications de commerce electronique dans un systeme informatique en grappe

Country Status (2)

Country Link
AU (1) AU5776900A (fr)
WO (1) WO2001001221A2 (fr)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
SE0200418D0 (sv) * 2002-02-13 2002-02-13 Ericsson Telefon Ab L M A method and apparatus for computer load sharing and data distribution
JP2003256225A (ja) * 2002-03-06 2003-09-10 Mitsubishi Electric Corp コンピュータシステム、障害対応方法及びコンピュータシステムを機能させるためのプログラム
US11303533B2 (en) 2019-07-09 2022-04-12 Cisco Technology, Inc. Self-healing fabrics

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5892900A (en) * 1996-08-30 1999-04-06 Intertrust Technologies Corp. Systems and methods for secure transaction management and electronic rights protection
US5774668A (en) * 1995-06-07 1998-06-30 Microsoft Corporation System for on-line service in which gateway computer uses service map which includes loading condition of servers broadcasted by application servers for load balancing
US6119105A (en) * 1996-06-17 2000-09-12 Verifone, Inc. System, method and article of manufacture for initiation of software distribution from a point of certificate creation utilizing an extensible, flexible architecture

Also Published As

Publication number Publication date
AU5776900A (en) 2001-01-31
WO2001001221A3 (fr) 2001-10-04
WO2001001221A2 (fr) 2001-01-04

Similar Documents

Publication Publication Date Title
US6453468B1 (en) Methods for improving reliability while upgrading software programs in a clustered computer system
US6446218B1 (en) Techniques for maintaining fault tolerance for software programs in a clustered computer system
US11443007B2 (en) System and method for managing network traffic routing
US9755990B2 (en) Automated reconfiguration of shared network resources
US7287179B2 (en) Autonomic failover of grid-based services
US9559938B2 (en) Method, system and apparatus for providing pay-per-use distributed computing resources
US9026655B2 (en) Method and system for load balancing
US20050108703A1 (en) Proactive policy-driven service provisioning framework
US20040054780A1 (en) Dynamic adaptive server provisioning for blade architectures
US7356592B2 (en) Method and apparatus for web farm traffic control
US7543060B2 (en) Service managing apparatus for keeping service quality by automatically allocating servers of light load to heavy task
US8533337B2 (en) Continuous upgrading of computers in a load balanced environment
US7237239B1 (en) Availability and consistent service semantics in a load balanced collection of services running different instances of an application
US20040243709A1 (en) System and method for cluster-sensitive sticky load balancing
US20070112956A1 (en) Resource optimisation component
US20080263401A1 (en) Computer application performance optimization system
US9628556B2 (en) Decentralized request routing
US7840674B1 (en) Routing messages across a network in a manner that ensures that non-idempotent requests are processed
WO2001001284A2 (fr) Formulaires intelligents pour un meilleur deroulement des operations
US20090265704A1 (en) Application Management for Reducing Energy Costs
US7673027B2 (en) Method and apparatus for designing multi-tier systems
US7716238B2 (en) Systems and methods for server management
US7680914B2 (en) Autonomous control apparatus, autonomous control method, and computer product
JP2007249445A (ja) クラスタシステムの負荷分散制御方法およびその装置
CN115698954A (zh) 管理故障转移区域可用性以实施故障转移服务

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CR CU CZ DE DK DM DZ EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG US UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
AK Designated states

Kind code of ref document: A3

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CR CU CZ DE DK DM DZ EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG US UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A3

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

AK Designated states

Kind code of ref document: C2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CR CU CZ DE DK DM DZ EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG US UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: C2

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

COP Corrected version of pamphlet

Free format text: PAGES 1/13-13/13, DRAWINGS, REPLACED BY NEW PAGES 1/13-13/13; DUE TO LATE TRANSMITTAL BY THE RECEIVING OFFICE

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase in:

Ref country code: JP