JP4653490B2 - Clustering system and method having interconnections - Google Patents

Abstract

A system and method is provided for a cluster system. The cluster inlcudes a plurality of nodes operating software instances. The nodes access files on one or more data storage devices over a network. The nodes are connected to each other over an interconnect networl. The interconnect network

Description

FIELD OF THE INVENTION This invention relates to node clustering. The present invention finds particular application to clustering systems and methods involving interconnections between nodes.

BACKGROUND OF THE INVENTION A cluster is a group of independent servers that work together as a single system. The main cluster components are processor nodes, cluster interconnect (dedicated network) and disk subsystem. Clusters share disk access and data management resources, but each separate hardware cluster node does not share memory. Each node has its own dedicated system memory, as well as its own operating system, database instance and application software. A cluster may provide a high degree of fault tolerance and incremental system growth of modules across a single symmetric multiprocessor system. If a subsystem fails, clustering ensures high availability. Redundant hardware components, such as additional nodes, interconnects and shared disks, provide higher availability. Such redundant hardware architecture avoids single points of failure and provides fault tolerance.

  In a database cluster, the CPU and memory requirements for each node can be different for each database application. Performance and cost requirements also vary from database application to database application. One factor that contributes to performance is that each node in a cluster needs to keep its health and configuration informed to other nodes in the cluster. This has been done by periodically broadcasting network messages called heartbeats across the network. Heartbeat signals are typically transmitted over a dedicated network, i.e., a cluster interconnect used for inter-node communication. However, if the heartbeat message is lost or delayed, an erroneous report that the node is not functioning may be made.

  In prior art systems, cluster interconnects have been built by installing a network card at each node, connecting them with appropriate network cables, and configuring a software protocol to span the entire wire. The interconnect is typically a low cost / low speed Ethernet card running TCP / IP or UDP, or a Compaq memory channel running RDG (Reliable DataGram). Alternatively, high cost / high speed proprietary interconnections such as Hewlett-Packard's Hyperfabric / 2 using the Hyper Messaging Protocol (HMP). A low cost / high speed interconnect will lower the cost of clustering for the user and reduce the running latency.

  The present invention provides a new and useful method and system for clustering that addresses the aforementioned problems.

SUMMARY OF THE INVENTION In one embodiment, a cluster is provided that includes one or more data storage devices and a plurality of nodes. Each of the plurality of nodes has data communication access to the one or more data storage devices. The interconnection bus provides an inter-node communication link between the plurality of nodes. Self-monitoring logic detects topology changes in the cluster based on signals on the interconnect bus.

  According to another embodiment, a method for communicating data in a cluster is provided. Here, the cluster includes a plurality of nodes in a communication network including one or more data storage devices, and each of the plurality of nodes controls software instances. The plurality of nodes further communicate with each other via an interconnection bus. A data request message is transmitted directly from the first node to the second node in the plurality of nodes via the interconnection bus. The data selected by direct memory access can be used when the selected data is available and is directly transmitted from the second node to the first node via the interconnection bus. Searched by node.

  Embodiments of the system and method are illustrated in the accompanying drawings, which are incorporated in and constitute a part of the specification, the embodiment being described with reference to the following detailed description and specific examples of the system and method. Help explain. It will be understood that element boundaries (eg, boxes or groups of boxes) shown in the drawings represent examples of boundaries. One skilled in the art will appreciate that an element can be designed as multiple elements, or that multiple elements can be designed as single elements. An element shown as an internal component of another element can be implemented as an external component, and vice versa.

Detailed Description of Illustrated Embodiments The following includes definitions of selected terms used throughout the disclosure. All singular and plural forms of each term are within the meaning of each term.

  As used herein, “computer readable medium” refers to any medium that participates in providing signals, instructions and / or data directly or indirectly to a processor for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media may include, for example, optical disks or magnetic disks. Volatile media can include dynamic memory. Transmission media can include coaxial cables, copper wires, and fiber optic cables. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications. Common forms of computer readable media are, for example, floppy®-disk, flexible disk, hard disk, magnetic tape or any other magnetic medium, CD-ROM, any other optical medium, punch card , Paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, carrier wave / pulse, or other computer readable Including any medium.

  As used herein, “logic” refers to hardware, firmware, software, and / or a combination of each for performing a function or operation and / or for inducing a function or operation from another component. Including but not limited to. For example, based on the desired application or need, the logic may include discrete logic or other programmed logic elements such as software controlled microprocessors, application specific integrated circuits (ASICs). The logic can also be fully embodied as software.

As used herein, a “signal” is one or more electrical signals, analog or digital signals, signal state changes (eg, voltage rise / fall), one or more computer instructions, messages, bits or bitstreams, Or other means capable of receiving, transmitting and / or detecting, but not limited to.

  “Software” as used herein refers to one or more computer-readable and / or computer-readable devices that cause a computer or other electronic element to perform functions, perform operations, and / or operate in a desired manner. Including but not limited to executable instructions. The instructions may be implemented in various forms such as routines, algorithms, modules or programs that contain separate applications or code from dynamically linked libraries. The software may also be implemented in various forms such as independent programs, function calls, servlets, applets, instructions stored in memory, parts of the operating system, or other types of executable instructions. Those skilled in the art will appreciate that the form of software depends on, for example, the requirements of the desired application, the environment in which it is executed, and / or the desires of the designer / programmer.

  FIG. 1 illustrates one embodiment of a simple clustered database system 100 in accordance with one embodiment of the present invention. Although two nodes are shown in this example, node 105 and node 110, a different number of nodes may be used and may be clustered in different configurations. A database cluster is used as an example, but the system is applicable to other types of clustered systems. Each node is a computer system that executes software and processes information. The computer system may be a personal computer, a server or other computing device. Each node may include various components and devices, such as one or more processors 115, operating system 120, memory, data storage devices, data communication buses, and network communication devices. Each node may have a different configuration than the other nodes. An example of a type of clustering system is “Method and Apparatus for Transferring Data from the Cache of One Node to the Cache of Another Node. ")", Which is assigned to the assignee of the present invention, the entirety of which is described in US Pat. No. 6,353,836, which is incorporated herein by reference for all purposes.

  With further reference to FIG. 1, the node 105 is used to describe an example configuration of nodes in the clustered database system 100. In this embodiment, the nodes are networked in a data sharing configuration where each node can access one or more data storage devices 125. Data storage device 125 may hold various files such as database files that may be shared by connected nodes in the cluster. The network controller 130 connects the node 105 to the network 135. The operating system 120 includes a communication interface between a software application executing on the node 105 and the network controller 130. For example, the interface may be a network device driver 140 programmed according to a selected communication protocol of the network 135.

Examples of communication protocols that may be used for network controller 130 and network 135 include Fiber Channel ANSI standard X3.230 and / or SCSI-3 ANSI standard X3.270. The Fiber Channel architecture provides a high-speed interface link for both serial communications and storage I / O. Other embodiments of the network controller 130 include Fast-40 (Ultra-SCSI), Serial Storage Architecture (SSA), IEEE Standard 1394, Asynchronous Transfer Mode (ATM), Scalable Coherent Interface (SCI) IEEE, among others. Other methods of connecting the storage device 125 and the nodes 105, 110 such as embodiments utilizing standards 1596-1992 or some combination of the above may be supported.

  Node 105 further includes a database instance 145 that manages and controls access to data held in one or more storage devices 125. Since each node in the clustered database system 100 executes a database instance, that particular node can access and process the data on the shared database in the storage device 125, so that the lock A manager 150 is provided. The lock manager 150 is an entity that is responsible for authorizing, queuing, and tracking locks on one or more resources, such as a shared database stored in the storage device 125. Before a process can perform an operation on the shared database, the process needs to obtain a lock that gives the process the right to perform the desired operation on the database. To obtain a lock, the process transmits a lock request to the lock manager. A lock manager is executed on one or more nodes in the network to manage the use of resources in the network system.

  A lock is a data structure that indicates that a particular process has been granted certain rights on a resource. There are many types of locks. Some types of locks can be shared by many processes, while other types of locks prevent any other locks from being granted on the same resource. A more detailed description of an example of a lock management system is entitled “Anticipatory Lock Mode Conversions in a Lock Management System” and is assigned to the assignee of the present invention in its entirety. Are found in US Pat. No. 6,405,274 B1, which is incorporated herein by reference for all purposes.

  A cluster configuration file 155 is maintained to track and manage on the network which nodes can access the storage device 125. The cluster configuration file 155 includes a current list of active nodes in the cluster, including identification information such as node addresses, node IDs, and connection structures (eg, adjacent nodes, parent-child nodes). Of course, other types of information may be included in such configuration files and may differ based on the type of network system. When a topology change occurs in the cluster, the node is identified and the cluster configuration file 155 is updated to reflect the current state of the cluster node. Examples of topology changes include when a node is added, when it is removed or when it stops operating.

  Still referring to FIG. 1, the database cluster system 100 further includes an interconnect network 160 that provides inter-node communication between the nodes 105 and 110. Interconnect network 160 comprises a bus that allows all nodes on the network to communicate bi-directionally with each other. Interconnect 160 provides an active communication protocol for exchanging messages and data with each node over the same bus. To be connected to the interconnect network 160, each node includes an interconnect bus controller 165 that may be a peripheral card that plugs into the node's PCI slot. The controller 165 includes one or more connection ports 170 for connecting cables between nodes. Although three connection ports are illustrated in port 170, a different number of ports may be used.

In one embodiment, the interconnect bus controller 165 includes firewire or i. It operates according to the IEEE 1394 protocol, also known as LINK. A bus device driver 175 is provided to allow the database instance 145 or other applications executing on the node 105 to communicate with the interconnect bus 160. Bus device driver 175 operates with operating system 120 to interface with interconnect bus controller 165 and applications. For example, database commands from the database instance 145 are translated by the bus device driver 165 into IEEE 1394 commands or open host controller interface (OHCI) commands. The IEEE 1394 OHCI standard defines standard hardware and software for connecting to the IEEE 1394 bus. OHCI defines standard register addresses and functions, data structures, and direct memory access (DMA) models.

  IEEE 1394 is a bus protocol that is easy to use and provides high-speed communication at low cost. The protocol is highly extensible and includes both asynchronous and isochronous applications, allowing access to a large amount of memory mapped address space and enabling peer-to-peer communication. Those skilled in the art will appreciate that the interconnect bus controller 165 can be modified to accommodate other versions of the IEEE 1394 protocol, such as IEEE 1394a, 1394b, and other future changes and enhancements.

  The IEEE 1394 protocol is a peer-to-peer network with a point-to-point signaling environment. Nodes on bus 160 may have several ports on them, for example port 170. Each of these ports functions as a repeater and retransmits any data packets received by other ports in the node. Each node maintains a node map 180 that tracks the current state of the network topology / configuration. The IEEE 1394 protocol, in its current form, supports up to 63 devices on a single bus, and connecting to devices is as easy as plugging into a telephone plug socket. Nodes and other devices can be connected immediately without first turning off the node or restarting the network. Management of the database cluster topology is described in more detail below.

  Using the interconnect network 160, the database 145 at the node 105 can directly request data, send and receive data, or send messages to a running database application on the node 110 or other nodes in the cluster. This avoids the situation where one or more intermediate steps or additional disk I / O is required, or a message or data packet must be sent to the storage device 125, which increases latency. Is done.

  FIG. 2 shows an example of an interconnect bus controller 165 based on the IEEE 1394 standard. This includes three ISO protocol layers: transaction layer 200, link layer 205 and physical layer 210. The layer may be implemented with logic as defined above, including hardware, software, or both. Transaction layer 200 defines a complete request-response protocol for performing bus transactions using three basic operations: read, write and lock. Link layer 205 is an intermediate level layer that interacts with both transaction layer 200 and physical layer 210 to provide asynchronous and isochronous transfer services for data packets. Components that control data transfer include a data packet transmitter, a data packet receiver, and a clock cycle controller.

The physical layer 210 provides an electrical and mechanical interface between the controller 165 and the cables that form part of the interconnect bus 160. This includes the physical port 170. The physical layer 210 also ensures that all nodes have fair access to the bus using an arbitration mechanism. For example, when a node needs to access the bus, it sends a request to its parent node, which forwards the request to the root node. When the first request received by the route is accepted, all other requests are rejected and canceled. The closer the node is to the root, the more likely it will be accepted. In order to resolve the resulting arbitration unfairness, the period of bus activity is divided into several intervals. Each node is transmitted once during an interval and waits until the next interval. Of course, other methods may be used for arbitration.

  Other functions of the physical layer 210 include data resynchronization, encoding and decoding, bus initialization, and signal level control. As mentioned above, the physical layer of each node also functions as a repeater, translating the point-to-point connection into a virtual broadcast bus. Standard IEEE 1394 cables provide up to 1.5 amps of DC power to keep remote devices "aware" even when they are turned off. The physical layer also allows nodes to transmit data at various rates over a single medium based on IEEE 1394. Nodes or other devices with different data rate capabilities communicate at slower device speeds.

  An interconnect bus controller 165 that operates based on the IEEE 1394 protocol is an active port and includes a self-monitoring / self-configuring serial bus. This is known as hot plug and play, which allows a user to add or remove devices even when the bus is active. In this way, nodes and other devices can be connected and disconnected without interrupting network operation. Self-monitoring / self-configuring logic 215 automatically detects topology changes in the cluster system based on changes in interconnect bus signals. The node bus controller 165 places a bias signal on the interconnect bus 160 when the node is connected to the bus. Adjacent nodes via self-monitoring logic 215 automatically detect bias signals that may appear as voltage changes. Thus, the detected bias signal indicates that a node has been added and / or that the node is still active. Conversely, the absence of a bias signal indicates that the node has been removed or has stopped functioning. In this aspect, the topology change can be detected without using a polling message transmitted between nodes. The self-configuration aspect of logic 215 is described in more detail in connection with FIGS.

  An application program interface (API) layer 220 may be included in the bus controller 165 as an interface to the bus device driver 175. This generally includes higher level system guidelines / interfaces that summarize data, end system design and applications. The API layer 220 may be programmed with desired features to customize communication between the database instance 145 (and other applications) and the interconnect bus controller 165. Optionally, the functionality of API layer 220 may be implemented in whole or in part within transaction layer 200 or bus device driver 175.

Referring to FIG. 3, one embodiment of a database cluster architecture 300 in which the system and method of the present invention can be implemented is shown. The architecture 300 is generally known as a shared disk architecture and is similar to FIG. 1 except that additional nodes are shown. Generally, in a shared disk database architecture, files and / or data are logically shared between nodes and each database instance has access to all data. Shared disk access is achieved, for example, by direct hardware connectivity to one or more storage devices 305 holding files. Optionally, the connection can be performed by using an operating system abstraction layer that provides a single view of all storage devices 305 on all nodes. Nodes A-D are also connected via node interconnect 160 to provide inter-node communication. In a shared disk architecture, a transaction executed on any database instance in the node can directly read or modify any part of the database on the storage device 305. Access is controlled by one or more lock managers as described above.

  Referring to FIG. 4, another embodiment of a cluster architecture that can incorporate the system and method of the present invention is shown. Cluster architecture 400 is typically referred to as a non-shared architecture. An example of a non-shared architecture is entitled “Hybrid Shared Nothing / Shared Disk Database System” and is assigned to the assignee of the present invention in its entirety for all purposes. U.S. Pat. No. 6,321,218, which is incorporated herein by reference. In a pure non-shared architecture, the database file is split between database instances running on, for example, nodes AD. Each database instance or node has ownership of a separate subset of data, and all access to this data is performed exclusively by this “owning” instance. The node is also connected with an interconnect 160.

  For example, if the data file stored in storage devices A-D includes an employee file, the data file controls the employee file for employee names where node A begins with the letters A-G, and node B is the employee Employee files for names H-N are controlled on storage device B, node C controls employee files for names "O-U" on storage device C, and node D is an employee on storage device D The file name “VZ” can be divided to control. A message requesting such data will be sent to access the data from other nodes. For example, if node D wants an employee file controlled by node A, a message requesting a data file will be sent to node A. Node A will then retrieve the data file from storage A and transmit the data to node D. It will be appreciated that the systems and methods of the present invention may be implemented on other cluster architectures and configurations, such as tree structures, and with other data access rights and / or restrictions as desired for a particular application. Let's go.

  FIG. 5 illustrates one embodiment of a methodology associated with the cluster system of FIG. 3 or FIG. This embodiment describes the direct transmission / reception of data between nodes using the interconnection bus 160. The illustrated elements represent “processing blocks” and represent computer software instructions or groups of instructions that cause a computer to perform operations and / or make decisions. Alternatively, processing blocks may represent functions and / or operations performed by functionally equivalent circuits such as digital signal processor circuits or application specific integrated circuits (ASICs). The figures as well as other illustrated figures do not show the syntax of any particular programming language. Rather, the figures illustrate functional information that can be used by those skilled in the art to create circuits and create computer software or a combination of hardware and software to perform the illustrated processing. Since electronic and software applications can include dynamic and flexible processes, the illustrated blocks can be executed in other sequences different from those shown and / or the blocks can be combined or additional It will be understood that it can be divided into various components. They can also be implemented using various programming techniques such as machine language, procedural, object-oriented and / or artificial intelligence techniques. The above applies to all methodologies described in this specification.

Referring to FIG. 5, a diagram 500 is an example of transmitting data between nodes using an inter-node interconnection network 160. If the node (requesting node) desires data from another node, a data request message is transmitted over the interconnect bus 160 to the destination node (block 505). The data request may be sent directly to one or more selected destination nodes by attaching a node name and / or address to the request. If the location of the requested data is unknown, the data request can be broadcast to each node in the interconnect network.

  When the data request is received by the appropriate node, the database instance determines whether the data is available on that node (block 510). If the data is not available, a message that the data is not available is transmitted to the requesting node (block 515). If the data is available, the data is retrieved from local memory by direct memory access (block 520) and transmitted to the requesting node via the interconnect bus (block 525). Remote direct memory access can also be implemented to perform direct memory-to-memory transfers. In this aspect, messages and data can be transmitted directly between nodes without having to transmit the message or data to a shared storage device. Inter-node communication reduces latency and the number of disk I / Os.

  FIG. 6 illustrates an exemplary methodology for reconfiguring a cluster architecture based on the IEEE 1394 bus protocol. When a node in a database cluster is added, removed, or stops functioning, the database cluster must detect the change and identify the node, and the cluster needs to be reconfigured appropriately. As described above, the interconnect bus controller 165 (FIG. 1) that operates based on the IEEE 1394 protocol is an active port and includes a self-configuring serial bus. In this way, nodes and other devices can be connected and disconnected without interrupting network operation.

  For example, when a node is added to a bus, the bus is reset (block 605). The added node's interconnect controller 165 automatically transmits a bias signal on the bus, and the adjacent node may detect the bias signal (block 610). Similarly, it can be detected that there is no node bias signal when the node is removed. That is, the adjacent node interconnect controller 165 may detect changes in signals on the interconnect bus 160, such as changes in bus signal strength caused by adding or removing nodes. The topology change is then transmitted to all other nodes in the database cluster. A bus node map is rebuilt in response to the change (block 615). In one embodiment, the node map may be updated in response to the change. The database instance is notified, which updates the cluster configuration file to keep track of active nodes for the lock manager (block 620). Of course, the sequence order shown may be implemented in other ways.

  Using the IEEE 1394 protocol, the interconnect controller 165 becomes an active port that includes the self-monitoring / self-configuration mechanism described above. When using this mechanism, the database cluster system can be reconfigured without the additional latency associated with the polling mechanism. This is because the node can detect changes in the topology substantially immediately. Active ports also allow cluster reconfiguration without having to power down the network.

FIG. 7 shows another embodiment for detecting and reconfiguring clusters. Each node monitors the interconnect bus to detect changes in the bus signal, such as the presence or absence of a bias signal (block 705). When a node detects a topology change (block 710), the node sends a bus reset signal over the bus and initiates a self-configuration mechanism. This mechanism managed by the physical layer 210 may include three phases: bus initialization, tree identification, and self identification. Active nodes are identified during bus initialization and a tree-like logical topology is built (block 715). Each active node is assigned an address, a root node is dynamically assigned, and the node map is rebuilt or updated with a new topology (
Block 720). Once the bus itself is configured, the node can access the bus. Once the database instance on each node is notified of the topology change (block 725), the database lock manager is reconfigured in response to the change so that the shared database can be properly managed across the cluster (block 730). .

  It will be appreciated that a network connection such as network 135 may be implemented in other ways. For example, this may include communication or networking software such as software available from Novell, Microsoft, Artisoft and other vendors, including TCP / IP, SPX, IPX, and twisted pairs, Can operate using other protocols over coaxial or fiber optic cables, telephone lines, satellites, microwave repeaters, radio frequency signals, modulated AC power lines, and / or other data transmission lines known to those skilled in the art . Network 135 may be connectable to other networks through a gateway or similar mechanism. It will also be appreciated that the protocol of the interconnect bus 160 may include a wireless version.

  Referring to FIG. 8, one embodiment of a heartbeat system for database cluster 800 is shown. A heartbeat system is a mechanism by which nodes periodically generate signals or messages that indicate that they are active and functioning. The mechanism also allows a node to determine the health state or state of other nodes in the cluster based on the generated signal. As shown, cluster 800 includes nodes 805 and 810, although any number of nodes may be connected to the cluster. The illustrated node may have a configuration similar to that shown in FIG. However, a simplified configuration is shown for illustrative purposes.

  Nodes 805 and 810 share access to a storage device 815 that holds files such as database files. The node is connected to the storage device 815 by the shared storage network 820. In one embodiment, network 820 is based on the IEEE 1394 communication protocol. To communicate with each other, the nodes 805, 810 and the storage device 815 include an IEEE 1394 network controller 825. Network controller 825 is similar to interconnect bus controller 165, and in one embodiment is a network card that plugs into each device. Alternatively, the controller can be fixed in the node. The network controller 825 includes one or more ports so that cables can be connected between each device. In addition, other types of network connections may be used, for example wireless connections based on the IEEE 1394 protocol or other similar protocol standards.

  With further reference to FIG. 8, each node includes a database instance 830 that controls access to files on storage device 815. As resources are shared between nodes in the database cluster 800, each node includes logic to inform other nodes of their health status and logic to determine the health status of other nodes on the network. . For example, the heartbeat logic 835 is programmed to generate and transmit heartbeat messages within a predetermined time interval. The heartbeat message is also called a status signal. The predetermined time interval may be any selected interval, but is typically on the order of a few milliseconds to a few seconds, for example on the order of 300 milliseconds to 5 seconds. Thus, if the interval is 1 second, each node will transmit a heartbeat message every 1 second.

In one embodiment, network load is used as a factor in determining the heartbeat time interval. For example, if heartbeat messages are transmitted as data on the same network, the frequency of heartbeat messages on the network can cause a delay in the data transmission process. FIG. 8 shows a network that may be affected by this situation, and FIG. 9 shows a network that reduces the amount of network traffic by implementing a heartbeat system on a different network. It will be further appreciated that the networks of FIGS. 8 and 9 may also be configured as a non-shared architecture.

  Still referring to FIG. 8, heartbeat messages from each node are collected and stored in a constant file 840. In this example, the constant file 840 is one or more files or areas defined within the storage device 815, which also holds shared files. Each node in the cluster 800 is assigned an address space in the constant file 840 and the heartbeat message is stored there. The space of the constant file 840 is typically equally divided and assigned to each node, although other configurations may be possible. Thus, the constant file 840 can be implemented as a separate file for each node rather than as a single file for the entire cluster, even though the file can be logically defined as a single data structure. A constant file may be implemented as a stack, array, table, linked list, text file, or other type of data structure stored in one or more storage locations, registers or other types of storage areas. When a node's constant space is full, when a new message is received, the oldest message in that space is pushed or overwritten.

  FIG. 9 shows another embodiment of a database cluster 900 and a heartbeat system. In this example, nodes 905 and 910 communicate with constant device 915 via constant network 920. The constant network 920 is a separate network from the shared storage network 925. Thus, the node accesses the shared file on the storage device 930 using a network bus different from the constant network. The constant network 920 can be part of an inter-node interconnection network as described above. The quorum device 915 includes data storage configured to hold a quorum file for storing heartbeat messages received from nodes in the cluster.

  Still referring to FIG. 9, nodes 905, 910 are connected to a constant device 915 and communicate with each other according to the IEEE 1394 communication protocol. Each node and constant device 915 includes an IEEE 1394 controller 935 that is similar to the controller described above. Since a separate network is configured for data communication with the file, each node includes a separate shared network controller 940 that communicates with the storage device 930. The shared network controller 940 can be an IEEE 1394 controller or other network protocol such as a Fiber Channel protocol. A database instance 945 within each node processes data requests via the shared network controller 940.

  The heartbeat logic 950 controls the heartbeat mechanism and communicates with the constant device 915 using the IEEE 1394 controller 935. With this architecture, adding or replacing the quorum device 915 within the existing database cluster 900 can be easily performed with minimal impact on the existing network. Also, since the heartbeat mechanism is processed via a separate network, reducing traffic on the shared storage network 925 allows for a quicker response to data processing requests. It will also be appreciated that the cluster of FIGS. 8 and 9 may include an inter-node interconnect network.

FIG. 10 illustrates an example methodology 1000 of a heartbeat system that runs on a constant file 840 or constant device 915, both referred to below as constant files. Once the quorum file is configured and activated in the database cluster, memory in the quorum file is allocated to each of the nodes in the cluster (block 1005). The constant file can be equally divided and further assigned to each node or other assignments can be defined. When the constant file becomes active, the constant file receives a heartbeat message from each node according to the IEEE 1394 protocol (block 1010). Each heartbeat message includes a node identifier, which identifies the node that transmits the message and a timestamp indicating the time of the message. Each message received by the constant file is then stored in the assigned location of that node (block 1015) and the process is repeated for each received heartbeat message.

  For each node, heartbeat messages are stored in a constant file in the order in which they are received. Thus, by comparing the last received timestamp with the current time, the system can determine which nodes are actively sending their heartbeat messages. This information may indicate whether the node is active. For example, it may be assumed that a problem may occur if a node misses a predetermined number of consecutive time stamps. Any number of messages may be stored for each node that contains one message. As described above, the heartbeat logic of each node is programmed to generate and transmit heartbeat messages at predetermined intervals. Thus, by reading data from the constant file, the logic can determine whether some intervals have been missed. This type of status check logic may be part of the heartbeat logic 835 or 950 and is described in more detail in connection with FIG.

  FIG. 11 shows an example methodology for determining the health or state of a node. As described above, the heartbeat logic includes logic for generating each heartbeat message at a predetermined time interval and transmitting the message to a constant file. At any desired time, the node's heartbeat logic updates its cluster configuration file to determine the current set of active nodes and either node has stopped functioning or is otherwise removed from the network. You can decide whether or not It is also possible to synchronize this decision across the entire cluster. Status check logic (not shown) can be programmed as part of the heartbeat logic to perform this task as follows.

  To initiate a status check, a constant file is read to examine time stamped information for each of the nodes (block 1105). Based on the time stamped data stored for each node, the logic may determine whether the particular node is still functioning based on the time of the last message written to the constants file (block 1110). By setting a threshold, it is possible to miss a predetermined number of time stamps before the decision indicates that a problem may exist. For example, a node may miss two consecutive time stamps, but if the third stamp is missed, the node may not function properly. The threshold value may be set to another value, for example, a value of 1.

If a node misses a specified amount of timestamp message (block 1120), it may not necessarily mean that the node has stopped functioning. Since the nodes are connected to the constant file according to the IEEE 1394 standard, additional status checks can be performed. As explained above, the IEEE 1394 bus is active, and each device connected to the bus can detect whether an adjacent node has stopped functioning or has been removed from the network. This additional information can help to better determine the health status of the node. The state logic may compare the time stamp information from the constant file with the node map data held by the IEEE 1394 controller.

  For example, if a node misses its timestamp (block 1120) and the node is not an active node in the node map (block 1125), it is assumed that the node is down or has been removed from the network. (Block 1130). However, if a node misses its timestamp but is still active in the node map, it may possibly hang up or some other delay may exist in the cluster (block 1135). ). In this case, the process optionally rechecks the quorum file for that node to see if a new timestamp has been received, a message indicating possible delays can be generated, and / or the node is active It can be determined whether it can be removed from the list of nodes.

  Referring back to decision block 1120, if a node does not miss its timestamp, the node is probably functioning properly. However, additional decisions may be made by checking whether the node is active in the node map (block 1140). If the node is active (block 1145), the node is functioning properly. If the node is not active (block 1150), there may be a network bus error. Thus, using information from both the constant file and the IEEE 1394 bus node map, a more detailed analysis of the health of the node can be determined. Further, in the cluster configuration shown in FIG. 9 in an embodiment where the shared storage network 925 is also an IEEE 1394 bus, two separate network node maps are maintained. Additional node maps may also be included in the comparison process and status check described above.

  Referring back to FIG. 11, a simplified embodiment can be realized. At decision block 1120, if a node fails to write its timestamp, the logic may declare that the node is not functioning and remove it from the cluster configuration file of the database instance. In this process, the node map is not examined.

  It will be appreciated that the various storage devices described in this specification include a constant device for assigning constant files and can be implemented in a number of ways. For example, the storage device may include one or more dedicated storage devices such as a magnetic disk drive or optical disk drive, a tape drive, electronic memory, and the like. A storage device may also include a computer, server, portable processing device, or similar device including a storage, memory or combination thereof for holding data. The storage device may also be any computer readable medium.

  Suitable software for implementing the various components of the system and method of the present invention includes the teachings and programming languages and tools presented herein, such as Java, Pascal, C ++, C, CGI, Perl, It is readily provided by those skilled in the art using SQL, API, SDK, assembly, firmware, microcode and / or other languages and tools. Components embodied as software include computer readable / executable instructions that cause a computer to operate in a predetermined fashion. The software may be a product and / or stored on a computer readable medium as defined above.

While this invention has been illustrated by describing its embodiments, which have been described in considerable detail, applicant's intent is to limit the scope of the appended claims to such details or to It is not limited by the method. Additional advantages and modifications will be readily apparent to those skilled in the art. The invention in its broader aspects is therefore not limited to the specific details, representative apparatus and specific examples shown and described. Accordingly, departures may be made from such details without departing from the spirit or scope of applicants' general inventive concept.

1 is a system diagram showing an embodiment of a cluster node according to the present invention. FIG. It is an example figure which shows the interconnection bus controller of FIG. It is a figure which shows an example of a shared disk cluster architecture. It is a figure which shows an example of a non-shared cluster architecture. FIG. 6 illustrates an example methodology for communicating data using an interconnect bus. It is a figure which shows an example of the methodology which detects a topology change. It is a figure which shows another example of the methodology which detects a topology change. It is a figure which shows another Example of the cluster containing a heartbeat system. It is a figure which shows another Example of a heartbeat system. It is a figure which shows an example of the methodology which hold | maintains a constant file. It is a figure which shows an example of the methodology which determines the state of a node using a constant file.

Claims (15)

A cluster,
One or more data storage devices;
A plurality of nodes each having data communication access with the one or more data storage devices;
An interconnection bus for providing a serial inter-node communication link using the IEEE 1394 protocol between the plurality of nodes ;
Status check means for checking the status of the plurality of nodes;
Constant storage means associated with the state check means ,
Each of the plurality of nodes is
An interconnect bus controller connected to one end of the interconnect bus;
A node map defining a node topology of the plurality of nodes in the cluster ;
Heartbeat logic for transmitting a heartbeat message within a predetermined time interval , wherein the heartbeat message includes a node identifier indicating a corresponding node;
The interconnect bus controller is configured to transmit a bias signal when a corresponding node is connected to the interconnect bus;
The interconnect bus controller includes self-monitoring logic based on IEEE 1394 for detecting topology changes in the cluster;
The self-monitoring logic is
Detecting addition or removal of other adjacent nodes based on a change in the bias signal on the interconnect bus;
Notifying other nodes of topology changes according to the addition or removal of other adjacent nodes,
Update the corresponding node map based on the topology change ;
The constant storage means holds the heartbeat message transmitted from each of the plurality of nodes in association with a node identifier included in the heartbeat message and a time stamp of the heartbeat message,
The state check means is based on whether a predetermined number of consecutive time stamps have been overlooked and whether each of the plurality of nodes is active in the inter-node communication link. A cluster that checks the state of each node .   The cluster of claim 1, wherein the change in the bias signal on the interconnect bus includes a change in signal strength.   The cluster according to claim 1, wherein the bias signal is a non-polling signal.   The cluster according to claim 1, wherein the inter-node communication link provides direct memory access of data between the plurality of nodes.   The cluster according to any one of claims 1 to 4, wherein the inter-node communication link provides an asynchronous message passing between the plurality of nodes.   The cluster according to claim 1, wherein each of the plurality of nodes is one computer.   The cluster according to claim 1, wherein the plurality of nodes are serially connected via the inter-node communication link. Each of the plurality of nodes is
A processor;
An application instance for managing and controlling access to data held in the data storage device executed by the processor;
The cluster according to claim 1, further comprising: a device driver that translates a command from the application instance into a command used on the interconnection bus.   9. A cluster as claimed in any preceding claim, wherein the one or more data storage devices are directly accessible by each of the plurality of nodes.   The cluster according to claim 1, wherein the one or more data storage devices are accessible by a node selected from the plurality of nodes. The plurality of nodes includes one or more database instances;
The cluster according to claim 1, wherein at least one of the one or more data storage devices is configured as a database.   The cluster according to claim 11, wherein the interconnect bus controller transmits data read from the database to a request node via the interconnect bus in response to a data request from another node. A method for communicating between a first computer as a first node and a second computer as a second node, wherein between the first computer and the second computer, The first and second computers are connected via a first network, and each of the first and second computers has a node map that defines a node topology of a plurality of nodes including the first and second nodes. ,
Said first and second computers transmitting data between said first node and said second node via said first network, said data using an IEEE 1394 based protocol Transferred via the first network, the first network being a serial communication link between nodes,
The first computer transmitting data between the first node and one or more data storage devices via a second network;
Said second computer transmitting data between said second node and said one or more data storage devices via said second network, said second network comprising said first network Separate from the network, wherein the first and second computers transmit a bias signal when connected to the first network;
The first or second computer detecting addition or removal of a first or second node based on a change in the bias signal on the first network;
The first or second computer updating the corresponding node map in response to the detected change ;
Transmitting a heartbeat message within a predetermined time interval;
Holding the heartbeat message transmitted from each of the plurality of nodes in association with a node identifier included in the heartbeat message and a time stamp of the heartbeat message;
Based on whether a predetermined number of consecutive time stamps have been overlooked and whether each of the plurality of nodes is active on the inter-node communication link, the state of each of the plurality of nodes is determined. And a step of checking .   14. The method of claim 13, further comprising: the first and second computers asynchronously transmitting data between the first node and the second node via the first network. Way to communicate.   The step of the first computer transmitting data read from the corresponding data storage device to the second node via the first network in response to a data request from the second node. 15. A method for communicating according to claim 13 or 14, comprising.

Images