JPH10105522A - Multicomputer system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve high usability, calculating processing and I/O performance in a cluster environment by preparing distributed streams and correcting a STREAMS framework so as to lead these distributed streams into various nodes inside the cluster. SOLUTION: The basic architecture of a STREAMS stack is a stream head 30 and a driver 44 having a module 32 of zero or more than it pushed at the upper part of the driver, and a control sled 34 and a physical cluster mutual connecting driver (P-ICS) 36 are provided at a minimum. The PICS 36 provides a high-speed mutual connecting link to use a light protocol of a little waiting time containing both the components of software and hardware. These mutual connections depend on a virtual line and the virtual line has ability for transmissively moving an instance from a certain application to the other one.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to the interoperation between networked computers, and more particularly to a distributed operating system used for a cluster of computer nodes.

[0002]

BACKGROUND OF THE INVENTION A node refers to a computer comprising one or more processors, local memory, input / output subsystems, and optional peripherals, which are potentially SMP or (Sy
mmetrical MultiProcessor) is a symmetric multiprocessor). A cluster is a set of two or more nodes that cooperate at some level to solve a problem. The nodes in a cluster interconnect and use different software and hardware solutions depending on price / performance requirements. Depending on the implementation, such as shared memory or message passing, the cluster
High availability, distributed applications do not stop even if a single point fails, level utilization of application load, effective use of cluster-wide resources, hardware sharing, increase of total computation and I / O bandwidth, mass storage and networking And other functions to increase access to resources.

[0003] STREAMS is a relatively lightweight message passing framework for implementing networking protocols and client / server applications, and has become the de facto industry standard. Prior art STREAMS mechanisms are based on UNIX Sys
tem V Network Programming, SA Rago (1993), Ch. 3 &
It is described in Section 9. For example, NFS, UDP / IP, TCP / IP, S
PX, NetBIOS, SNA and DLPI are all STREAMS
The framework is used to define network protocols implemented by many vendors.
Applications based on these protocols and services, if successful in clustering, can benefit from the above benefits provided by the cluster.

[0004]

Accordingly, there is a need to understand the requirements of the cluster infrastructure and to modify the STREAMS framework so that applications and cluster functions can be performed in the STREAMS framework environment.

[0005]

SUMMARY OF THE INVENTION It is an object of the present invention to provide high availability, improved computing and I / O performance, and single point management in a multi-computer environment (ie, a cluster environment in general). In order to solve the problem of large-scale users such as, for example, it is an object of the present invention to enable the operation of STREAMS in such an environment.
In accordance with the present invention, the STREAMS framework is modified so that distributed streams can be created and these distributed streams can be introduced to nodes in the cluster.

Another object of the present invention is that all of the work of the present invention, in both user and kernel space,
It is transparent to the user application. To this end, the STREAMS framework is modified so that software does not need to be aware of the existence of STREAMS in a clustered environment.

As a means for solving the above problems, the present invention provides a multi-computer system having a distributed STREAMS function. The multi-computer system includes a computer including one or more system processor units, at least two nodes each having a local memory and an input / output subsystem, interconnected via a data communication interconnect subsystem. A STREAMS messaging mechanism running on each of the system processor devices in the cluster and used in implementing one or more of the networking protocols, client / server applications and services. An operating system, including a software application running on at least one of the node's system processor units under the control of the operating system for performing tasks or solving problems Means for creating a distributed STREAMS instance independent of the application and the STREAMS messaging mechanism on a first initiating node of the plurality of nodes, a networking protocol on the first node, a client / Distributed STR to the second target node to perform selected tasks of a software application on a second target node of the plurality of nodes transparently to server applications and services
Means for migrating at least a part of the EAMS instance is provided.

[0008]

DETAILED DESCRIPTION OF THE INVENTION I. Overview of Embodiments A cluster is a set of computers of any size interconnected via high speed communication links. The cluster itself is connected to a non-cluster node via standard network links such as FDDI, ATM or 100VG hardware links and drivers installed on a subset of the cluster nodes that provide the gateway service.
It is also possible to connect to nodes. These standard networking links can use standard networking protocols such as TCP / IP for communication, while the cluster interconnect preferably uses a custom lightweight protocol.

A node is a computer composed of one or more processor units, local memory, input / output subsystems and, optionally, peripheral devices such as bulk devices, and is used to solve problems. . A distributed stream (hereinafter simply referred to as a distributed stream) is a stream divided between two nodes at a point designated in advance. this is,
Generally, it takes place between the module and the driver. For example, the TCP / IP protocol is TCP or UDP
Are on one node and the IP is on the other node. The rules for determining the split are generally based on the content, location, and performance considerations of the state information to be maintained. In the following description, TCP / IP is referred to as 1 for ease of understanding and generality of implementation.
Use as an example.

A distributed STREAMS is a STREAMS having any of the following features / attributes. * The entire application and STREAM stack can reside on the same node as in a non-cluster environment, but leverage the STREAMS-related framework infrastructure to implement cluster functions such as load leveling, high availability, etc. You can enjoy the benefits. S
The TREAM stack can run on a different node than the node running the application accessing it. * STREAM can be divided at the module / driver / streamhead level into multiple components, each running on a different individual node within the cluster. * Each of the end points of a STREAMS-type pipe can run on different nodes within the cluster. * The STREAM stack can migrate, in whole or in part, from one node to another.

The present invention relates to a control thread, a preview feature set, a P-ICS driver and preferably an S-IC.
Implement distributed STREAMS in a cluster that uses a system component called the S-driver. A thread is a software processing element that provides a set of functionality through the execution of application instructions on a computer. The control thread is a special thread that acts as a third party communication and management point for distributed STREAM (functional details are described below). S-ICS
The driver is a STREAMS software interconnect driver that serves to move messages between any of the components of the application's STREAMS stack, control thread, P-ICS and preview function. P-ICS is a physical cluster interconnect that provides the basic communication network. The preview function
STREAMS Dynamic Function Replac
ement) function, which is enabled through the STREAMS function, and has the function of examining messages to multiplex messages between various components in the system.

FIG. 6 illustrates a distributed stream, an interconnect driver, and a separate middleware software thread. The middleware included in this figure is a distributed STR
It is important to note that while making EAMS feasible, the process described is completely independent of middleware implementation. This is a great advantage, as it allows to work with multiple middleware suppliers as long as the communication interface is standardized.

As shown in FIG. 6, the stream is split on at least two nodes and utilizes middleware running on any one of the two nodes or on a third node. The interconnect driver consists of an intelligent STREAMS type driver (S-ICS) and (non-ST
REAMS is further divided into non-intelligent physical drivers (S-ICS) that can be reused for inter-node communication. Intelligent STREAMS type driver (S-ICS) is a middleware and STREAMS
Includes a protocol for communicating with kernel daemons that handle requests / responses.

The basic stream creation algorithm for creating the configuration shown in FIG. 6 is as follows. This algorithm utilizes the new in-kernel STREAMS interface. Device via normal open ()
Open a file. Since the file system is viewed as a single system from the application's point of view, no application changes are required, but it is necessary to determine the correct node to perform the open.

When the file system performs an open, VFS (virtual file system) is the final call to the STREAMS framework open routine. Use the framework's autopush and configuration functions to open the interconnect driver transparently instead of opening the target driver (in this case, the IP). This involves remapping the interconnect driver device number to a target device number stored for later use (remappi
nng). Kernel remapping actually occurs when a driver is introduced into the system during a system boot. The STREAMS framework needs to be aware that the driver has been configured for clustering. Next, STREAMS performs the opening of the interconnect driver as it normally does for any driver. At this point, there is only one stream head and one driver instance on the node where the application currently resides.

The next step is to perform an open on the node where the actual driver resides. To do this, the stored device number is re-associated with the node address. STREAMS type interconnect driver
Communicate with the middleware via the appropriate protocol to determine the target remote node and all appropriate communication information. This communication is either configured as part of the interconnect driver's open routine or is initiated through a transparent ioctl that tells the driver to investigate and take the necessary action. The choice is a matter of implementation, but it should be noted.

The interconnect driver then composes a message to be sent to the target node at the known address. Within this node is an in-kernel daemon or control thread that listens for such requests and reacts to the arrival of messages. This daemon interprets the request and (1)
Choose to process the request itself, (2) perform thread creation to handle the request, or (3) hand off the request to an existing thread that already multiplexes multiple target driver instances. And run.

At this point, the thread that handles the request
Perform an in-kernel streams_open () for the target node using the original device number retrieved from the application open request. streams_open () opens the IP driver as if the IP driver were on the same node as the application.
Once the IP is opened, the stream head
The address is returned to the initiating node's interconnect driver,
Using this address in combination with the address of the remote node creates a unique addressing tuple within the cluster without having to create another independent naming scheme. This tuple is used later for stream migration and verification. At this point, the application must be able to return from open () and continue processing without modification. It will be appreciated that the above process does not require any modifications to the user application, TCP module or IP driver. This is an important feature for successful cluster configuration.

Stream migration deals with migrating one or both of the two split distributed streams in a cluster to a new node. The implementation of the migration must be transparent to the application and must not require changes to the components of the stream stack. Using the same example as in the case of creating a distributed STREAMS configuration described above, and giving two points that lead to a failure, the first point is the node where the IP driver exists. If this node fails for any reason, the connection currently running via this node should not fail and should not be aware of the failure. Recognizing that the interconnect driver has lost connection for that node, it initiates a recovery protocol using the stream creation process described above.

With respect to the upper half of the stream, it is often necessary to migrate applications for various reasons, such as load leveling or system maintenance. Because the upper half cannot process the information, which can cause timing and data corruption issues, the upper half stream will remain until the transition is complete and the lower half no longer sends incoming data to the upper half. This type of migration is relatively difficult because and applications must be balanced.

Prior art embodiments involving TCP / IP partitioning on different nodes rely on modification of the actual relevant modules, thus increasing performance, supportability, maintenance costs, increased development time, increased costs, etc. Affects directly. This is an unnecessary load, as it limits the modules and drivers that can be easily executed within the cluster. To remove this load, the following process is provided.

The process of migrating TCP (used as an example because it is applicable to any module) in a distributed STREAMS takes advantage of the key properties of the STREAMS framework combined with new ioctls, The TCP module is migrated transparently without realizing that a migration has occurred and without requiring changes to the module itself. This process requires one function developed by the module / driver developer. This function is used to associate data structures and messages with the structure pointed to by q-> q_ptr. The actual details of this function are not required at this time, but this is a necessary step in moving the stream transparently.

The basic stream transition is as follows.
One independent thread within node A or the application itself must be designed to transition itself or a connection from one node to another, but is deciphered by the STREAMS framework where the transition should occur Issues a new ioctl. This ioctl contains all the information needed to move the stream to the new node.

The framework provides the interconnect driver with io
Issue a ctl to notify the remote driver processing thread to perform the migration and consider the migration to be flow controlled. Next, the framework uses local
Freeze the stack and execute the simplification function on the q_ptr structure of each module, which is a module private data structure. This simplification function returns all the information needed to duplicate this structure and associated memory on the target node.

Once that information has been collected, the framework sends it along with all the framework specific information such as stream head status and messages to the daemon on the target node. Then the daemon
Instantly create a stream stack or wait for the process to be re-established, then rebuild the stream according to the open nodes. For example, this is NFS
If you have an in-kernel application like
The stream can be completely reconstructed immediately and the NFS process will be notified of its creation and re-initialization. If this is a user space socket application, the process waits for the migration to complete before performing the rebuild. Once the stream stack is rebuilt, the new interconnect driver instance notifies the remote node that it can now receive the message, and communication begins anew. All of these are transparent to the applications, modules and drivers involved.

Similarly, if the node on which the IP driver runs fails, the user application must continue operation without modification. To do so, the interconnect driver must recognize that a failure has occurred and re-establish its connection on the new node via the open process described above.

The above process provides the following advantages in a cluster environment: The stream is split and distributed among multiple nodes without requiring changes to the drivers or modules involved. In the past, without significant modifications to these components, stream distribution that would add complexity and cost effectiveness was not possible. Unmodified means shorter development time, reduced support and problem resolution time, and reduced test time. Furthermore, because well-designed STREAMS drivers and modules work together in a manner similar to that on a single node, developers do not need to learn how a cluster environment works. This means that users of the present invention can publish the cluster environment to third party independent software suppliers more quickly and at lower cost than prior art systems.

Streams can be migrated between nodes without the need for modifications to drivers or modules, thus removing complexity and performance loss from the current mechanism in use. The requirements for a migration are that the migration is a normal STRE for the driver or module.
It is limited to being a single independent function in the execution of AMS framework communication. The system utilizes the P-ICS driver without requiring modifications to the control thread or the underlying technical implementation in the S-ICS.
The present technology provides a set of common data structures and interfaces that achieve P-ICS independence. This allows the present technology to transparently enjoy the benefits of future P-ICS improvements. A STREAMS-type implementation of a protocol or application stack can be incorporated, implemented, and executed transparently, ie, without modification, within a cluster.

The basic STREAMS framework is D
DI / DKI (ie device driver interface / driver kernel interface Dev)
Performed without any modifications to the iceDriver Interface / Driver Kerneru Interface) routine. Furthermore, all S
TREAMS framework messages, commands,
Record / Manage Driver, System Call, Dynamic Function Replacement
(Dynamic Function Replacement), Dynamic Function Registration (Dynam
ic Function Registration) and stream head operation all operate as in a non-distributed environment. This allows the stack to be designed using the usual single-system paradigm, and the designer understands and takes into account the cluster configuration, except for the cleanup function (if supported). No need to develop. Distributed STREAMS uses VFS (virtual file
It can be implemented without modification of the VFS to ensure portability to other operating systems that implement STREAMS under (System).

The design of the present invention does not depend on a protocol. This allows the design to be used for various networks or other applications, meaning that no implementation modifications are required. Some of the preview features may require minor modifications, but such modifications are primarily related to solving the problem. For example, the IP connectivity preview feature, which is specific to the global port management solution, is not required to implement basic TCP / IP using the techniques of the present invention.

The design of the present invention can take advantage of a variety of different P-ICS driver embodiments without requiring a design change in the S-ICS driver or control thread.
Thus, the design of the present invention makes it possible to utilize a new technology or communication path paradigm of shifting from receiver-based communication to sender-based communication which offers a number of simplifications and performance improvements.

This design shows how all aspects of the STREAMS framework can be achieved within a distributed environment. This includes command, logging (strlog), management (S
AD), pipes, system calls, DDI routines, etc. In most cases, the STREAMS framework does not require much modification because the controlling thread, S-ICS driver and some preview functions perform most of the work. This allows for rapid integration of the present technology into STREAMS embodiments other than Hewlett-Packard. In addition, the S-ICS driver and control thread are independent of the implementation of STREAMS, except that they require dynamic function replacement, so their designs can be ported to other vendors' platforms. There must be.

There is no need to modify the design to allow STREAMS-type modules and drivers to operate within a distributed environment. This is because Sun and Loc
us system seems to require extensive modifications to allow protocol modules like TCP, UDP and IP to work,
This is an important point. This is where the design of the present invention differs from any previously proposed. The design focuses on changes in the various components that exist outside of the modules and drivers supplied by the application, thereby speeding up the module or driver import and support and increasing cost efficiency.

The present design also eliminates the need to modify the VFS (Virtual File System) to perform a distributed driver open. This design proposes a unique and protocol independent open and migrate algorithm.

II. Detailed Description of Embodiments 1.0 Overview 1.1 Cluster Setup Example The cluster shown in FIG. 1 provides a unique cluster solution that cannot be easily duplicated using the prior art distributed application techniques. 2 is an example of a cluster based on a combination of a number of new technologies to be formed and a conventional technology. Remote client applications can use this cluster to access distributed databases used to implement highly available data warehouses. The client application can utilize a global web browser such as Netscape to create a query request and verify the results. Browsers are unique to traditional client software in that they have a consistent interface and operating standards that make it easier for remote users to learn and execute to validate data.

The number of nodes used to implement the various parts of this solution in a cluster depends on the size of the data warehouse and the number of clients accessing the data simultaneously. For example, a small data warehouse with hundreds of gigabytes of data can be implemented using four to eight nodes for the data warehouse. On the other hand, a large data warehouse consisting of terabytes of data can serve as an Internet gateway if there are a large number of remote clients and a considerable number of nodes must serve as database servers. It will be implemented with 16 to 32 nodes, including nodes. Each Internet gateway node has 4
Two 622 Megabit ATM links can be used to communicate with remote clients. All nodes are
Interconnected using a cluster interconnect solution consisting of a combination of hardware, software and firmware technologies.

The example shown in FIG. 1 utilizes a cluster 20 of four small nodes, including nodes A, B, C and D. Nodes A and B implement a distributed database providing a data warehouse, and nodes C and D
Serves as an Internet gateway and query engine that provides access to remote clients. The actual technology used to implement the interconnect is independent of the distributed STREAMS,
There are, of course, preferred techniques that further simplify the design and implementation of AMS. This independence is a design advantage as new technologies can be incorporated transparently,
A lower cost solution with a relatively long service life as an application protocol is realized, and various cluster services can be built on the basis of distributed STREAMS.

Assume the following for the cluster: * Clusters are interconnected via software / hardware on which STREAMS and stream-type drivers run. * The operating system instance on each node must be multi-threaded and multi-processor capable. * Clusters may also have a middleware framework against which STREAMS seeks or supplies information. Such communication takes place via a suitably designed interface. A middleware framework, such as a controlling thread, is needed when making decisions about the various operations that STREAMS performs. Note: Although the STREAMS design itself is independent of the middleware, the actual extent to which the design embodiment utilizes this middleware is highly dependent on the embodiment. This is to be recalled in understanding what the control thread is and how the control thread is used to perform some of the middleware tasks depending on the embodiment.

A distributed stream is a STREAM that can operate in a clustered environment, for example, in the following manner. * The entire application and stream stack may be able to reside on the same nodes as in a non-cluster environment, but by utilizing the STREAMS-related cluster framework infrastructure, load leveling, high availability, etc. It is possible to enjoy the advantages of the cluster mechanism as described above. * The stream stack can run on a different node than the node running the application accessing it. * At the module / driver / streamhead level, a stream may be divided into multiple components, each potentially running on an individual node within the cluster. * For stream-type pipes, both ends of the pipe may be located on separate nodes in the cluster.

The important point is that the stream is simply decoupled from the application accessing it in a way that would not normally exist if the stream were running on a single node. This separation occurs for a number of reasons described below. The stream is distributed for (but not limited to) any of the following reasons: To take advantage of the features of the clustered mechanism, applications have a cluster 20, which appears as a single system, performing and managing them within a single cluster, with high availability, under load average and hardware sharing. Have to have sex.
Side effects of distribution are reduced CPU overhead and higher overall cluster performance. These concepts will be described in detail in the following sections using examples of clusters.

1.2 Hardware Sharing The cluster 20 implemented according to the present invention allows for significant hardware sharing. Hardware sharing offers many advantages over prior art system configurations. Each node can be measured in terms of processing power, memory, I / O backplane, etc., thus increasing hardware choice and flexibility for the task being performed. In other words, a cluster need not be a set of homogeneous nodes. In the example of the cluster of FIG. 1, the configuration of the gateway nodes C and D is a 4-way SMP, a 32-bit processor, two SCSI 4 GB hard disks, and 25
A 6 MB memory, two 622 Mb ATM network cards, and two interconnect cards if HA is a problem, but one interconnect card. The database server configuration is, for example, a 4-way SMP, a 64-bit processor with a large address space, a Fiber Channel connected disk array with terabyte capacity disks, and two interconnect cards.

The purchase / maintenance cost is lower than the service life of the cluster. In example cluster 20, only gateway nodes C and D need to have a 622 Mb ATM card installed. This reduces the number of cards and reduces fixed card costs over the life of the cluster, which reduces costs both in the short and long term, and also reduces the number of communication lines that must be installed. Let it. Communication lines that can support 622 Mb ATM cards are very expensive, and each line has repeated use and maintenance costs that may be difficult to amortize over the life of the cluster.

The security is improved by dividing the cluster. The three main parts in this example, the remote client, the gateway and the database server all run on different nodes. Such an embodiment facilitates both physical and software security.

Cluster subdivision also reduces the number and type of failure points. For nodes A and B, the points of failure are limited to mass storage, database software, distributed applications and cluster interconnects. For nodes C and D, the points of failure are mass storage (although on a very small scale if these nodes do not require large capacity), gateway hardware, networking protocol stacks, query engines, Limited to Internet software and cluster interconnect. In each case, the reduction in failure points
The solution of the present invention provides a single point of database or gateway management, so the field of application of Murphy's law is small, fault isolation is improved, debugging is easier, and cluster complexity is reduced. This has the effect of reducing management and support costs.

Also, within the cluster environment of this embodiment, hardware sharing improves overall cluster performance and efficiency by keeping the TCP / IP stack running only on the gateway node. This is achieved by using the standard RPC (Remote Procedure Call) function delivery paradigm to send only the results of the remote execution to each node. In a conventional distributed client / server application, each node typically runs a TCP / IP stack and then communicates with remote clients using packets sent to the gateway node. The problem with the traditional approach is that it takes thousands of cycles per packet to perform all of the work associated with creating, processing, and routing packets through the gateway. It is. The consumption of such a large number of cycles per packet is not optimal given the number of possible remote clients and the amount of data circulated,
This leads to a decrease in cluster processing performance. Because the cycles spent on networking are not consumed on the database server, the user does not achieve maximum system efficiency for the price paid. In addition, the instruction cache requires frequent switching between database applications and networking code, resulting in frequent cache misses that can degrade processor performance. Implementing a network protocol over a cluster interconnect increases packet latency by increasing collisions between system resources (such as memory, processors, I / O buses, timers, etc.) and the interconnect I / O card. This leads to reduced overall cluster performance for distributed databases and increased response time for remote applications. Note: Latency is considered a key in building high performance clusters, so low latency is generally interpreted as meaning faster response time and higher throughput / performance.

If the interconnect solution is TCP or UD
If P had to support all of the transmission protocol capabilities required by it, it would be too complex, resulting in poor processing performance and high design costs. Instead of building a client / server application between the database application on node A and the remote Internet application, the present invention
By using EAMS type TCP / IP, TC
Applications that require a P / IP connection to exist only at node C are distributed and transparently accessible from applications on node A using the standard RPC function delivery paradigm. Function delivery basically sends the execution of a function from one location to another, so that, for example, the cluster in this embodiment is a node C
And D can be offloaded from nodes A and B for all of the expensive TCP / IP processing cycles. This should be interpreted as an increase in system performance for nodes A and B. Details of how such points are achieved within the STREAMS framework are described below.

1.3 Load Averaging Load averaging is used to smooth the workload between nodes in a cluster. In cluster example 20,
Load averaging is used in two areas: database servers and Internet gateways.
For database server nodes, if the database is correctly distributed, the probability of requesting each server node must be the same. Unfortunately,
Probability does not necessarily reflect reality. If requests create "hot spots" in the database, cluster nodes are not used equally and overall cluster performance may be degraded. Similarly, an unbalanced number of remote clients are
However, if the cluster arrives via node C instead, the cluster response time to client queries may increase. In such a situation, the solution of the present invention introduces a load averaging policy (i.e., policy) and migrates the application, application instance, data, connection, or any combination thereof to the lightly loaded node Let it. The following example:
It shows how such a transition can be performed using a STREAMS-type pipe.

STREAMS type pipes have at least 2
A bidirectional queue that can be used to send and receive data between two execution coordination threads. Practical examples range from as simple as command line processing to as complex as multiple clients communicating with a server application using a pipe endpoint. Applications use STREAMS-type pipes instead of memory-type pipes when additional filtering or pre-processing of data is required. Memory-based pipes cannot do this by pushing the filtering module to the center of the pipe.

Within the scope of the example cluster 20, a database server application can consist of a set of cooperating threads that communicate via pipes. When a cluster becomes unbalanced,
Migrate some database components or some database server instances to other nodes within the cluster. In a distributed environment of the prior art, this approach is to use different communication paradigms such as sockets, or to understand how to switch to a new paradigm when making the transition.
It means you have to redesign your application. Such a design change is unacceptable to many users because it is expensive and reduces the number of applications that can take advantage of clustering. To solve this problem, in embodiments of the present invention, the application can continue to use the pipe. (STREAM-type pipes can transparently replace memory-type pipes, even if additional functionality is not available). When an imbalance is detected, the pipe endpoint or the entire pipe, and the associated execution thread, are migrated to a new node within the cluster. All of this transition takes place without having to modify even a single instruction line in the application.

1.4 High Availability High availability is a commonly used expression to describe software or hardware solutions that perform fault-tolerant operations without necessarily providing all of the complexity or cost of a fault-tolerant system. Extending STREAMS to allow distribution or migration as described above provides the ability for software and hardware within the cluster to recover from most single points of failure. For example, in FIG. 1, if a remote Internet application is communicating with an application on Node B via a Node D gateway, and the Node D gateway fails, a node that transparently handles all application activity The recovery mechanism on C can be triggered. Similarly, if the application includes synchronization points on Node A and Node B, then if either node fails, the remote client application runs unnoticed that recovery is taking place. To continue. That is, the application continues to operate normally without detecting that a problem has occurred. Note: Using distributed STREAMS,
Recovery can be performed from some single point of failure, where the component is primarily a stateless entity such as DLPI. For state maintenance components such as TCP, distributed STREAMS cannot be used to recover from a single point of failure in the event of a node failure. In such a case, the remote client application must detect that the connection has been lost and perform a restart. This point may be tolerated by users now, but will not be tolerated in the future. Presumably, distributed STREAMS will utilize the Brevix study, a project of the Hewlett-Packard Company laboratory, and its prototype solution. Brevix is not just a system disruption,
Defines a mechanism to set up specific traps to invoke error handling routines. Such an error handling routine is executed by the client application and the ST.
It could be used to migrate a REAMS instance to another node in the cluster. As this work is extended to include more traps, the use of the "panic" function will be replaced by an evaluation routine that determines whether it is safe to migrate the application before the system goes down. If the distributed STRE
AMS will be able to provide an unparalleled high availability solution in the clustered market.

[0051] Brevix is limited in the number of system traps it can handle. The trap is
Trap 15 for data segment failure and trap 18 for handling memory protection and unaligned memory references
Usually limited to 26, 27 and 28. The advantage of Brevix is that it can be performed on a subsystem-by-subsystem basis, so that the recovery mechanism can rely on the recovery mechanism to determine whether it is safe to migrate components that depend on that subsystem. Will be able to judge. Many kernel subsystems call the function panic () when they cannot advance or when something wrong is detected. If panic () were changed from just having a message string to having a policy parameter, recovery and migration would be performed with more intelligence. This policy parameter will include suspicion of the entire subsystem and consideration of the impact on other subsystems and recovery steps.

1.5. Single System Viewpoint Because the cluster 20 is composed of multiple nodes, the cluster is inherently complex and potentially difficult to manage on a node-by-node basis. However, many problems are much easier to solve if the cluster is viewed as a single system, at least from an application perspective. For example,
If an application uses TCP / IP to tie itself to a particular port, and then wants the application to migrate to another node for some reason, another port for that port on the target migration node You must ensure that no applications are already connected. Otherwise, no migration can be performed. A solution to this problem is to use cluster middleware to manage the port space of the entire cluster, thereby preventing potential problems.
The distributed STREAMS design of the present invention provides techniques, algorithms and STREAMS framework modifications that eliminate the need to modify existing modules or drivers, allowing applications to run transparently on any combination of nodes without modification To be able to

1.6 DLKM and Single System Perspective Problem The operating system uses a dynamically loadable kernel module (hereinafter referred to as a dynamically loadable kernel module).
(indicated by the acronym modules), each node within the cluster can self-configure to only load and execute the modules for the currently running application. it can. In a distributed STREAMS environment, this is achieved by having a flat file (accessible for each node) that describes how to load the STREAMS subsystem. Thereby, STREAMS can bring in the necessary modules and drivers and load and configure STREAMS-type subsystems such as TCP / IP for the application being executed. From a single system perspective, there must be a mechanism by which each node can obtain not only configuration data for the entire cluster but also node-specific configuration data.

For the DLKM to work, each node must include a flat file that contains information on the list of nodes from which the STREAMS subsystem can be loaded. The STREAMS initialization function allows this list to be processed and communicated via an existing connection management protocol, which allows the initiating node to determine which nodes can download the subsystem and to determine the actual download Can be executed.

The above flat file also has a STR
Includes a default STREAMS driver that must be loaded whenever EAMS is loaded and a parameter set for loading the driver. minimum,
The clone, strlog and sad drivers must always be loaded. These may not be needed until the application attempts to open the drivers, but almost any STREAMS-type driver will use them directly or indirectly, so load them during initialization. It is easier and faster. The file also contains a list of STREAMS-type subsystems that may need to be loaded and their corresponding server nodes. The basic idea is to pre-package enough information so that these subsystems do not need their own flat file. Modify str_install () to read this information via communication with the server node,
By having the current data structure downloaded,
This works. The difference will be that str_install () does not actually load the subsystem. str_inst
all () allows subsystem loading to be faster and with less information (i.e. in a new flat file)
It will only create the required STREAMS substructure as done. Therefore, every event is S
Since it occurs via TREAMS, there is no need for the STREAMS subsystem to have to create a new load mechanism. In addition, since the environment is a cluster, loading subsystems can maintain HA quality for it.

1.7 Design Issues The present invention provides a solution to a number of design issues associated with distributed STREAMS and STREAMS-type applications. To understand how the design of the present invention works and the reasons for the design of the present invention,
The following is a list of design issues.

From the viewpoint of a single system of a cluster,
There is only one set of device files that the application can open. The problem to be solved is how to determine which node is the correct node to open the device since the potential corresponding target hardware or software may not be present on all nodes. In addition, how to determine the correct node based on a device file name that does not convey such information? Such a situation is illustrated in the hardware sharing example above.

If different parts of the stream-type stack had to operate on different nodes, how would these components be created and interconnected without requiring component design changes? Do you? This is an important goal for creating an open cluster environment that encourages developers to port their modules and drivers to a cluster.

If the stream should transition from one node to another, for load averaging, high availability, or whatever the reason, how to maintain state and correctness without modifying the component design. Do you make the transition? (Note: For some cluster mechanisms,
Each component may require additional functionality to take advantage of these features, while others have default behavior and do not require modification or additional code). Feasible STREAM Asynchronous with Transition Effort
You need to pay attention to the Sput and service routines. Strl, a DDI / DKI standard utility
For modules or drivers that use og (),
How do administrators send messages to the single cluster log driver on which they run strace? Also, how do you distinguish between logging for one node and logging for another? Also, how do you do that without losing information or modifying a module or driver? (HP, Sun,
Many STREAMS embodiments (such as OSF, Mentat, Unixware) utilize many different queuing synchronization levels. These synchronization levels are key in ensuring proper queue access and operation.
How and how to provide those capabilities within a cluster?

When creating a stream-type multiplexer, how to interconnect components when unnecessary overhead and performance degradation occur? What if one half of the multiplexer migrates to the new node but the other half cannot? There are a number of other problems besides the above,
Before EAMS can be an effective solution within a cluster, it provides a basis for understanding what must be solved.

1.8 Design Rules There are three design rules to consider when evaluating a distributed STREAMS design. The first rule states that any modifications to the STREAMS framework will require a non-distributed STREMS
It should not cause performance degradation for AMS applications. Developers tend to add features here and there to get the satisfaction of some customers. Typically, a single mechanism does not make a significant sacrifice in terms of performance degradation, but over time, the mechanism continues to expand, leading to severe overall performance degradation such that the accumulation of performance degradation affects the entire customer. Sometimes. Thus, adding distributed STREAMS functionality should not be designed to add performance degradation as experienced by the entire customer base. The temptation to spend cycle time here and there for convenience must be repelled as much as possible.

The second rule is that no STREAMS module or driver should require modification to be able to operate within a distributed STREAMS environment. This means that third-party developers can develop their software on the target platform,
It is key to reducing the overall time and cost of developing, deploying and supporting applications for a cluster environment. In the past, cluster software developers have performed extreme acts, such as modifying the main path of code to add a check to see if cluster-specific operations must be performed immediately. This type of modification is not only unacceptable in terms of performance and cost, but also limits cluster usage and reduces the appeal to customers. The only potential exception to this rule is that a module or driver may need additional functionality to migrate an instance from one node to another. This functionality does not require modification of existing code, but adds the functionality that the STREAMS framework uses to migrate the private data of modules / drivers that STREAMS normally remains unaware of. The module / driver functionality is S
TREAMS Dynamic Function Replacement and Reg
It can also be increased using istration. This is the subject of U.S. patent application Ser.No. 08 / 545,561 and U.S. Pat.
08 / 593,313.

The third rule is to make the distributed STREAMS solution as independent of middleware as possible. The design must limit as much as possible the points and situations where communication with the middleware occurs. This independence from middleware allows for a more flexible design, as well as a faster transition to new middleware technologies with lower costs in terms of time to market, time to entry and overall product support.

2.0 Design Overview The basic architecture of the STREAMS stack shown in FIGS. 2 and 3 is a streamhead 30 and driver 44 with zero or more modules 32 pushed on top of the driver. Although shown and accessed as a single stack instance in this figure, more complex stacks can be created using software multiplexers that have the ability to combine multiple stream stacks into a tree-like structure. With this in mind, the details of the distributed STREAMS design are described below.

The design of the present invention includes at a minimum a control thread 34 and a physical cluster interconnect driver (P-ICS) 36. In addition, there may be one or more instances of a STREAMS-type driver called Software Cluster Interconnect Driver (S-ICS) 38. Before describing the details of these components,
We examine two potential design configurations and how they can be used to solve various cluster problems. Two configurations are shown in FIGS.

P-ICS provides a high-speed interconnect link that uses a lightweight, low-latency protocol that includes both software and hardware components. These interconnects are virtual circuits for all intents and purposes. Virtual circuits provide a number of conceptual benefits as well as implementation benefits, but at least have the ability to move instances from one to another transparently to the application. In addition, the virtual circuit eliminates the need to maintain protocol specific information within the stream or physical interconnect driver, or within the STREAMS framework.
This can simplify overall design and implementation, improve performance, and increase code reuse and flexibility. It should be noted that the simple architecture shown in FIGS.
It is not meant to limit the number of MSs or cluster mechanisms available for each stack to those shown.

The first configuration shown in FIG.
X having a plurality of DLPI ATM drivers 40, 42 coupled under an X (ie, IP multiplexer) 44
Includes CP / IP stack. That is, this is the STRE
This is a standard configuration found in AMS type TCP / IP embodiments. In addition, there are three other components.
The first two are the control thread 34 and the P-ICS 36
It is. Third is a set of preview functions 31, 33 that can increase TCP and IP operations based on which cluster mechanism this stack uses. TCP
For, this can be a cluster-wide port management scheme, while for IP, it can be a high availability set of functions that handle error and card fault conditions. In any case, this is the control thread and the P-IC
Optional functionality that requires access to S. Note: Since all stream-type stacks are known to the controlling thread when they are created, all stacks must at least be able to migrate between nodes without modifying the non-clustered embodiment. It must be noted that the stack is required to provide a pair of functions for each component that has associated private data.

In the second configuration of FIG. 3, a TCP stack is shown having a plurality of DLPI ATM drivers linked below an IP multiplexer, in which case one DLPI instance 42 is actually a different node Running on To enable this operation, the S-ICS linked below the IP multiplexer
Driver 38 is responsible for sending data from IP to remote DLPI instance 42. This stack may be created this way, or for example, DLPI
If a detects a card failure and sends an M_ERROR message, this stack may be created as a result of the high availability recovery scheme. The message is caught by the IP extension function, notified to the controlling thread, and a recovery policy routine is invoked. In this situation,
The application will not notice the card failure and will continue to operate as before. How this is achieved will be described later.

[0069] In both configurations, the middleware entity is an optional component and will not be specifically described here (middleware will be described later). In these configurations, the controlling thread uses this T
It has enough information to coordinate different mechanisms and recovery mechanisms across all nodes running CP / IP. This coordination operation will be described in detail when a mutual exchange between components is described in a later section.

2.1 Overview of the Control Kernel Thread In order to transparently establish and migrate streams in the cluster, the first to communicate with the STREAMS framework, other control thread instances, and the cluster-wide management middleware. The existence of a three-party entity must be allowed. This is achieved by creating one or more kernel management threads.
These threads have at least the following responsibilities / abilities: Assuming that the stream is normally created as shown in the first configuration, the controlling thread will detect its creation and notify the STREAMS framework of it so that the STREAMS framework can perform the function increment. Take responsibility. The stack notifies the STREAMS framework using str_install (). Command st
r_install () is used to install a STREAMS driver or module into the kernel. The parameters passed to this function define items such as sync level, stream tab entry, etc. For clustering, a new parameter containing the policy set to be activated is added. Since even non-cluster ports of a driver or module require recompilation, the driver or module automatically retrieves the default set of policies without changing the embodiment. The default policy allows stack migration for load averaging and high availability.

When a stream is being created with components residing on different nodes within the cluster, the controlling thread is responsible for creating components on remote nodes. Control thread is remote DLP
Responsible for creating I instances and local stream heads. If modules need to be pushed to this driver instance, the controlling thread is also responsible for pushing those modules.

The controlling thread communicates with the stream instance on the local node by using the ST in the kernel.
Use the REAMS interface. STREAM
The preferred form of the S interface is
Packard HP-UX Releases 10.10 and 1
0.20. The reason for keeping the controlling thread in the kernel is performance and security. The in-kernel interface does not need to go through the system call interface excessively to communicate with its partner. As for security, it maintains a routing table and administrative structure protected from other threads and snooping. Does this mean that control threads cannot be realized in user space? Since the answer is not, and the problem is the ability to send the message, not where the message passes, implementation in user space does not require design changes for this solution.

The control thread is the S-ICS for the node.
Responsible for establishing instances. There may be only one instance for all control threads, in which case they either share the stream head pointer address, or each thread can share information held in the S-ICS. It may correspond to the functionality supported by each thread. The controlling thread is responsible for passing streamhead specific messages.

The control thread is responsible for passing messages between components and performing the necessary processing flow control. The controlling thread is involved in the stream transition. If a node fails and some of the stream instance components were running on that node, the controlling thread participates in error recovery related to the high availability solution. The controlling thread communicates with the middleware across the cluster as one of two locations in the cluster that has in-depth knowledge of the middleware and how to communicate with it, seeking information and changing what it manages. Middleware can be updated. By restricting this information, STREA
The MS framework maintains middleware independence, thereby increasing the flexibility of solution design for new middleware technologies. Details of how all of these things are achieved are described in the next section.

2.2 Outline of S-ICS The S-ICS 38 is a key to making the STREAMS framework design solution independent of middleware.
Component. The S-ICS is a stream type software driver that exists on the physical interconnect driver P-ICS36. Embodiments of the present invention rely on P-ICS implementation specific dependencies for easy integration into the STREAMS framework and because this driver provides some unique and standard mechanisms to simplify stream migration and recovery. STREAM
The S-ICS driver was selected as stream type (ie, stream based) so that it could be isolated to a single point from the S framework point of view.

At a minimum, the S-ICS 38 has the following capabilities /
Take responsibility. * S-ICS is responsible for creating distributed stream components. * S-ICS examines middleware and checks low level STR
For EAMS framework and application specific information, if the information is not stored in the controlling thread, it is responsible for updating the middleware. * S-ICS is P-ICS, STRE if necessary
Handles messages between the AMS framework, control management thread and middleware. * S-ICS improves processing performance and provides distributed STREAM
To simplify module and driver integration within the S environment, application-specific cache information is potentially maintained. These caches may also contain information such as routing tables for packets arriving at nodes that may not have a running target application.

2.3 Example of Component Operation To illustrate how all of the above components work together, take the most common area where distributed STREAMS applies, namely TCP / IP, as an example. consider. This example is shown in FIG. 4 as two alternative configurations. The way in which the stream is initially created is described in a later section.
The discussion will focus on how the stream works and what design solutions are used to address the above three design rules.

2.3.1 Non-split Stack TCP / IP In the first configuration of FIG.
0, 32, and 44 are typically created from a stack perspective, but with additional processing. Upon opening, the control thread 34 is notified that a stack is being created. When the STREAMS framework's open () code is executed, the controlling thread recognizes that the stream head address and driver have been opened and that the modules have been pushed. The controlling thread then examines each driver / module to determine if there is an associated cluster policy set. Next, the control thread 3 gives the stack appropriate functionality according to those policies, so that the stack can utilize the mechanism provided in the cluster.

Once enabled, the controlling thread only intervenes or reacts according to what is sent to it and the policies set. Therefore, in many cases TC
The P / IP stack operates as in a non-cluster environment, and the performance of the application looks as if it were. Clearly, this configuration meets all of the design rules.

2.3.2 Split Stack When the TCP / IP stack is distributed, there are two different configurations, each with a trade-off. The first option splits the stack between TCP and IP modules. The advantage of this split is that when the stack needs to be migrated to a new node because the TCP maintains state and the IP does not, no IP update is required and only the TCP needs to be migrated, thus the migration Work load is reduced. Furthermore, if the node running the IP fails, a new IP instance can easily be established on another node in the cluster without affecting the TCP module or application. For stream type TCP, there is a TCP default queue, such as a connection indicator. The default queue may require that additional code be added to ensure that a connection indicator is sent to the correct target node. Such a situation would occur if the application was migrated to a new node. If the stack were to be split at this level, the code needed to perform this default queuing would be TCP at least in terms of deciding where to send the message in the cluster.
It could also be reused in the case of / IP processing.

The disadvantages are as follows. This option is
Not independent of protocol embodiment. In most cases, superficial modularity appears to exist, but it does not. When this partition is used, the design must take into account the special form of communication between these two modules and the prejudice regarding private data and access to it. In other words, this option is T (far from essential)
Requires in-depth knowledge of CP / IP stack implementation. This design requires that the S-ICS be almost completely protocol dependent, instead of having to be completely 95% or more protocol independent.

Many of the S-ICS cannot be reused for other transmission providers. This is because the clustered TCP / IP stack and other protocols
This means that the cost of developing and maintaining the stack increases unacceptably. In order to route a message to its destination, there must be a mechanism that can contact the cluster middleware for route updates. Since TCP and IP are located in the middle of the stream stack, they are not allowed to pause,
Requests to potentially different nodes can cause significant problems with the execution of the stack, especially if packet processing still occurs and is located on the interrupt control stack. Partitioning at this level means that many distributed stack instances in the cluster, one for each connection
Derive one instance.

The second option is to set the stack to IP / DLP
It is divided at the I level, and both the TCP and IP modules are constantly maintained on each node. This form is shown in FIG.
(B). The advantages of this design are as follows. The design is more than 95% protocol independent. Stream-type TCP is used by TPI (ie,
It is enough to understand that it supports the interface Transport Provider Interface). This TPI uses SNA, OSI, Netware, A using DLPI connected under the module / driver of the transmission provider.
It is supported by various transmission providers such as ppletalk. Many S-ICS can be reused in the case of other protocol stacks and are potentially sufficiently generalized to be completely protocol independent. This reduces solution costs, increases flexibility, and reduces time to market for new technologies in a transmission stack that is populated in a cluster environment.

According to an embodiment, the number of distributed stacks decreases with splitting at the IP / DLPI level. If IP is implemented as a multiplexer, it has only one DLPI instance per I / O card. If splitting is done at this level, each remote DLPI is linked under the IP and there is no limit on the number of connections through this single link (actually the S-ICS is a linked driver but correct, predictable) Which DL on the remote node to be able to perform message routing
Record enough information to understand if you are communicating with the PI instance). Division is TCP / IP or U
Assuming done at the DP / IP level, there is one interconnect per TCP connection and this setting is based on IP / DLPI
Occurs more frequently within a cluster than partitioning at the level, but degrades cluster performance and increases migration costs.
Because the DLPI is stateless, the DLPI portion does not need to be migrated if a stack transition is desired.

If the DLPI instance fails, the TCP / IP part does not have to be migrated to a new DLPI instance. If the card is local to the stack, the new DLPI instance simply informs the new or existing DLPI that it has the following additional IP address assigned to it. DL
If the PI is a remote instance, the fault handling uses the same process, but in this case has the S-ICS used as its DLPI instance. This can be achieved by linking all DLPI instances within a cluster for each node, or the configuration code can select which DLPI to use as a fault-tolerant component.
This depends on the embodiment.

The disadvantages are as follows. S-ICS requires more development effort and time for protocol independence, but this effort is offset by the ability to reuse the same design for different stacks. If the default TCP queue requires a routing processing module, this module needs to be created, albeit at a higher cost. Of course, this code is S-ICS
Being a subset of the code, the design could fully incorporate the S-ICS code and keep the cost within an acceptable range. Most of the 5% dependence that cannot be ruled out is in this regard.

According to the examination of the advantages and disadvantages described above,
The use of the second option in FIG. 4B is preferred. In the next section, we will see how one problem related to the single system perspective of the cluster, global port mapping, is solved using the second alternative, and the solution is transferred to other protocol stacks. It describes how it is applied.

Note: In FIG. 4, three numbers 1, 2, and 3 are shown as symbols. These figures are each associated with the following notes regarding potential performance optimizations that may be used if precautionary measures are added. 1. Assuming a cluster with multiple versions of the second option in FIG. 4B, adding the possibility that an application accessing the TCP / IP stack part can migrate from node to node. if you can,
Before all intra-cluster routing is updated to reflect the new location of the stack, the possibility exists that the packet can be routed to the node. In such a case, the above point is recognized when the IP determines that the correct TCP or UDP instance is not on this node using its fanout table. In such a case, the packet is re-routed via the S-ICS and sent to the new node. Since the IP will have finished processing the packet by that time, if we just send the packet back to the correct node and back to the IP, there is no reason to repeat this process and the impact / recovery of the process present in the stack There is no reason to worry about Thus, packets can be successfully sent directly to a specific TCP or UDP instance on the new node, depending on the characteristics provided by the control cluster. As mentioned above, this also eliminates IP reprocessing problems and improves overall stack performance. The details of this rerouting will be described later, but in short, the fan-out table has a
Contains the UDP target queue set. This is accomplished by changing the TCP or UDP target queue (read queue) to be an S-ICS write queue, which when putnext () is executed,
Activate the S-ICSput routine instead of the DPput routine. This eliminates the need for message scanning or IP code modification, but requires an understanding of this basic IP implementation design and the generation of code that can modify this fan-out table accurately. It is needed (this is the main point where you really need to understand how the stream-based IP is implemented and why this design can achieve 95% protocol independence). 2. As mentioned above, the first option is to use S to improve performance.
-Empower direct messaging between ICS and IP. This again requires implementation dependent knowledge.
In addition, the first option requires an embodiment that handles the possibility that the S-ICS decides to add additional preventive code for a remote node failure. 3. The message is sent to S-ICS and DL
Performance can be improved by pushing directly between PIs. The same considerations apply as before. Either way, the fact that the message does not need to pass through the stream head and there is no need for a potential context switch to the controlling kernel thread produces significant performance gains. Either emphasis on performance enhancement or emphasis on complexity and the addition of proactive code is chosen depending on the implementation for each protocol.

FIG. 5 shows how packets are forwarded when the second option of FIG. 3 is implemented. The controlling thread 34 accesses the local S-ICS 30 and the local stack instances 30, 32, 44 by using a hash function that determines a stream head address to store in its local data structure 46. The local stack instance also
It has a pointer 48 to the management S-ICS instance 38, which can be used to direct messages to the S-ICS instance without going through the controlling thread. Similarly, the S-ICS does not need to access the control thread to send data to the DLPI driver 42.

2.3.3 Global Port Correspondence In order to create the above configuration, global port correlation is performed in some cases. This function controls the assignment of TCP ports throughout the cluster, rather than just on a node-by-node basis. One way to do this is to simply divide the port space by port space size per node / number of nodes in a cluster port. Although simple to implement, it does not provide enough flexibility or adapt to the functions that each node in the cluster may be performing.

A more suitable solution is a list showing which ports are active, which ports are available, and where the ports are actually used in the cluster, as shown in FIG. Cluster that holds
This is to create a middleware control thread 50.
This approach allows an application to move to any node within the cluster, while still allowing the number of ports per node to fluctuate as needed, yet without worrying about port duplication Make it possible. Note: FIG. 6 illustrates the operation of both non-split and split stacks. A TCP or UDP instance can run on the local DLPI link 40 or the remote DLPI link 42, but the algorithm is essentially the same. TCP /
For an IP cluster implementation, it may be best to link all DLPI instances under each IP during cluster initialization. This allows for faster recovery from a card failure, and the S-ICS on each node automatically knows how to transmit large amounts of packets between nodes when a transition occurs, thereby reducing the S-ICS.
-Eliminates the need for ICS updates and eliminates the need for path failure recovery.

The following settings are made for each node in the cluster. The middleware may reside on separate nodes, but if its data cache does not have enough information, as shown in FIG.
CS may consult middleware for packet routing. Of course, S-ICS always maintains the freedom to simply discard packets. If the S-ICS routes to another node, the packet will be placed directly on the TCP or UDP read queue and will not pass through IP again.

The packet arriving at the IP flows as follows. The packet arrives at the IP lower MUX (multiplexer). The IP examines the packet and uses the IP fanout table 52 to determine which queue to send it to. Fanout table is <port, q
ueue> There is nothing other than the mapping function. When a packet is bound to a local TCP or UDP instance, IP is directly putnext (tcp_queue, mp) or putnext
Execute (udp_queue, mp). If the packet has an entry in the fanout table rather than for the local endpoint, the packet is putnext (S_
Execute ICS_0 queue, mp). If there is no fanout table entry, the IP will either drop the packet or
Otherwise, the packet may be forwarded to the S-ICS lower MUX if the IP transmission function is enabled. The S-ICS examines the packet for embedded routing information and determines a route by referring to its data cache. If S-ICS is MUX, which S-ICS
Use the knowledge of whether the ICS instance sent the data to find the path.

Essentially for this solution, there are three components: an extended scan function set, a control thread that manages this effort, and a middleware thread. However, this middleware thread may actually be the controlling thread, as the controlling thread is involved in almost every administrative aspect. The only disadvantage is that the controlling thread may not be protocol independent as desired, and design a middleware thread specific to the protocol stack involved, while keeping the controlling thread completely protocol independent. Things are easier and faster. Regardless of how this is actually implemented, the pertinent functionality and implementation outlines are described below.

1. When a transmission endpoint, or TCP or UDP endpoint in this example, is created, the STREAMS Dynamic Func described in US patent application Ser. No. 08 / 545,561.
Use the tionReplacement (dynamic function replacement) feature to put, service, and
Extend the close routine (details of this will be described later).
These functions are extended to proactively check the messages transmitted to and from the stack. put and servi
For the ce routine, T for the write queue function
T_BIND_REQ and T_UNBIND_REQ messages for PI and T_BIND_ACK, T_UNBIND for write queue function
M_P which may include _ACK and T_ERROR_ACK messages
Targets TROTO messages. As for the close routine, if it has been called and the endpoint has not been released, some work needs to be done. These functions look for specific message types or activities and take action if conditions are met. STREAMS Dynamic instead of modules pushed onto the stack
Using Function Replacement
(This is an alternative to using this technique) eliminates an extra set of put routines (read and write) that are invoked for every message, resulting in improved processing performance. Note: All other messages are processed immediately using the original put or service routine, so only the function call overhead is the only performance penalty.

2. A control thread, the same thread as described above, selectively cooperates with middleware and extended functions. Continuing with the same example as above, the endpoint is open and the function is extended. Extended wr when application sends T_BIND_REQ TPI message to TCP / UDP module
The itput routine detects this occurrence and tentatively redirects the message to the controlling thread. After the message has been redirected, processing continues. Returned to the application
Since the application request is not really completed until either T_BIND_ACK or T_ERROR_ACK is received, processing can be continued. This is an asynchronous event and all TPI embodiments and transport providers support this operation.

3. The controlling thread retrieves the binding information, in this case the relevant port information, and looks up middleware threads that may be running on different nodes within the cluster. 4. The middleware thread checks the binding information and determines whether the specified port is available. If the specified port is not available, return an error to the controlling thread and the controlling thread takes the appropriate action. If the port is available, mark the port as "in use" and notify the remaining components of the cluster, the rest of the controlling thread, that the port is in use and the node it is using.

5. The control thread of each node issues a binding request to the IP. For this connection, the fan-out table <port, read queue address> is extended to
The ad queue address (read queue address) is actually an S-ICS write queue address. The S-ICS is also notified of the binding and updates its data cache to reflect this new routing information. Thus, each node has an IP instance containing the bound port address, which is used to route packets arriving at the node where the application is not currently running (potentially application migration or With recovery, a DLPI component failure occurs at some point and a new instance is started on another node).

6. When the cluster completes the join operation on each node, the middleware thread notifies the control thread of success or failure. If successful, the controlling thread uses the in-kernel STREAMS interface to write the original binding message to the write queue of the transmission module (ie, TCP or UDP). This module and its IP instance perform the necessary binding operations and either T_BIND_ACK or T_ERROR_AC
Generate K. When T_ERROR_ACK occurs, the controlling thread is notified that the binding operation was being processed, so the controlling thread is notified that the binding has failed, and the controlling thread notifies the middleware thread of the fact that the Unbind all other nodes for the port. If the connection continues, the stream stack will operate normally regardless of whether the components are distributed or not. That is, the techniques described above can be applied to streams that have been successfully created and whose components are running on the same node.

7. The application then closes the stream instance or issues an unbind command. In each case, the extended routine detects it and takes action. If the application closes the stream stack, the close routine sends a message to the controlling thread, which notifies the local S-ICS instance to clean up its cache. Furthermore,
The controlling thread also notifies the middleware thread, which notifies all other nodes in the cluster and issues a de-coupling instruction to the relevant IP instance. This instruction causes the S-ICS instance at each node to clean up its data cache. If the application issues a disassociation request instead of a close request, the message is as if the binding instruction waited for the controlling thread on all nodes to execute the disassociation instruction before the extension function continued. The above algorithm is the same except that processing is suspended. Table 1 below is a sample of a bind preview code.

[0101]

[Table 1] tpi_bind_w_preview (q, mp) {union T_primitives * tpi; if (mp> b_datap-> db_type! = M_PROTO) (* q-> q_qinfo-> qi_putp) (q, mp); else {tpi = (union cast it) mp-> b_rptr; if (tpi-> PRIM_type == T_BIND_REQ || tpi-> PRIM_type == T_UNBIND_REQ) {/ * Send message to control thread * / route_msg (tpi_ct_endpoint, mp, write_queue);} else (* q-> q_qinfo-> qi_putp) (q, mp);}}

Within the control thread, a plurality of communication points are monitored, one of which is associated with this activity. The controlling thread performs the necessary tasks described above and invokes streams_putmsg () to write the message to the correct queue. If the operation fails, the controlling thread creates T_ERROR_ACK and putnext (tp_read_queue, mp)
Execute If the operation succeeds on the cluster, the controlling thread returns the original message putnext (tp_write_queue).
ue, mp). Table 2 below shows a sample of the preview function on the reading side.

[0103]

[Table 2] tpi_bind_r_preview (q, mp) {union T_primitives * tpi; if (mp> b_datap-> db_type! = M_PCPROTO) (* q-> q_info-> qi_putp) (q, mp); else {tpi = (union cast it) mp-> b_rptr; if (tpi-> PRIM_type == T_ERROR_ACK) {/ * Notify join thread and notify control thread and perform * / / * cleanup * / route_msg (tpi_ct_endpoint, mp , read_queue);} else (* q-> q_qinfo-> qi_putp) (q, mp);}}

For the close processing, a close function as shown in the following sample of Table 3 is executed.

[0105]

[Table 3] tpi_close (q, mp) {mblkP mp; / * Create a close * / / * message to notify the control thread that mp = create_close_msg () * / route_msg (tpi_ct_endpoint, mp, read_queue); / * Wait for a message indicating that the cluster has been cleaned up * / / * Wait for the control thread * / recv_msg (q, mp); / * Invoke the original close routine * / (* q -> q_qinfo-> qi_putp) (q, mp)}

This function executes a cooperative operation with the control thread. The port has been released and all S
-The controlling thread informs the other thread control that a close has been performed for the ICS and the routing table must be updated to reflect that the given queue is no longer valid. If a race condition exists, the S-ICS contacts the control thread which must check for the availability of the port, and informs the S-ICS to abandon all messages for this port for the time being.
Tell ICS.

2.4. Port Mapping Optimization In the above description, the control thread was used to pass the binding request to the middleware thread and return the result. This communication allows the controlling thread to maintain a protocol-independent state, which slows down the port mapping function and creates a large number of connections, such as a web browser. In some cases, the speed at which these connections are established is important. To solve this problem, the controlling thread can be made somewhat protocol dependent by recognizing that it is executing a bound instruction and must invoke a protocol specific processing function.

In this embodiment, the TCP / IP processing function is defined as follows. There are 64K ports available for applications. About 5K of this port
Is reserved for the application and allows the protocol to perform port decisions. The processing function performs local optimization while taking this point into account. When an application requests any port, the processing function allocates a port from the local pool of any port that is split between the cluster gateway nodes. In this case, there is no need to adjust the port assignments between the nodes, but only to note which ports are currently in use and establish later connections. That is, it performs weak coupling of the entire cluster and handles routing requests as needed. As needed, because the application may set a "lifetime" period with options that may be set during initialization. For example, web browsers make potentially many connections per multipart document and each connection is short-lived. Well-designed browsers signal that the connections that clustering can utilize to reduce port management overhead are short-lived. If a node runs out of the local "any address" pool of ports, the node issues a request to borrow a port from another node or invokes a protocol that rebalances the available ports within the cluster. I do. For the rest of the port space, the controlling thread must adjust the port allocation and routing information and perform cleanup on connection decoupling or closing. Table 4 below shows a pseudo-algorithm that implements this.

[0109]

[Table 4] if (if the binding address is "any address") {if (unless the current node has an available port) {at least if there is an available port within the range of the cluster Group notify to rebalance the port space so that the starting port is guaranteed to get one port
Start the order. If no ports are available, putnext (OT
T_ERROR_ including port busy error condition by performing HERQ (transport provider write queue), mp)
Issue an ACK to prevent the local transmission provider from receiving the join request. } If the connection is not known to be ephemeral, or if there is a policy to always notify all nodes that know the state of the port about all binding-related activities, appropriate action can be taken. like,
Advertise the remote node and node address of the port. This must be achieved using a reliable broadcast protocol. } else {<About a specific port> It is necessary to adjust the port assignment among all nodes that grasp the port status. To do this, group no
Use the tify instruction. The group notify instruction allows low-latency packets to be reliably sent to the node set, and each node sends a response back to the initiating node. In this example, the request is a request for a port address and associated node, and the response is an acceptance or rejection of the request. When a node responds with a trust, it will locally (1) bring the port in use, (2) create a join instance to update the fanout table if the connection is long lived, and (3) route Update the specified information. If the node responds with a rejection, the requesting node assumes that the port is in use. No further assumptions are made regarding the execution of the reject node. }

In the above scenario, there is a potential for a race condition where two nodes try to bind the same port address. This is marked by a group notify indicating only partial success for each initiating node. When this occurs, either an implementation failure is selected, or a random backup algorithm is run to retry the integration instruction and either is chosen. Random backup can be achieved by using the same random number generation routine that includes a node-specific key that allows some prediction. This allows one application to succeed and another to fail.

If the binding fails, the initiating node notifies the nodes that responded to the port grant of the failure, and those ports perform the necessary cleanup. If the interconnection cannot tell which node failed, the initiating node must inform all nodes that it did not perform the port binding. The remote node then checks the source address and port to determine if cleanup is needed.

With partial permission, there is a race condition between one application and another application for binding to the same port. If this happens, there are two possible solutions: Each node looks at the grant-to-reject ratio, and if many of the nodes have granted the join, the port wins. The losing node issues a T_ERROR_ACK containing the port busy condition to the application and an error message to the node licensed as the port owner to clean up their tables and allocated resources. The winning node has enough time to notify other nodes, ie
After waiting a time interval of (time to process error * N) + (time to send error cleanup completion message * N), initiate a join request to the node that originally failed.

Another option is for each node to notify all authorized nodes that the merge has failed and needs to be cleaned up. Each node then waits for a node-specific random amount of time (uses a random number generator with node-specific keys, so that each node can be replayed if needed for debugging or testing purposes) Pattern exists). When the timer starts, the join instruction is started again and the result is checked.
If there is another collision, the maximum collision counter is used to determine if another attempt should be made. This method can result in both applications failing to join, but this is very unlikely.

If that is not clear, TCP / IP needs to have a larger port space if the cluster handles thousands of connections simultaneously. This requires a protocol change approved by the IETF. Desirably, some way to remedy this problem will be developed in the future, but until then it would be necessary to place an artificial limit on the number of connections a cluster can have at any one time.

2.4.1 Further Optimization of Port Mapping It may be possible to further optimize the port mapping solution by performing the binding only when the application has migrated. In the cluster embodiment of the present invention, at least one IP per DLPI driver
There is an address. This means that there is no possibility that the packet will arrive at another node or interface other than the one that the application is currently using. This is true for the combined lifetime unless the application migrates to a new node. In such a case,
To ensure that all packets arriving at this node are forwarded to the new node, the S-ICS must create a new routing path and the fan-out table contains the <port address, S-ICS write It must be updated to have a queue> entry. If all of this is true, apart from notifying other nodes which ports are in use, there is no need to create a fanout table entry for each combined application, and There is no need to perform any additional cleanup on de-association or close instructions other than notifying the cluster that it is available.

2.5 Use of Stream Dynamic Function Replacement Within a cluster, a stream stack instance operates in the following modes. * The stream stack may not be aware that it is part of a cluster. The stream stack may operate as if it were a stand-alone node, may be migrated, etc., but may not be aware of the cluster functionality itself, or may not necessarily have the ability to utilize such functionality. Maybe not. * Recognizing that even though the entire stream stack is running on a single node, it is part of a cluster,
It has the ability to utilize cluster functions such as global port mapping functions. This does not mean that the implementation of the stream stack was directly modified. It is only potentially expanded. * The components of the stream stack are TCP /
As described in the IP / DLPI splitting embodiment, it may be performed on two or more nodes within a cluster.

In the latter two modes, it is necessary to extend the different stream functions associated with each module or driver. The STREAMS dynamic function replacement function allows all instances of a module or driver to be extended, or the extension is performed on a per-stream queue. As far as the embodiment of the global port mapping mechanism shown in FIG.
It would probably be easiest to increase functionality at the TCP or UDP module level by using the application (potentially the controlling thread) via the SAD driver to perform the function replacement step. See US Patent Application No. 08
/ 545,561. This change has been made to the system /
This can be achieved as part of cluster initialization.

On the other hand, it is undesirable for the cluster function that every instance of the stream stack must be constantly modified. In such a case, it is necessary to apply the extension later based on the cluster-specific information. To accomplish this, it is only necessary to extend the open routine for every instance of the module or driver. The new open examines specific information about the cluster (such investigations are allowed because the STREAMS open routine can pause) and the entire stack or specific modules / drivers in the stream stack being created are further Determine if you need an extension.

In this embodiment, the TCP / TCP divided at the IP / DLPI level as shown in FIG.
IP / DLPI stack. This example illustrates a potential performance improvement. That is, the message is sent directly from DLPI to S-ICS without having to pass through the local stream head, and is then redirected by the controlling thread to the remote node. This is achieved in two forms: (1) The DLPI read queue q_nex field is re-routed to point to the S-ICS write queue. Such a rerouting is possible, albeit complicated for an S-ICS implementation. (2) The DLPI stream head read queue put routine is extended to preview the data and then re-route. The second method is preferred for the following reasons.

The queue is not changed in any way, and the basic functionality and message flow remain the same.
That is, the message flow from the DLPI driver to the stream head is as before, which reduces potential support and debugging issues. If the first method were to be performed, the S-ICS would need to determine whether the message should be reflected or routed to the controlling thread for processing. Some messages, such as M_SETOPTS, require modifying the local streamhead parameters along with the remote streamhead parameters. This not only complicates S-ICS design and implementation, but also increases the amount of protocol-specific information that S-ICS must know.

Using the second method, it is possible for the controlling thread to select the type of preview to be performed for each module / driver. This can be done in two forms: Either (1) the cluster designer writes the code for the preview function class based on where the stack is split, such as a generic DLPI class, or (2) each module / driver developer has a stream specific to the protocol. Define a head preview function. In any case, the focus of the preview code is on the immediate need of the stack, thereby keeping the control thread independent of the protocol,
It is possible to shorten design and development time and improve flexibility. One example is when the DLPI driver detects a hardware problem, in which case the DLPI driver generates an M_ERROR message. If the DLPI streamhead preview function does not require anything to be invoked to handle this condition, the message is sent to the S-ICS like a normal message. However, with respect to the driver's DLPI class, or specific driver like DLPI over ATM,
With respect to type, if the cluster designer determines that the controlling thread must perform something for the high availability recovery scheme, such as migrating a DLPI instance to a new card or a new node, the preview function will determine the stream head. It will invoke the put routine and give the message to the controlling thread. The controlling thread then determines whether the message needs to be forwarded to the application based on the message type and the configuration configured to execute for this class of stream module / driver.

STREA such as M_DATA or M_PROTO
For messages that do not require MS framework actions, the preview function automatically executes putnext () on the S-ICS write queue. This avoids having to store the message at the stream head, makes the controlling thread context-switchable, and examines the message only to make sure that the controlling thread does not need to be involved. Table 5 below is a sample program code of the preview function.

[0123]

[Table 5] dlpi_sth_read_Preview (q, mp) {switch (mp-> b_datap-> db_type): case M_HANGUP: case M_ERROR: / * Even if the control thread sends only a request * / / * The acknowledgment ack is the control thread The control thread fires a local ioctl instruction, as routed via. * / case M_IOCACK: case M_IOCNAK: case M_ERROR: / * Invokes the standard stream head read put routine, * / / * The control thread recovers what to send or * / / * in the M_HANGUP / M_ERROR case. To determine if it should be started. * / (* q-> q_qinfo-> qi_putp) (q, mp); break; case M_DATA: case M_PROTO: case M_PCPROTO: default: / * Any message type not listed * / / * automatically Sent to * / if (canput (q-> q_ptr-> s_ics-> write_queue) putnext (q-> q_ptr-> s_ics-> write_queue, mp); else {/ * The upper S-ICS mux is flow controlled, Process * / initiate_flow_control_recovery (q, mp);}}}

The s_ics pointer is stored within a private data structure (q-> q_ptr) that defines the stream head associated with this DLPI instance. s_ic
The s pointer is the S-ICS associated with this instance
Streamhead address, the message is stored in the S-ICS write queue that sends the message to the remote node. Note that how the routing information is embedded in the message is shown in the function sample above.

3.0 Control Thread Design Although the outline of the control thread has been described in the previous section, the details of the control thread are described in this section. However, it should be understood that the final design will depend on the target operating system, the STREAMS implementation and the implementation based on the platform on which the distributed STREAMS is implemented. This section also highlights the high-level design, the data that the control thread must track, and the way the control thread interacts with other components. The details of control thread design and interaction for distributed stream creation, stream migration and policy management are described in a later section.

Note: In the previous section, the control thread 34
It has been shown to operate directly on P-ICS and S-ICS. These two drivers have completely different interfaces and potentially different operating characteristics. For example, one is a communication mechanism managed by the sender and the other is a communication mechanism managed by the recipient. Such differences can derive design complexity and potential performance tradeoffs. If such a difference is large, the controlling thread can also be implemented as two threads with a shared data set to appear as a single execution thread. One thread is S
It manages activities related to TREAMS, and the other manages cluster management activities such as port space and device driver management. This embodiment improves protocol independence of drivers and code associated with STREAMS.

3.1 Control Thread == Middleware Can the control thread 34 replace the middleware component 50 within the cluster? The answer is "can". The control thread is basically implemented as distributed middleware with the described functionality. This point has already been described in the description of the global port mapping optimization (of FIG. 6), but will be explained as much as possible in subsequent sections if enhancements to handle this functionality are needed. .

3.2 Creation The control thread 34 is created as follows. If the controlling thread must be running before the system completes its initialization, the thread must be created as part of the standard system initialization process. This is the case when the node performs configuration work on its own based on the requirements of the system. For example, if the system has to load a particular networking subsystem, and this subsystem is part of or relies on a cluster facility, the controlling thread will call this subsystem before it is initialized. Must be operational.

If the controlling thread does not need to start up the system, it can be started at any time via the start cluster command, but the following must be considered: If the clustering mechanism includes subsystems that are launched at system initialization, these subsystems are not affected by clustering, or if they are, they will not allow control threads to communicate with them. It must be converted into some form that can be done. One example of this is a node with a running NFS service that combines a set of ports. Assuming that NFS is running for a stream-based transport provider, information about the port must be notified to the middleware thread, and if the packet is routed correctly to this node or Appropriate ports must be tied on other nodes in the cluster to ensure that they are sent correctly. In addition, put, ser
The function substitution described above needs to be performed for the vice and close routines.

In general, the controlling thread must operate as a real-time execution thread to ensure that the cluster mechanism utilizing this thread receives a timely response to the request. Typically, this is performed as part of thread creation, ie, in the manner in which the thread is started or via a set priority command.

This suspension of the thread will lead to the cessation of STREAM-related clustering activity on that node, unless steps have been taken at creation time to allow the thread to resume. Such steps are as follows. (1) Allocate a known data structure within the kernel and store its address in a kernel global variable. (2)
Within this data structure, the controlling thread must store the following information: That is, addresses of all stream heads opened or assigned to this control thread, addresses of middleware interface devices (communication with middleware is
(Via pointers or interconnect descriptors) and policy information communicated to the thread. (3) If the communication of the source management is used for the communication with the middleware thread, some mechanism is required to maintain the user space. This may involve notifying the cluster coupling management subsystem how to keep this data and how to restore it for a new thread of control.

The node structure depends on how the nodes are addressed and how routing is achieved within the cluster. The above structure includes what is considered to be essential as a field, but it is possible to add others in the course of implementation. The given flags are the minimum required. Table 6 below is a sample node structure definition.

[0133]

[Table 6] #define NODE_INACTIVE Ox0000 #define NODE_ACTIVE Ox0001 #define NODE_IN_ERROR 0x0002 struct node_t {int32 node_flags; / * Remote node status flag * / node_address_t node_address; / * Remote node address * / node_route_t node_route; / * Local node * / Struct node_t * next, * prev; / * next or previous node in list * /} #define NODE_HASH_SIZE 1024

The structure shown below includes a structure type specific to P-ICS. That is, the functions of P-ICS and P-ICS
Unless the ICS knows what it wants to communicate with, it cannot show the contents of these structures. At a minimum, this structure must include enough endpoint definitions for message processing,
Includes state flags and generic function vectors that may be implicitly invoked by the S-ICS to activate the operation. That is, a P-ICS embodiment, a migration policy when the S-ICS needs to migrate from one PICS instance to another, or a P-ICS.
The S-ICS remains as ignorant as possible about recovery policies and the like on how to recover from S errors or interconnect failures.

[0135]

[Table 7] struct p_ics_endpoint_t {uint32 p_ics_state; uint32 p_ics_flags; int32 (* p_ics_memory_alloc) (); int32 (* p_ics_memory_free) (); int32 (* p_ics_open) (); int32 (* p_ics_close) (); int32 (); int32 (* p_ics_detach) (); int32 (* p_ics_ctl) (); int32 (* p_ics_send) (); int32 (* p_ics_recv) (); int32 (* p_ics_misc) ();}; typedef p_ics_endpoint_t P_ICS_ENDPOINT p_ics_data {P_ICS_ENDPOINT p_ics_endpoint; P_ICS_MIGRATION_POLICY p_ics_migration_policy; P_ICS_RECOVERY_POLICY p_ics_recovery_policy;}

In the above sample, p_ics_endpoin
Defines a function vector that represents the P-ICS endpoint and a set of possible actions. p_ics_migration_poli
cy is a policy structure that defines the content of the policy, the timing of activating the policy, and the set of function vectors for executing the policy. p_ics_recovery_policy is a policy structure that defines the content of the recovery algorithm, the timing when the remote node fails, the timing when the local P-ICS instance fails, and the function vector to execute the policy.

[0137]

[Table 8] #define CTL_S_ICS_TBL_SIZE 64 #define CTL_STH_HASH_SIZE 512 #define CTL_HASH_FUNC (dev) do {\ & ctl-> ctl_thread_streams [(((unsigned (major (dev) ^ (unsigned) minor (dev)) &(CTL_STH_HASH_SIZE); (0) #define CTL_S_ICS_HASH_FUNC (dev) do {\ & ctl-> ctl_s_ics_streams [((unsigned (major (dev) ^ (unsigned) minor (dev)) &(CTL_S_ICS_TBL_SIZE);} while (0);

[0138] The assumption underlying the embodiment is that the stream heads may be interconnected via the sth-> sth_next field. Otherwise, implementation requires a different hash bucket management strategy.

[0139]

TABLE 9 struct thread_management {struct sth_s * ctl_thread_streams [CTL_STH_HASH_SIZE]; struct sth_s * ctl_s_ics_streams [CTL_S_ICS_TBL_SIZE]; struct inter_con_descriptor * ctl_middleware_descriptor; struct policy * ctl_known_policies; struct preserve_policy * ctl_preserver_policy; struct node_t ctl_cluster_nodes [NODE_HASH_SIZE]; struct p_ics_data * ctl_p_ics_instance; struct driver_management_policy_t * ctl_driver_policy;} struct thread_management ctl_thread_global_data;

In the above sample, ctl_thread_strea
ms is a stream head pointer representing a distributed component (eg, a distributed DLPI instance) operating on this node. ctl_s_ics_streams is a stream head pointer representing a local S-ICS instance. ctl_middleware_descriptor is self explanatory. ctl_known_policies is a set of policies that describes how this controlling thread should operate. These structures are mechanisms that allow the controlling thread to hide basic protocol-specific requirements from the outside world. ctl_preserver_policy is the self-preservation policy described above. ctl_cluster_nodes is a node table created at the time of control thread initialization. This data can be derived from the interconnect driver from which the data was downloaded as part of the initialization process. The ctl_p_ics instance is P-ICS data necessary for the control thread to communicate with another control thread within the range of the cluster. ctl_driver_policy is
Describe driver classes and how to handle them.
This data is downloaded during control thread initialization. The example policies are for determining where the driver should be opened and whether the stream stack requires splitting between nodes.

Once a thread is created, it accesses the existing cluster mechanism policies and sets up the appropriate substructure. Policy to specify how specific tasks should be performed, such as how and where to download cluster area information, how to determine which node is the current cluster management node, etc. Is used. This information is stored at a known location on each node or downloaded via broadcast to a running middleware thread within the cluster.

If the cluster node is a multi-processor system such as SMP, and that node manages many distributed streams, it may be necessary to create multiple control thread instances. . Each instance may be responsible for an individual protocol stack, or for a subset of instances of each protocol stack if space is too large for a single thread to handle, or It can be designed specifically for different clustering aspects, such as one control thread handles strlog (), one control thread handles SAD, and one control thread handles protocols. The form of partitioning and the number of control threads required is highly dependent on the embodiment. FIG. 7 shows the data structure layout and interconnections. It should be noted that S-ICS 38 points to such a data structure from its private data structure area.

3.3 Control Thread Configuration The control thread 34 can be configured using any of the following techniques. The controlling thread must have at least a default mechanism that can communicate with the P-ICS 36 (of FIG. 4). This communication mechanism depends on the P-ICS embodiment and requires that the initialization setup protocol be included as a microprogram.

Instead of broadcasting, the control thread 34
May rely on a less complex but error-free mechanism of reading the file system bootstrap configuration data. This data specifies how the thread should behave, where the middleware, if any, is, and any other necessary initial steps. Of course, the configuration file can also contain everything the thread needs to know. Such files are managed by a single management node from which they are distributed within the cluster.

Another mechanism is to bootstrap the thread 34 to a point where the thread can communicate with other similar control threads or middleware. The controlling thread detects other threads or middleware 50 by executing "ARP" and identifying itself and its current state. Thereafter, the controlling thread responds to the content sent to it.
For example, when the controlling thread opens S-ICS or local stream components, it can look up the data and policies needed to meet those requirements for those components. This capability allows the controlling thread to meet local requirements while remaining fairly implementation independent.

3.4 Inter-Component Communication The following section describes a distributed STREAM that interacts with the controlling thread.
The communication between the main components of S will be described in detail. Each clause is
Includes samples that show how problems are solved through such communications.

3.4.1 Middleware Communication The amount and timing of communication with the middleware 50 (of FIG. 6) depends on the services provided by the middleware and the policy requirements for the distributed STREAMS stack. This is a design decision factor for keeping the distributed STREAMS solution middleware independent. At a minimum, the areas of stream creation, stream distribution and error recovery require communication. In order to show how the communication works, the manner of using the cluster, such as the hardware sharing, load averaging, high availability and single system perspective described above will be discussed below.

Hardware Sharing : When a hardware sharing cluster is defined, at a given point in time, not all nodes in the cluster are active. That is, the available cluster hardware is fluid and no nodes can depend on it. This means that the cluster hardware configuration must exist in software entities that reside on one or more nodes within the cluster, and that dynamic (static) decisions determine the location and time of information. It means that there must be a protocol. This can be achieved within the control thread, but its form can complicate the design and make the implementation and data unavailable for other non-streamed cluster mechanisms. Middleware improves this situation.

In the illustrated hardware sharing example, the entire stream stack resides at nodes C and D, the data exchange with the TCP / IP stack relies on function delivery, and the application Executed on B. Application is TCP / IP
To open an instance (the algorithm for opening is described below), the application must interact with a control thread that contacts the middleware thread. The controlling thread sends the middleware 50 by sending messages indicating the type of hardware being accessed, the type of application being executed, and additional modifiers such as the remote target network address and application priority. Find out. This information is used to create a middleware policy request. The middleware examines this request and determines at which node the request is being satisfied. This algorithm is shown in FIG.

Although the actual content of these messages depends on the middleware, it does not exclude that the controlling thread is independent of the middleware. To maintain independence, the controlling thread may also hide all of the above information by creating a function vector that performs the general message and action to convert between middleware-specific messages. This technology uses STR
It has been used in EAMS and many operating systems and is very common in object-oriented applications. The following operations must be supported for distributed streams: * Acquire middleware location and middleware failure recovery policy. * Acquire the cluster node configuration and related hardware. The node configuration includes items such as where STREAMS drivers and modules are located, the control threads running on that node, and which STREAMS services are supported. * Obtain the cluster node routing table. * Get a list of active cluster nodes. Since this list is dynamic, the controlling thread may periodically verify this information, or the middleware may update all controlling threads of the change. * Obtaining policies that determine how to perform node disaster recovery and other activities related to high availability. Obtain policies specific to the running STREAMS stack, such as where the stack is split, where the stack transitions, and whether dynamic function replacement should be used to improve performance.

In order for the above to be achieved without using a middleware thread, the control thread needs the following configuration data: * Which nodes contain stream drivers and modules. * Which nodes are currently active and fully operational and have the necessary resources and capabilities to meet the default requirements. * How to access the device file and associate it with the dev_t structure. * How to handle protection / security issues for applications that require driver access. * What is the current system load on the target node set and what are the criteria for selecting a specific target? A round-robin scheme can be used, but individual nodes can be overloaded even without load data. * What are the addresses of all S-ICS in the system,
What are those abilities? * What should the control thread interact with to meet this request?

Load averaging : Load averaging requires that the stream-type stack be able to migrate to different nodes in whole or in part according to the thread. The control thread uses the middleware to determine a migration policy, a stack division location, a target node, and the like. The majority of this information is determined at the time the stack is created, but the timing to transition and the transition goal are usually the only dynamic elements. FIG. 9 illustrates a potential message exchange between the control thread 34 and the middleware 50.

High Availability : For high availability, the controlling thread (not the middleware) is the recovery starter. This is necessary because recovery can be the result of a middleware failure and the controlling thread needs to re-establish communication with the new middleware. If a middleware failure occurs, the following steps must be taken according to the failure detection method.

1. If the controlling thread had initiated a request or pending request timeout, the controlling thread checks its middleware recovery policy to determine the next action to take. This may vary depending on the middleware embodiment. The following is an example of a policy.
(A) Broadcast a message to a control thread on another node to detect a failure and determine if it is unique to that node. If so, the controlling thread needs to notify the dollarware via one of the other controlling threads to initiate recovery. (B) Check the node list to determine whether the middleware is running there. In that case, re-establish communication and proceed to the next node. If neither node is running middleware, it consults the list of alternate middleware nodes and notifies that node to initiate and perform middleware recovery.

2. The controlling thread may also know of a middleware failure with a new middleware instance created as a result of a previous step. In that case, the new instance will notify all known control threads of the change and prevent the recovery process from being triggered.

3. The controlling thread may be notified via S-ICS that the middleware can no longer be found. In that case, the controlling thread starts its recovery mechanism. When the recovery is completed, the control thread notifies the S-ICS of the new instance,
The ICS continues its processing. Note: S until recovery is completed
-It would be straightforward to design the S-ICS and the control thread so that the ICS forwards all middleware requests to the control thread. Some activities are streamed
This method is an S-IC because it is inherently simpler to design on the stream head rather than within the stack.
S is simplified.

If a stream stack component fails, the interaction with the recovery mechanism and middleware is dependent on what is notifying and where the fault has occurred. For this, the following steps must be taken. 1. The control thread 34 may be notified by the middleware 50 that the node has failed. This is the case when middleware takes an active approach to disaster recovery. TCP / IP / DLPI shown in FIG.
In the case of a configuration, whether TCP or TCP / IP depending on the split position of the stream stack, if the node executing the upper part of the stream fails, the recovery policy simply closes the DLPI instance. , Update all S-ICS and control thread data structures. For another stream stack, the recovery policy may be to reconnect this stream component to another node as if one endpoint of the streamed pipe was reconnected to a new application. In this case, the middleware may include a new point of attachment in the message sent. This would be one way to implement application migration in the case of a stream-type pipe.

2. The controlling thread may be informed that the S-ICS can no longer route requests to remote nodes. The controlling thread invokes the recovery policy that was being downloaded when the stream stack was created. This policy is communicated to the middleware and new remote nodes are possibly determined. Naturally S-IC
S can also perform this operation, depending on the desired location where recovery should be performed. Performing the recovery in the S-ICS may be faster. This is because there is no context switch to the control thread, but a timing problem may occur between the S-ICS and the control thread based on the form in which the data is divided and reflected in each component. The controlling thread may be notified by the stream stack component itself that there is a problem. For example, if a component sends an M_HANGUP or M_ERROR message, the stack is essentially useless until the problem is resolved. If an M_ERROR message is issued, it indicates that the driver cannot communicate with the hardware. If the controlling thread knows that the hardware has been replicated on the local node, it simply creates a new instance of the stream component and updates the relevant data. The controlling thread may also start a new instance by checking its hardware knowledge policy and executing an algorithm that opens a distributed stream.
Alternatively, the controlling thread may seek out middleware for alternative hardware and start the same algorithm. In each case, based on the manner in which the data is split and reflected, the control thread needs to notify the middleware, other control threads, and the S-ICS about the location of the new stack component.

Single System View : If the cluster is to maintain a single system view for the entire cluster resource, there must be a middleware mechanism to translate the generic resource criteria list into a specific resource identification list. In the hardware sharing described above, a reference message was generated by the controlling thread and sent to the middleware. The middleware decrypts this message and responds with a specific resource instance that meets this requirement. The controlling thread can then continue its operation. For more information on how to generate these messages,
This will be described later together with the opening of the distributed stream.

3.4.2 S-ICS Creation and Communication The control thread is responsible for creating, managing and destroying S-ICS instances via the in-kernel STREAMS interface. Table 10 below
Is sample code for the control thread to create an S-ICS instance that is a stream type driver that can be duplicated.

[0161]

Table 10 / * Determine device identifier for S-ICS dry * / if ((open_dev = str_dev_lookup ("/ dev / s_ics", & err)) && err) {/ * If device file does not exist Record an error and end the process. * / / * If the operating system supports dynamically loadable * / / * kernel modules, * / / * explicitly start the subsystem load of this driver * /} / * device identifier Open the driver using * / / * find the stream head * / if ((sth = streams_clone_open (open_dev, O_RDWR, & error)) == NULL) / * Record the error and use the S / ICS driver major / Attempt error recovery, such as * / / * detection of minor numbers. * / / * Hash bucket entry * / / * using streamhead address and remember address * / struct sth_s * bucket; bucket = CTL_S_ICS_HASH_FUNC (sth); bucket-> sth_next = sth;

If the S-ICS is implemented as a non-cloning driver, the streams_clone_open () call
Replaced by open () call. The rest of the code remains unchanged. Each control thread has one or more S
-May create an ICS instance. The number of instances created is based on:

If the controlling thread is assigned a particular task, such as module and driver strlog () activity processing for one node, only one SI
Just create a CS task.

The majority of the messages between the distributed stream components are mainly M_DATA and M_PROM, and if the control thread is not involved, the control thread will increase the bandwidth throughput / efficiency and the distributed STREAMS
At least one per processor or cluster interconnect card to improve the solution MP scale
One would like to create one S-ICS. A distributed stream that is managed based on something simple, such as a target node with which to communicate, or based on a hardware card, such as a managing DLPI instance.
An S-ICS instance can be assigned to a component. This improves the efficiency and performance of the system cache.

If the implementation of S-ICS cannot be implemented in a protocol-independent manner, for example, almost all stream-based transport providers use TPI from an application perspective and DLPI from a hardware interface perspective. If, while using, there are some situations that cannot be solved in a general way, the SI
It is necessary to perform CS implementation. This is S-ICS
S-ICS, such as a routing cache, not only means that many of the designs cannot be reused
Means best performed on a protocol-by-protocol basis. This effect may be such that some of the S-ICS are simplified and others are not.
For example, not all protocol stacks may require a highly available solution, or may have different routing schemes that lead to a simplified design. Note: STREAMS dynamic function replacement
S-ICS using (Dynamic Function Replacement)
These problems can be solved in a general form of extending the functions of the above. STREAMS Dynamic Function Registration (Dynamic
Using Function Registration) [3] to decrypt the message and perform any pre / post-processing via the S-ICS to include or remove information that assists the S-ICS in its processing Help them decide if they can do it. The controlling thread pre-stores a number of S-ICS in a quick lookup cache for additional dynamic allocation requests. Pre-storage allows the controlling thread to respond quickly to remote requests and stack transitions.

The following steps must be taken for each S-ICS created. 1. The controlling thread conveys cluster substructure data such as node tables, node routing tables, and middleware communication endpoints. The S-ICS uses this data to create a data cache. in addition,
The controlling thread may convey additional S-ICS specific information if the S-ICS has the specific operating characteristics described above. 2. The controlling thread determines the S-ICS read / write queue address. Next, the control thread notifies each of the stream head preview functions of the distributed stream components to which this S-ICS passes data. This allows the stream head function to
Data such as DATA and M_PROTO messages can be bypassed via the stream head.
In the preview function above, these messages are sent directly to the S-ICS write queue.

3. Similarly, the control thread notifies the S-ICS of the distributed stream of the stream head address and its sth->wq-> q_next and sth->rq-> q_next addresses. The S-ICS uses this data to build a fanout table. The fanout key is
The node identifier and stream head address embedded in the message, providing a unique, cluster-wide identification tuple. Steps 2 and 3 are used to improve performance by eliminating the need to include a control thread if not needed.

4. To be able to achieve the performance enhancements described above, the controlling thread recognizes that all components involved in this bypass must remove incomplete activity, disables the stack and disables the stack. A flag used during the close process must be set in the stream head to make sure that the completion activity is canceled. If this code were not provided in the STREAMS embodiment, the system would be in a race condition that could lead to a system panic.

5. If the controlling thread detects that the middleware has failed, it initiates recovery and notifies the S-ICS of the new instance. Similarly, if the S-ICS detects a middleware failure, it notifies the control thread, which initiates recovery and updates the S-ICS of the new middleware instance. When the S-ICS is created and initialized, the communication between the controlling thread and the S-ICS depends on the capabilities provided by the cluster mechanism and the distributed STR.
Each component of EAMS depends on the level implemented. The following is a brief description of potential communication lists between the controlling thread and the S-ICS and how and under what circumstances they are implemented.

When the stream stack is opened and its components are distributed between at least two nodes, the controlling thread contacts the controlling thread of the remote target node to create a remote stack component and create an S-ICS data Initialize the cache. To achieve this, one communication protocol needs to be applied between the two thread instances and between the thread and the S-ICS instance. In addition, the control thread needs to coordinate middleware knowledge and updates. This is a bit complicated and will be covered in another section.

Some of the performance tuning ideas presented in this design technically violate the ideals of how STREAMS modules and drivers should be implemented, and how messages should be sent in the stack. .
This is not unusual, as many module and driver designs are usually poor and such violations must be made to achieve the goal with minimal effort.
For example, in the case of a networking protocol, a protocol originally implemented in a non-STREAMS stack is applied to a STREAMS stack. Within this design, it violates the normal routing of messages by enabling the routing of messages between certain stacks and the S-ICS stack, i.e., only via the stream head. . The main reason for not allowing such a violation is the close competition.

When the stack is being closed, individual components are removed from the stack in a top-down fashion from below the stream head to the lowest driver. Before each module / driver is fetched, their associated close routines are called so that each component can perform private data cleanup (pointed to by q_ptr) and erase current messages. It becomes possible. Also, to ensure that there are no pending references to any queues or messages before the module / driver structure is released, all pending
The STREAMS framework cancels put and service routines. If such routines were not canceled, they could reference invalid pointer addresses that would cause system corruption.

The put or servic function of one stream stack, whether original or extended, puts a message on another stream stack and puts or puts the servicing of its queue instance.
By invoking the ce routine, if a close is in progress, the reference pointer, such as the address of the target queue, may become invalid and the system may be broken. To prevent this, a close protocol used for the two stacks needs to be developed. In this case, the S-ICS, the control thread and the streamhead preview function that performs the routing must be adjusted.

[0174] The close protocol described below is divided into IP and DLPI as shown in FIG.
An example of the TCP / IP / DLPI configuration being performed between the two is used. If the DLPI stack is being closed, a message is sent from the S-ICS read queue via its put and service routines to the DLP
A close race condition occurs only if it has been sent to the I write queue. To prevent this, the control thread from which the close operation was initiated must confirm that the close is pending, cancel any pending messages, and set the target DLPI before the close can begin.
The S-ICS must be notified that the current operation on the instance should be confirmed as completed. SI
The CS is notified via the ioctl because the ioctl does not end until the S-ICS has completed all cleanup.
The S-ICS performs this cleanup by using an "active" counter and a "closed" flag in its private data structure associated with the target DLPI. This counter is incremented each time a put or service service routine is started and decremented upon completion. When the counter reaches zero, the S-ICS can confirm that all in-flight activities have been completed, and thus can safely remove all messages from the queue targeted for this instance. .
Table 11 below is a sample S-ICS read / write routine showing these changes. Counter is STR
It should be noted that there is no extra spinlock to add, as controlled by the EAMS framework synchronization mechanism. Since the close flag can also be set at any time and never cleared, no spin clock is required.

This structure corresponds to each S-ICS instance q-
> part of the S-ICS private data structure stored in q_ptr.

[0176]

[Table 11] struct target_queue_t {queue_t * target_read_queue; queue_t * target_write_queue; int32 target_pending_actions; int32 target_flags; / * Other additional information * /} / * Sample of put routine modification * / int32 s_ics_read_put (q, mp) {struct target_queue_ * target_queue; struct s_ics_q_ptr * s_ics_data; / * Local S-ICS data * / s_ics_data = (struct s_ics_q_ptr *) q-> q_ptr; / * If M_DATA or M_PROTO, send the instance * / / * immediately to correct * / if (mp-> b_datap-> db_type == M_DATA || mp-> b_datap-> db_type == M_PROTO) {if ((target_queue == find_target (mp, s_ics_data)) == NULL) {/ * Start error recovery * /} else {if (target_queue-> target_flags & TARGET_CLOSING) {freemsg (mp); return;} target_queue-> target_pending_actions ++; putnext (target_queue-> target_write_queue, mp); target_queue-> target_pending_actions--;}} else / * Send * / / * messages to the controlling thread for all other message types It should be whether or not to judge. * / switch (mp-> db_type) {case M_PCPROTO: / * Execute check and handle message interruption * / case M_IOCTL: / * Execute check and handle message interruption * / case M_FLUSH: / * Execute check * / .... / * Add as many cases as necessary for S-ICS processing * / freemsg (mp); return;}}

If the S-ICS instance is being closed, the S-ICS instance manages to ensure that all current and in-progress actions are completed or prevent reference to an invalid queue address. All of the DLPI instances that have been running must be notified. Routing to this queue address
The solution is simple because the stream head read put routine is only performed if it has been extended to preview and send data. That is, remove the extension function from the stream head, or
DLPI instance from one S-ICS to another SI
When shifting to CS, the extended function target queue information is either updated. S to remove the function
Uses the same technology used for TREAMS Dynamic Function Replacement. There is no need to maintain a counter or close flag since the stream head has no read service routine. Thus, when the stream head is obtained to perform this activity, all of the active calls have been completed and all in-progress activities have been updated with the stream head pu.
It is guaranteed to be performed using the t routine.

There are at least three areas of communication that must be addressed for the implementation of a distributed stream. They are ways to perform splitting of existing streams, handling of operational flow control issues within a cluster, and handling distributed and remote stream heads.
These details will be described later.

3.4.3 Local Stack Component The local stack component is responsible for passing messages between components and performing the necessary flow control.

3.4.4 In-Kernel Stream Interface The controlling thread utilizes the in-kernel STREAMS interface to communicate with the S-ICS it controls and the various module / driver stacks. The controlling thread uses the following in-kernel interface instructions: * Streams open () and streams close (): S-ICS
Used to create / destroy instances and distributed stream components. * Streams_read () and streams_write (): used to send out messages generated by the controlling thread. These routines use M_DA via allocb ()
The built-in recovery function bufcall () that generates TA mblks, which means that the controlling thread does not need to handle this problem
have. However, to determine where to start
Since the M_DATA message must be scanned, the M_DATA message as a communication mechanism for distributed stream management has a limited use. * Streams_putmsg () and streams_getmsg (): used to send out messages mblks created by the controlling thread. In this case the control thread is of course allo
The cb () failure must be handled, so the controlling thread either uses the new routine allocb_wait (), which returns as soon as memory is allocated, or maintains a pool of mblks that manage itself. Or maybe. This pool is very small and only small in size, about 2-3 messages, as control threads need only be supplied in case of emergency. The actual primary allocation scheme is mblk reuse, because the majority of the messages sent to it are mblk.

By using mblk, the control thread can execute M_CTL, M_DELAY, M_RSE,
Communicate directly with the S-ICS or remote S-ICS via M_PCRSE, M_STARTI, M_STOPI or M_HPDATA messages. This is possible because most modules and drivers use these message types in a limited way, so that the S-ICS and streamhead preview functions can use this particular subset of messages without performance degradation. Is detected, and all other messages can be automatically transmitted. In addition, these utilities allow the controlling thread to directly recognize messages such as M_ERROR,
A recovery mechanism can be implemented and appropriate action taken, such as not sending a message to the application. * Streams_ioctl (): Used for all ioctl processing. The kernel interface is the kernel ioctl
Defines a new set of and how to use them. st
Most of the standard STREAMS defined in reamio (7) are supported.

4.0 S-ICS Design S-ICS is a stream-type driver, preferably a duplicable driver. S-ICS is a software driver because S-ICS does not control physical hardware devices. Although the controlling thread is illustrated as communicating via an in-kernel interface, the communication can be performed using a standard system call interface. The choice is made in consideration of processing performance and potentially security. Further, when a plurality of control threads operate at the same time, it is more difficult to balance the load of the S-ICS and perform data sharing among them because it requires additional code and a shared memory segment. is there.

The above points do not affect the S-ICS design, except that the node table and P-ICS instance data must be maintained within the S-ICS and not the controlling thread. S-ICS is P
-Contains a function vector describing the ICS-specific operation functions
Communicate with P-ICS using the P_ICS_ENDPOINT structure.
By limiting communication with these functions, SI
The CS basically remains independent of the P-ICS embodiment. The S-ICS maintains a routing table and some data tables used to send messages between the target queue it is managing and itself. The remaining uses improve performance by eliminating the need to move the majority of messages via the controlling thread and leaving them to handle only exception situations. Since the tables are updated so as to be always synchronized based on the method of updating the routing table, the S-ICS does not require much error processing.
Thus, the S-ICS must be involved in the creation and migration of all distributed STREAMs when components are split between nodes. The above points regarding S-ICS will be described in more detail below.

4.1 Creation and Destruction This is described in the section on creating and interacting with control threads.

4.1.1 Driver Open Routine Design Like all drivers, the S-ICS 38 has an open routine. Since the stream type driver is allowed to sleep during the open, the S-ICS performs the following steps including the sleep. 1. If S-ICS is implemented as a replicable driver, it needs to get a new minor number that is returned to the caller as part of the dev_t structure.
This minor number is an integer that represents an index into the array of potential S-ICS device structures. In such a case, the array element is marked in use and the open routine initializes it. S-IC
Assuming that S is implemented as a non-duplicatable driver,
It is already active and it is not permissible to perform multiple opens at any one time. In such cases an error must be returned.

2. Assuming that the P-ICS dev_t is obtained via either a default value or a known function call that returns a dev_t structure, the S-ICS uses the P-ICS protocol specific to the embodiment. Can be used to connect itself to this hardware driver (which makes this a generic function call for all subsystems accessing the P-ICS using data concealment). If the P-ICS is dormant during this operation, the connection is preferably completed during the open. If not dormant, it is a simpler and more flexible way to have the controlling thread check the middleware for the best P-ICS instance for this node (this allows the middleware to It is possible to balance multiple P-ICS among various other subsystems running for a node). 3. At this point, the S-ICS must allocate all private data structures, locks, etc., and store the control address in the q_ptr field of its queue. The potential layout for this information is shown in the next section. 4. If there are no errors, return the minor number.

Table 12 below shows the S including the above steps.
-A sample of the ICS open routine. The actual implementation details are left to the individual developers of this driver.

[0188]

[Table 12] int32 s_ics_open (q, devp, flag, sflag, credp) {int32 minor; struct p_ics_attach attach; / * Find available minor number * / if ((minor = find_minor_num (q, devp, flag, sflag )) == 0) return (0); / * Determine P-ICS and execute connection * / if (streams_P_ICS) {if ((error = p_ics_attachment (DERIVE_DEV_T (streams_P_ICS), & attach)) {release_minor_number (minor) ; return (error);}} / * Private data structure and S-ICS * / / * Allocate fanout table to initial state * / if ((q-> q_ptr = allocate_s_ics_structures (q, minor, flag) ) NULL) {if (streams_P_ICS) detach_p_ics (attach); release_minor_number (minor); return (ENOMEM);} return (minor);}

4.1.2 Driver Close Routine Design The S-ICS driver close routine is almost the reverse of the open routine. When the S-ICS is closed, any modules or driver queues that were directly communicating bypassing the stream head are notified of the pending close command, the extended stream head is properly adjusted, and all It is assumed that the in-flight request instruction has been canceled or routed to another S-ICS instance. Thus, the close routine simply undoes what the open routine did. The sample code in Table 13 below is an example of performing such a step.

[0190]

Table 13 int32 s_ics_close (q) {int32 minor; struct s_ics_data * sp; / * Write private data to a known S-ICS structure * / sp = (struct s_ics_data *) q-> q_ptr; sp = (struct s_ics_data *) q-> q_ptr; minor = sp-> minor; / * If P-ICS is still connected, disconnect it * / if (sp-> p_ics_attached) detach_p_ics (sp-> attach); / * Release private data structure * / free_s_ics_structure (sp); / * Release minor number * / release_minor_number (minor);}

4.1.3 Write_put Routine Design Table 14 below shows sample code for the write_put routine.

[0192]

Table 14 int32 s_ics_write_put (q, mp) {struct target_queue_t * target_queue; struct s_ics_q_ptr * s_ics; struct rem _ s_ics_descrp * target_sics; struct p_ics_data_t * p_ics_data; s_ics = (struct s_ics_q_ptr *) q-> q_ptr; / * If not from M_DATA or M_ * PRO or from the controlling thread * / / *, pull out the target S-ICS and send it immediately * / if (mp -> b_datap-> db_type == M_DATA || mp-> b_datap-> db_type == M_PROTO || mp-> b_datap-> db_type == M_PCPROTO ||! from_controlling_thread (mp)) {/ * S derived from mblk − Find ICS instance * / if ((target_s_ics == find_s_ics (mp, s_ics_data)) == NULL) ｛/ * Initiate error recovery and notify source via M_ERROR * / mp-> b_datap-> db_type = M_ERROR; set_m_error (mp, ENXIO); target_queue = target_queue_index [mp-> b_datap-> source_queue_index]; / * Update in-process action * / Target_queue-> target_pending_actions ++; putnext (target_queue-> target_write_queue, mp); target_queue-> target_pending_actions--;} else {if (p_ics_s_ics_status & S_ICS_FLOW_CONTROLLED) {p_ics_s_mp_fls_ics_fls return;} mp = s_ics-> p_ics_instance-> p_ics_memory_alloc (mblk_header_size); CONVERT_mblk (mblk, mp); db = s_ics-> p_ics_instance-> p_ics_memory_alloc (dblk_size); CONVERT_dblk (dblk, db); _ remote__REME ); (* s_ics-> p_ics_instance-> p_ics_send) (rem, mp, db); / * Release the message if no transmission error is detected * / freemsg (mp);} else / * All other messages Take the appropriate action for the type. * / / * For example, for M_IOCTL, the controlling thread is likely to update the underlying structure data * / / *, which must be reflected in the local S-ICS data * / / *. Other messages depend on the protocol applied between the controlling thread and * // * S-ICS. * / / * Note: Use the M_DATA or * / / * M_ * PROTO messages to communicate with the S-ICS as this may confuse the routine. * / switch (mp-> db_type) {case M_IOCTL: / * Execute check and process message * / break; case M_FLUSH: / * Execute check and process message * / break; ... / * Process * / Default: freemsg (mp); return;}}

4.2 S-ICS Private Data S-ICS Instances 38, 38A. . . 3
8N maintains a private data structure in the q_ptr field of its queue. Table 15 below is a sample of that private data structure and associated structure.

[0194]

【Table 15】 #define MAX_FREE_FUNC 10 #define MAX_FREE_ARG 10 struct target_queue_t {struct sth_s * target_sth; struct dev_t dev; uint32 cluster_route_id; uint32 target_queue_index; queue_t * target_read_queue; target_flags; / * Current target state / ability * / short free_func_index [MAX_FREE_FUNC]; short free_func_arg_index [MAX_FREE_ARG]; target_recover_t rem_target_recovery_policy; struct target_queue_t * next, * prev; / * Hash in the same bucket * / / * /} #define TARGET_HASH_SIZE 1024 / * Maximum / minimum hash array collision * /

Target_sth: The stream head associated with this queue instance. This address is
-Maintained when ICS needs to perform stream head updates, such as updating sth_cluster_route_id. dev: device identifier associated with the target stream head. The device identifier is sent initially in mblk during a system or in-kernel stream interface call. Then it is retrieved and the stream head is updated with the cluster path identifier. cluster_route_id: an index into the S-ICS routing table. target_queue_index: an index into target_queue_hash that uniquely identifies this target queue. target_write_queue and target_read_queue: local queue addresses where messages are to be stored.

Local_read_queue and local_write_queu
e: Local queue address indicating the source of the message. Using these addresses, cluster_rout
Hash the remote target index if e_id was not defined in the mblk. This occurs if the mblk was created by a module or driver for local communication. One example of this is during the configuration of the DLPI driver under IP. Many control messages can occur and the S-ICS determines which D
It is necessary which IP to understand how to send data to know whether to send the data to the LPI. target_pending_actions: a counter used to prevent a close race condition. free_func_index and free_func_arg_index: mblk_hea
This is explained in the description of der. rem_target_recovery_policy: A policy structure returned as part of the distributed stream initialization to allow the remote node to understand what steps to take if this target cannot be found due to out-of-sync of the routing table. is there.

The routing table can be out of sync during the migration so that the policy is as simple as contacting the controlling thread and finding the new location of the remote component. When a request comes from a remote node, if the target queue has transitioned between the S-ICS and the associated real target queue address, which is the local node address,
It contains the target queue index used to find the correct S-ICS instance. The use of a table index such as a uint32 value avoids pointer coercion or concerns about whether the node is 32-bit or 64-bit.

[0198]

[Table 16] struct target_queue_index_t {struct s_ics_q_ptr s_ics_instance; / * Different S-ICS processing * / / * Back reference * / struct target_queue_t * target_queue; / * Pointer to real target * /} #define TARGET_MAX_INDEX 65536

A route table is a structural array that defines remote route entries. There is one entry per remote target queue. The index to this entry is targ
Stored in the initiating stream head within the et_queue element and within the stream head. Each entry includes at a minimum the fields of Table 17 below.

[0200]

[Table 17] #define REMOTE_INDEX_ACTIVE Ox0000 #define REMOTE_INDEX_INACTIVE OxOOO1 #define REMOTE_INDEX_MIGRATING OxOOO2 #define REMOTE_INDEX_IN_ERROR OxOOO3 #define REMOTE_INDEX_IN_FLUX OxOOO4 struct s_ics_rem_route_t {int32 remote_index_state; int32 remote_target_index; struct node_t * remote_node; struct rem_s_ics_descrp * remote_s_ics; struct sth_s * source_sth;}

Remote_index_state: reflects the current known state of this element. When the normal state is active, no additional checks need to be performed when routing messages. If the element is in a flowing or transitioning state, the operation may need to be interrupted for a short time.
This interruption can be performed by disabling the module / driver and forcing the operational flow control to be unable to send any messages. In-flight messages need to be cached for a short time. This is independent of the S-ICS and streamhead preview functions, and once the state is removed, all processing proceeds normally.
remote_target_index: An index into the remote target_queue_index array that defines the actual remote target queue read / write address. This index is swapped when a distributed stream is created, migrated or closed. In that case, the routing table is updated to reflect the current state.

Remote_node: points to the node table managed by the controlling thread. Using this information, P-
An ICS header is created. remote_sics: remote S-ICS address and P-IC
Points to the structure that defines how to get there via S. This structure may be pointed to by a number of remote index elements. source_sth: The stream head address of the local stream component. This address is maintained so that error handling is redirected to the appropriate streamhead instance.

[0203]

[Table 18] #define REM_S_ICS_ACTIVE OxOOOO #define REM_S_ICS_INACTIVE OxOOO1 #define REM_S_ICS_IN_FLUX OxOOO2 struct rem_s_ics_descrp {P_ICS_S_ICS_ADDR p_ics_s_fs_ics_ics_ics_data_ints

P_ics_s_ics_address: points to a structure that describes address information specific to P-ICS implementation. This structure allows the P-ICS to target a remote S-ICS.
Contains a set of functions used to create and manage headers. p_ics_s_ics_state: The current state of the remote S-ICS. p_ics_s_ics_data_buf: Data buffer description structure. If the P-ICS uses a recipient management paradigm, this may be a pre-allocated memory pool managed by the P-ICS. If the sender management paradigm is used, this is the remote target address range and p_ics_s_ic
s_offset is the offset into this memory pool to actually put the data. This buffer pool
Implemented by functions defined within the P_ICS_ENDPOINT structure.

The S-ICS private data structure is:
Includes structure types that cannot be known until the driver is executed. At least define and st in Table 19 below
The ructure (structure) must exist.

[0206]

[Table 19] #define S_ICS_INACTIVE Ox0000 #define S_ICS_ACTIVE Ox0001 #define S_ICS_CLOSING OxOOO2 #define S_ICS_MIGRATION OxOOO3 #define S_ICS_IN_ERROR OxOOO4 #define S_ICS_MIDDLEWARE_ERROR OxOOO5 #define S_ICS_ORPHANED OxOOO6 #define S_ICS_OUTBOUND OxOOO7 #define S_ICS_INBOUND OxOOO8 #define S_ICS_FLOW_CONTROL OxOOO9 struct s_ics_q_ptr {int32 s_ics_flags; struct p_ics_data * p_ics_instance; struct middleware_t * s_ics_middleware; struct target_queue_index_t target_queue_index_hash [TARGET_MAX_INDEX]; struct target_queue_t target_queue_hash [TARGET_HASH_SIZE]; struct node_t * node_tbl_ptr; struct sth_s * s_ics sth [S_ICS_HASH_SIZE]; struct s_ics_rem_route_t remote_route_tbl [TARGET_MAX_INDEX]; struct s_ics_rem_route_t s_ics_route_tbl [REM_MAX_INDEX ]; S_ICS_MIGRATION_POLICY s_ics_ migration_policy; S_ICS_RECOVERY_POLICY s_ics_recovery_policy; S_ICS_CONTROL_POLICY s_ics_control_policy;}

S_ics_flags indicates the state of each S-ICS instance and the contents of execution. p_ics_instance: P-IC defined by the previous structure
This is S instance data. s_ics_middleware: Describes how and how to contact middleware and how to handle middleware migration / failure via standard functions. target_queue_index_hash: An index array used to specify the actual target queue structure. This is not a hash array, but there is actually only one element per index. target_queue_hash: A hash table that contains all of the target queue elements. There may be multiple target queues per bucket so that the function of finding the element and updating this hash must be able to follow a doubly linked list. node_hash_tbl: A hash table containing node definitions (potential nodes per bucket).

S_ics_sth: Each upper multiplexer SI
6 is a hash table that contains all of the local stream head addresses associated with a CS instance. This is tracked to find the queue address stored in the stream head for the running measurement instruction. remote_route_tbl: As described above. s_ics_route_tbl: S-ICS routing table. This table describes where all S-ICS instances are in the cluster and how to get there. This is necessary if the message has to be routed for a different node if the application or stream component is to be migrated. If a node fails, the S-ICS will:
Examine the controlling thread or middleware to update its routing table. These updates are triggered by the S-ICS in the system to update all these other S-ICS for changes rather than reacting to path error issues. Note: This structure uses the same definition as remote_route_tbl, except that it does not use index values.

The remaining three elements are policy structures. The contents of these structures depend on the S-ICS embodiment and will not be described in detail here. For each policy, at a minimum, there must be a function vector that the S-ICS can call to handle a particular state.

In order to exchange data between two nodes, the S-ICS must be able to convert the mblk header to and from a byte string transmitted as part of the message sent. This means that various non-pointer related fields must be assembled into a new structure. Table 20 below is one example of a structure.

[0211]

[Table 20] #definePRROT_NORMAL 0x00 #define PROT_ROUTE_ERROR OxO1 struct mblk_header {unsigned short protocol_msg_type; unsigned short b_flag; unsigned char b_band; unsigned char db_ref; unsigned char db_type; unsigned short db_flag; unsigned int source_queue_index; unsigned int target_queue_index}

Protocol_msg_type: This message is S
Indicates whether a successful exchange between ICSs, error recovery or something else. Error recovery may occur if the target queue index embedded in the mblk being sent cannot be found. The error return message contains the source and target queue indexes, and the S-ICS needs to examine the message and determine if it needs to look up a new path, raise an error, or drop the message. b_flag, b_band, db_ref, db_type, db_flag and db_s
ize: Just the value retrieved from mblk, used to reconstruct the mblk header on the remote node. free_func_index: An index into the free (free) function address array. The source node verifies the remote node mapping index and embeds it in the header. Required to handle esballoc () and its variants when a driver or module defines an alternative free function for their data. free_func_arg_index: Same setting, but esballo
Since the argument passed to c () may be an offset pointer to the encapsulated data, or a pointer to a structure that provides the actual argument for the free function, this index May require a specific interpretation. Therefore, further research and consideration is needed to make a good design.

Source_queue_index: can be interpreted in two forms. If the mblk is created as part of a streamhead related routine, such as in a system call, the routine will embed the sth_cluster_route_id in this field. This field is usually the S-ICS route table index so that the route search is quick.
The exception is when the S-ICS cannot update this field, in which case the field has been initialized to be the device identifier (dev_t) associated with this stream head. The routing function can use this field to determine the routing table entry. During error recovery, source_queue_index in mblk is
It is a correct routing table index, not a device identifier. target_queue_index: index into the remote target queue hash table. A data structure that handles policies and the like may include a function vector that is executed via the S-ICS. This function vector allows the S-ICS to maintain the protocol independence that component designers need to develop new functionality. If no function is defined, SI
CS discards the message or blocks the distributed stream if this is a problem and issues an M_HANGUP or M_ERROR message to inform the application of the problem. FIG. 10 shows the S-ICS data structure. FIG. 10 also illustrates data structure interaction with the controlling thread.

4.3 S-ICS Routing The S-ICS 38 routes messages using a unique addressing scheme that spans the entire cluster.
The details of this method are as follows. First, the outgoing message will be described. 1. write, putmsg, putpms passing through the stream head
g, ioctl, streams_putmsg, streams_write and strea
Each sending function call such as ms_ioctl is performed. allocb
Mblk was allocated via () or streams_pu
After the mblk is received from the calling function by tmsg, STREAMS returns mp->b_datap-> source_queue_in
Store sth-> sth_cluster_route_id in mblk header in dex. When the stream is created, the path identifier is sth-> sth
Initialized to _dev. The supported S-ICS routing algorithm understands any form. The framework ends the processing of mblk and sends it downstream.

2. When the S-ICS receives the mblk via the write put routine, the S-ICS receives mp->b_datap-> source_queu
Check e_index and apply hash function. This hash function performs a range check, and verifies whether or not the path identifier corresponds to the array index in the path table. If that doesn't work, apply another hash function and make this a path index by treating the index as dev_t. If that fails, an error recovery mechanism is applied.

3. When the path index is determined, mp-
>b_datap-> target_queue_index is remote_target_index
Is updated using. The source_target_index remains updated so that error handling for messages initiated from remote nodes can be performed. Next, the mblk is sent to the P-ICS, where it is converted to an embodiment-dependent format and sent to the remote S-ICS.
Also, the remote node / S-ICS identifier is retrieved from the routing table and sent to the P-ICS.

4. The remote P-ICS examines the incoming data and determines if it is for the STREAMS subsystem. Otherwise, send a message to the correct target subsystem or thread, such as the controlling thread. Note: If using source management communication, the target address is known and all that the P-ICS needs to do is correct message arrival if the application has signaled to do so. Notify the subsystem. When an interrupt routine is used, it translates the message to mblk and determines which S-ICS to activate.

5. (via read queue put routine)
The remote S-ICS examines the target_queue_index to determine if the index is valid, whether the associated queue has been closed, or has been migrated to a new node. If valid, forward the message to the target queue.
The S-ICS need only hash the registration queue associated with it on every node. These queues are combined with the node table to create a set of correspondences from one end of the stack to the other.
For example, in a split TCP / IP / DLPI stack, messages are sent to IP and DL via S-ICS.
Transferred between PIs. Since the S-ICS is involved in setting up this configuration, it knows which identifiers and queues correspond to each other. Note: Stream-type stacks typically fragment packets by duplicating the mblk header using dupb (), and simply adjust b_rptr and b_wptr to reflect the fragment location. Identifier is mb
Fragmentation is not an issue as it is stored as part of the lk header. The same applies to making duplicates for retransmission.

For an incoming message, ie, a message generated by a remote component, such as a remote DLPI instance, the routing is performed as follows. 1. For each incoming message, the stream head read queue preview function determines whether the message should be forwarded to the controlling thread or directly to the relevant S-ICS. When a message is sent to the controlling thread, the preview function invokes the sth_rput (q, mp) function,
Execute the same framework processing as in the non-cluster environment. If the message goes to the S-ICS, the preview function returns the SI in its sth_cluster_data field.
It finds the stream head corresponding to the CS and uses it to select the route at the time of the message.

2. The preview function is actually putnext (s_ics
Before executing _sth-> wq, mp), embed the index value in the range of mblk. The preview function returns the actual route table and information that were notified when the distributed stream was created.
Index sth-> sth_cluster_route_id sourc
Store in e_queue_index. Depending on the embodiment, this path index should not be changed, as the mechanism does not need to update the stream head of this component. Any route changes should be limited to updating the routing table automatically recognized by the S-ICS.

3. Since the S-ICS is not concerned with the origin of the message, but only with the source and target index values, the rest of the process is similar to the routing case described above.

For error recovery, if the S-ICS (read queue put routine) determines that target_queue_index is not valid, the following processing is performed. target_que
ue_index is always valid unless the stack has been closed or the stack has transitioned. In both cases, the source S-ICS and the local S-IC
A race condition exists between S. When a close or transition occurs, the S-ICS notifies all remote S-ICS of the route change. This can be implemented in one of two ways depending on the following embodiments. The first method is that the S-ICS simply broadcasts the message simultaneously, and all nodes are notified of it,
Assume that CS is updated. The problem is that there is no guarantee that the message in transit will be correctly routed using the new information.

The second method is that the S-ICS determines that a route update is about to occur
, And all the S-ICS being executed are completed, and the process waits for a route update. All S
-When there is a response from the ICS, the S-ICS updates the route and waits for confirmation. In essence, S-ICS uses a second / third phase commit protocol to ensure that all S-ICS are always in sync. Obviously, the second method eliminates the need for error recovery, but path updating requires more action and time. However, this could be tolerated if the route is not updated very often (this is not the case for stream creation as the required S-ICS will always be signaled to be in sync). . Thus, the selection criterion of the second method is where complexity is added and how much time has to be spent in each field.

If a broadcast mechanism (multiple reporting mechanisms are used, since only the S-ICS relevant to the path need understand the change), the S-
The ICS needs to generate an error response to the originating S-ICS. Further, the S-ICS needs to hold information for a certain period of time. That time is the amount of time to ensure that all routing tables are updated so that the S-ICS can inform the remote S-ICS what the fault is. This is optional. Because the mechanism can always return a path failure message, S_I
This is because the CS can check the route table and check the state of the control thread on the distributed stream. The controlling thread may actually be performing fault error recovery as in the case of a node fault, in which case the S-ICS may issue a message. Next, the control thread holds the data until the recovery is completed and controls the operation flow of the starting stream (this point will be further described later).

The only potential problem with such a routing solution is the M_CTL used for inter-module communication.
This is the case when a module or driver generates an outgoing message, such as a message or an M_PROTO / M_DATA message used to issue a hardware driver binding request. In such a case, the message does not include the source and target index values. There are two ways to solve this problem: The first is to simply prevent problems by not splitting the stack between stream components communicating in this fashion. However, this is possible but not practical. The second method uses STREA to record additional information.
Modifying the MS framework measurement routine
(These routines are related to the autopush (1M) function and I_LIN
K, related to I_PLINK, I_PUSH and I_POP ioctls).
In addition, based on the way the stream stack is measured, the P-ICS
It may be necessary to implement the S-ICS as an Nx1 multiplex driver for multiple S-ICS instances feeding a lower multiplexer located above. This point will be described below using an example of TCP / IP / DLPI.

An IP component can also be implemented as a multiplexer for one or more DLPI instances linked directly below it. If the stack is split at the IP / DLPI level, the IP read / write associated with this DLPI instance is recorded. This may happen if the stream is initially created as a distributed stream during an I_LINK operation, or if the stream is split after creation, the IP and DLPI read / write queues will be inserted. Updated to reflect certain S-ICS. Details of the division after the creation will be described later. In either case, the queue address is recorded in the S-ICS private data structure and used as a hash queue to determine the correct path. Now IP and S-IC
This is possible because there is a one-to-one correspondence between S instances.

The sample code in Table 21 below shows the macro used to determine which S-ICS is based on the stream head since private data is accessible to all multiplexer instances. Including.

[0228]

[Table 21] s_ics_private_data = (struct s_ics_data *) sth-> wq-> q_ptr; RECORD_QUEUE (upper_sth, s_ics_sth, s_ics_private_data);

The conditions under which the S-ICS must be implemented as a multiplexer include how the S-ICS is used, how many different protocol stacks are supported by one S-ICS,
It depends on how much S-ICS can maintain protocol independence, how much loading is observed, and so on. This is a matter of judgment, but regardless of implementation, the general concept and design remains the same.

Table 22 below is a sample code that the S-ICS handles both outgoing and incoming cases for message routing. This function uses the message pointer to retrieve the source queue address index. If this index is not retrieved,
The S-ICS starts the error recovery algorithm and must find the path in another way. If the source is found, the S-ICS finds the target queue index using it as a hash key. This target queue index is embedded in the message. The function then uses the target queue index to find the remote node (if the data structure managing the index is cross-linked, this operation is a simple search). Using the node address, the corresponding S-IC
The S instance address is determined and used to send a message to the remote node.

[0231]

Table 22 struct s_ics_instance * find_s_ics (mp, s_ics_data) {struct target_queue_t * target_queue; uint32 source_queue_index; struct node_t * node_address; struct s_ics_instance * s_ics; struct sth_s * sth; mblkP update_mp; Retrieve * / source_queue_index = mp-> b_datap-> source_queue_index; if (! VALIDATE_INDEX (source_queue_index)) {if (route_index = VALIDATE_DEV_ID (source_queue_index)) {/ * Need to update streamhead with correct route index * / / * There is. In this sample code, this update is done using the M_SETOPTS message, which sends the updated route. * / / * Using a regular hash function, the * / / * local streamhead address of this stack is found, * / / * the message is stored directly in the streamhead read queue, and * / / * Therefore, sth rput () updates sth-> sth_cluster_route_id. * / update_mp = allocb (sizeof (struct stroptions)); mp-> b_datap-> db_type = M_SETOPTS; sop = (struct stroptions *) update_mp-> b_rptr; sop-> so_flags = SO_CLUSTER_ROUTE_UPDATE; sop-> cluster_route = route_index; sth = STH_HASH (source_queue_index); putq (sth-> sth_rq, update_mp); goto route_msg;} else {/ * There may be * / / * messages created by the module / driver. Using the local S-ICS data * / / * determine what is the source queue and find the * / / * target_queue element by examining the local_read_queue * / / * or local_write_queue fields. * / if (! (remote_index = FIND_TARGET_QUEUE (s_ics_data))) {/ * Unable to route message, check control thread or discard message * /}}} route msg: / * On error Returns NULL. * / / * Return remote_sics structure address if no error * / return (& s_ics_data-> remote_route_tbl [remote_indexl.remote_s_ics);}

This function uses the message pointer to retrieve the local target queue address. This address is used to hash the target_queue structure, which is then referenced to handle the close race condition.

[0233]

Table 23 target_queue_t find_target (mp, s_ics_data) {uint32 target_queue; / * Retrieve the target queue index and * / / * return a validated target queue or NULL. * / target_queue_index = mp-> b_datap-> target_queue_index; return (VALIDATE_TARGET_QUEUE (target_queue_index));}

When the transition or closing operation has occurred,
If the message is in transit before the initiating S-ICS is updated (potential timing issue), FIND_TARGET_QUEU
E fails. Each local S-ICS instance is considered current for all operations when the mapping function is applied. Each S-ICS instance is updated whenever a close or transition occurs. If a packet is in transit, a time interval is taken and the updated information is not applied. Thus, it is up to the receiving S-ICS instance to interpret the address and take the appropriate action upon success or failure.

Thus, to enable the S-ICS to route messages within the cluster, the S-ICS is modified where modules or driver instances reside and their locations. Need to be informed of when In the example of global port mapping, the S-ICS may open or later stream the stream when a join, unjoin or close command is initiated, or when a distributed stream is created. About, must be notified. Thereby, the S-ICS has sufficient information about all routes and generally eliminates the need to consult middleware and update its route cache. FIGS. 11-13, 14 and 15 illustrate message routing within a cluster range.

4.4 Modification of S-ICS Related Stream Data Structure In the preceding sections, several functions have been defined that retrieve routing information from the mblk header. This information is embedded in the header and reduces the amount of information that is accessed and distributed within the cluster. To achieve this, the following data structure needs to be modified. The datab structure is modified to add additional routing addresses. SV at present
As an R4.2 DDI / DKI version, the datab structure has a number of padding words to maintain cache line alignment of the data. These embedded words are replaced with the following values: uint32 target_queue_index; uint32 target_queue_index; These fields are reused to pass the stream head path identifier on outgoing messages. The path identifier is sth-> sth_d when the stream head is assigned
This is the default value for ev, but is stored in the mblk generated at the stream head. This is done as shown in Table 24 below.

[0237]

[Table 24] if ((mp = allocb (size, pri)) == NULL) {/ * Execute error recovery which is usually bufcall () * /} mp-> b_datap-> source_queue_address_index = sth-> sth_cluster_route_id ;

The performance penalty is minimal, ie, one load and store. It is not worth keeping the streamhead flag to see if the above has been performed, since the cost of checking exceeds the allocation cost. However, the framework maintains a stream head flag to indicate whether a particular queue address should be recorded in a given structure during a measurement operation.

[0239]

[Table 25] #define F_STH_CLUSTER_INTERCONNECT Ox2OOO #define F_STH_CLUSTER_CTL_CHECK Ox4OOO struct sth_cluster_data_t {queue_t * read_queue; queue_t * write_queue;}

The read queue and the write queue are:
Updated during the framework measurement operation to reflect the actual queue above this SICS. For example, since the IP is a multiplexer and may generate messages directly to the driver, which IP set in the queue is being used to generate the message to be able to determine the remote DLPI target You need to know. In some cases this is obvious and in others it is not. For example, when an IP sends a message via its incoming fanout table to the remote DLPI via S-ICS, the S-ICS cannot see q_next to determine which IP sent the message.
This is because this may not be accurate or sufficient to find the correct DLPI depending on the IP embodiment. The structure shown in the following Table 26 is added in the stream head.

[0241]

[Table 26] struct sth_s {... struct sth_cluster_data_t * sth_cluster_data;}

The question now arises as to how to adjust the queue addresses that exist at different nodes within the cluster. Adjust in some cases and not in others. The addresses themselves are not exchanged, but the indexes for structures that can be used to find the actual address are exchanged. For example, when a message is sent to a SICS with a defined source_queue_address_index, the S-ICS uses this index to find the remote target queue address index. This target queue address index is an index into the remote table that defines the queue location.

4.5 Component Communication Communication that can be performed between the S-ICS and the component has been described above. The following sections discuss the details of such contacts from an S-ICS point of view and how they are designed. Further consider how the S-ICS 38 communicates with the PICS 36.

4.5.1 Control Thread Interaction Although the interaction between the two components has been described above, some important points are described below. As far as the control thread 34 is concerned, the S-ICS 38 is just a stream-type driver, so the control thread is nothing more than a stream-type application. Communication takes place using a standard message paradigm.
S-ICS uses a standard message type when it needs to send a message to the controlling thread. However, when the control thread needs to send a message to the S-ICS, the control thread cannot use the standard message type because it cannot identify the source due to stream head bypass.

The control thread uses the in-kernel STREAMS interface to communicate with the S-ICS. Because of this, the controlling thread can specify the message type, since those in use cannot usually be accessed from user space. If the controlling thread was implemented as a user space application,
All S-ICS communications will have to pass data using transparent ioctls. This is because only those S-ICS can be clearly distinguished from inter-module communication.

S-ICS retrieves its node table and P-ICS access information directly from the controlling thread data structure. The S-ICS obtains the addresses of these structures during its creation. By having the S-ICS access the same information that is read-only, the controlling thread does not lock and also transfers new information to all S-ICS.
This data can be updated without having to notify the ICS directly. In this way, the route update or node status can be automatically reflected on all S-ICS. This is not possible if the control thread is implemented in user space. In such an embodiment, there must be a message exchange protocol that handles such updates. Note: The S-ICS can also update the state of the node directly to indicate if the node has failed or the path to it has failed since it is usually the first to be detected.

When the node state is a 4-byte quantity, PA_RIS
This method will not require any locking, since 4-byte writes in an architecture like C is an atomic operation. Within the scope of the routing algorithm, the S-ICS has the ability to probe the controlling thread if a message routing instruction has failed. This fault occurs at the local node where the message was generated. Depending on the occurrence of this fault, source_queue_index, local_read_queue or local_write_qu of local S_ICS
Since the eue information cannot be used (because it is undefined), fetch remote_target_index. The only thing the S-ICS can do to address this problem is to notify the controlling thread that a route failure has occurred and provide some data and messages it has received. In most cases, such problems will never occur if the S-ICS is implemented as a multiplexer in such a way that the upper multiplexer instance is unique to the distribution point.

4.5.2 Middleware Communication The S_ICS indicates that the necessary routing information
When it does not exist inside, it communicates only with the middleware 50.
For example, if the S-ICS detects that a node has failed, the S-ICS may invoke a recovery policy that allows a new remote instance to be automatically created on another node. This remedy does not require the local control thread to participate in the S-ICS as it searches the middleware for an alternate path.
This check is similar to the case where the controlling thread checks the middleware for an address, with the difference that
The point is that the S-ICS provides invalid route information so that the middleware is not updated and the same route is not returned as an alternative route. Could all this be done in the controlling thread? The answer is possible, ST
Depending on the content of the service provided via REAMS and the mode in which S_ICS is implemented, this could be implemented more easily.

4.5.3 P-ICS Communication The P-ICS 36 is a cluster-dependent technology.
S is designed to be treated as a "black box". Communication must be performed using a well-defined API that includes instructions that are limited to the small "glue" layer. This API converts a generic S-ICS instruction into a specific P-ICS instruction. At a minimum, the following instructions must be provided: Data must be able to be sent to and received from S-ICS endpoints within the cluster given a particular node and path data. This means that the interconnect header must be generated automatically by the P-ICS and is not the responsibility of the S-ICS. The P-ICS 36 must support scattering / collection. The scatter / gather operation eliminates the reclassification in question to improve performance and simplify transmission of the mblk header and associated data. That is, when data arrives, the header and the S-ICS must attempt to determine the content of the data.

Therefore, the data buffer must be treated as a set of adjacent bytes. Thereby,
The S-IC does not need to copy data to or from the mblk. For outgoing messages, S-IC
S creates a small buffer area that defines the mblk header so that it can be recreated on the target node. In addition, the P-ICS is given an mp-> b_rptr address and length (mp->b_wptr_mp-> b_rptr). They are,
Treated as raw data for transmission and as the second part of the scatter / gather message. For an incoming message, the S-ICS allocates an mblk via esballoc () and allocates an mblk header pointer that points to all data sent as the second part of the message.
The S-ICS writes the header information contained in the first part of the message into the mblk header. For incoming and outgoing, the mblk being sent may be a message set linked via the b_cont pointer. For transmission, the mblks are all the same message data so as to define the scatter / collect data portion, allocate a small block of data for each mblk header, and only perform sending out all of the header data and user data. Must be type. On arrival, when the message set is decrypted, an mblk header will be allocated for each part and each data part will be esballoc ().

If scattering / collection is not supported,
Except for performance issues, except for the need to modify the protocol headers and address the problem
The above algorithm will remain the same. The mblk header has, for example, a format as shown in Table 27 below.

[0252]

[Table 27] struct mblk_header {unsigned short b_flag; / * original b_flag * / unsigned short b_band; / * original b_band * / unsigned short db_type; / * original message type * / unsigned short m_flags; / * Mblk header flags to interpret * / int32 data_size; / * Actual data difference * / int32 target_queue_index; / * Target queue index table * / / * Index to ** int32 mblk_seq_number; / * Match data Serial number * /

The data now needs a protocol header to be added to it. This requires one of the following: Either require the use of a stream head write offset to pre-allocate header space so that headers can be added, as is a common STREAM action, or reduce the performance but add data to the new buffer along with the protocol header. Need to be actually copied. Table 28 below is an example of a protocol header format.

[0254]

[Table 28] struct data_header {int32 mblk_seq_number; / * * Serial number to match mblk * / unsigned short data_flags;}

If the P-ICS cannot guarantee packet delivery, such as handling congestion control, the control thread and possibly the S-ICS will need to establish a recovery protocol. This recovery protocol may depend on a higher-level protocol that performs recovery from lost messages. For example, if IP fragments are lost,
IP times out and performs its normal error recovery based on the transmission provider requirements. Should be as above, if possible, but for messages such as port updates between control threads, the control thread requires a timeout mechanism and an error recovery mechanism. Of course, this problem does not occur if the P-ICS is managed by the sender.

4.5.4 Flow Control To control how much data should be sent to and received from software components or cards, the STREAMS module and driver invoke flow control somewhere. STREAMS includes a built-in flow control mechanism and several DDI routines for determining flow control status. In a distributed STREAMS environment, these flow control and state functions are:
It must work as intended by the developer without changing the module or driver. To accomplish this, the S-ICS and streamhead preview functions must reflect the appearance of each local stream component of the next module in the stack (if the stack is not split across multiple nodes, The flow control is the same as in a non-clustered environment and does not need to be changed.) The description here is also for TCP / IP / DLP where division is performed between IP and DLPI.
Use the I stack. First, consider the flow control for DLPI.

DLPI has flow control in both directions.
For the sending case, DLPI cannot overrun the hardware. Therefore, so that the hardware can keep the number of backtracks to a minimum,
DLPI sets its high and low thresholds. That is, the ratio between the high and low thresholds corresponds to a large number (not one or two) of packets corresponding to data.
Low ratios cause excessive flow control situations, potentially leading to the majority of data being processed via service routines that add overhead in terms of routing and scheduling. The only thing that DLPI does to trigger flow control is to remove messages from its queue. As a result, the upper module stops the computer () and stores the message locally in a queue, since the upper threshold is exceeded. In a cluster, the S-ICS would have to write a message somewhere and notify the remote node that the DLPI instance is being flow controlled to stop message transmission. If the S-ICS instance is one-to-one associated with the DLPI, the S-ICS can locally queue the message and continue receiving messages until its high threshold is exceeded. SI
The setting of the CS high threshold is important for avoiding control message exchange between S-ICS. The following points must be performed: 1. If the S-ICS high threshold is reached, S-ICS
Receiving of in-transit messages for is continued and they are stored in a queue. 2. The S-ICS sends a low latency message indicating flow control to the remote S-ICS and updates the routing table p_ics_s_ics_status to be flow controlled. 3. The remote S-ICS upper multiplexer instance also putqs to ensure that its high threshold is crossed.
Place the flow control state by storing mblk on its write queue via (q, mp). mblk is
The mblk may be a zero-length mblk whose (b_wptr_b_rptr) indicates that it contains a large amount of data but no data is actually present. This mblk is pre-allocated when the S-ICS is created and stored locally to avoid bufcall () processing.

4. For remote flow control, the local DLPI releases its queue and the local S-ICS queue. When the local S-ICS low threshold is reached, or when it detects that the S-ICS has been re-enabled, the S-ICS sends another low latency message and the flow control no longer exists. Without
Signals that the remote S-ICS can start sending messages. 5. The remote S-ICS starts sending messages when its control mblk is found. In that case, remote S-ICS
Stores it locally, and the module /
Make the driver instance reusable.

If the threshold is carefully selected, S-
The number of control messages exchanged between ICSs is minimal. For incoming messages from the local DLPI to the remote IP, DLPI putnext () the message to the upstream stream component. In this case, it is the local stream head and not the remote IP. DL
If the PI performs a put (), it can determine that the local stream head (rather than the remote IP) receives the message. More precisely, only those messages that must be recognized by the controlling thread are actually stored at the stream head, and all others must be transmitted directly to the S-ICS.
This point is solved as follows.

1. When the driver executes canput () or a variant thereof, the status of the local S-ICS write queue or remote IP is returned. This is achieved as follows. (a) The stream head preview function executes canput () on the S-ICS write queue. (b) Regardless of the success or failure of canput (), the preview function always puts a message on the S-ICS write queue. (c) When canput () fails, the preview function returns the local stream head
Modify the QFULL flag. This will cause all future DLPIcanput () to fail. (d) SI
When CS is no longer flow controlled, it
Notified to local stream head via _queue-> target_sth. This is possible whether the S-ICS is designed with a one-to-one correspondence or a multiple-to-one correspondence. In the latter case, S-ICS is
Target qurur_index-> s_ics_instance to associate with it
From target_queue_index, and target_queue-> ta
Update rget_sth directly.

2. When the remote S-ICS 38A is flow-controlled, only the local S-ICS 38 is flow-controlled. The remote S-ICS uses the upper mux instance ca
Flow control is performed when nput () fails. When this occurs, the remote S-ICS stores the message on its read queue. The remote S-ICS may choose to notify the local S-ICS that it is flow controlled or that it will continue to receive messages until it reaches the upper threshold. The choice of this upper threshold is important in keeping the control messages generated between the S-ICS to a minimum and keeping the application receiving data as quickly as possible.

3. When the local S-ICS receives the flow control message, it can enforce local flow control to use the same mblk technology used on the outgoing path. In this embodiment, the DLPI is designed to be able to handle this situation, and a mechanism is applied that can guarantee that the already received message will be sent out with the release of flow control. desirable.

4. When the flow control for the remote IP is released, the IP makes the remote S-ICS reusable, and the remote S-ICS then releases the remote SI to release the flow control.
Either immediately generate a control message for CS or wait until the lower threshold is reached. The advantage of generating the message immediately is that the amount of time required to release the local message is very small compared to the time to generate the message, send it out and start the data transfer again. Therefore, the delay caused by the stack and the application is kept secret.

The following are some points to keep in mind. Control messages between S-ICS instances must have low latency and must be reliably sent between nodes. When these messages disappear,
The stack remains flow controlled and the application remains in a wait state. If the P-ICS loses a message due to congestion control, the S-ICS needs a request response protocol to handle this situation (which naturally reduces processing performance). This protocol is
It also requires a timeout that is used to cause the request to be retransmitted if the response is not returned quickly.

S-ICS control messages must have low latency. If the P-ICS does not support such a concept, the control message may be chained after the bulk data transfer. If this is possible, SI
S-ICS upper and lower thresholds must be set so that CS does not use too much memory. S-ICS
Design complexity may also increase.

If the P-ICS provides a management mode based on the data source, the control message can always be sent out if the source has a usable buffer space. do not do. Furthermore, the source management scheme generally allows for low-latency messages because the receive buffer can be subdivided or dedicated buffers can be provided for these tasks. Thus, the use of source-managed interconnections makes distributed STREAMS easier to implement.

Although not part of the flow control, the DDI routines noenable () and enableok () are used to control whether the module / driver processes the message. These routines check the QNOENB flag. Normally, this flag is only used on the local queue being accessed, not the next queue on the stack. If used for remote access (although not desired), the S-ICS needs to use similar control message logic for the QFULL flag.

5.0 Creation of Distributed Stream Distributed STREAM consists of (1) a STREAM that exists on one node but uses the mechanism of the entire cluster, (2) a stream executed on a node different from the application, (3) They belong to one of four categories: streams with components split across multiple nodes, and (5) streams that include stream-type pipes where each endpoint runs on a different node. The following describes how each of these types is created.

5.1 Cluster-wide mechanism stream This type of stream is shown in the global port mapping example above. The various components of this stream stack are extended to communicate with the controlling thread or possibly the S-ICS, depending on what services are required. To implement such an extension, the controlling thread must be notified that the stream has been created so that the appropriate extension can be performed. This is achieved as follows.

1. The application opens the stream type driver by opening the file. The format of this file is, for example, / dev / driver_name. If the cluster has provided a single system view of the file system, the open program retrieves the node-specific dev_t structure. dev_t defines the major and minor numbers of the device being opened. By examining this structure, the STREAMS framework makes sure that the driver's stream tab entry qi
Execute the corresponding driver open routine stored in the nit structure.

2. When clustering is enabled through the creation of a control thread, the global structure within the STREAMS framework is set to TRUE and the master control thread's address data is set to TRUE. Becomes effective. During open processing, the framework program checks this global structure. If FALSE, the open process proceeds as in a non-clustered environment. If TRUE, the open program passes this controlling thread the dev_t being opened and requests that it respond.

3. The controlling thread checks dev_t and compares dev_t with the driver assumed to be processed. If the driver does not require special handling, the controlling thread simply returns control to the caller and the open program proceeds as in the prior art. If special, the controlling thread invokes the associated driver open policy.

4. The driver open policy is how the open should be performed, whether the stream should be created as a split stream, the stream is created and all modules are autopushed (automatically queued). Indicates what function should be invoked when is completed. For this stream, this policy indicates that the open should be performed locally and that the following functions must be invoked once the stream stack is completed.

5. In the case of the global port mapping example, a function for extending TCP and IP code is called. Immediately before the framework completes all of its open tasks, this function is invoked (this is performed without the controlling thread having to understand any specification of the function, i.e., the function is Independent of the thread).

This type of stream does not require changes to the driver or module embodiments, and the controlling thread may remain unaware of the underlying protocol implemented by this stack.

5.2 Different Node Streams If the stream stack exists on a different node than where the application is running, the controlling thread 34 determines where this node is and establishes an interconnect. Must be able to. This can be achieved as follows (see FIG. 16).

1. The open program proceeds as in the prior art, but dev_t returned by VFS is
-That of ICS. In addition, this dev_t uses a special device major number similar to the duplicate major number. As with the duplicate device, the minor number is the encoded major number of the open real device driver. The controlling thread knows that the S-ICS is open and retrieves the actual major number by using the minor (dev_t) call.

2. Next, the control thread 34 examines the major number open policy and determines how to determine the node to execute the open. The policy is, for example, one of the following. * Examine the local data structures and find the controlling thread and the corresponding node identifier constructed during the cluster initialization step. * Which thread has the required driver,
Broadcast requests to other control threads to determine which is the best choice in terms of load,
Or * Contact a middleware entity that must have similar data. In each case, the controlling thread will, depending on the embodiment, target node address,
The remote control thread address and the remote S-ICS address can be determined.

3. The control thread 34 contacts the remote control thread 34A and passes it the driver name, ie dev_t, which contains the minor number, now the major number.

4. The remote control thread executes streams_open () in the kernel of the given driver. This open performs the configured autopush activity as in a non-clustered environment. When the remote thread completes the open, it returns a message containing the remote S_ICS address to be used and the associated remote_target_index.

5. The controlling thread now recognizes that it is possible to perform the open and uses standard framework open code to execute the local S-ICS.
Perform an open on the driver.

6. Similar to the function extension described above, the controlling thread stores the function vector to be executed before finally returning from the open routine. The invoked functions send messages to the local SICS using the remote_target_index. This is done for two purposes. First, it allows the local SICS to change the associated dev_tin in sth-> sth_cluster_route_id with the correct route table index. Second
In addition, it allows the remote SICS to update its routing table and signal, if necessary, that a message is being sent between two nodes in the stream. The open operation has been completed as described above.

The advantages are as follows. STREAMS
System calls do not require modification for operation.
No increase in complexity. The coordination work between nodes does not increase. There are no potential new timing issues. Even select () and poll () work as before. When implementing other operating system components,
Similar to S_ICS can be constructed to provide the same type of function dispatch. If this is not done, the file_ops could be re-assigned to be STREAMS file_ops at open time as in a non-clustered environment. Other components could operate as in the alternative environment. When system calls are executed in a non-clustered environment, no performance degradation occurs due to configuration operations.

Since the configuration makes use of existing functionality and algorithms, there is no additional work to support it. Synchronous system calls do not cause undue complexity or modification to any component, since the actual waiting is done on the local system rather than remotely. For example, if an application requests a read of 2000 bytes and only 1000 bytes are received, the system call will pause until the remaining 1000 bytes are received or an interrupt occurs. In an alternative embodiment described below, the migration software needs to retrieve the local stack frame variables and store that information so that the system call can resume in the same state. By generating system calls on the local node via the S-ICS, these system calls do not need to be restarted and operate as in the prior art.

Node failure detection and recovery may be easier to implement transparently to the application because system calls are performed at the local node. This means that local stack frame variables can be reused to restart system calls once the stack is recreated. The configurations described below may not have this capability. getmsg (), getpmsg (), putmsg (), putpmsg (),
poll () and select () do not require modification to operate within this configuration. These functions work even if the stack is local. These functions are selected because only select () is concerned with the file_ops function vector and the rest works as normal system calls.

Disadvantages are as follows. There is a potential for more components to send and receive data, which can cause performance problems. The configuration is ST
There may be two system calls that are not the same for REAMS and non-STREAMS devices, and thus are processed. This is not necessarily undesirable, but means that the troubleshooting solution needs to be aware of this situation.

Alternative Embodiment: The above scenarios and system configurations are not the only ways to implement the present invention. Alternative system configurations and algorithms are discussed below with reference to FIG. This alternative configuration is created as follows. 1. The VFS needs to be modified to recognize that the target device is at a remote node, and a mechanism needs to be created to redirect the open to that node. 2. On the remote node, there must be a kernel thread that performs the actual open, as well as the controlling thread. 3. For the most part, the open proceeds normally, and the controlling thread is involved only when the local component needs to take advantage of the cluster-wide mechanism.

This advantage is as follows. At least on the surface, few components are involved in sending the message. This configuration and implementation may be preferred for non-STREAMS drivers.

The disadvantages are as follows. All STRE
AMS system call and processing functions require modification to handle the movement of data to intermediate structures for routing to remote nodes. For example, read () and wr
ite () uses copyin () and copyout () to move data from user space to kernel space via the uio structure. The user space address that this structure contains must now be changed to a kernel space address and the copyin
() And copyout () must be replaced with lbcopy () with kernel sid. There must be a new kernel thread to handle all synchronization requests, which can lead to excessive context switching leading to performance degradation. This is because the P-IC will have all of the data needed to satisfy the request until it can be processed by the local stream stack.
This is necessary because S cannot be held.

This configuration requires that distinct middleware entities be involved in handling migration and management issues. The above arrangement only required that the execution of these tasks depend on the controlling thread. specfs layer
(I.e. special file interface code layer)
Unless the VFS function vector is replaced with a cluster-specific function vector instead of downgrading the decision function, the non-cluster environment suffers a performance degradation in determining whether it is a cluster. Which configuration to use depends on the modification of the remaining operating system for clustering and the actual processing performance. As a first step, it is desirable to implement an initial system configuration because the work is small and does not require coordination with other operating systems. The rest of the operating system will decide to create an S-ICS equivalent to handle this situation and will be able to avoid problems that may be encountered. If this is not done, the non-clustered environment suffers from performance degradation, which is unacceptable.

5.3 Component Split Stream A split component stream is just a special instance of a different node stream configuration, and thus its open processing is just that special instance. The steps are as follows. 1. Step 1 is the same. 2. When the controlling thread checks the major number open policy, it knows that a split stream configuration should be created. The controlling thread contacts the remote target node and passes it the dev_t with the driver name, the minor number, which is now the major number. As in step 4 above, the remote control thread opens the driver and configures the auto
Perform push activity. When this is complete, the stream
The lower part of the stack has been built. 3. As in the previous case, the local control thread
Open CS, the only difference is that the upper part of the stack, ie any local module, is now
It is pushed onto the CS, the stack is completed, and the relevant message is generated and processed.

There is no significant difference. An example of doing this is to use the TCP / IP / DLPI stack with IP / DLP
That is, division is performed not at the I level but at the TCP / IP level. This allows the IP to be opened if the local DLPI is directly connected below the IP.
As mentioned above, such a division is not preferred, but should be considered in the case of a multiplexer. The partitioning at the preferred IP / DLPI level can also be used to apply to multiplexers whose components reside on different nodes.

It should be noted that the multiplexer opens two devices and uses the I_LINK or P_LINK ioctl
It has been created by combining them using calls. In this case, one component is local, eg, IP, and the other is remote. Thus, the above-described open processing is applied. Opening for IP is done on the local node. DLP
For I, the opening occurs at a remote node with a local stream head, which is actually for S-ICS. When the open processing is completed, the application executes a normal link instruction, and communication between components is performed.

5.3.1 Creating a Split Stream from a Single Stream Stack For load averaging and potential hardware sharing, it is possible to split the stack at some point into multiple components running on different nodes. May be required. In most cases, by using the open algorithm part described above and by using the transition function,
This is achieved. The basic approach is as follows.

1. The controlling thread is notified that the stream stack needs to be split into its components. The message indicating the split also indicates the node where the split component resides (local and remote nodes, or all remote nodes).

2. The controlling thread uses the STRE in the kernel.
STREAM via streams_ioctl () to stream head using AMS interface
Issue framework specific ioctls. The ioctl freezes the stream, making the invocation and creation of all necessary components dependent on where the split is to take place. In the case of a split between IP and DLPI, all that must be collected is the private data and state of the TCP or UDP module. The ioctl returns a migration structure (described below) containing this information. Note: IP / DLP
For splitting at the I level, the IP is finally flow controlled on the incoming path, in which case the IP informs the DLPI to start dropping packets or stop sending this address.

3. When the ioctl returns, the controlling thread migrates the component to each node. 4. In the case of hardware sharing, some components remain local but upper components migrate,
That is, the DLPI part remains, but TCP / IP moves to a new node. Alternatively, the local hardware card probably fails and the DLPI is recreated on the new node. In these situations, more steps must be added.

5. If DLPI is being migrated, I
P needs to have DLPI connected below it. this is,
The controlling thread replaces the old DLPI with the new remote DLPI
Request to replace with an S-ICS instance that can communicate with This can be achieved in one of two ways: (1) The unbind / bind instruction is executed by the control thread, in which case IP to D
It must recognize the initialization message to the LPI and take appropriate action. Or (2) the control thread issues an ioctl to the S-ICS, and the S-I
Modify the framework structure on IP to point to CS. The first approach is probably the safest and easiest to implement. Note: Once the DLPI has migrated, an ARP must be issued so that the remote driver understands the new mapping of physical addresses to logical addresses.

6. When TCP / IP is migrated, the control thread executes the same steps as in the case of the global mapping. The controlling thread creates a SICS upper mux instance, ensuring a unique path determination, then performing a local join and updating the IP fanout table to point to this S-ICS. This ensures that all incoming packets are correctly routed to the remote TCP / IP via the S-ICS.

7. The controlling thread then performs a message exchange with the remote node, activates the path,
Check the _target_index, close the local TCP / IP stack and release their current resources. TC
When the P / IP is migrated, the other VFSs are assumed to be updated by the VFS migration software. If there is only one connected, TCP including socket
Or whatever accesses the IP migrates its instance specific data and recreates its state on the target node. This requires cooperation between the controlling thread and the VFS or socket migration software, at least until the point at which the streamhead mapping structure is recreated.

8. If an error occurs, since the stack is still present until the final confirmation message, the controlling thread clears the allocated resources and simply releases the TCP / IP component. This causes the connection to continue executing as in the prior art.

5.4 Stream-type Pipes Stream-type pipes create problems that ordinary stream drivers do not experience. That is, the stream-type pipe does not include driver information that can be used to obtain the remote node address. For this reason, it is impossible to transparently open a pipe having an endpoint on a different node without external interference. There are two approaches. The first approach is to create a pipe as usual and migrate one endpoint to another node. The second approach is to remap the pipe routine to a cluster-specific function that opens a pipe endpoint on each node. With either approach, there must be an external entity that seeks node data and performs the appropriate work. The question of whether this is a worthwhile approach may be raised regarding the second approach. The implementation of the first approach is preferred because in the first approach, a given migration function is exploited and the application using the pipe receives both endpoints and sends one endpoint to the other application. Therefore, the design of this embodiment is
Assume that the () code produces two stream descriptors with two file descriptors, two file pointers, and corresponding read / write queues cross-coupled to each other. Details of the pipe transfer will be described later.

6.0 Stream Migration A STREAM migration is a global or component-by-component stream stack migration from one node to another. The migration details are described below using two configurations, where the entire stack resides on a node independent of the application and when the stack is split between two nodes.

6.1 Simplification Functions For each module or driver to be migrated,
Developers need to develop two functions that are completely module / driver specific. Within STREAMS, each queue is a pointer (q_pt) to a private data structure that developers can use to store embodiment-specific data for that queue.
r). Since this data structure may contain state-specific information, ie timers, retransmitted packets, states, etc., this data is migrated to the new node and must be used to re-establish the module / driver in its original state. Must. Developers need to write these cleanup functions because STREAMS cannot know where it is pointed.

Such a function must have the following functions. The function must have two capabilities: collecting private data and restoring private data. The data stored in mblk is S
Migrated by the TREAMS framework. The function must be able to be invoked without restrictions by the STREAMS framework. The function must use two parameters: a queue and a pointer to the cleanup data structure. The function uses the queue address to dereference q_ptr and OTHERQ (q)-> q_ptr, which are used to retrieve private data held in the queue. Each function is mb
Allocate enough memory with MALLOC () to hold all of its private data not included in lk. The allocation must use the M_WATOK flag to confirm that the allocation is successful. This memory is freed using FREE ().

Within this allocated memory area, the function lays out all data continuously using ABI (Application Binary Interface) format. For example, if there are four integers and one pointer to a private structure, then the integers must be copied to an integer multiple of 4 bytes of memory and the contents of the pointer copied byte by byte. This allows the data to be transferred as a stream of bytes and reorganized on the remote node. The function must cancel all in-progress timers and record the time remaining. When the component is migrated, the function must restart the timer. The function is bufcal in process
The l () operation must be canceled and the request reissued on the new node. Since bufcall () uses a recall function, this function must be present on the new node. The function address, as shown in Table 29 below,
Stored in the queue_t structure.

[0307]

Table 29 struct queue {struct qinit * q_qinfo; / * Procedures and limits for queues * / struct msgb * qfirst; / * Head of message queue * / struct msgb * q_last; / * End of message queue * / struct queue * q_next; / * next queue in stream) * / struct queue * q_link; / * link to scheduling queue * / void * q_ptr; / * to private data structure * / ulong q_count; / * Weighted character counter on q * / ulong q_flag; / * Queue status * / long q_minpsz; / * Minimum packet size accepted * / long q_maxpsz; / * Maximum packet size accepted * / ulong q_hiwat; / * High threshold for flow control * / ulong q_lowat; / * Low threshold for flow control * / struct qband * q_bandp; / * Band information * / unsigned char q_nband; / * Number of bands * / unsigned char q_pad1 [3]; / * reserved * / struct queue * q_other; / * pointer to the other Q in the queue pair * / int32 (* q_marshall_func) (queue_t * q, struct marshall * marsh_ptr); int32 (* q_demarshall_func) (queue_t * q, struct marshall * marsh_ptr); QUEUE_KERNEL_FIELDS};

The marshall data structure is shown in Table 30 below.
It is as follows. Note: Not all fields are filled by the simplification function, some are filled by the SREAMS framework.

[0309]

[Table 30] struct marshall {struct streamtab * str_tab; caddr_t str_frame_wk; caddr_t q_private_data; mblkP read_mp; mblkp write_mp; mblkP read_mp_private; mblkP write_mp_private;

Str_tab: current stream tab entry for this module / driver. The Current Streams tab reflects changes in dynamic function replacement. This is filled out by the framework. str_frame_wk: all S for this component
Points to the memory containing the TREAMS framework data. This information includes the local stack frame structure for system calls, synchronization queues, etc. being processed. q_private_data: Pointer to the structure allocated by the simplification function. It must include both read and write q_ptr data. read_mp and write_mp: mblk chains for messages stored in the queue waiting to be processed. These are created by the framework. read mp private and write_mp private: mblk chains currently held by the module / component. These may be messages waiting for retransmission. Note: All mblks in transition are b_next / b
Combined using the _prev field. All mblks bound via the b_cont pointer are assumed to be part of one message.

When the structure is completed, the controlling thread transmits the data to the remote node, where the inverse function is applied to the data and the structure is recreated on that node. The developer can recreate the read and write queue q_ptr structures and q_private_data through FREE ()
Is responsible for releasing.

6.2 Other Entire Stack In the previous section, two embodiments were shown. In an alternative embodiment, there are no components of the local STREAM framework, so the migration consists of migrating the stack normally and simply updating the VFS layer on the two nodes. Therefore, the following description focuses on the first configuration and describes how it is migrated. This will generally be used as a basis for the transition.

1. The controlling thread gets the S-ICS write queue and causes the flow control on the "write" path to be placed on the local S-ICS instance. This prevents further messages from being sent from this node to the remote node. 2. The controlling thread notifies the remote controlling thread that a migration will take place and uses the remote target index to identify the component. 3. The remote control thread disables the driver's local streamhead and disables the streamhead preview function. Also, a QFULL flag in the stream head is set. Between these two steps, all downstream activity is stopped sending data to the local S-ICS or local streamhead.

4. The remote control thread invokes the normal migration algorithm for these components. When this is complete, the remote control thread sends a message to the initiating control thread that a transition is currently occurring. 5. When the migration is complete, the remote control thread sends a message to the initiating thread to confirm the migration status and new routing table data. 6. The controlling thread then uses the new information to
Update the CS route table. 7. At this point, the controlling thread releases the flow control and allows the S-ICS to send out the message. Release of flow control allows sending a message to the remote S-ICS,
Can send messages from the stream stack.

6.3 Migration Algorithm There is no real difference between migrating the entire stream stack and migrating a single distributed component. The transition steps are as follows: 1. The controlling thread gets the S-ICS write queue and causes the flow control on the "write" path to be placed on the local S-ICS instance. This prevents further messages from being sent from this node to the remote node.
To do this, the control thread utilizes a flow control mblk stored within each S-ICS instance. The control thread is on the S-ICS write queue
Execute putq (wq, mp) and write this state in the S-ICS.

2. The controlling thread informs the remote controlling thread that a migration will take place and sends the remote target index (r
emote_target_index) to identify the component. The remote target index is determined by retrieving a route table entry. Routing table
To retrieve an entry, the controlling thread needs to identify the S-ICS that manages this stream stack. The S-ICS address is found in the local component stream head along with the cluster_route_id. Using these components, the correct routing table entry is indexed and the data is retrieved.

3. The remote control thread disables the driver's local streamhead and disables the streamhead preview function. The remote control thread uses the corresponding targe based on the remote_target_index
Identify the relevant stream head by finding the t_queue entry (this algorithm is described above). ta
The target_sth is extracted within the range of rget_queue. Using streams_loctl () in the kernel, the remote control thread reverses the process to set up the preview function. This prevents low level modules / drivers from bypassing the stream head and putting messages on the local S-ICS.

4. The remote control thread also sets the QFULL flag within the stream head read queue. This prevents lower-level modules or drivers from trying to store messages in the stream head. When this condition is detected, the modules / drivers are flow controlled or discard the message as part of their standard protocol instructions. In any case, the message no longer flows upstream.

Next, the remote control thread contacts the target remote node. The remote control thread causes the target remote control thread to execute a normal open instruction and create the same stack configuration that exists at this node. When this is completed, the target node responds with appropriate messages and data to initiate the data exchange.
Note: When the target node control thread creates the stack,
Causes the STREAMS framework to get all synchronization queues for the stack instance. This prevents any activity from occurring on these queues until the transition is completed.

6. At this point, the remote control thread is convinced that migration is possible, although not guaranteed. The remote control thread gets all stack synchronization queues starting from the lowest component.
This causes "inverted" because all in-flight service routines stop running, and the service routine cannot get the status that the service routine has completed its execution and the required synchronization queue. "Place in one of two states: placed in execution path. When this happens, the STREAMS framework moves all in-flight service routines and associated messages to the target stack. Note: When a STREAMS implementation executes service routines outside the process area, those routines and associated data are retrieved and migrated. Also, if a software interrupt is used to execute a service routine, all service routines need to be restarted as new software interrupts when the transition is completed. This means that there must be a list with service routine requests until the stack is restarted.

7. The controlling thread now allows the completion of all disabled system calls. All disabled system calls are frozen, their state is recorded and dispatched to a new node for restart. If the application is stuck while the stack is moving, the framework must perform content switching within the local system. To do this, based on the embodiment of the file system, for example, update vnode v_rdev to point to the dev_t associated with the S-ICS, and likewise update vnode v_stream to S-
Replace with the stream head pointer of the ICS instance. These steps add the associated file pointer f_data field pointing to the vnode structure to the S-ICS
S-ICSvnod generated during open sequence
Achieved by replacing e instances.

8. The controlling thread organizes STREAMS framework data for this stack.
This includes the results of all intermediate system calls, stream head flags, internal system call processing structures,
Synchronous queue data and pending requests, all queue flags, all dynamic function replacement data, all dynamic function registration data, all messages and their bandwidths,
Includes all modifications related to M_SETOPTS and so on. This data is sent to the target control thread which rebuilds the stack framework. 9. Next, the controlling thread invokes a simplification function for each stream component.

10. The structure is passed to the target control thread, which invokes a simplification function to flip the process on this stack. In so doing, it may not be possible for the target node to regenerate all of the data and messages due to potential memory constraints. This is handled as follows. If the migration fails, the remote control thread detects the failure, restores state, and cleans up allocated resources. next,
Wait for the starting control thread to select a new target node and start a new process. Another method is STRE
Let the AMS framework create a memory recovery algorithm. This increases the work, but reduces the transition complexity for the controlling thread.

11. When the target node has completed its operation, the target node notifies it to the control suite via the remote control thread. The remote control thread then tears down the existing stream stack and cleans up all resources. The target control thread then releases all synchronization queues, allowing the stack and pending system calls to resume.

12. The controlling thread then updates the S-ICS routing table with the new data and releases flow control by removing the mblk. This allows communication to start from a new node. If the controlling thread detects an error at any point, it will undo what was done and restart the stack. This is possible because the transition process does not change any aspect of the stack until the transition is complete. Note: In many situations, migration can be caused by node or hardware failure. This means that the controlling thread is the initiating entity and not the cluster management thread. Furthermore, depending on the stack configuration,
Some migrations may be performed between the control thread that detects the failure and the S-ICS where the application coexists. The recovery operation may include going to a new interface card on the same node or a new node. In any situation, all of the above steps may or may not be relevant to any transition situation, so these steps must be selectively applied on a case-by-case basis.

7.0 Stream Synchronization Hewlett-Packard and many other vendors' S
The TREAMS embodiment currently supports a variety of different levels of automatic framework synchronization. Create their own synchronization mechanism for data, and what
These levels have been created to relieve module or driver developers from having to control whether put or service routines execute in parallel. These levels have five levels: Queue, Queue Pair, Module, Other and Global. These are defined as follows:

The queuing synchronization level provides most of the parallelism. It serializes access to the queue so that only one instance of the queue's put or service routine is executed at a time.
Different queue instances have their put or servi
The ce routine may be executed in parallel on different processors. The queue pair synchronization level is read put, re
Serialize access to read / write queue pairs so that only one of the four functions ad service, write put or write service is performed at a time. It is also possible that different queue pair instances execute in parallel on different processors.

The module synchronization level serializes access to all of a module's queue pairs or instances. That is, only one instance
readput, read service, write put or write servic
Only the e-routine is executed at one time. Different modules can execute in parallel on different processors. "Other" synchronization provides a group of different modules to execute only one function at a time. This means a cooperative task between multiple drivers / modules. When used in practice, it can provide a simple synchronization mechanism for complex applications. Global synchronization does not provide parallel processing. Only one of the modules configured as global synchronization
Two modules are allowed to run at once.

The most important feature common to all synchronization levels is that without the need for additional locking mechanisms,
Each level provides a different degree of information sharing. For example, "other" synchronization allows different modules to easily share data without having to coordinate mechanisms such as spin locks. The framework provides a simple identifier that indicates what is accessible, so that all accesses are automatically serialized without using additional spinlocks or signals.

The actual implementation of these synchronization levels is achieved by processing the parent synchronization queue, which is stored at a different location in the stack or kernel. For example, queue and queue pair synchronization store the parent's synchronization queue that is local to the queue, while module synchronization stores the parent synchronization queue address in the fmodsw table. "Other" and global synchronization are unique only to the local kernel instance. Thus, the cluster configuration need only distribute drivers and modules that support queue or queue pair synchronization. In this case, this does not remove other levels from the operating system, but they can only operate within the local node.
For example, the TREAMS type TCP ARP module
Because it uses module-level synchronization, it cannot be distributed. In most cases, this is not a problem. This is because the IPs that communicate with the ARP are on the same node and the target drivers are not distributed.

8.0 Driver / Module Switching Table When a module or driver is loaded into the system, its stream tab structure and configuration data are
Stored in a global switch table for quick and easy access by the REAMS framework. In most embodiments, the drivers are stored in the dmodsw table, and the modules are stored in the fmodsw table. In most cases, these tables are completely independent and do not need to be synchronized within a cluster. The exception is when a module or driver uses a module, global or "other" synchronization level. For module synchronization, this is usually not a problem.
Because most modules do not require data to be synchronized between all nodes, they only require synchronization between instances local to a particular node. This is a problem for global and "other" synchronizations. These levels are
Provide independent modules and drivers to coordinate activities. To support these levels, ST
The REAMS framework needs to provide a distributed locking mechanism. Implementation of such mechanisms is not difficult, but they add latency and complexity to all operations.

When creating such a mechanism, the following must be considered. Assuming that the synchronization lock is now remote and the operation is being performed separately from the interrupt stack, the request is a ST that records the acquisition request and later performs that execution.
It must be stored in the REAMS daemon's queue.
If the underlying P-ICS does not guarantee message delivery, the distributed locking mechanism requires the current lock holder to wait for an acknowledgment message from the target node before formally releasing the lock. I do.
This requires the current holder to record a retransmission timer and handle potential message loss or node failure. Periodically, as there is a single point of failure for such a model, the current holder needs to simultaneously notify other lock management threads where locks are currently and are all well. This is done like a heartbeat, allowing another thread to perform recovery if a heartbeat is detected.

A switching table can also be used to store and retrieve additional information that may be useful. For example, by examining the stream tab structure, it can be determined whether the driver is a multiplexer. This can be included in the open policy that determines the appropriate node to perform the open. Further, by examining the cluster configuration data, it can be determined whether the lower mux is on the same node and whether this is important. The controlling thread also examines the stream tab and autopush data stored within the SAD to determine what the stack will ultimately look like,
It can be determined whether it is appropriate to split the stack or to generate the entire stack on a node. Switching table entries can also be a global repository for errors, H / A and migration recovery policies. These policies can also be updated by the cluster management thread, as well as how dynamic permutations are performed.

9.0 Distributed STRLOG STREAMS provides a log driver that allows a module or driver to send out logging messages. These logging messages are used for error logging or debugging.
DDI is a standard utility that performs this, strlog
() Is defined. Within the kernel, strlog ()
Write a message to the driver's read queue and schedule a log driver to process it. stre
The log driver stores the message on its stream head read queue until either rr or strace is executed by the user thread to retrieve the message. To avoid consuming all of the kernel memory, the log driver limits the number of messages it stores and releases all excess messages.

Within the scope of a cluster showing a single system perspective, strerr or strace is run on one node, and the cluster manages all log data for all nodes.
Returns the message and identifies the node that generated the log message for quick fault isolation. The problem is strlog
(2) A method of delivering all cluster node messages to a specific node without modifying the definition or usage. The solution is shown below with reference to FIG.

When the strace is issued, the instruction
Open the driver and synchronously retrieve messages from the driver until it is stopped using getmsg (). This is performed as follows. 1. The log driver implementation is changed as follows. This may be interpreted as a violation of the cluster design rules, but is an exception because the log driver is part of a STREAMS implementation. (This also applies to SAD drivers). The log driver is converted to a multiplexer driver. The lower part is basically the current log driver embodiment, and no changes are required to operate in this environment. 2. The upper part of the upper mux, whether or not the node is within range of the cluster, is the STREAM in the kernel
Downward mux instance using S interface
str_dev_lookup (), streams_open (), and streams_io
Create ctl (I_LINK). 3. The control thread that recognizes that the upper mux is open contacts all control threads within the cluster managing the log driver and establishes a separate connection and S-ICS instance for each node. Establish and connect each S-ICS under the upper mux. by this,
Allows the upper mux to pre-pend the node identifier for each log message before delivering it to the command.

4. When a connection is first made, each node automatically forwards all pending messages to the current node. New messages are automatically routed via standard message routing algorithms. note:
A component can make excessive strlog () calls, causing excessive kernel memory consumption. In a non-clustered environment, the log driver is arranged to discard the message when N are queued. Also in the upper multiplexer, N * M
It is necessary to formulate that no more than the node message is allowed and more are discarded. 5. If logging is used frequently, the controlling thread may pre-execute an S-ICS connection between all nodes and possibly one or two cluster management nodes. This allows the cluster to have a STREAM component
Messages can be automatically collected, and their capabilities can be incorporated into management mechanisms such as Hewlett-Packard's Openview.

10.0 Distributed SAD SAD is an abbreviation of STREAMS Administration Driver and means a stream management driver. The SAD is created to perform STREAMS driver management on the node. The primary use of SAD is to handle autopush requirements and query the system to determine if a set of modules is within range of the system.
Within the scope of a cluster, applications using SAD run on local nodes or across the cluster. If running only on the local node, the application and the SAD would not need to change. When performing cluster-wide operations, both the application and the SAD require modification.

A management application such as autopush (1m) uses iocts to communicate with the SAD driver. Assuming a single system perspective is maintained, the implementation of ioctls must be propagated throughout the cluster. log
As with the driver, the SAD may be configured in a multiplexer or a one-off approach to do message distribution may be used since the SAD has no ongoing messages. Using this approach, SA
The D driver is modified during cluster initialization using dynamic function replacement.

The modified put / service routine notifies the controlling thread that takes action and returns a result. Because SAD uses ioctls, applications can safely wait until M_IOCACK is generated. Communication is performed using an in-kernel STREAMS interface to a known stream position known at initialization. In most cases, this task is very simple and can be included in a cluster management solution.

11.0 Distributed Streamhead Requirements When a stream stack is split between multiple nodes, the streamhead content and related functionality must be reflected between these nodes. Thus, the controlling thread examines all stream head specific messages and takes local actions, such as recovery from the M_HANGUP situation, or records a new stream head write offset and sends the message to the remote node Do one or the other. The following is a list of message types and options, and the corresponding actions taken in the controlling thread.

M_HANGUP: When this message arrives,
sth_rput () places the stream head in an error state (ENXIO). It also restarts pending system calls and polls for events. Within the range of STREAMS, stream head flag F_STH_CLUSTER_CTL_CHEC
The code must be extended to verify whether K is set. If set, do not perform any normal steps, store the message in a queue, restart the control thread, and perform error recovery, or simply send the message to the remote node and reply. Have them wait.

M_IOCACK and M_IOCNAK: used by control thread or application ioctl
Is the response to Sth_rput () must be modified to check whether the controlling thread wants to use it. If it is set, the controlling thread restarts, checks the route data and ioc_id fields and
Determine if l belongs to it. If not set, it examines the target path, streams_pu
S to forward message to remote node using tmsg ()
-Send message to ICS.

M_COPYIN and M_COPYOUT: These are clever messages. They are used to pass addresses to the stream head and request that data be copied in one direction. Since the target address space is not aligned because it is on a different node, the controlling thread and the S-ICS must cooperate to operate properly. Because these messages can only be generated during ioctl processing, this information can be used to solve the problem. For any message, the driver specifies a target address to copy data to and from cq_addr.

In the case of M_COPYIN, the format is one message block with the target address. To do so, the controlling thread records at the target address the fact that the driver is executing an M_COPYIN instruction. next,
The controlling thread uses the message as an M_CTL message so that the S-ICS knows it is a special request.
-Transfer to ICS. The remote S-ICS receives this message, allocates local kernel memory of cq_size, updates cq_addr, and sends the message as an M_COPYIN request to the local streamhead performing copyin (). When this is complete, the S-ICS sends this data back to the initiating S-ICS that sends the data to the controlling thread. Next, the control thread moves the data to the driver's original address.

With respect to M_COPYOUT, the driver uses M_DATA
The data is supplied within the range. This message chain is forwarded to the remote S-ICS once the controlling thread has written it in the M_CTL message,
The CS recognizes the existence of a special processing request. S-ICS
Target cq to reflect the new local address
Update _addr and send a message to the stream head to execute copyout ().

M_IOCTL: Used by the stream head when implementing a pipe. In that case, M_IOCNAK must be generated and returned to the remote sending application. It should be noted that the S-ICS forwards these messages directly to the controlling thread to determine it. M_ERROR: Similar to M_HANGUP, there may be a recovery policy. If the flag is set, the control thread makes a decision; otherwise, the process proceeds as usual. M_SIG and M_PCSIG: Used by drivers and modules to generate signals for applications. If the controlling thread does not use the signal, the message will be sent automatically and will not be interpreted. The flags are checked to determine whether to process in sth_rput ().

M_PASSFP: This message may not be meaningful, as it is assumed that the file pointer of the pipe is moved from one endpoint to another in order for I_RECVFD to take place. If the pipes are on different nodes, there is no need to transfer this information, the file pointers are only node-specific, and have no function of addressing between nodes. M_FLUSH: These messages must be processed locally and remotely. The streamhead preview function or control thread must send the message. M_SETOPTS: There are a few options that the driver may set on the stream head. If the remote driver sets these options, the data needs to be reflected or modified to reflect local values. Depending on the embodiment of the STREAMS, the stream head preview function or the controlling thread modifies this message with the correct value while executing copyb (), while transferring the Perform interpretation using the modified values. The following is a list of fields and applicable actions.

SO_READOPT, SO_WROFF, SO_MINPSZ, SO_MA
XPSZ, SO_MREADON, SO_MREADOFF, SO_ISTTY, SO_ISNTTY,
SO_NDELON, SO_NDELOFF, SO_BAND, SO_TOSTOP and SO
_TONSTOP: sent to remote stream head as it is,
No local modification is required. SO_HIWAT and SO_LOWAT: Need to update local streamhead values, which are sent unmodified to the remote streamhead. Those values are recorded and while the flow is controlled, the local node reacts as in a non-clustered environment with respect to the amount of data that is allowed to reside on the local stream head.

SO_COWENABLE and SO_COWDISABLE: Not very meaningful within a cluster environment because the driver is remote and the message being processed arrives in the kernel. Whether these should be sent depends on whether the driver / module needs knowledge of the specific actions to take. SO_FUNC_DISABLE and SO_FUNC_ENABLE: Used to control dynamic function registration. The controlling thread is M_SETO
The index within the PTS must be checked and mapped to the remote function registration array. If the array is not set up to have the same layout for each module / driver, the message needs to be updated. In addition, the referenced function must be within the scope of the remote kernel image, and to recognize this, the driver / module calls str_instal
Must be introduced through l (). These bits cannot be interpreted locally because the data conversion or copyin / checksum is performed at the remote node based on the calling application.

12.0 Effects on Copy Avoidance Copy avoidance and copy_on_write (COW) operate in a distributed STREAMS environment, as in the prior art. Copy avoidance and copy_on_write must be able to be used in a distributed STREAMS environment and the current V
No changes are required to the M embodiment. In order to show the possibility, copy avoidance, that is, page remapping will be described as an example. Page remap is a function that remaps a kernel page from one space to another, for example, from kernel space to user space or from user space to kernel space. Within a distributed STREAMS environment, there are two STREAMS stacks that communicate to the driver. (Note: For stream stacks, if functions are just shifted to remote nodes without passing through the local stream, copy avoidance and COW are not possible, leading to performance issues. Must be taken into account when choosing a). The node where the application resides is the physical driver and is in communication with the local interconnect driver that creates paged data that can be easily remapped to user space. The remote node where the actual application driver resides creates paged data where possible. The in-kernel thread must send this data to its local interconnect driver instance, so there is no need to remap the data (if necessary, existing kernel-to-kernel page remapping functionality can be used. ).

With respect to COW, since the interconnect driver must be developed to always have this function,
It is possible to have this mechanism on the application node. For remote nodes, there is no user memory space block to lock, so copy_on_wr
ite is not required. Socket that checks whether the driver supports checksum offload (checksum_off_load) that enables execution of copy_on_write and determines
For applications like this, this can be done without modification, but STREAMS must ignore the result. The simplest way to do this is for the interconnect driver to send an M_SETOPTS message containing the new bits. With the message, the stream head recognizes that the STREAMS is distributed, and sets the F_STH_COW flag within the range of the stream head as one of the actions to take.

13.0 MP vs. UP Emulation Hewlett-Packard and many vendor STRs
EAM embodiments support multiprocessor (MP) and uniprocessor (UP) emulation drivers and modules. Hewlett-Packard's STREAMS does not support mixed MP and UP modules / drivers for stream stack instances. Therefore, in the embodiment of the present invention,
When the UP module is pushed onto the MP driver,
Drivers and all other modules are translated to use UP emulation. Conversely, from the stack, which potentially contains all MP modules / drivers,
The process of removing the P module is not performed.

In order to split the UP stream stack, it is necessary to send the conversion information executed for open () or I_PUSH ioctl to the remote stack segment. To do this, a new streamhead ioctl, I_UP_CONVERT, needs to be created. This ioctl is created by using M_IOCTL. This M_IOCTL is a stream_ioctl (I_UP_CONVE) that invokes, by the controlling thread, an algorithm that converts all existing modules and drivers for the UP.
RT). Of course, a simpler solution is to not support UP emulation in such an environment.

14.0 Dynamic Function Rep
lacement) Dynamic function replacement is used for various tasks. This functionality does not require modification to continue working within the cluster. The only exception is when the stream is split between nodes. In this case, the alternate stream tab
The index to the entry is not the same between nodes. This problem is solved by having the controlling thread perform the action on the correct node using the corresponding index value for that node. This is ioct
l This is implemented by modifying the embodiment as shown in Table 31 below.

[0356]

Table 31 case I_ALTSTRTAB_ACTIVE & Oxff: / * Enable to call this ioctl on the lower mux * / close_out_check ((RWEL_ERROR_FLAGS & (-F_STH_LINKED)), sth); error = copyin (* (caddr_t *) data , buf, sizeof (struct straltactive)); if (sth-> sth_flags & F_STH_CLUSTER_CTL_CHECK, I_ALTSTRTAB _ACTIVE) {error = func_replace_probe_ctl (sth-> ctl_address, buf); ioctl_dequeue_osr_wakeup (sth, osr); osr_altstractive (osr); break; case I_ALTSTRTAB_FUTURE & Oxff: close_out_check (RWHL_ERROR_FLAGS, sth); error = copyin (* (caddr_t *) data, buf, sizeof (struct straltfuture)); if (sth-> sth_flags & F_STH_CLUSTER) (sth-> ctl_address, buf, I_ALTSTRTAB_FUTURE); error = func_reg_probe_ctl (sth-> ctl_address, buf); ioctl_dequeue_osr_wakeup (sth, osr); return (error);} error = osr_altstrfuture (osr); break;

[0357] func_replace_probe_ctl () sends and receives messages using a buffer that has been updated to reflect the appropriate index value. The controlling thread actually examines the remote controlling thread, executes instructions on that node, and returns a result. Note: Since this code is not on the main data path, no performance degradation is observed from the application or other stacks. Also, since everything happens at the stream head level, threads can sleep while waiting for the response of the controlling thread.

15.0 Source-Type Interconnection Requirements No significant changes need to be made to operate a distributed stream in a source-type interconnect environment. STREAMS
Will work normally as far as memory management and use of allocb () / freeb () are concerned. When attempting to send a message, the data needs to be iomapped so that the interconnect has an IOVA (ie, I / O virtual address).
This mapping operation requires about twelve instructions, but can be avoided by creating a few tlb entries assigned to the interconnect kernel instructions.

Since the data always exists in the kernel,
There is no need for a secure operation as in the case of being in the user space. If it is a COW (copy_on_write) buffer,
The user data is already reserved via the COW call so that there is no difference. On the receiving side, the S-ICS driver allocates and reserves enough memory to satisfy the distributed STREAMS requirements. All user data and mblk headers arrive at this memory and es
balloc () is executed. The free function is an interconnect API call that frees memory and acknowledges receipt for the remote node.

The page remap function does not work between the local node and the application because the receive buffer area is reserved by the interconnect. To solve this problem, modify the application so that the S-IC
Provide a receiving buffer area so that S can recognize this stream instance. This requires the application to send an additional ioctl indicating the address and buffer length. STREAM
S can continue to allocate mblks from this area using esballoc (), but must mark them as already existing in the user area. The streamhead routine verifies that such a case has occurred and skips the data movement step within the routine. To prevent performance degradation, the streamhead routine should be simpler, so it only remaps STREAMS f_ops to be a cluster-specific vector that understands this behavior. Similar changes can be made on the write side. note:
For sockets, this can be done in a similar fashion, depending on whether STREAMS can move the data.

The only problem is interconnect_ctl_block, but the S-ICS sets up memory in the kernel and modifies it so that other interconnect APIs can recognize it. In most cases, source-based interconnect embodiments
It has many advantages over the FC4 embodiment. It does not suffer from congestion control. It is source-managed and does not overrun the buffer. It does not drop packets. It has a built-in low-latency mechanism and a group notification mechanism so that global port mapping and other database synchronization instructions can be implemented faster and more easily. Overall, the source architecture is the best architecture that implements this environment, while other architectures require additional design and complexity to address the issues described above.

As described above, the principles of the present invention have been described with reference to the specific embodiments. However, it is apparent that the structure and details of the present invention can be modified without departing from such principles. Would.

The present invention includes the following embodiments as examples. (1) A multi-computer system having a distributed STREAMS function, wherein one or more systems
A cluster comprising at least two nodes each having a computer including a processor unit, a local memory, and an input / output subsystem, interconnected via a data communication interconnect subsystem; An operating system running on each of the processor units and including a STREAMS messaging mechanism used in the implementation of one or more of networking protocols, client / server applications and services; A software application running on at least one of the node's system processor units under the control of the operating system, and a first initiating node of the plurality of nodes. A means for creating a distributed STREAMS instance independent of the application and the STREAMS messaging mechanism; and a plurality of networking protocols, client / server applications and services on the first node transparently. Means for migrating at least a portion of the distributed STREAMS instance to the second target node to perform a selected task of a software application on a second target node of the nodes. system.

(2) The computer in which the STREAMS message transfer mechanism provides a bidirectional data path between a user process and a device driver via a module which is an intermediate element that can be dynamically added or deleted. STREAMS located in each of
A data structure, wherein the device driver resides in a system kernel portion of the operating system and includes peripheral devices for enabling data received by the device driver to move toward a user process. Operating the device driver to control and transfer data between the kernel and the peripheral device, wherein the STREAM data structure includes a stack stored in local memory and controls execution of module and driver functions. The multi-computer system according to (1), wherein the multi-computer system holds a set of function pointers for performing the operation. (3) The above STREAM according to the cluster parameter
A set of previews including control thread means included in the operating system for communicating with and controlling the S message transfer mechanism and a dynamic function replacement function for multiplexing messages between multiple components of the system. Means for defining a physical cluster interconnect driver or P which provides basic communication functions between nodes in the system under the control of the control thread and in accordance with the preview function.
(2) wherein the device driver includes ICS.
2. The multi-computer system according to claim 1.

(4) Application STREAMS
The above (3) further comprising means for defining a STREAMS software interconnect driver or S-ICS for moving messages between at least two components of the system components: stack, control thread, P-ICS and preview function. )
2. The multi-computer system according to claim 1. (5) The operating system and the S-IC
The multi-computer system of claim 4, further comprising means for defining a middleware thread that operates to process and pass messages between S. (6) The means for creating a distributed STREAMS instance is a file system located in the operating system of each node and executing an open function in response to a request from the application to initiate an open command for a STREAMS messaging mechanism. Means for storing the device number of the interconnection driver in the first initiating node in association with the target device number, having a function of opening the selected driver on the first node, and providing the STREAMS message transmission mechanism with Means included, means for re-associating the stored device number with a node address for the second target node, and a function of transmitting a message including the application open request and the stored device number to the target node. Holding Means located in the interconnections driver, and a multi-computer system according to including a file system, located within the target node has the ability to open the target driver in response, the (4) in the message.

(7) The multicomputer of (6), wherein the operating system on the target node includes means for returning a message containing the stream head address for the target driver to the interconnect driver of the initiating node. ·system. (8) The means for migrating at least a part of the distributed STREAMS instance to the second target node includes a means for transmitting at least a part of a stack on the first node to the second node. The multi-computer system according to (2). (9) A means for transmitting an ioctl indicating execution of migration to the STREAMS message transmission mechanism,
The TREAMS message transfer mechanism uses the ioctl
Sending a message to the target node including information on which stack from which node the stack on the starting node can be used to regenerate the stack on the target node, The multi-computer system according to the above (8). (10) The multicomputer of (9) above, wherein said STREAMS message transfer mechanism includes a private data structure for a module represented by a stack on a target node, performing a simplification function on a portion of said data structure. system.

(11) A computer including one or more system processor units, at least two nodes each having a local memory and an input / output subsystem, interconnected via a data communication interconnect subsystem. A STREAMS messaging mechanism running on each of the system processor devices in the cluster and used in implementing one or more of the networking protocols, client / server applications and services. An operating system, including a software application running on at least one of the node's system processor units under the control of the operating system to perform tasks or resolve problems. Comprising a ® down, in a multi-computer system having a distributed STREAMS function, S
Starting a control thread on each node in the cluster, including determining where the TREAMS driver is located within each node, and determining which mechanism is associated with each driver; Setting a flag in the STREAMS message transfer mechanism on each node to indicate that clustering has been performed, and each driver to uniquely represent a particular driver on one node in the cluster. Assigning a file name encoding a major number table identification and a minor number parameter that selectively identifies at least one driver on the remote node and a local mechanism on the local node; and File name above cluster A step of transmitting definitive to other nodes through the file system, STREAMS to open the driver the file name is marked
executing an open function and checking its filename to confirm that it is a clustering facility, and if so, passing its initial major and minor numbers to the controlling thread.
A method for creating a data structure, wherein the controlling thread uses the major and minor numbers to look up the device and mechanism represented by it, retrieves new major and minor numbers, and stores those numbers in a set of mechanisms. Sending back to the STREAMS messaging mechanism along with the identifier, if the initial major and minor numbers are associated with a mechanism on the remote node, the STREAMS messaging mechanism on the local node is a STREAMS software interconnect driver on the local node That is, the S-ICS is opened, the S-ICS driver notifies the control thread on the remote node of the open request, and the control thread sets the initial state and sets the STREAMS software interconnection on the local node. Operating to establish a driver i.e. S-ICS, control thread on a remote node, for creating distributed STREAMS instance of STREAMS data structure instance on the local node on a remote node, performing an internal STREAMS open,
Distributed STREAMS data structure creation method.

(12) If the initial major number and minor number relate to a mechanism on the local node,
The method of claim 11, further comprising the step of opening the designated local driver so that the STREAMS messaging mechanism at the local node enables the local mechanism associated with the driver. (13) The STREAMS messaging mechanism provides to each of the computers to provide a bidirectional data path between a user process and a device driver via a module which is an intermediate element that can be dynamically added or deleted. An STREAMS data structure to be located, wherein the device driver resides in a system kernel portion of the operating system, so that data received by the device driver can move toward a user process. Operating the device driver to control a peripheral device to transfer data between the kernel and the peripheral device, wherein the STREAM data structure includes a stack stored in a local memory, a module and a driver function. Run At least a portion of the distributed STREAMS instance to the second target node by holding a set of function pointers to control and transmitting at least a portion of a stack on the first node to the second node Further comprising the step of:
The method according to the above (11). (14) The method according to (13), wherein the STREAMS stack is executed on a node different from a node on which an application accessing the STREAMS stack is running. (15) The method according to (13), wherein the STREAM is divided at a module / driver / streamhead level into configuration components running on different individual nodes in the cluster. (16) The (1) above, wherein the distributed STEAMS data structure includes a STREAMS-type pipe having two pipe endpoints each running on a different node in the cluster.
The method according to 3). (17) The method according to (13), wherein the STREAMS stack is migrated, in whole or in part, from one node to another. And (18) detecting a fault condition on said second node to initiate error recovery.
The method described in. And (19) performing a simplification function on a portion of the data structure to gather information needed to replicate a private data structure for a module represented in a stack on the target node. The method according to 13).

(20) A multi-computer system having a distributed STREAMS function, the computer including one or more system processor units, each having a local memory and an input / output subsystem, and having a data communication interconnect. A cluster consisting of at least two nodes interconnected via a subsystem, and one of a networking protocol, a client / server application and a service running on each of the system processor units in the cluster; Or ST used for multiple implementations
An operating system including a REAMS messaging mechanism; a software application running on at least one of the node's system processor units under the control of the operating system for performing tasks or solving problems; Means for creating, on a first initiating node of the plurality of nodes, a distributed STREAMS instance independent of the application and the STREAMS messaging mechanism; and
An operating system running on each of the processor units and including a STREAMS messaging mechanism used in implementing one or more of the networking protocols, client / server applications and services; A software application running on at least one of the node's system processor units under the control of the operating system and a first initiating node of the plurality of nodes
Means for creating a distributed STREAMS instance independent of the application and the STREAMS messaging mechanism; and controlling communication between the distributed STREAMS instance on the first node and the STREAMS messaging mechanism on the second node. Distributed ST with cluster mechanism including preview function
Means for use on said second node under control of a REAMS instance.

(21) The computer in which the STREAMS message transfer mechanism provides a bidirectional data path between a user process and a device driver via a module which is an intermediate element that can be dynamically added or deleted. STREAM placed in each of
A peripheral device including an S data structure, wherein the device driver resides in a system kernel portion of the operating system, and enables data received by the device driver to move toward a user process. To transfer data between the kernel and the peripheral device.
The driver operates, the STREAM data structure includes a stack stored in local memory, holds a set of function pointers to control execution of module and driver functions, the application and the S
Although the entire TREAMS is on the same node,
The multi-computer system of (20) above, wherein the EAMS messaging mechanism is utilized to exploit the advantages of the cluster function of hardware sharing, additive averaging and high availability. (22) the operating system includes threads defined in software processing elements that are stored in a local memory of one of the nodes and provide a set of functionality through execution of application instructions on a computer; The above thread is distributed STR
The multi-computer system of claim 20, including a control thread defined by a dedicated thread that serves as a third-party communication and control point for the EAM.

[0371]

By implementing the distributed STREAMS of the present invention in a cluster environment of a multi-computer system, high availability in which a distributed application does not stop even if a single point fails, and the entire cluster resource by leveling the application load , Effective use of hardware, increase in the sum of computation and I / O bandwidth, and increase in access to large-capacity storage areas and networking resources.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating one example of a computer node cluster in which high speed digital data is interconnected by an interconnect mechanism in accordance with the present invention.

FIG. 2 is a block diagram showing details of a part of the cluster of FIG. 1 and showing mode switching and mutual operation of nodes C and D;

FIG. 3 is a block diagram illustrating an alternative embodiment of FIG.

FIG. 4 is a block diagram showing details of a portion of the cluster of FIG. 1, showing an alternative configuration for splitting the stream stack for TCP / IP operation on Node A and Node B.

FIG. 5 is a block diagram showing a data structure packet routing process in the configuration of FIG. 4 (B).

FIG. 6 is a block diagram corresponding to FIG. 4B showing a packet routing process in a plurality of instances for performing global association of a plurality of ports under the control of a middleware thread.

FIG. 7 is a block diagram of a data structure used to implement the packet routing process of FIGS. 5 and 6.

FIG. 8 is a block diagram showing a message flow between a control thread and middleware.

FIG. 9 is a block diagram showing a message flow between a control thread and middleware in the process of FIG. 6;

FIG. 10 is a block diagram illustrating an example of a data structure used to implement the S-ICS of FIGS. 5 and 6;

11 is a block diagram illustrating the status / flow of a process for identifying a remote index using the data structure of FIG.

FIG. 12 is a block diagram illustrating the state / flow of a process used by the S-ICS of FIGS. 5 and 6 to identify a target queue in the data structure of FIG. 10;

FIG. 13 is a block diagram illustrating the status / flow of a process for identifying a target queue index using the data structure of FIG. 10 to implement the preview function of FIG. 3 on node D.

14 and FIG. 15 and FIG. 2 and FIG.
DLPI or IP via ICS and S-ICS
4 is a flowchart of an operation for specifying a message route to a server.

FIG. 15 is a cross-sectional view of FIG.
DLPI or IP via ICS and S-ICS
4 is a flowchart of an operation for specifying a message route to a server.

FIG. 16 is a block diagram illustrating an alternative embodiment in which a process with a recovery mechanism opens a driver in a STREAMS framework distributed across two nodes of the cluster of FIG. 1;

FIG. 17 is a block diagram illustrating a second alternative process of opening the driver of the STREAMS framework on only one remote node in the cluster of FIG. 1 without a recovery mechanism.

FIG. 18 is a block diagram illustrating a single system implementation of STREAMS logging with a log multiplexer in the system of FIG.

[Explanation of symbols]

Reference Signs List 20 cluster 30 stream head 31, 33, 35 preview function set 32 TCP / UDP 34 control thread 36 physical cluster interconnect driver (P-ICS) 38 software cluster interconnect driver (S
-ICS) 40, 42 DLPI 44 IP MUX (Mux) 46 Local Data Structure 48 Pointer 50 Middleware Thread 52 Fanout Table

Claims (1)

[Claims] 1. A multi-computer system having distributed STREAMS capabilities, comprising: a computer including one or more system processor units; each having a local memory and an input / output subsystem; A cluster consisting of at least two nodes interconnected through a system; and a networking protocol running on each of said system processor units in said cluster;
STREA used in the implementation of one or more client / server applications and services
An operating system including an MS messaging mechanism; a software application running on at least one of the node's system processor units under the control of the operating system for performing tasks or solving problems; Means for creating, on a first initiating node of the plurality of nodes, a distributed STREAMS instance independent of the application and the STREAMS messaging mechanism; a networking protocol on the first node; To the second target node for performing selected tasks of a software application on a second target node of the plurality of nodes transparently to applications and services Multicomputer system comprising: means for shifting at least a portion of the serial distributed STREAMS instance, the.