KR20060090810A - Group-to-group communication over a single connection and fault tolerant symmetric multi-computing system - Google Patents

Abstract

A system enabled for reliable and ordered data communication between two sets of nodes with atomic multi- point delivery and multi-point transmission, for example, extending TCP/IP is described hereon. When multiple nodes must be delivered with data, the delivery is performed atomically. A system enabled for fault- tolerant symmetric multi-computing using a group of nodes is also described hereon. A symmetrical group of nodes networked using a reliable, ordered, and atomic group-to-group TCP communication system is used in providing fault-tolerance and single system image to client applications. The communication between the client and the group is standards based in that any standard TCP/IP endpoint is able to seamlessly communicate with the group. The processing load is shared among a group of nodes with transparent distribution of tasks to application segments. The system is fault-tolerant in that if a node fails remaining replicas if any continue service without disruption of service or connection.

Description

GROUP-TO-GROUP COMMUNICATION OVER A SINGLE CONNECTION AND FAULT TOLERANT SYMMETRIC MULTI-COMPUTING SYSTEM}

The present invention relates to network communication between n to n points in a network, where n is any integer value.

For optimal resource utilization flexibility and reduced management costs, the industry demands a solution based on the "utility computing" model. The model can add as much processing power and storage capacity as needed and dynamically supply resources to meet changing needs. Existing mainframe solutions are out of reach of companies because of their high cost. There are many high performance, low cost "blade servers" and networking technologies available on the market. However, no solution yet exists that can efficiently and flexibly aggregate these resources and operate in a wide range of applications to meet utility computing needs.

The client-server paradigm is common in the industry due to the simplicity that clients request and servers respond. A common communication protocol used between clients and servers in a communication network to enable this paradigm is transmission control protocol / Internet Protocol, or simply "TCP / IP". In a communication network, a client (or client system or machine) sees the server (or server system or machine) as a single logical host or entity. A single physical server often does not effectively meet the needs of numerous clients. As a result, the failed server makes the client unable to run.

In order to deal with the defects of a single physical server, a cluster configuration with many servers operating in parallel or grid to provide services to clients has been developed using a load balancer. This configuration offers potential advantages such as fault-tolerance, low cost, efficiency and flexibility that are comparable to mainframes. However, these or other benefits are not widely realized due to inherent limitations and the lack of a standard platform in most applications.

In addition to physical clustering, existing software systems have attempted to introduce clustering at the operating system and application levels. However, a disadvantage of this software configuration, including instances in which clustering has been introduced into applications, is that they limit the use of such applications. Similarly, operating system level clustering is attractive, but existing efforts in this field are not yet successful due to the many abstractions that need to be virtualized.

Compared to the clustering of physical servers, software applications and operating systems, network level clustering offers some attractive advantages without worrying about these problems. For example, the ability to treat clusters of server nodes as a single virtual entity is a useful requirement in client server programming. In addition, the ability to easily create virtual clusters from a pool of nodes adds to the greater utilization and flexibility of mainframe classes.

Existing network level clustering platforms should be generic and applicable to a wide range of applications. These applications range from web servers, storage servers, database servers, and scientific grid computing. These existing network-level clusters enable a set of computational power and capacity of nodes to be evenly distributed according to the application rate. Current applications should be able to operate with minimal no or change. However, existing network level clusters have only limited success.

The degree of success of the Symmetric Multi-Processor (SMP) architecture can be attributed to the simplicity of the bus, making the location of the processor and memory transparent to the application. In clustering too, the transparency of the node location and the transparency of the node itself can be attributed to the simplicity of the virtual bus connecting the server nodes. However, these existing systems lack the ability of the client's application to directly tap the bus for efficiency. Similarly, buses based on User Datagram Protocol (“UDP”) packet broadcasts and multicasts lack the guarantee of data transmission, leading to application level clustering.

TCP / IP is the most used protocol with transmission guarantees by industry. Data Transmission Guarantee of TCP Virtualized TCP is desirable for the required transmission guarantees and ubiquity. However, TCP's support of only two endpoints per connection limits its potential. An asymmetrical configuration of processing elements / nodes with predetermined tasks, such as distributing incoming requests across a cluster, is inherently inflexible and difficult to manage or load balance. Asymmetric nodes often fail and bottleneck frequently at a single point. In order to succeed in multi-computing (MC), a symmetric configuration as opposed to an asymmetric node configuration is required.

Another problem with asymmetry in a client server environment is latency. Switches and routers employ special hardware to reduce latency as data passes through. When data has to go through the node's UDP / TCP / IP stack, the serious latency of copying and processing is added. Therefore, to achieve optimal performance, the system must avoid passing data through intermediary nodes with an asymmetric structure. However, if the server node's CPUs must handle a large amount of network traffic, the application's throughput and processing must bear this. Thus, existing systems must use hardware accelerators such as specialized integrated circuit chips or adapter cards to improve application performance and reduce latency at the endpoint. This increases the cost and complexity of the system.

Low cost barriers are highly demanded in many industries. Solutions with a fixed number of additional hardware components are costly due to the lack of flexibility, lack of ability to recover and complexity. Today's solutions provide high availability by quickly switching services to standby servers after a failure. Because standby systems are passive, their resources are not available due to high costs. Simple but powerful form of failover by replication, it maintains service through connection without interruption in case of node failure.

In a traditional cluster, active nodes perform tasks and passive nodes update later on changes. In many instances, there are fewer updates compared to other tasks, such as queries. Machines are best used when loads are shared between all replicas, while updates are reflected in replicas. Updates must be synchronous and made in the same order for consistency. Atomic delivery ensures that data is sent to all destinations before the client receives a TCP ACK signal indicating data reception. If a failure occurs, the rest can continue service while avoiding connection interruptions that affect failover. Non atomic replication lacks usefulness. In particular, each generates a response when the client's request is received by the service. Since the client sees the server as a single entity, it must be guaranteed that only one instance of the response goes back to the client. Similarly, even if multiple client replicas attempt to send the same request, it should be guaranteed that only one instance is sent to the server. Existing systems often fail to provide this atomicity. Thus, there is a lack of usability and overcoming obstacles to avoid disconnection.

Another problem with existing clustering systems is load balancing. As with any other system, the ability to balance the load between nodes is necessary for optimal application performance. However, existing clustering systems have limited support for standard load balancing schemes such as, for example, round-robin, content hashed, and weighted priority. do. In addition, many existing clustering systems cannot support the implementation of an application of a particular load balancing scheme.

Many services have load levels with many variations in time-dependent clusters. The running process needs to move to stop the active server. Existing cluster systems lack the support to add or remove nodes from the cluster in a way that is easy to perform without service interruption.

There have been many attempts to deal with network level virtualization. However, each attempt still has significant drawbacks. For example, one existing solution is a device for load balancing in a cluster of web servers that are popular in the industry. This load balancing device, also disclosed in US Pat. Nos. 6,006,264 and 6,449,647, switches incoming client TCP connections to one server in a server pool. The existing server for this process is Microsoft's network load balancer software that broadcasts or multicasts client packets to all nodes as switches or routers. However, once a connection is mapped, the same server handles all client requests for the life of the existing one-to-one TCP connection.

The problem with the traditional system above is that when a service consists of different types of tasks running on a node, any mapped server that does not have all the services the client requests over the connection will not provide a complete solution due to service failure. . This limits the use of such a system to provide a web page where only one task of serving the page is replicated to many nodes. In addition, the mapping of devices externally implemented on the server also creates bottlenecks and a single point of failure. Also, replication is not supported because the connection has only two endpoints. Therefore, such updates using single-ended TCP are not reflected in the replica and thus have a significant limitation in usability.

To address the shortcomings of the existing system above, other traditional systems have been attempted to distribute the client's request via a connection to a node providing another task. Ravi Kokku et al. Presented this system in his article "Half Pipe Anchoring." Half pipe anchoring is based on backend forwarding. In this schema, when a client's request reaches a cluster of servers, the designated server accepts the request and sends it to the optimal server after the data has been tested. The optimal server given the connection status information then responds directly to the client after replacing the address with the original target address. Here a single TCP endpoint is dynamically mapped to nodes to distribute requests. This schema is an example of an "asymmetric" approach where an intervening node intercepts data and distributes it based on the data content.

Another traditional system for building asymmetric structures is presented in two white papers from EMIC Networks. This existing system uses proprietary protocols that allow a given node to intercept and capture incoming data and forward it to multiple nodes. Sometimes only one node is allowed to send data, the data is sent to the designated server first, and then the server sends the data to the client. Here, also a single endpoint is dynamically mapped and the TCP connection is terminated at each node where replication is initiated. This schema is another example of an "asymmetric" attempt at which each node intercepts and replicates data.

The two schemas described above maintain the TCP definitions of both endpoints, even if they are mapped from different nodes. Replication in this existing schema is done at the application level using proprietary protocols. In addition, this existing schema adopts an asymmetric node structure. The selection node here acts as an application level router that distributes requests. However, this asymmetry limits the versatility as mentioned by Aaron et al. In "Scalable Content Aware Request Distribution in Cluster Based Network Servers". These limitations include single points of failure, bottlenecks in data throughput, lane performance and lack of location transparency due to high latency.

Therefore, a method and a symmetric system are needed that use the current definitions of the two endpoints of TCP to provide m to n connections. (M, n may or may not be equal to any integer.)

The above mentioned and other requirements extend TCP's current host-to-host communication range to group-to-group communication, more specifically the symmetrical definition of the current definition of two connection endpoints. This is accomplished by extending the definition of two groups of endpoints connected by configured nodes. Each endpoint has the right to receive and transmit independently and in parallel while maintaining the ordered transmission of TCP. Data is passed to the entire group or subset, depending on the configuration. Only the required target endpoints are required to receive data before sending an ACK of TCP to the peer group.

In one embodiment, the present invention allows addressing a cluster of nodes as a single virtual entity with one or more IP addresses. Communication between client and server groups is strictly standards-based in that any standard TCP / IP endpoint must be able to communicate seamlessly with the group. Data is atomically passed to endpoints that terminate at nodes in a symmetrically organized group.

A filter installed on the endpoint filters out data that arrives independent of the application segment. Data delivery to application segments is dynamically controlled by appropriately configured and installed filters. In addition, the filter optionally performs the placement of incoming data directly into the target application memory without the intervention of a copy.

The inputs and outputs over the connection are separated by the nodes receiving and transmitting each other independently and in parallel. All transmissions are sequential according to the TCP specification and transmission control between nodes is ordered based on the round robin method between group nodes, the round robin method between requestors, or application specific schema. Nodes retransmit in parallel and do not need additional synchronization between them to do this.

Application functionality is distributed across group nodes for versatility and load sharing. To accomplish this, the applications are logically partitioned, each operating as a subset of the application's functionality. Incoming requests that arrive at the TCP connection are forwarded to the segment residing on the group that effectively distributes the load. By passing only a specific set of requests to an application instance, logical segmentation is possible without changing the application code. Application segments can move dynamically from node to node without disrupting connectivity.

In addition, a node can communicate with other nodes in the group by forming a connection with a virtual entity represented by the group. This provides all the above features for communication between group nodes.

This system is failover in that, if a node fails to run an application, the set of application replicas remaining in the group can continue service without interrupting connection and service. Nodes are dynamically added to or removed from a group to maintain a certain quality of service that is transparent to the application. For load balancing or node removal purposes between nodes, the system transparently moves active services and redistributes tasks within groups.

An application running on the nodes of the group can view and operate the rest of the group as a single virtual entity that simplifies client / server application programming and resource management. One embodiment of the present invention allows separating an application into one or more segments that operate independently through a group node, in a transparent manner to the application without requiring code changes.

The system shares the processing load among groups of nodes by dynamically and transparently distributing incoming tasks through connections to various application segments. A single request that arrives over the connection is serviced by a combined multi-segment operation, allowing for the proper distribution of processing or computations across nodes. This system allows multiple instances of a segment to operate in parallel. Requests are delivered to selected instances based on schemas such as round robin, least loaded node, affinity based, content hashing, and so on.

Incoming requests are atomically forwarded to a multi-segment instance for failover. The results are optionally compared and one instance is the output. In the event of a segment / node failure, the remaining segment instances continue to service without interruption.

The system allows for flexible external management of the system by distributing tasks in a finely controlled and configured filter at the connection endpoint. When removed, responsibility for the load of the node is shifted to another node selected using a schema such as least-load, round-robin, or application-specific schema. The system automatically and dynamically adds resources from pool to group to meet changing needs. Similarly, nodes are automatically and dynamically removed and fed. The system automatically adds or removes resources while maintaining a certain quality of service.

The features and advantages described in this specification are not all-inclusive, and in particular, many additional features and advantages will be apparent to those skilled in the art in view of the drawings, specification, and claims. In addition, the languages used in the specification are selected in principle for readability and educational purposes and are not selected to limit the subject matter of the invention.

Other advantages and features of the present invention will become more readily apparent from the following detailed description of the invention and the appended claims with reference to the drawings.

1A is a generalized diagram of a communication system constructed in accordance with one embodiment of the present invention.

1B illustrates a block diagram of a communication system according to an embodiment of the present invention.

Figure 1c shows a block diagram showing the configuration of a high-level component for the implementation of the communication system according to an embodiment of the present invention.

1D illustrates a block diagram of a low level configuration component implementation for optimal performance of a communication system in accordance with an embodiment of the present invention.

2 shows a block diagram of a hardware configuration of a high level component for implementation of a communication system according to an embodiment of the present invention.

3A is a flowchart of a processing path of input data when connected according to an embodiment of the present invention.

FIG. 3B illustrates the remainder of the flowchart of FIG. 3A for a processing path of input data upon connection according to an embodiment of the present invention.

3C is a flowchart illustrating filtering of incoming data upon connection according to an embodiment of the present invention.

4 is a flowchart illustrating data transmission through a connection while limiting a maximum transmission size per one time for fairness between nodes according to an embodiment of the present invention.

5A illustrates a block diagram of a request / grant schema for active dispatch head allocation in accordance with an embodiment of the present invention.

5B illustrates a flowchart for processing a request for a dispatch head according to an embodiment of the present invention.

6 is a block diagram illustrating a virtual window schema for a peer group window advertisement according to an embodiment of the present invention.

7A illustrates a block diagram of a computing system for a communication system in accordance with an embodiment of the present invention.

7B illustrates a block diagram of a computing system for a communication system that provides lightening of the load of the main processor in accordance with one embodiment of the present invention.

7C illustrates a block diagram of a computing system for a communication system that provides lightening of the load of the main processor on dedicated hardware / accelerator chips in accordance with one embodiment of the present invention.

8 shows another generalized diagram of a communication system according to an embodiment of the present invention.

9 illustrates a data transfer and acknowledgment scheme between a client group and a server group according to an embodiment of the present invention.

10 illustrates a logical view of an implementation in accordance with one embodiment of the present invention.

Figure 11 illustrates a generalized diagram of a symmetric multicomputer system with failover, load distribution, load sharing, and a single system image in accordance with one embodiment of the present invention.

The present invention includes, for example, a system capable of reliable and ordered data communication between two sets of nodes with atomic multipoint forwarding and multipoint forwarding extending TCP / IP. The present invention extends TCP's concept of reliable host-to-host communication to include symmetric group-to-group communication, which maintains TCP's specification for data traffic between groups. Furthermore, the present invention extends the definition of two endpoints of a TCP connection to include at least two groups of endpoints communicating over a single connection.

In the present invention, the endpoint of the connection is terminated at the group node. When data needs to be delivered to multiple nodes, delivery is done atomically. Optionally, a single data instance is sent for data originating from multiple nodes. As will be described later, each endpoint consists of a receivehead and a sendhead that operate independently.

In one embodiment of the invention, a node comprises a data processing device such as a general purpose computer or the like, or another device with software having the function of a microprocessor or data processing device for network related communication. . A group refers to a set of one or more symmetrically configured nodes. An application segment refers to an application or a segment of an application provided with other application segments operating on various group nodes. An application consists of one or more application segments and an application segment consists of one or more processes.

The sending head represents the sending end of a TCP connection, which controls the data transmission and maintains the transmission state at the node. The receiving head represents the receiving end of the TCP connection, which controls the data reception in the connected state and maintains the data receiving state at the node. The active dispatch head represents a dispatch head designated to have the most recent transmission status information, such as, for example, a data sequence number and a sequence number of a recent acknowledgment.

The bus controller represents a node that controls and / or coordinates the connection establishment and termination process with the peer group. Signals represent messages exchanged within a group of nodes via a logical bus. If the source and the target of the signal are in the same node, the signal is not sent, even if it is the effect of receiving the signal internally. The endpoint of the connection represents a stack, such as TCP, that exchanges data with peers in sequential order based on previously agreed pairs of sequence numbers. The endpoint of the connection has an output data stream start point and an input data stream end point. A request represents an optional segment of an incoming data stream, eg, a service request from a client.

System overview

1A illustrates a communication system according to an embodiment of the present invention. The communication system includes a TCP connection 130 connecting the first group 120 and the second group 160. For example, first group 120 has first, second and third member nodes 100a, 100b, 100c, and second group 160 has first and second member nodes 150x, 150y. Has) Member nodes in these two groups are configured symmetrically in that each node has the same access to the TCP connection and operates independently and in parallel. The first data stream 110 and the second data stream 111 may flow between the first group 120 and the second group 160 of the communication system.

The first application segment 135 and the second application segment 136 build a server application at 120. The first application segment 135 has replica sets 135x and 135y, and the second application segment 136 also has replica sets 136x and 136y. Application segment replicas 135x and 135y are run through nodes 100a and 100b, respectively, and replicas 136y and 136x are run through nodes 100b and 100c, respectively. The client application of group 160 is composed of application segments 151 with replicas 151a and 151b.

The application segments 135 and 136 of the first group 120 communicate with the segment 151 of the second group 160 via the connection 130. The two data streams 110 and 111 of the connection 130 follow the TCP protocol. Connection 130 has three different connection endpoints 130a, 130b, 130c in first group 120 and two different connection endpoints 130x, 130y in group 160 for the same connection.

Each group 120, 160 is assigned a respective group Internet Protocol ("IP") address 121, 161. A group sees each other as a single entity while it is made up of nodes. Communication between the two groups 120, 160 is addressed via group IP addresses 121, 161, respectively. When a request from segment 151 arrives at first group 120, it looks like data from group IP address 161. Similarly, the second group 160 sends data targeting the group address 121.

The endpoints 130a, 130b, 130c of the first group 120 may be set such that one or more application segment replicas 135a, 135b, 136a, 136b are delivered with an incoming request. Examples of other policies where data is delivered to an application segment include all replicas, one replica, all application segments and selected application segments, and round robin request distribution based on a hashing schema to map requests to specific nodes. Target and weight based priority determined based on the content. This one schema change ("write") request is sent to all replicas of the application segment and the "read" request is sent to only one selected replica.

Each of the endpoints 130x or 130y in the second group 160 may send a request to the server group 120. At the endpoints 130a, 130b, 130c of the first group 120, one or more receiving heads receive data depending on the setting. The endpoints 130a, 130b, 130c of the first group 120 may transmit response data received by the endpoints 130x, 130y of the second group 160. Application processes that wish to receive certain or all incoming data are guaranteed to have received it before acknowledging the client with the receipt of the data. In order to maintain the TCP's sequential order of data transmission, TCP sequence numbers are assigned in a sequential order before the data transmission begins.

Optionally, the replicated data output by replicas 151a, 151b of second group 160 is reduced to a single instance to be delivered to first group 120 by the communication system. Similarly, the replica output of the application segments 135, 136 of the first group 120 may also be reduced to one. The replicas of 135a, 135b, 136a, and 136b do not necessarily produce an output, because in many cases the request is only sent to one replica, depending on the setting.

The communication system according to the present invention provides advantageous atomic client / server requests and responses. That is, they enable multiple processes to exchange data between two groups over a single connection, sending or receiving as a continuous sequence of bytes.

The protocol between groups 120 and 160 is TCP and data is guaranteed to be delivered in a sequence of orders that had been sent in accordance with existing TCP. When targeting multiple endpoints, data is guaranteed to be delivered to all target endpoints before the client sends a TCP ACK segment indicating receipt of the data. Optionally, when the replica output should be reduced to the transmission of a single copy output, the output is guaranteed to be atomic in that the data is transmitted if all nodes output the same data. However, if the results do not match, the application can optionally select the output for transmission based on multiple agreements, corrections, or successful results.

With application segmentation, application processes are typically delivered with only a selection of incoming data streams for processing. For example, a request that arrives at the second data stream 111 may be forwarded to select an application segment. The order of delivery of data to the application process shall be guaranteed to be the order in which it is sent as specified by RFC 793. In other words, all progress data arriving in the stream must be successfully delivered to the target's endpoint before specific data can be delivered to the application segment.

1B, the first group 120 includes first, second, and third nodes 100a, 100b, and 100c. The connection 130 between the first group 120 and the second group 160 has outgoing and incoming data streams 110, 111. Each node 100a-c has a group to group communication stack 130a-c, respectively. Data transfer to all nodes goes through switches 141a-c associated with each node 100a-c. No assumptions have been made by the underlying hardware regarding the guarantee of delivery for switches 141a-c. This is because popular hardware technologies like Ethernet are unreliable. It is possible by basic hardware devices that the data transfer to each node 100a-c or a subset thereof is optional or there is no transfer at all.

Incoming data is switched to standard TCP / IP stack 140a-c or group-to-group communication stack 130a-c by switches 141a-c based on IP addresses and / or ports. The application process 142 of the node 100 communicates using the standard TCP stack 140. Application segments 135x, 135y 136x, 136y communicate with group communication stack 130a-c, respectively. Bus 105 carries a control signal that controls and coordinates the operation of group 131. The range of signals transmitted via the control bus 105 is limited to the first group 120. The virtual bus 143 is composed of control signals of the bus 105 connected to the first and second data streams 110 and 111 and the group 120. This bus is tapped directly by peer group TCP connection 130.

An alternative to virtual bus 143 is point-to-point communication between nodes, which has the advantage of better bandwidth usage. However, this requires that each node in the communication system keep an eye on the other nodes and their addresses and roles. In one embodiment, the logical bus model is preferred over control messaging due to location and identity transparency.

1C illustrates a connection of endpoint 130a in accordance with one embodiment of the present invention. In general, switch 141 directs data to a standard TCP stack or group to group communication stack Internet Protocol (“IP”) input 171. For split IP packets, 170 performs recombination before forwarding to 171. If the input packet is not split, it is passed directly to the input content filter 171 after a simple basic consistency check. The input content filter 171 examines the input data content and / or packet header to determine if it contains data to be delivered to the application segment (eg, 135x, 135y, or 136x).

If the communication system decides not to pass the packet anymore, the packet is discarded and memory is freed without further action. Otherwise, input content filter 171 marks the segment of the packet that passed to the application. The packet is passed to IP input processing layer 172 to complete validation including checksum calculation and other consistency checks. Any malformed packets are no longer processed and discarded. The resulting packet is forwarded to group to group TCP layer 173. The group-to-group TCP layer 173 functions with group nodes (eg, 120, 160) and controls the receipt of data to meet TCP's specification requirements, such as acknowledgments for peer groups. The group to group TCP layer 173 maintains the input TCP state of the connection and passes data through the data path 137 to the socket. Data path 138 represents the transport data path from the socket interface to the stack.

The user socket exports data activating the output content filter 174. In one embodiment, the output content filter 174 is not installed and therefore performs no operation. Failover filters synchronously compare the data to be transmitted with the outputs of other replica segments and send a single output instance. The choice of output instance sent to the peer group depends on the policy set in the filter, such as the same output, multiple agreements, correction results or successful operation outputs. In the event of a transport segment instance failure, the replica receives and continues to transmit without interrupting the connection. Upon receiving a successful output instance from the peer group, the replica discards the data and frees the memory. Output content filter 174 forwards the data to group TCP output layer 175 for transmission. The group TCP output layer 175 controls the data transfer and maintains the transfer state with the group nodes. The group TCP output layer 175 works with its group node to send data to the peer group in sequential order as defined by TCP. Group TCP output layer 175 forwards to transmit data to IP output layer 176. IP output layer 176 later performs standard IP functions on the data and passes the data to device driver 177 for data transmission.

If the result of the output comparison by the output content filter 174 indicates that the output produced by the nodes is different, the subset replica is considered defective and is no longer serviced over the connection, leaving endpoints serviced without interruption of the connection. To last. In embodiments excluding endpoints, this exclusion is based on a schema that agrees with the results that many of the endpoints exclude others. Alternatively, the exclusion of the endpoint can occur where the operation failed. Exclusion of an endpoint can be any application specific schema that can be programmed with a filter. In the event of an endpoint failure of a data transfer, the replica endpoint completes the transfer without interrupting the connection.

1D illustrates a connection endpoint 130 of a group-to-group communication stack in which a content process examines input and output data in accordance with one embodiment of the present invention. Content filtering is a function of the content processor. The content processor determines a location where application memory data must exist, an order of data, a time for notifying an application of receiving a pull request, and the like. When working with the network interface device driver 177, data is copied by the direct memory access controller 196 between the network interface 193 and the application memory 190.

While inspecting the incoming new request data, the content processor allocates memory 192 to the application space. The allocation size is the size of the complete request data from the peer, typically application specific. The remaining data in the request does not require allocation if memory for the complete request has been allocated. Output data 193 is allocated by the application itself. In addition, there is a copy of the segment of request / response data 194, 195. This schema application data is copied directly between the network interface I / O buffer and the application memory without mediating the associated memory copy.

2, the first group 120 is a set of servers comprised of nodes 100 (100a, 100b, 100c). The second group 160 may consist of a set of client nodes 150 (150x, 150y). The nodes 100, 150 in each group 120, 160 are connected to each other via a connection device 180. The access device 180 may be configured as a broadcast / multicast apparatus. For example, an Ethernet bus or a layer 2/3 switch. The network 189 may be an existing network. For example, it may be a local area network (LAN) or the Internet through which two groups of nodes are connected. The network 189 is not needed if both peer groups are connected directly through the connecting device 180.

In one embodiment, the communication system of FIG. 2 includes one or more network interface ports 185a, 185b, 185c at server nodes 100a, b, c. The communication links 187a, 187b, 187c and 188a, 188b, 188c connect the node 100 and the device 180. Input data reached through connection endpoint 130 is replicated to connection 188a, 188b, 188c by layer 2 or layer 3 multicast or broadcast capability. Reaching data is forwarded to ports 185a 185b and 185c. Data transfer by 180 or related hardware ports or links is not guaranteed. The data delivered by the first group 120 via 187a 187b, 187c are independent of each other and thus operate in parallel. The data delivered to the peer group via 187a, 187b, 187c need not be visible at 120. As data enters through connection 130, the signal transmitted over logic bus 105 is replicated to link 188a, 188b, 188c by device 180. The data delivered to the logical bus 105 of FIG. 1B is visible to the server nodes 100a, 100b, 100c.

signal

In one embodiment of the present invention, the signal may have access identification information common to the group. Moreover, the signal may also have a source and target identifier. The target identifier may be one or more nodes or an entire group.

In one embodiment of the invention, the signal in the communication system comprises an IACK signal. This is an input acknowledgment signal that acknowledges input data from peers instead of groups. The IACK contains a PUSH flag indicating whether the acknowledged sequence number, the remaining bytes of data expected from the peer group, the window update sequence number, the latest best time stamp, and whether the received active sendhead should forward the peer group TCP ACK. Include. The REQSH signal may require a recent dispatchhead assignment that contains the request and is designated as the active dispatchhead. The target address may be an entire group.

The GRANTSH signal contains a message with active dispatchhead status information, bus time, a list of nodes for which REQSH has been acknowledged, and the most recent IACK information of the announcement. The target of this signal is to assume the active dispatch head after updating the status information. The IACKSEG signal contains an input data acknowledgment sent on behalf of the segment. It may have the same or similar information as the IACK signal. The REQJOIN signal is passed to the application segment, requesting to join the service over the connection. The LEAVE signal is sent requesting that service interruption of the connected application segment be allowed.

The ACKLEAVE signal grants the service interruption permission of the connected application. The RESET signal is sent when requesting to reset a connection. The CLOSE signal is delivered when requesting the termination of an output stream of a connection by an application segment. The ACKCLOSE signal acknowledges receipt of a CLOSE request.

Connection establishment and termination

Existing TCP state diagrams are known. A flow chart describing this state diagram is shown and described in a book entitled Richard Steven's "TCP / IP Illustrated Volume I and Volume II." This content is incorporated herein by reference. In addition, the TCP / IP protocols and specifications are discussed in RFC 793 and RFC 1323, the relevant parts of which are incorporated herein by reference.

Nodes are selected to act as bus controllers during connection establishment and termination. The bus controller coordinates and controls the process and communicates with peer groups instead of groups. By default a static bus controller is selected, while the application program optionally selects a bus controller as needed. Bus controller functions can be assigned to nodes in a round robin fashion to distribute the load resulting from controlling the bus to group member nodes. Alternatively, the bus controller can be dynamically selected based on a hash value for an incoming sequence number or a combination of source IP address / port address. In the schema where the segment with the lowest ID is assumed to be the bus controller role, the bus controller's responsibility is assigned to the round robin method during the replica when the replica of the segment is available.

In general, there are four types of connection operations in TCP. Each form follows a different set of state transitions. When a group initiates a connection with a peer group, it is called active initiation. On the other hand, the connection process initiated by the peer group is called passive initiation. Similarly, when the termination of a connection is initiated by a group, it is called active termination, and when the termination is initiated by a peer group, it is called passive termination.

Passive connection establishment

When a synchronization ("SYN") request arrives from the peer group with passive initiation, the bus controller issues a REQJOIN signal requesting an application segment to join the connection service. The bus controller responds to the peer group with a SYN request with an ACK for the received SYN signal. If the peer group acknowledges the SYN request sent on behalf of the group, the group node driving the application segment responds with IACKSEG. If all required group nodes combine the connection service with the IACKSEG signal, the connection is considered established and data transmission can begin.

Active connection establishment

In active initiation, for connections initiated from a group, the bus controller specifies group nodes and sends a REQJOIN signal. Next, initiate the connection process with the peer group by sending a SYN request instead of the group. Upon receipt of a SYN request from a peer group with an ACK for the bus controller SYN, the group node sends an IACKSEG indicating receipt of a valid ACK from the peer group. When receiving an IACKSEQ from the requested node, the bus controller issues an ACK for the SYN request from the peer group and the connection is considered established.

Passive connection termination

In passive termination, upon receipt of a FIN segment from a peer group, the node sends an IACKSEG signal indicating receipt of the FIN. When an IACKSEG is received from all requested segments, the bus controller responds to the peer group with an ACK for the received (completed) FIN. When the node has finished sending data, it sends a LEAVE signal indicating that it wants to stop the connection. If the LEAVE request signal arrives from all group nodes after FIN receipt, the bus controller issues a FIN request to the peer group. The bus controller emits an ACKLEAVE signal and upon receiving it the target of the signal node enters the CLOSED state. When the ACK for the transmitted FIN request is reached, the bus controller enters the CLOSED state.

Active connection termination

In active termination, when the application segment wants to terminate the data transfer and close the connection, they send a CLOSE signal. Upon receiving a CLOSE request from all group nodes, the bus controller issues a FIN request to the peer group. When receiving a FIN request from a peer group, the node issues a LEAVE request. When the LEAVE signal from the group nodes and the ACK for the transmitted FIN are received, the bus controller enters the TIME_WAIT state.

Data entry via connection

As shown in Fig. 3A, when the data packet reaches the node, it is checked whether the packet is targeted to the group address (311). If the packet is a TCP fragment, then a fragment recombination operation is performed (314) to make a complete TCP segment upon arrival of the last fragment of the TCP segment. In most cases, TCP segments are not split. Therefore, this operation does not occur.

If the TCP segment is not targeted to the group, the standard TCP / IP stack is handed over to the TCP segment for further processing (312).

As shown in FIG. 3B, upon receipt of data, the receiving head of the group performs filtering (315) and verifies whether there is data targeted to the application segment on the node (316). And discard any data that is not relevant to the application. After filtering, the resulting data packet is checked for time stamp validity, checksum validity, and validity of other TCP / IP parameters (317). All invalid packets are immediately discarded. The receivehead updates to reflect all valid packets examined. By performing checksum after filter processing and verification of other detailed packets, the computational overhead of discarded packets can be avoided.

Next, it is verified (318) that all data preceding the received data has been delivered to the appropriate application segment. The data is then immediately passed to the application segment (320). If required by the specification, the TCP ACK segment is forwarded to the peer group. However, if there is preceding data in acknowledgment pending, the data is queued waiting for an acknowledgment (319).

If a segment instance fails during receipt of data, the remaining instance continues to receive and acknowledge control. This allows the application to continue service without interruption in the event of a node failure.

Filter data

Referring to FIG. 3C, the receiving head maintains the state of the input data such as whether subsequent data is needed to determine the target, the start of a new request, whether it is delivered to the application, or the request is ignored. Since the packet may contain one or more requests or partial request data, it is verified whether the packet has residual data to be processed (330). If no data is left, the filtering process is complete.

The current state is verified when there is data remaining in the packet to be filtered (331). If the current state indicates that the request data should be discarded, it is scheduled as if discarded to the maximum of the request data in the packet (332) and verifies if there is more residual data (330). Similarly, if request data is accepted and must be delivered to the application segment, the remainder of the request data in the packet is scheduled for delivery. All delivered data must be check-sum calculated once, timestamp and packet header verified (333) and invalid packets are immediately discarded (336).

When the current state indicates the start of a new request, an application specific filter is triggered to determine the data target (334) and the current state is updated to reflect the verification result. If the lack of sufficient data prevents the filtering code from determining the request target, it is combined with data immediately following from a recombination queue that holds the data arriving in any order. If there is still not enough data, the remaining data enters the recombination queue and repeatedly checks when enough data arrives. If enough data is found, step 330 is repeated to filter the data.

Data entry acknowledgment

If atomic data delivery to multiple endpoints is required, an acknowledgment of the received data is sent only when all endpoints receive data positively. The target endpoint receiving the data sends an IACK signal over the bus indicating receipt of data in TCP order. After verifying that all requested nodes have received specific data, if the TCP specification is met, the active sendhead sends a TCP ACK segment to the peer group.

Data output through connection

Multiple endpoints in a group can transmit data in TCP order. Therefore, it is necessary to assign consecutive sequence numbers to segments of data to be transmitted. It is also necessary to maintain consistency of the transmitted data to avoid mixing distinct requests / responses from the endpoint. For this purpose each complete request / response data is treated as a record by the transmitting node.

4, when the application process writes data (385), a new transport record is created (386). When more than one write request data needs to be sent without sending any other intermediary data to the peer, the MORETOCOME flag is set until the arrival of the last write. If the transmitting node is not the active sendhead 387, the active sendhead request is sent with the REQSH signal when the preceding request is not acknowledged or pending (388). When the active dispatching head state arrives with the GRANTSH signal directed to the node (389), the active dispatching head repeatedly checks whether it is assumed to be the active dispatching head after updating the last information from GRANTSH (387).

After becoming the active dispatching head, the node with the data to send allocates a new transmission sequence number to the record in a sequence of steps and transmission begins (390). If no more data is expected in the continuation of the transmitted write operation, and there are no more records waiting to be allocated using the transfer sequence number (392) or the maximum transfer limit has been exceeded (393), the active sendhead is the sendhead. The request is then sent to the next request node.

The next node to which the sending head is to be granted is determined by selecting the node by numerically close node-id in a clockwise manner from the REQSH requestor, the highest priority sending head requester, a round robin or a list of application specific schemas. However, if more transport records wait for the assignment of sequence numbers, step 387 will be repeated to send the remaining data.

active Shipping head  Assignment

5A illustrates a schema for active dispatch head assignment. Node 100a issues a REQSH 105r signal requesting the active dispatch head role and active dispatch head 100c sends the role to the requestor with a GRANTSH 105t signal with the necessary status information. The REQSH signal is sent by node 100a. Node 100b does not respond to the dispatch head and ignores REQSH. When there is a request, the node 100c, which is the active sending head, responds to the 100a request with a GRANTSH signal because the sending head to be granted is empty.

Upon receipt of the GRANTSH signal the requesting node 100a assumes an active dispatch head. The GRANTSH signal contains a list of active requestors maintained by the group. Upon checking the GRANTSH signal 105t, node 100b verifies the checklist of the requestor operating in the signal to check if its request for an active dispatch head has been acknowledged. When acknowledged, the retransmission of the REQSH signal is turned off.

When a node grants an active sendhead to another node, if it has outstanding data to be sent, it adds itself to the requester's list to avoid further request signals. Another way to send a REQSH-like signal to all nodes is to send them directly to the same target as the active sendhead node. The advantage of this approach is smooth bandwidth usage but lacks location transparency.

Referring to FIG. 5B, when the REQSH signal arrives at the active sendhead node (551), if the sendhead cannot be granted (553), the requestor id is entered into the requester list (554). However, the GRANTSH signal is sent to the requestor if the sending head can be granted (555). GRANTSH is a list of pending requesters and acts as an input acknowledgment signal for all outstanding REQSHs. The sending head grants itself to acknowledge the receipt of a REQSH without granting to another node. When GRANTSH arrives at the target node, the requestor's list is added to the requestor's local list. The GRANT signal is ordered by assigning a very simple incrementing sequence identification number for each instance of the sendhead except for retransmission.

Within a group TCP  Round with time stamp Trip  time( Time stamp and round - trip -time: RTT ) Calculation

Most TCP implementations follow RFC 1323. This details the method of measuring the round trip time using the time stamp. Round trip time is typically measured by subtracting the time stamp of the acknowledged data server from the host's real time. In order to identify packets that are not valid because they are wrapped in sequence numbers, the specification simply requires increasing timestamps.

It is necessary to match the specification for several nodes of various types with different hardware timers. The ideal solution would be to have the nodes have perfectly synchronized time, but this is very difficult. In one embodiment, the time stamp is synchronized with a timestamp that increases monotonically by synchronizing the time of the node that sent the data with the time of the node that last sent the data. This synchronization ensures that data is always sent with a timestamp value that is equal to or higher than the previous TCP data segment timestamp.

The implementation of the schema is done here. Nodes holding real time (often local time) are implemented along with a hardware clock that increments its value at fixed intervals. A group-wide real time clock, called bustime, must be implemented for each TCP connection accessed by the node. The bus time at any node is calculated as

Bus Time = Local Time-Base Time

The base time can be any value initially selected and calculated by the bus controller. Each time an active sendhead is granted to a node, the recipient's bustime is sent with a GRANTSH signal. Upon receiving the GRANTSH signal, the node to which the active sendhead is granted adjusts the bus time as follows.

If the bus time is less than the bus time received with the recipient's dispatch head:

Bustime = the recipient bustime (ie the bustime from the GRANTSH signal)

Although the node's bus time does not fit perfectly with the above formula because of the GRANTSH propagation delay, it satisfies the need for a simply increasing timestamp value. By selecting the granularity of the bus times higher than the sent timestamp granularity, errors caused by timestamp collisions during simultaneous retransmission by the node can be reduced. For example, when the timestamp is 10 millisecond granularity and the bus time is 1 microsecond granularity, the error coefficient decreases from 1 to 1/10000. For precise round trip calculations, the base time at the time of transmission enters the transmission record by the sending head. To calculate the minimum latency of the signal, a fixed time value is added to the grant bustime at the GRANTSH target node. Using bus time as the timestamp, the round trip time of the packet is calculated as follows:

Round Trip Time = Bus Time-Timestamp

Within a group TCP window update

Windows is the amount of data the endpoint can accept into memory. Existing TCP, with only two endpoints to have information, agreed that optimal performance could be easily performed. As many endpoints are involved, it is important that each of the different memory sizes and unpredictable data targets achieve optimal use and performance.

This section describes the schema in which a single virtual window size in a group scope is used for effective window operations within a group. The sending head is responsible for updating the peer group with window information instead of the group. The virtual node size is initially assigned to the group node. The node updates the window with the dispatch head by sending an input sequence number of data once read by the application. The active dispatchhead updates the peer group with the window obtained by subtracting the amount of outstanding data to be delivered to the application from the virtual window size of the group scope.

The window update is piggybacked with the IACK signal to reduce the number of window update signals. To further reduce the number of window update signals and TCP segments, the specified window size is maintained in addition to the virtual window. The data reaching the total of these windows may be pending data acknowledged and acknowledged by the group at any time. When a node emits an IACK acknowledging receipt of data of less or equal size than the specified window, and when a lot of ongoing data is read by the application, an updated window, such as an IACK sequence, is used, as many data are read by the application. do. Since the window update is done with the IACK, it avoids the additional window update signal that is otherwise required. This technique is optionally set or reset.

Referring to FIG. 6, an unacknowledged input sequence is indicated by 610, and the maximum data sequence number expected to be advertised to a peer group is indicated by 620. 619 indicates the maximum specified window sequence to which a window update can be sent. 611, 612, 613, 614 show the window sequence of data received by nodes 100a, 100b, 100c. 615 is the amount of data that can be sent by the peer group. 617 and 618 show window updates sent by nodes 100a and 100c, which are sent with an IACK in association with 611 and 612. The maximum advertised window is shown as 621 and the maximum specified window is shown as 622.

group TCP Lab Around ( wrapped - around ) In sequence  For protection

In high-speed networks such as 10 Gigabit Ethernet, TCP's current 32-bit sequence number wraps around for a short time. A signal delayed with a wraparound sequence number, such as an IACK, can be considered to be valid by mistake when the sequence number is used to confirm the data input. We use a scheme that maps a 32-bit TCP sequence number to a 64-bit TCP value that takes into account the time at which the 32-bit sequence number wraps around from the beginning of the connection. The 64-bit sequence values used within the group are mapped back to the 32-bits used with the peer.

To map a 32-bit sequence, separate the 64-bit sequence into two 32-bit values, where the lower 32 bits represent the TCP sequence used seamlessly with the peer, and the upper 32 bits wrap how many times the sequence wraps around since the connection was initiated. Count that number The lower 32 bits are used to map 64-bit values to 32-bit sequence numbers. In another schema, the IACK is sequenced even though the overhead is the same.

application Segment Instance  And replication

Many instances of the segment allow further distribution of loads between nodes. Loads between replica segments can be shared by passing requests to segments using round-robin, least-loaded, hashed, and affinity-based schemas with filters.

Segment replicas allow for failover. If a replica fails during input, it will continue service without interruption if there is a replica left. This is achieved by replicas that keep a consistent eye on input. The segment controller enables coherence between replicas with atomic input delivery. The new segment controller may need to be selected after the failure.

If a replica fails while sending data, the remaining replica can continue to service without interrupting the connection. Each replica agrees with an instance of the output and the dispatchhead state is shared before the transfer begins. Each replica frees memory and data acknowledged by the peer group.

Each application segment is free to choose the remaining number of replicas. The dynamically selected node as a segment controller adjusts the replica IACK to form a segment IACK. The selection of segment controllers can be based on round robin, minimum load, hash or static nodes. All the operations described, such as connection establishment, termination, and window management, have already been described with the corresponding schema. When the segment replica agrees to the receipt of a particular input segment, the controller responds on behalf of the replica. Unlike replicas, there is no need to include a controller when the load is balanced between segment instances.

When the replica receives the data, the replica sends an IACK indicating receipt of the input data. When a segment controller monitoring an IACK from a replica determines all replicas that have received a particular input in the order in which it was sent, it sends an IACK instead of a segment and initiates a client ACK. This IACK acts as an acknowledgment to the replica to send data to the application socket or to take atomically necessary actions. The selection of the segment controller is static on the selection replica, as if it had a round robin or lowest replica ID for the request.

Node-based group-to-group communication

7A is a block diagram of a typical computer, the elements of which are suitable for implementing the invention. A group-to-group communication stack is performed by the system's processor.

main CPU To group communication to reduce the load on the computer

7B is a block diagram of a computer, the elements of which are suitable for implementing elements of the invention while offloading the main processor when processing certain elements. The group-to-group communication stack has its own processor and is offloaded to the adapter card.

Group to Group Communication in Integrated Circuits

7C is a block diagram of a computer, the elements of which are suitable for implementing the elements of the invention while offloading the main processor when processing certain elements of the invention for a dedicated hardware / accelerator integrated chip. Offloading is required for most processing, otherwise the group-to-group communication stack must be fully or partially implemented by the main CPU.

8 is another embodiment of the present invention. In this embodiment, the hardware device replicates a single TCP connection endpoint to multiple endpoints. The node group is indicated by 802. Connection 801 has an input stream 826 and an output stream 825, respectively. The device 820 here is outside of the nodes 800a, b, and c. Each server node has connection endpoints 801a, b, c of the same connection 801. Device 820 replicates a single connection 801 to three endpoints 801a, b, c of node 800. Device 820 has four ports 816, 817, 818, 819 while port 819 is linked to connect to the peer device. The device is a single point of potential failure and adds extra network hop.

9 illustrates atomic data transfer and acknowledgment schema between a client group and a server group where two data segments 910 and 911 must be delivered to two nodes 902 and 904. 901 is a client group and 902, 903, 904, 905 and 906 are server group nodes. 910 and 911 describe TCP where data segments are sent by client group 901. Although 910 and 911 are potentially available to all server group nodes, they are only forwarded to nodes 902 and 904 as determined by the input filtering system in a programmable instance. Reference 912 describes a TCP ACK segment sent from server group sendhead 903 to a client group. The TCP ACK segment is not transmitted when the data segment 910 arrives, but the ACK segment is sent by the server group to the client in accordance with the TCP specification that another packet must be acknowledged when the second data segment arrives. The flex controller 902 transmits an IACK signal 914 indicating atomic delivery only upon receipt of a PIACK (Plex IACK) signal 913 indicating an acknowledgment of the same data segment in the required flex / replica 904. . The 902 does not send a PIAK because it is the controller's responsibility to send an IACK indicating the atomic delivery of data. When receiving the IACK signal 914, 903 with the sendhead sends a TCP ACK segment 912 to the client group. In addition to transmitting the client ACK segment when another TCP segment arrives, the ACK segment may optionally be sent at the end of every complete request input. The client ACK segment is also sent when an exceptional condition is detected, such as out of order of the arriving segments and segment latency. Thus, client and server groups immediately resend synced and lost TCP segments. If the server node sends a PIACK and fails to receive an IACK for it, the server node retransmits the PIACK and the receiving head responds with another IACK representing the latest sequence of the node's input data.

10 shows a logical view of an implementation where input data is shared as on a bus but output data is switched. 1000 is the input data stream from the peer group. 1010 is a logical half-bus that shares input only using multicast or shared media such as Ethernet. 1020, 1021, and 1022 represent busin endpoints corresponding to nodes 1030, 1031, and 1032, respectively. 1040, 1041, 1042 are output endpoints that are fed to the IP switching device 1050 of Layer 2 or Layer 3. 1060 represents the aggregated output generated by nodes 1030, 1031, 1032 generated for input 1000. 1000 and 1060 form a single connection input and output, respectively.

road Sharing and  road Balancing

11 illustrates a symmetric multicomputer system in accordance with an embodiment of the present invention. The server group 1112 is composed of a plurality of nodes 1100a, b, c, d, e, and f. Input stream 1110 of TCP connection 1109 has multiple endpoints 1110a, b, c, d, e, f that span group nodes. Similarly, output stream 1111 of the same connection consists of endpoints 1111a, b, c, d, e, f.

This application consists of three segments 1113, 1114, 1115 that operate across an entire group of two instances for each application segment 1113a, b, 1114a, b, 1115a, b. By programming the communication system, segments are delivered with specific tasks based on criteria such as data to manage and actions to perform. By configuring data delivery in such a way as to deliver a specific subset of requests for services to a particular instance of the application, segments of the application are achieved and in many cases without code changes to existing applications. The application is partitioned in various ways. For example, the segment may be divided based on a hashing algorithm based on access information or data content such as a type or type of request that can be processed, or a sequence number. It is also possible for an application to be divided into segments by programming to other segments.

Group nodes are paired like replicas 1100a, b, 1100c, d and 1100e, f, and each pair operates as two instances of application segments 1113, 1114, and 1115, respectively. When segment 1100a fails, pair 1100b continues service without interruption. If the failure of instance 1100a occurs during transmission, another instance 1100b will deliver the rest of the response to the peer without interruption of service. Similarly, new application segment instances are added to the group to increase the failover due to the added instances available to continue service in the event of a failure. This can be done, for example, by forming a new process driven application segment instance and adding it to a group to properly distribute the request.

In one mode of operation, a non-empty subset of a group is delivered with the request in a specific order, such as round robin and weight-based priority, which distributes the request into essentially non-empty subsets to balance the load across the nodes. .

In one mode of operation, one or more replicas are delivered with the task and after the task is completed, results from the instance are sent over the connection regardless of other replicas. In another mode of replica operation, more than one replica will be delivered with the same task. Relevant replicas then perform the actions in parallel and drive the results. Output filters installed on the output stream of the group-to-group communication system compare the results and a single instance of the results is sent to the peer group so that the group appears to be a single entity for the peer group. The choice of output instance delivered to a peer group depends on the policy set, such as the same output in the filter, multiple agreements, corrective results or successful behavior outputs. The choice of policy depends on the application. If the segment instance fails to transfer, the replica takes over and continues the transfer without interrupting the connection.

When the output comparison result by the output content filter indicates that it is different from the output produced by the node, the subset replica is considered an error and is excluded from further service over the connection, and the remaining endpoints continue to service without interruption of the connection. In embodiments in which endpoints are excluded, this exclusion is based on a schema that most of the endpoints agree to prevent others from coming in. Optionally, the exclusion of endpoints occurs when the operation fails. Exclusion of endpoints can also be from application specific schemas that can be programmed by filters.

In another mode of operation, a replica is conveyed with an operation that ends in a state change, such as data modified in memory and storage. In this way, the replica remains consistent. When an operation occurs between replicas, such as a read operation that does not affect consistency, the task is delivered to only one instance of the replica. This allows for load balancing between replicas.

Adding and Removing Nodes

Filters in the endpoint connections of TCP group-to-group communication systems enable fine grain control of data delivery to application segments. By dynamically configuring filters, specific tasks are delivered to specific nodes, enabling external control through the delivery of task requests to nodes. The flow of requests to the application segment is controlled like a switch for good task distribution between nodes.

Nodes can be added to a group at any time. Newly added nodes can share load from existing and new connections. On an already existing connection, the node joins the service and begins to receive the tasks arriving there. The necessary load between nodes is balanced by the movement of tasks.

To remove a node, the node's load responsibility is transferred to another node selected using a schema such as minimum load, round robin, or application-specific schema. Being removed, the node is completely free while waiting for small tasks to finish without receiving new tasks. When long progress tasks are involved, movement of tasks such as system level process movement is used. The entire situation of an application process, such as the stack, uses process movement so that data open files are transparently moved to another node. The nodes communicate with other nodes in the group by establishing a connection with the address of the virtual entity represented by the group. This provides all the aforementioned features for communication between group nodes.

Automatic feeding

The system automatically and dynamically adds resources from the pool into groups to meet changing needs. Similarly, nodes are removed and fed in dynamically and automatically. The system monitors the quality of service delivered to clients and maintains a certain quality of service through the addition and removal of resources. The operation can be done outside the system and potentially transparent to the peer group.

Those skilled in the art with respect to the present invention will understand that it is possible to design additional other structures and functions for group-to-group communication in a single connection in accordance with the principles of the present invention. Thus, while specific embodiments and applications of the present invention have been illustrated and described, it will be understood that the invention is not limited to the precise structure and components described herein. It will be apparent to those skilled in the art that various modifications, changes and variations can be made in the arrangement, operation and details of the method and apparatus herein without departing from the scope and spirit of the invention as claimed.

Claims (14)

In a method of communication between nodes for atomic delivery of data, Determining a node to transmit data; Sending a single instance of data; And Serializing transmission control for data transmission. An apparatus for communication between groups in a network, the apparatus comprising: a. A first uniquely addressed group and a second uniquely addressed group, wherein the first group includes a plurality of endpoints and each group comprises at least one node; b. A communication protocol having a single logical connection enabled for communication between each group; c. Send data to the second uniquely addressed group that is enabled and to allow data to the nodes of the first uniquely addressed group to effectively communicate between non-empty subsets of each group. And a plurality of endpoints enabled for the purpose of communication. The communication device according to claim 2, wherein data communication between each group does not pass through each node of each group. 3. The communications device of claim 2 wherein the endpoint comprises a protocol stack. 3. The communications apparatus of claim 2 wherein the node comprises one of a group of processes or devices. 3. The communications device of claim 2, wherein each group is enabled to address another group by a unique address. The communication device of claim 2, wherein the communication protocol comprises a Transmission Control Protocol (TCP). An apparatus for communication between groups in a network, the apparatus comprising: a. A first uniquely addressed group and a second uniquely addressed group, wherein the first group includes a plurality of endpoints and each group comprises at least one node; b. A communication protocol having a single logical connection enabled for communication between each group; c. Send data to the second uniquely addressed group that is enabled and to allow data to the nodes of the first uniquely addressed group to effectively communicate between non-empty subsets of each group. And a plurality of endpoints enabled for the purpose of communication. 9. A communications device according to claim 8, wherein data communication between each group does not pass through each node of each group. 9. A communication device as claimed in claim 8, wherein the endpoint comprises a protocol stack. 9. A communications device according to claim 8, wherein said protocol guarantees data transmission in the order in which they were sent by said group. 10. The communications apparatus of claim 8 wherein the endpoint resides in a node. 9. The communications device of claim 8 wherein the endpoint resides outside of the node. 9. The communications device of claim 8, wherein the membership of the two groups varies over the duration of the connection.

Images