WO2001027726A1

WO2001027726A1 - System and method for fault-tolerant clock synchronization using interactive convergence

Info

Publication number: WO2001027726A1
Application number: PCT/US2000/027193
Authority: WO
Inventors: Thomas Gilbert Roden, Iii
Original assignee: Alliedsignal Inc.
Priority date: 1999-10-08
Filing date: 2000-10-03
Publication date: 2001-04-19
Also published as: JP2003527667A; ATE239245T1; EP1228412B1; DE60002501T2; DE60002501D1; AU7848700A; US6801951B1; PT1228412E; CN1409837A; CA2387056A1; EP1228412A1

Abstract

A method for synchronizing nodes in a network is described that utilizes an interactive convergence technique. The technique utilizes communications protocol IEEE 1394 to broadcast each node's clock value to the other nodes in the network in step 301. Each node applies a voting algorithm to the set of broadcasted clock values to determine a voted clock value in step 303 and each node's clock is set to that voted clock value in step 305. When a node's clock value is close in value to another node, those nodes are considered to be synchronized in step 302. The set of clock values to be used to determine the voted clock value consists of those nodes that are synchronized. The technique is implemented on hardware separate from the node's hardware and can be implemented on a field programmable gate array.

Description

SYSTEM AND METHOD FOR FAULT-TOLERANT CLOCK SYNCHRONIZATION USING INTERACTIVE CONVERGENCE

FIELD OF INVENTION

The present invention relates to the field of computer synchronization and in particular to a system and method for synchronizing multiple nodes on a network that utilizes a convergence technique. BACKGROUND OF INVENTION

For any network system, synchronization of each node within the network is important to ensure that the network functions smoothly. Nodes that are not synchronized can cause disturbances within the network as miscommunication can occur between the nodes.

Any synchronization implementation needs to be as reliable as possible. However, the more reliable that a synchronization implementation is, usually the more complicated the implementation of that technique tends to be as well.

For example, prior synchronization algorithms have been based on the concept of interactive convergence v. through local timestamping of broadcasted messages. The synchronization algorithm had to infer the value of the remote node's clock to arrive at a voted and globally consistent clock. This approach, while proven to be fault-tolerant, is inherently complex and resource intensive .

In addition, implementing a synchronization algorithm mainly through software on the node' s processor imposes requirements on the hardware of the node itself. The node could be required to have high quality hardware to implement the synchronization software to be able to maintain high reliability of the synchronization procedure. This high quality requirement greatly increases the cost of the implementation overall and sometimes is unnecessary to perform the intended functions of the network.

As such, there exists a need for a synchronization method to effect synchronization of the nodes of a network that will have sufficient features to ensure reliable synchronization while reducing the complexity of the implementation and lowering the high quality standards for hardware. SUMMARY OF THE INVENTION

This invention utilizes a hardware implementation separate from the network node to synchronize each network node. Synchronization of the network is implemented in a Redundancy Management System (RMS) that can interface with common communication techniques within the network in conjunction with a Field Programmable Gate Array (FPGA) to implement the synchronization algorithm. The invention also exploits the ability of some communication protocols, such as IEEE 1394, to periodically broadcast their local clocks, forming a distributed global database and simplifying the synchronization process. The algorithm uses interactive convergence techniques to arrive at a globally consistent clock.

In accordance with one embodiment of the invention, a method for synchronizing nodes in a network is described that comprises the steps of broadcasting the clock value of each node on the network; determining a voted clock value based on a set of the clock values that were broadcast; comparing the clock value of each node to determine which nodes are synchronized with each other; resetting each node's clock to the voted clock value; and setting flags to indicate which nodes are synchronized. In accordance with another embodiment of the invention, a system to synchronize nodes in a network is described that comprises a clock broadcaster to broadcast the clock values of each node in the network to all other nodes in the network; a clock voter to determine a voted clock value based on a set of the clock values that were broadcast; a clock setter to set the clock values of each node in the network to the voted clock value; an array of clock timers to store each of the clock values that have been broadcast; a clock comparer to determine which nodes are synchronized based on the clock values stored in the array; and a synchronization indicator to designate the nodes that are synchronized.

In accordance with another embodiment of the invention, an apparatus to synchronize nodes in a network is described that comprises a communication interface associated with each node in the network, the interface capable of broadcasting a clock value of the associated node and receiving the clock values that have been broadcast; a plurality of logic gates associated with each node; the logic gates arranged to be able to determine a voted clock value based on a set of the clock values that have been broadcast and arranged to be able to determine which nodes are synchronized by comparing the clock values; and a memory register associated with each node to store flags indicating the synchronization status of each node. BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a layout of a prior art network that utilizes one embodiment of the present invention.

Fig. 2 is a diagram of the functional components of one illustrative embodiment of the present invention.

Fig. 3 is a flow chart depicting the synchronization process in accordance with one embodiment of the present invention .

DESCRIPTION OF THE INVENTION Fig. 1 depicts a typical network system that utilizes my invention. Network nodes 101 are connected through communication buses 103 to form the network system. Associated with each network node is a redundancy management system (RMS) 105 that performs many fault tolerant functions including synchronization of the nodes 101 within the network. The RMS 105 can utilize logic gates to implement certain synchronization functions through a field programmable gate array (FGPA) .

The communication buses 103 between the nodes can utilize communication protocols such as IEEE 1394 that can be utilized in the present invention. IEEE 1394 is a hardware and software standard for transporting data at

100, 200, or 400 megabits per second (Mbps) . One useful feature of the IEEE 1394 communication protocol is the ability to periodically broadcast the nodes' local clocks .

Fig. 2 depicts the functional components of one illustrative embodiment of a system 201 in accordance with my invention and comprising a part of the redundancy management system 105 shown in Fig. 1. In accordance with my invention, all of the structural elements of system 201 are implemented by a field programmable gate array. The system 201 has a clock receiver module 203 that accepts clock values broadcasted from the other nodes in the network. The clock receiver module 203 stores each clock value into a timer that corresponds to that node. The clock receiver module 203 maintains an array of clock timers 204 for all the nodes in the network. Once stored, each clock value is incremented according to that node's internal tick frequency.

The system 201 also has a voter module 205 that determines the correct clock value to be used by all of the nodes in the network. The voter module 205 applies a voting algorithm to the clock timers in the array to arrive at the voted clock value. Each node applies the same algorithm in that determination. Since each node will receive the same clock values from the other nodes in the network and apply the same algorithm, each voter module 205 will arrive at the same voted clock value.

The voting algorithm used is a fault-tolerant mid- value select algorithm. The voted clock is the average of the extreme values of set Xf as shown in the following equation :

Set X_f is not necessarily all of the clock values in the array. Depending on when the voting takes place, the set of clock values can be restricted to those clock values pertaining to nodes that are synchronized with another node. In the initial synchronization round, all of the nodes can potentially be included in the set. After it has been determined that some nodes in the network are synchronized, only those nodes will be considered in the voting algorithm.

In addition, faulty values can be excluded from the operative set. The following equation describes the formation of the set of clock values to be used.

V \X\ ≥ max(2, 3/) = reduce(x)

The set X has a minimum number of clock values greater than or equal to the maximum of either two clock values or three times the number of faulty clock values in the set. Faulty values are clock values from nodes that are misrepresentations of its clock value and have a large variation from the other clock values. Discarding the faulty extreme values from the set X forms the reduced set X_f.

System 201 has a clock setting module 207 that directly interacts with the node's clock. Clock setting module 207 is capable of obtaining the clock value of the node and broadcasting the clock value to the other nodes. Clock setting module 207 also can set the local clock of the node to the voted clock value.

An in-sync detector module 209 is also part of system 201. The in-sync detector module 209 compares the clock values to determine the synchronization status and manages the flags that indicate whether a node is synchronized with other nodes.

Synchronization is based on the concept of an atomic period composed of a finite number of indivisible ticks. The atomic period determines the synchronization interval and the number of ticks determines the synchronization precision. The smallest clock adjustment that can be made is one tick, therefore the number of ticks in an atomic period ultimately determines the synchronization jitter. There is one synchronization cycle and hence one clock adjustment per atomic period. Synchronization jitter is substantially determined by the accuracy of the crystal oscillator driving the local timers and by the broadcasted clock latency.

When the in-sync detector module 209 for a node determines that it is synchronized with at least one other node, the clock for that node is reset to a starting clock value (typically 0) and the in-sync detector module 209 sets its "IN_SYNC" flag to true. A node is in sync with another node when the difference between its clock, measured in hardware timer ticks, and the other node's clock is less than an application- dependent tick tolerance. Fig. 3 depicts a flow chart that describes the operation of the invented system in synchronizing the individual nodes of the network. In the first step 301, each node will broadcast its clock value to all the other nodes. During step 301, each node will be receiving the clock values of the other nodes.

Broadcasting the clock value to other nodes occurs periodically after a set time interval has passed. That time interval should be large enough to allow clock values from all the nodes to be broadcast and received. The invented system utilizes the IEEE 1394 protocol's ability to periodically broadcast the nodes' local clock. Although the local clock is broadcast after a set time interval pursuant to the IEEE 1394 protocol, the time it is broadcast is controlled by the invented system since it sets the nodes' local clock which triggers the broadcast. Along with each clock value broadcast, each node will broadcast its in-sync flag. If synchronization is occurring for the first time, the value of the in-sync flags is set to false.

After receiving the clock values and in-sync status from the other nodes, the next step 303 is to calculate the voted clock value. The voting is usually performed near the middle of each atomic period. Each node then resets its clock value to that voted clock value in step 305.

In step 307, each node compares the clock values from all of the nodes in the network to determine the in- sync status. The clock values to be compared are the values that have been stored in the corresponding timers and not the clock values that have been readjusted in step 305. By comparing these stored values, each node can determine which nodes of the network have synchronized clock tickers. The steps outlined in Fig . 3 are continually repeated to ensure synchronization . By restricting the set of clock values by which the voted clock value is determined to those nodes that are synchronized with each other, each iteration of the steps will bring the voted clock to a value consistent with the most nodes .

The steps detailed in Fig . 3 can be implemented in hardware through a FPGA . By having a separate hardware system to implement the synchronization process , the quality requirements on each node' s processors are not as high and can be set at the level sufficient to perform the main purposes of the network .

Implementing the synchronization process on a FPGA can be accomplished by encoding the following function onto the gate array .

FUNCTIONS } SynchronizeO

SYNC TIME tick_pos; COUNT in sync; tickjpos = REMAINDER(g_local_clk, SYN_PREC); IF{1} (tickjrøs .EQ. PERIOD_START)

IF{2} (g_first_to_sync .EQ. TRUE) g_local_clk = STARTING_CLOCK _VALUE; g_first_to_sync = FALSE; END IF{2} g_in_sync_flag = g_next_sync_flag; IF{3} (g isten .LT. LISTEN TME) INCREMENT(g isten);

END IF{3} ELSE IF{ 1 } (tick_pos .EQ. SYNC_RUN .AND. g_rcvd[OWN_NID] .EQ. AS_RCVD .AND. g isten .EQ. LISTEN_TIME) Reset_Sync_Flag(g_local_clk); Compute ClockO;

TEMPORARY COUNT i = 0; TEMPORARY SYNC_TIME delta; in sync = 0;

WHILE{1 } (i .LT. NUM_NODES) g_rcvd[i] = NOT_RCVD; delta = Get_Delta(g_global_clks[i], g_global_clks[OWN_NID]); IF {4} (delta .LE. SYN_TOL)

INCREMENT(in_sync); END IF{4}

INCREMENT(i); END WHILE{1 } IF{5} (in_sync .GE. 2)

Set_Sync_Flag(g_local_clk); IF {6} (g_nodes_in_sync .NE. 0) g_next_sync_flag = IN_SYNC; END IF{6} ELSE{5} g_next_sync_flag = NO_SYNC; g_fιrst_to_sync = FALSE;

END IF{5} END IF{1 }

END FUNCTION{ 1 }

FUNCTION{2} Get_Delta(SYNC_TIME clkl, SYNC_TIME clk2) RETURNS SYNC TIME

Reset_Sync_Flag(clk 1 ) ; Reset_Sync_Flag(clk2);

RETURN ABSOLUTE_VALUE(clkl - clk2);

END FUNCTION {2}

FUNCTION{3} Compute_Clock()

SYNC_TIME local_delta, voted_clk; SYNC TIME clk_array[NUM_NODES]; SYNC TIME in_sync_clk_array[NUM_NODES]; COUNT msgs_rcvd, in sync msgs rcvd, i, j, k; BOOLEAN_FLAG is_in_sync = NO SYNC; i = 0; j = 0; k = 0; msgs_rcvd = 0; in_sync_msgs_rcvd = 0; WHILE{1 } (i XT. MAX NODES) IF{ 1 } (g_rcvd[i] .EQ. WAS_RCVD) IF{2} (Test_Sync_Flag(g_global_clks[i]) .EQ. TRUE) in_sync_clk_array[k] = Reset_Sync_Flag(g_global_clks[i]); INCREMENT(k);

INCREMENT(in_sync_msgs_rc vd) ; IF{3} (i .EQ. OWN_NID) is_in_sync = IN_S YNC;

END IF{3} END IF{2} clk_array[j] = Reset_Sync_Flag(g_global_clks[i]); INCREMENTO); rNCREMENT(msgs_rcvd);

END IF{1 } INCREMENT(i); END WHILE{1 } IF{4} (in_sync_msgs_rcvd .GE. 2)

IF{5} (g_nodes_in_sync .EQ. 0 .AND. is_in_sync .EQ. IN_SYNC) g_first_to_sync = TRUE;

END IF {5} g_nodes_in_sync = in_sync_msgs_rcvd; SORT(in_sync_clk_array);

IF{6} (in_sync_msgs_rcvd .EQ. 4) voted_clk = (in_sync_clk_array[l] + in_sync_clk_array[2]) / 2; ELSE IF{6} (in_sync_msgs_rcvd .EQ. 3) voted_clk = in_sync_clk_array[l];

ELSE IF {6} (in_sync_msgs_rcvd .EQ. 2) voted_clk = (in_sync_clk_array[0] + in_sync_clk_array[l]) / 2; END IF {6} ELSE{4} SORT(clk_array);

IF{7} (msgs_rcvd .EQ. 4) voted_clk = (clk_array[l] + clk_array[2]) / 2; ELSE IF{7} (msgs_rcvd .EQ. 3) voted_clk = clk_array[l]; ELSE IF{7} (msgs_rcvd .EQ. 2) voted_clk = (clk_array[0] + clk_array[l]) / 2; ELSE IF{7} (msgs_rcvd .EQ. 1) voted_clk = clk_array[0]; ELSE IF{7} (msgs rcvd .EQ. 0) voted_clk = local_clk; END IF{7} END IF{4} local_delta = voted_clk - g local clk; g local clk = g_local_clk + local delta; END FUNCTIONS }

The code is presented as a block-structured pseudo-language with the following conventions:

• Language keywords and relational operators are UPPERCASE and are presented in courier bold TYPEFACE.

• Constants are UPPERCASE and are presented in courier TYPEFACE.

• Comments begin with a double-slash (//), continue to the end of the line, and are presented in the courier italic typeface.

• Global variables are prefixed with a g_ as in g local clk.

• Flow control statements are annotated with a numeric value enclosed in curly braces for readability (i.e. IF{ 1 } (expression) begins an IF flow control statement and END IF{ 1 } ends the statement).

• The language operators are as follows:

SORT(array) Sorts array yielding array[n] < array[n+l] ...

REMAINDER(A, B) Returns the remainder of A ÷ B ABSOLUTE_VALUE(A) Returns the integer absolute value of A ■ INCREMENT(A) Increments A, i.e. A = A + 1

A .EQ. B Equivalence, returns true if A ≡ B

A .NE. B Not equal, returns true if A ≠ B

A .LT. B Less than, returns true if A < B

A .GT. B Greater than, returns true if A > B ■ A .LE. B Less or equal, returns true if A < B

A .GE. B Greater or equal, returns true if A > B

A .AND. B Logical AND, returns true if A Λ B

A .OR. B Logical OR, returns true if A v B The first section of the pseudo-code defines constants, special types and global variable declarations. The comments within this section describe the purpose of each declaration.

CONSTANT TRUE = 1; //Boolean true value CONSTANT FALSE = 0; // Boolean false value CONSTANT SYN_PREC = 1000; // Ticks per atomic period CONSTANT PERIOD START = 0; // Tick count at an atomic period

// boundary

CONSTANT STARTING_CLOCK_VALUE = 0; // Clock start value for initial

// operating set

CONSTANT SYNC_RUN = SYN_PREC * 0.10; // Sync function delay within an

// atomic period

CONSTANT NUM NODES = 4; //Number of nodes in system CONSTANT LISTEN JTIME = 2; //Listening time on startup in

// atomic periods

CONSTANT OWN NID = HDWR_SETTING; //Set by hardware strapping EQUIVALENCE WAS_RCVD TRUE; // WAS_RCVD is boolean true EQUIVALENCE NOT_RCVD FALSE; //NOT RCVD is boolean false EQUIVALENCE IN_SYNC TRUE; //IN SYNC is boolean true EQUIVALENCE NO_SYNC FALSE; //NO_SYNC is boolean false TYPE SYNCJTIME IS INTEGER : 32 BITS; // The clock type (32 bits) TYPE BOOLEAN_FLAG IS UNSIGNED : 1 BIT; //A boolean flag TYPE COUNT IS INTEGER : 32 BITS; //An integer count (32 bits) SYNCJTIME g_local_clk = 0; // The local node 's timer

SYNCJTIME g_global_clks[NUMJNODES]; // The clock database

BOOLEAN_FLAG g_fιrst_to_sync = FALSE; //Indicates that this node is a

// member of the first operating

//set

BOOLEAN FLAG g in_sync_flag = FALSE; // This node is in sync BOOLEAN FLAG g next sync flag = FALSE; // This node will be in sync at the

// next atomic period boundary BOOLEAN FLAG g_rcvd[NUM_NODES] = FALSE; //Indicates that a clock message

// was received from a node. COUNT g nodes in sync = 0; // The number of nodes that this

// node believes are in sync COUNT g_listen = 0; // Startup delay counter

The present invention is not to be considered limited in scope by the preferred embodiments described in the specification. Additional advantages and modifications, which will readily occur to those skilled in the art from consideration of the specification and practice of the invention, are intended to be within the scope and spirit of the following claims.

Claims

What is claimed is: 1. A method for synchronizing nodes in a network comprising the steps of: broadcasting the clock value of each node on the network; determining a voted clock value based on a set of said clock values that were broadcast; comparing the clock value of each node to determine which nodes are synchronized with each other; resetting each node's clock to said voted clock value; and setting flags to indicate which nodes are synchronized.

2. The method as claimed in claim 1, wherein said set of clock values comprises nodes that are indicated to be synchronized by said flags.

3. The method as claimed in claim 1, wherein said determining step is implemented on a field programmable gate array.

4. The method as claimed in claim 1, wherein said broadcasting step is implemented on a field programmable gate array.

5. The method as claimed in claim 1, wherein said broadcasting, determining, resetting, comparing and setting steps are continuously repeated.

6. The method as claimed in claim 1, wherein said broadcasting step is implemented through communications protocol IEEE 1394.

7. The method as claimed in claim 1, wherein each node in said network separately performs said broadcasting, determining, resetting, comparing and setting steps.

8. The method as claimed in claim 1, wherein said determining step utilizes a mid-value voting algorithm.

9. The method as claimed in claim 1, wherein said comparing step is implemented on a field programmable gate array.

10. The method as claimed in claim 1, wherein said setting step is implemented on a field programmable gate array.