WO2002029529A1 - Method and system for synchronizing clocks on networked computers - Google Patents

Method and system for synchronizing clocks on networked computers Download PDF

Info

Publication number
WO2002029529A1
WO2002029529A1 PCT/US2001/030918 US0130918W WO0229529A1 WO 2002029529 A1 WO2002029529 A1 WO 2002029529A1 US 0130918 W US0130918 W US 0130918W WO 0229529 A1 WO0229529 A1 WO 0229529A1
Authority
WO
Grant status
Application
Patent type
Prior art keywords
clock
system
processor
networked computers
data
Prior art date
Application number
PCT/US2001/030918
Other languages
French (fr)
Inventor
Marcus Matuszak
Mark La Count
Rod Ylst
Original Assignee
Mindtronics Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00Details not covered by groups G06F3/00 – G06F13/00 and G06F21/00
    • G06F1/04Generating or distributing clock signals or signals derived directly therefrom
    • G06F1/14Time supervision arrangements, e.g. real time clock
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00Details not covered by groups G06F3/00 – G06F13/00 and G06F21/00
    • G06F1/04Generating or distributing clock signals or signals derived directly therefrom
    • G06F1/12Synchronisation of different clock signals provided by a plurality of clock generators
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04JMULTIPLEX COMMUNICATION
    • H04J3/00Time-division multiplex systems
    • H04J3/02Details
    • H04J3/06Synchronising arrangements
    • H04J3/0635Clock or time synchronisation in a network
    • H04J3/0638Clock or time synchronisation among nodes; Internode synchronisation
    • H04J3/0658Clock or time synchronisation among packet nodes
    • H04J3/0661Clock or time synchronisation among packet nodes using timestamps
    • H04J3/0667Bidirectional timestamps, e.g. NTP or PTP for compensation of clock drift and for compensation of propagation delays
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04JMULTIPLEX COMMUNICATION
    • H04J3/00Time-division multiplex systems
    • H04J3/02Details
    • H04J3/06Synchronising arrangements
    • H04J3/0635Clock or time synchronisation in a network
    • H04J3/0676Mutual

Abstract

A method and system for synchronizing clocks on networked computers (206) is provided. This invention works with modem high-speed workstations and networks to improve clock synchronization between the computers. This invention employs techniques, which accurately predict the phase and frequency noise for each synchronization source and network path. Techniques are included which reduce the impact of delay spikes and oscillator wander while speeding up initial convergence. Also included are improved clock discipline processes, which can operate in frequency-lock, phase-lock and hybrid modes. This invention is specifically adapted to synchronize the clocks of host computers, routers, servers and workstations on LANs, WANs and the Intranet, WAN or LAN.

Description

METHOD AND SYSTEM FOR SYNCHRONIZING CLOCKS

ON NETWORKED COMPUTERS

Background of the Invention Field of the Invention. This invention relates to methods and systems for synchronizing the clocks of computers on a network. More specifically, this invention relates to methods and systems for clock synchronization of networked computers that is adapted specifically to the synchronization of host computers and routers and active network components in an intranet, WAN or LAN. Description of Related Art. A variety of computer clock synchronization techniques have been proposed. Many of these techniques are well known in the art. These prior systems, however, are unable to provide the desired accuracy necessary for high speed "real-time" Intranet, WAN or LAN computing. The reader is referred to the following U.S. patent documents for general background material. Each of these patents is hereby incorporated by reference in its entirety for the material contained therein.

U.S. Patent No. 4,276,594 describes a digital computer with the capability of incorporating multiple central processing units that uses an address and data bus between each central processing unit and from one to fifteen intelligent composite memory and input/output modules.

U.S. Patent No. 4,280,221 describes a digital data communication system that includes a data source and a source interface, a digital data bus, for transferring encoded information from one data source to one or more receivers, each having a receiver interface.

U.S. Patent No. 4,530,091 describes a packet switching system in which a remote real-time clock is accurately synchronized to a standard real-time clock via X.25 controlled high-speed transmission links.

U.S. Patent No. 4,683,530 describes an interconnecting transparent serial bus for extending a parallel CPU domain to a parallel peripheral module domain that includes a bi-directional serial protocol for transferring information between the CPU and one or more peripheral module controllers. U.S. Patent No. 4,685,106 describes a high rate multiplexing system, which is capable of operating on multiple channels while ensuring synchronization of the system operation.

U.S. Patent No. 4,918,689 describes a method and circuitry for encoding and decoding signals using combined time-signal path diversity techniques that trade-off bandwidth for autonomy of communications and simplified signal processing.

U.S. Patent No. 5,020,081 describes a communication link interface having an assembly register which is loaded in response to a transmitter's clock and unloaded in response to a receiver's clock connected to the interface.

U.S. Patent No. 5,054,020 describes a general purpose communications processor for communicating data between an asynchronous data terminal and a synchronous data communication facility.

U.S. Patent Nos. 5,144,692 and 5,369,749 describe the functions of two virtual operating systems merged into one physical system. U.S. Patent No. 5,341,368 describes a digital switching system that includes a digital switching matrix having an input bus and an output bus; a plurality of information systems with each information system having an information bus for transmitting and receiving information using a protocol which is different from a protocol used by the information bus of each of the other information systems.

U.S. Patent No. 5,347,512 describes a telecommunication system having a multiport receiving switch with a central control circuit for distributing information received over a multichannel link from a multiport sending switch to at least one of a plurality of various applications in the telecommunication system. U.S. Patent No. 5,440,759 describes a modular networking switch system for a land mobile transmission trunked communication system provides transmission trunked communications with a plurality of transceivers and networks the plurality of transceivers with remote transceivers in one or more external communication networks. U.S. Patent No. 5,471,617 describes a method of managing a plurality of networked manageable devices, which include at least one file server having a system board, a drive array subsystem associated with the file server and a server manager installed in the file server for monitoring the system board from a manager console using a management information base. U.S. Patent No. 5,491,799 describes a system communication interface unit that provides uniform communication between hardware and software units of a computer system. U.S. Patent No. 5,517,532 describes a clock phase and frequency distribution system which uses a standing sine wave and provides substantially simultaneous significant crossing instances everywhere in the system while using low power and not requiring bus termination or precise control of transmission path characteristics. U.S. Patent No. 5,574,849 describes a synchronized data transmission between elements of a processing system, where two identical streams of multi-bit symbols are received by a pair of storage elements.

U.S. Patent No. 5,583,856 describes a large capacity switch architecture, which uses a split interface. U.S. Patent No. 5,604,771 describes a system and method for transmitting computer data by converting and modulating the data into a sound file, allowing the sound file to be stored for later retrieval and transmission.

U.S. Patent No. 5,623,357 describes a method and apparatus for establishing an intelligent demarcation between facility and terminal segments of asynchronous optical network.

U.S. Patent No. 5,655,149 describes an integrated network switch system, which can uniquely and consistently detect and designate each processor in a bus without the relative position information, and allow a dynamic reallocation of the processors according to their designations. U.S. Patent No. 5,657,317 describes a hierarchical communication system in which wireless local area networks exhibiting substantially different characteristics are employed in an overall scheme to link portable or mobile computing devices. U.S. Patent No. 5,657,440 describes a method and means for asynchronous remote data duplexing at a distant location from copies based at a primary site storage subsystem.

U.S. Patent No. 5,689,689 describes a multiprocessor system that includes a number of sub-processor systems, each substantially identically constructed, and each comprising a central processing unit, and at least one I/O device.

U.S. Patent No. 5,696,903 describes a hierarchical communication system in which two wireless local area networks exhibit substantially different characteristics are employed to link inherently portable or mobile computer devices. U.S. Patent No. 5,701 ,502 describes a method and system for merging the functions of two virtual operating systems into one physical system.

U.S. Patent No. 5,748,627 describes an integrated network switch with an elastic linear buffer.

U.S. Patent No. 5,751 ,955 describes a method of synchronizing a pair of central processor units for duplex, lock-step operation by copying data into a corresponding location of another member.

U.S. Patent No. 5,790,536 describes a hierarchical communication system, arranged in a spanning tree configuration, is described in which wired and wireless communication networks exhibiting substantially different characteristics are employed in an overall scheme to link portable or mobile computing devices.

U.S. Patent No. 5,790,776 describes a multiprocessor system that includes a number of sub-processor systems, each substantially identically constructed, and each comprising a central processing unit and at least one I/O device, interconnected by routing apparatus that also interconnects the sub-processor systems.

U.S. Patent No. 5,815,583 describes a method for communicating information from a first integrated circuit to a second integrated circuit. A sync signal is transmitted between the two integrated circuits to indicate the start of a transmission of a frame of information from the first integrated circuit to the second integrated circuit.

U.S. Patent No. 5,828,664 describes an integrated network switch that operates in either a full-frame or half -frame switching mode on a circuit-by-circuit basis.

U.S. Patent No. 5,838,894 describes a computing system that includes a pair of central processor units structured to operate in substantial synchronism to each execute the same instruction at substantially the same moment in time of identical instruction streams to achieve a logical central processor unit with fail-functional operation.

U.S. Patent No. 5,857,113 describes an integrated network switch system, which can uniquely and consistently detect and designate each processor on a bus without the relative position information, and allow a dynamic reallocation of the processors according to their designations. U.S. Patent No. 5,907,690 describes a control system for controlling a very large number of remotely located input and output devices that has a single CPU coupled through a host interface module to a data and control transmission line. U.S. Patent No. 5,912,888 describes an apparatus and method for enabling bilateral transmission of digital data between a local area network and telephone company networks employing both analog and digital telephone lines.

U.S. Patent No. 5,933,607 describes a digital communication system, protocol and method that facilitate information transfer that includes user and control information from Continuous Bit Rate and/or non-Continuous Bit Rate signal sources.

U.S. Patent No. 5,949,776 describes a hierarchical communication system in which wireless local area networks exhibit substantially different characteristics and are employed in an overall scheme to link portable or mobile computing devices. U.S. Patent No. 5,968,154 describes a serial arbitration method and system for rapidly and accurately identifying a station with the highest priority when a plurality of stations with different transmission rates are simultaneously requesting the use of a bus in a multi-point communication network where a plurality of communication stations share a common serial bus. U.S. Patent No. 6,006,069 describes a point-to-multipoint or two-way communications system that uses a nodal transmitter located in anode with a plurality of nodal antennas radiating different polarization signals about the node.

U.S. Patent No. 6,115,763 describes a data processing system, integrated circuit device, program product, and method that uses a service interface to provide external access to a plurality of cores integrated into an integrated circuit device.

Summary of the Invention It is desirable to provide a method and system for synchronizing the clocks of networked computer devices. It is particularly desirable to provide a method and system for highly accurate synchronizing the clocks of personal computers, workstations, hosts, servers, routers and the like connected over the Intranet, WAN or LAN to facilitate the shared computational procession of the networked computer devices. Accordingly, it is an object of this invention to provide a method and system for networked computer clock synchronization that provides high degrees of accuracy.

Another object of this invention is to provide a method and system for networked computer clock synchronization that permits operation at much longer poll intervals than is presently available.

A further object of this invention is to provide a method and system for networked computer clock synchronization that can synchronize the clocks of host computers and routers and active components connected over the Intranet, WAN or LAN. A still further object of this invention is to provide a method and system for networked computer clock synchronization that provides improved clock synchronization on local area networks (LAN) as well as wide area networks (WAN).

It is another object of this invention to provide a method and system for networked computer clock synchronization that can synchronize via radio, satellite or modem and/or via a hierarchical subnet.

It is a further object of this invention to provide a method and system for networked computer clock synchronization that provides high reliability through the use of redundant servers and diverse network paths. It is a still further object of this invention to provide a method and system for networked computer clock synchronization that reduces clock jitter.

A further object of this invention is to provide a method and system for networked computer clock synchronization that mitigates multiple sources. Another object of this invention is to provide a method and system for networked computer clock synchronization that avoids improperly operating servers.

A still further object of this invention is to provide a method and system for networked computer clock synchronization can function with all known computers, workstations, and server platforms available, from PCs to Grays, regardless of operating system as long as they are connected with a TCP/IP protocol.

Additional objects, advantages and other novel features of this invention will be set forth in part in the description that follows and in part will become apparent to those skilled in the art upon examination of the following or may be learned with the practice of the invention. The objects and advantages of this invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims. Still other objects of the present invention will become readily apparent to those skilled in the art from the following description wherein there is shown and described the preferred embodiment of this invention, simply by way of illustration as one of the modes best suited to carry out this invention. As it will be realized, this invention is capable of other different embodiments, and its several details, and specific electronic circuits, are capable of modifications in various aspects without departing from the invention. Accordingly, the drawings and descriptions should be regarded as illustrative in nature and not as restrictive.

To achieve the foregoing and other objectives, and in accordance with the purposes of the present invention, a system of a number of methods or techniques are combined and/or employed, including a clock filter technique, an intersection method, a clustering method, a clock discipline technique, and prediction functions.

Brief Description of the Drawings In order to show the manner that the above recited and other advantages and objects of the invention are obtained, a more particular description of the preferred embodiments of this invention, which is illustrated in the appended drawings, is described as follows. The reader should understand that the drawings depict only present preferred and best mode embodiments of the invention, and are not to be considered as limiting in scope. A brief description of the drawings is as follows. Figures la, lb and lc are top-level network diagrams showing typical computer networked devices incorporating the clock synchronization methods and system of this invention.

Figure 2 is a top-level block diagram of the preferred embodiment of the system of this invention.

Figure 3 is a diagram showing the preferred protocol data format of this invention.

Figure 4 is a block diagram of the dataflow analysis of this invention. Figures 5 a and 5b show the clock filter technique and filtering results preferred in this invention. Figures 6a and 6b are time-offset plots showing the performance of the preferred clock filter of this invention.

Figure 7a is a timing diagram of the intersection process of the preferred embodiment of this invention. Figure 7b is a process flow chart of the preferred intersection process.

Figure 8 is a process flow diagram showing the steps of the clustering process of the preferred embodiment of this invention.

Figure 9 is a block diagram of the clock discipline technique of the preferred embodiment of this invention. Figure 10 is a block diagram of the prediction functions of the preferred embodiment of this invention.

Figures 11a and 1 lb are plots of the phase and frequency response of the preferred clock discipline techniques of this invention.

Figure 12 is a timing diagram of the preferred clock discipline techniques of this invention.

Figure 13 is a block diagram of the error budget of the preferred embodiment of this invention.

Figure 14a is a state diagram of the preferred clock state machine of this invention. Figure 14b is a plot of time vs. frequency showing the time, frequency, and poll interval response for initial conditions 500-PPM frequency error and 100 ms time error. Figure 15 is a plot of the Allan deviation vs. time interval of the synchronization of this invention.

Figures 16a and 16b are plots comparing the FLL weight, PLL weight and Hybrid modes of the poll intervals. Figure 17 is a plot showing the standard error vs. poll intervals for various network paths.

Figure 18 is a plot of the measured performance of the synchronization of this invention.

Reference will now be made in detail to the present preferred embodiment of the invention, examples of which are illustrated in the accompanying drawings.

Detailed Description of the Invention This invention is a method and system for synchronizing networked computers, including workstations, personal computers, hosts, routers, hubs or any active component, thereby providing substantially improved clock accuracy. For the purposes of this patent disclosure the term "active component" shall be interpreted to mean: an electronic device employing a time oscillator for clock generation. The networks compatible with this invention include, but are not necessarily limited to local area networks (LANs), wide area networks (WANs), and any Intranet configuration. This invention makes use of a number of methods and/or techniques, including a clock filter technique, an intersection method, a clustering method, a clock discipline technique, and prediction functions, which cooperate to accomplish the purpose of this invention. The present embodiment of this invention, reduces clock error to the low microseconds thereby increasing clock accuracy, in the active component, by about a factor of ten when compared to known present synchronization methods, while permitting network communication at much longer poll intervals.

Figures la, lb and lc are top-level network diagrams showing typical computer networked devices incorporating the clock synchronization methods and system of this invention. In figure la the connection between Intranet, WAN or LAN primary servers lOla-f is shown connected 103a-f to campus secondary servers 102a- c. As shown, the campus secondary servers 102a-c are typically in communication 104d-i with each other and "buddy" servers 104a-c in other subnets. Typically, in the preferred embodiment of this invention, the primary servers 101 a-f are synchronized to a Uniform Time Clock (UTC) via a radio signal, a satellite link and/or a modem connection. While the preferred synchronization of the secondary or campus servers 102a-c is accomplished via a hierarchical synchronization subnet. Figure lb shows the connections 106a-d between the campus secondary servers 102a-d and the department servers 105a,b. The department servers 105a,b are in communication 107b with each other as well as other department servers 107a, 107d and "buddy" servers 107c, 107e in other subnets. The department servers 105a,b are preferably synchronized via a hierarchical synchronization subnet. Figure lc shows the connections 109a-c between department servers 105a-c and a typical workstation 108. The preferred workstation 108 synchronization is accomplished via a hierarchical synchronization subnet. This networking system, shown in figures la, lb and lc, provides high reliability because of the use of redundant servers and diverse network paths. In is present preferred embodiment, a Unix daemon program, portable to nearly every type of workstation and server platform currently available, from PCs to Grays, including computers running a wide range of computer operating systems, including but not limited to Unix, Windows, and VMS. The time synchronization of this invention flows from primary servers lOla-f, to the other servers 102a-d, 105a-c and 108 and clients.

Figure 2 is a top-level block diagram of the preferred embodiment of the system of this invention. This diagram is provided to show the cooperation between several of the processes or processors of this invention, which cooperate to accomplish the purposes of this invention. Remote Servers (RS) 201 are provided in communication with Peer Processes or Processors (PP) 202. The Peer Processes or Processors 202 are in turn in communication with the System Process or Processor (SP) 203, which in turn is in communication with the Clock Adjustment Process or Processor (CAP) 204. The Clock Adjustment Process or Processor 204 provides a feedback signal 205 to the Peer Processes or Processor 202. Each Peer Process or Processor 202 runs independently a poll interval determined by the System Process or Processor 203 and the Remote Servers 201. The System Processes or Processors 203 run at poll intervals as determined by the network phase jitter and local clock oscillator frequency wander. The Clock Adjustment Process or Processor 204 runs at up to 10,000 intervals per second to discipline the local clock phase and frequency. Additional processes and processors are used to provide name resolution and authentication services. Multiple Remote Servers 201 / Peers 206a-c and Peer Processes/Processors 202 / Filters 208a-c are used to provide redundancy and diversity. Each Peer 206a-c is in 2-way communication 207 a-c with a Filter 208a-c. The Filters 208a-c select the best sample from a window of eight clock offset samples. The System Process/Processors 203 includes Intersecting and Clustering processes/processors 210 as well as a Combining process/processor 212. The Intersecting and Clustering processes/processors 210 receives 209a-c the selected best sample from the Filters 208a-c. The Intersecting and Clustering processes/processors 210 serve to pick the best subset of peers and discard outliers. This picked best subset of peers is provided 211 to the Combining process/processor 212, which computes the weighted average of offsets for best possible accuracy. Receiving 213 the weighted average of offsets, from the Combining process/processor 212, is the Clock Adjustment Process/Processor 204. The Clock Adjustment Process/Processor 204 includes a Loop Filter 214 and a Local Clock Oscillator (LCO) to implement a hybrid phase/frequency (P/F) feedback loop to minimize clock jitter and wander. The Loop Filter 214 receives 213 the weighted average of offsets and produces a filtered signal 215 to the Local Oscillator 216. The Local Oscillator 216 produces the feedback signal 205.

Figure 3 is a diagram showing the preferred protocol data format of this invention. The Protocol Header Format 300 is 32-bits wide and has variable length. A Cryptosum block 318 has a first word 330, a Root Delay word 307, a Root Dispersion word 308, a Reference Identifier word 309, two Reference Timestamp words 310, two Originate Timestamp words 311, two Receive Timestamp words 312, two Transmit Timestamp words 313, an Extension Field 1 (which is optional) 314, an Extension Field 2 (which is also optional) 315. Following the Cryptosum block 318 is an optional Authenticator block 319 that includes a Key/Algorithm Identifier word 316 and two or four Message Hash words 317. The first word 330 is further provided with six fields; LI 301, NN 302, Mode 303, Strat 304, Poll 305 and Prec 306. LI 301 is a leap-warning indicator. VN 302 is a four-bit version number. Mode 303 indicates the operating mode. Strat 304 indicates the stratum (a number between 0 and 15). Poll 305 indicates the poll interval in log base 2 format. Prec 306 indicates the precision in log base 2 format. The preferred Timestamp Format 320, used in the Reference Timestamp 310, Originate Timestamp 311, Receive Timestamp 312, Transmit Timestamp 313, is composed of two 32-bit fields: Seconds 321 and Fraction 322. The value of the Timestamp Format 320 is in seconds and a fraction of a second since 0:00 am January 1, 1900. The present version 4 Extension Field 323, used in the Extension Field 1 314 and the Extension Field 2315, is composed of first word 331 consisting of a Field Length field 324 and a Field Type field 325. Following the first word 331 is an Extension Field block 326. The last field of the Extension Field block 326 is padded to a 64-bit boundary. The optional Authenticator block 319 is defined as shown 332, with a first word 327 for use in versions 3 and 4, a second word 328 used in version 4 only and an authentication only word 329. The present preferred authenticator uses DES-CBC or MD5 cryptosum of the header plus extension fields in version 4.

Figure 4 is a block diagram of the dataflow analysis of this invention. Each server process 401 a-c calculates the server variables offset Θ, delay Δ, and dispersion E relative to the root of the synchronization subtree. As each server calculated message 402a-c arrives at the peer processes 403a-c, the peer processes 403a-c update peer offset θ, delay δ, phase dispersion εr and frequency dispersion ε (in version 4) from the timeslips and clock filter processes. At system poll intervals, the clock selection and combining processes 405, having received 404a-c the updates of peer offset θ, delay δ, phase dispersion εrand frequency dispersion ε from the peer processes 403a-c, updates 406 the system variables 407: server variables offset Θ, delay Δ, and dispersion E. Generally, dispersions ε and E increase with time at a rate that depends on the specified frequency tolerance.

Figures 5a and 5b show the clock filter technique and filtering results preferred in this invention. The preferred clock filter technique uses the following equations to calculate the desired Offset θ and Delay δ, as shown in figure 5a: Offset Θ = ! 2 [(T2 - Tι) + (T3 - T4)]

Delay δ = (T4 - T - (T3 - T2). The most accurate clock offset θ is measured at the lowest delay δ, at the apex 503 of the wedge diagram of figure 5b, which plots Offset (in seconds) 501 verses Delay (in seconds) 502. Phase dispersion εris the weighted average of offset differences over the last eight samples and is used as an error estimator. Frequency dispersion ε^ represents the clock reading and frequency tolerance errors and is used as a distance metric. The synchronization distance, λ = ε/+ δ/2, is used as a distance metric and a maximum error boundary, since the correct time θ0 must be in the range of: θ - λ <= θo <= θ + λ.

Figures 6a and 6b are time-offset plots showing the performance of the preferred clock filter of this invention. Figure 6a, which is a plot of Offset, in seconds, 601 verses time, in days, 602 of the raw absolute clock offset data in semi- log coordinates for a network communication path between the east coast and the west coast over a period of six days, showing the relatively amount of clock offset variability. Figure 6b, which is a plot of Offset, in seconds, 603 verses time, in days, 604 of the data offset samples processed by the clock filter process of this invention. By comparing figure 6a with figure 6b, the reader can see that this invention reduces the standard of error in clocks by a factor of about ten and that this invention is particularly effective at removing error spikes.

Figure 7a is a timing diagram of the intersection process of the preferred embodiment of this invention. The DTS correctness interval 701, which is the intersection that contains points from the largest number of correctness intervals. The NTP correctness interval 702, is defined as θ - λ <= θ0 <= θ + λ. The intersection process of this invention requires that the midpoint of the intervals be in the intersection of the DTS correctness interval 701 and the NTP correctness interval 702. The preferred steps of this process, shown in figure 7b are as follows. For the purposes of this process, c and d are counter variables; m is the number of clocks;/is the number of falsetickers. As shown in figure 7, A 703, B 704, and C 705 are truechimers, while D 706 is a falseticker. Initially, the falsetickers/ (which is the number of presumed falsetickers) and counters c and d are set 707 to zero. Next, scan 708 from the far left endpoint: if a lower endpoint is encountered 709, adding 710 one to c and, if an upper endpoint is encountered 711, subtracting 712 one from c. If a midpoint is encountered 713, one is added 714 to d. A test 715 is made to determine if c >= m -/and \f d >= m -/, process success is established and the procedure is exited 716. If the 715 test is negative,/is incremented 717 by one. A test 718 is made to determine if/<= /2, if this test is positive, then failure is established and the process ends 719. Otherwise, the same process is performed starting from the far right endpoint 720. Figure 8 is a process flow diagram showing the steps of the clustering process of the preferred embodiment of this invention. Initially, the survivors of the intersection process are sorted 801 by increasing the synchronization distance, n is defined as the number of survivors and nmin is the lower limit. For each survivor -.,, the select dispersion (weighted sum of clock differences) is computed 802 between ■-,• and all others. Setting 803 smaxXa be the survivor with the maximum select dispersion, relative to all other survivors, and s„ύn to be the survivor with minimum sample dispersion, that is the clock differences relative to past samples of the same survivor. A test 804 is of whether smax<- _.m;„ or n <= nmi„ is made. If test 804 is not true, then the survivor smax is reduced 805 by one and the process returns to step 802. Otherwise, if test 804 is true, then the resulting survivors are processed 806 by the combining process to produce a weighted average that is used as the final offset adjustment, ending this process.

Figure 9 is a block diagram of the clock discipline technique of the preferred embodiment of this invention. As shown in this figure, N</ 904 is a function of the phase difference between the 901 ΝTP, time phase, and the LCO, local clock oscillator output 913; Vs 906 depends on the stage chosen and on the clock filter shift register, x 910 and y 911 are the phase update and the frequency update, respectively, computed by the prediction functions. In the preferred embodiment, the clock adjustment process runs once per second to compute Vc, which controls the frequency of the local clock oscillator 902. The local clock oscillator 902 phase is compared to the NTP phase to close the feedback loop. Receiving the NTP 901 signal and the output of the local clock oscillator 902 is a Phase Detector 903, which outputs the Vd 904 signal. The N 904 signal is received by a Clock Filter 905, which outputs the filtered signal V- 906. The filtered signal Ws 906 is received by a Phase/Frequency Predictor 908, which outputs the x 910 and y 911 signals that are received by the Clock Adjuster 909. The Clock Adjuster 909 provides the Nc that is the input signal to the Local Clock Oscillator 902. The Clock Adjuster 909 and the Phase/Frequency Predictor 908 are components of the Loop Filter 907. Figure 10 is a more detailed block diagram of the prediction function 908 of the preferred embodiment of this invention. Vs 906 is the phase offset produced by the clock filter 905 process, x 910 is the phase correction computed as a fraction of V, 906 by the Phase Corrector 1001. YFL 1005 is the frequency adjustment computed, by the FLL Predictor 1002, which receives Vs 906, as the average of past frequency offsets, y^ 1004 is the frequency adjustment computed by the PLL

Predictor 1003 of V, 906. yPLL 1004 and yFLL 1005 are combined by a summer 1003a according to a weight factor computed from past prediction errors to produce y 911.

Figures 11a and 1 lb are plots of the phase 1105 and frequency 1106 response of the preferred clock discipline techniques of this invention. Figure 11a shows the simulated time phase error in response to a step frequency change of 2 ppm and an initial condition of 1024 second poll interval. Offset 1101 is shown verses time 1102. Figure lib shows the simulated time frequency error in response to step frequency change of 2 ppm and also initial conditions of 1024 seconds poll interval. Offset 1103 is shown verses time 1104. These figures, 11a and 1 lb, show the typical response to a 2-degree (C) ambient temperature change, which is not unusual in general computing environments. The preferred way to reduce these errors is using a PPS discipline or and IRIG driver. Figure 12 is a timing diagram of the preferred clock discipline techniques of this invention. In the preferred embodiment the Unix adjtime() slews frequency at a net rate -R — φ 1210 PPM beginning at A 1203. Slew continues to B 1209, depending on the programmed frequency offset S 1207. Offset continues to C 1204 with frequency offset due to error φ 1211. If ε, which is adjustment between A 12-3 and B 1209, 1208 is less than x, then

R => φ + S and σ <= ( 1/ φ + 1/(R - φ)) x For ε 1208 = 100 μs, φ = 200 PPM, S = 200 PPM, this requires R => 400 PPM and σ <= 1 second, σ 1202 is the adjustment interval between A 1203 (and θ 1201) and C 1204. +S 1205 is shown relative to the 1 1206 axis. Figure 13 is a block diagram of the error budget of the preferred embodiment of this invention. The following is a definition of the notation used in figure 13. Constants (peers A and B) System variables

PA , PB - maximum phase error Θ - system clock offset

ΨA , ΦB - maximum frequency error Δ - root delay. Sample Variables E - root dispersion

T\ , -T2 , -T3 1 -T - protocol timestamps εs - select dispersion x - sample clock offset ε - system dispersion y - sample roundtrip delay Packet Variables (from peer B . z - sample dispersion Q' - peer system clock offset τ - time interval since last update Δ' - peer root delay

Peer Variables E' - peer root dispersion θ - peer clock offset τ' - time interval since last update δ - peer roundtrip delay εr- phase dispersion εf- frequency dispersion w - clock filter weight The Sample Variables 1301 are used to calculate the Peer Variables 1302 according to the equations shown. The Peer Variables 1302 are summed 1303, 1304 with the Packet Variables 1308 to calculate the System Variables 1305. The System Dispersion 1307 is calculated using the sum 1306 of Peer Variables 1302 and System Variables 1305. Figure 14a is a state diagram of the preferred clock state machine 1400 of this invention. This state machine 1400 is designed specifically for fast initial convergence and recovery from very large disruptions. The initial state UNSET 1401 progresses to the HOLD 1403 upon detection of the first set 1402. The state machine 1400 stays in the HOLD 1403 state, a minimum of five updates to allow clock time and frequency to converge, and until a θ < 128 ms and τ = 0 sec. status 1404 is detected, at which point the machine 1400 progresses to the SYNC state 1405. So long as θ < 128 ms and τ = 0 sec. status 1406 remain true the machine 1400 remains in state SYNC 1405. When θ > 128 ms and τ > 900 seconds status 1407 is true the machine 1400 progresses to SPIKE state 1408. Once in the SPIKE state 1408, the machine 1400 transitions to the HOLD state 1403 if θ < 128 ms and τ = 900 sec. status 1409, and alternatively transitions to the SYNC state 1405 if θ < 128 ms and τ = 0 sec. status 1410. Figure 14b shows the time 1413, frequency 1414, and poll interval response

1415 for initial conditions equal to 500 PPM frequency error and 100 ms time error on a graph having Frequency (PPM) or Time (ms) axis 1411 verses a Time (hour) axis 1412.

Figure 15 is a plot of the Allan deviation 1501 vs time interval 1502 of the synchronization of this invention operating on a variety of clocks and/or servers. Specifically, the IEN 1503 of Torino, Italy is shown, the USNO (US Naval Observatory) 1504 Washington, D.C.; PEERS (19 non-local time servers in Europe, Japan, Australia, North America and South America) 1505; BARN (local time server on Dcnet) 1506; LAN (free running clock via Ethernet) 1507; PPS (free running clock via a PPS signal) 1508, and NOISE (free running synthesized clock) 1509. The "V shaped curves, 1507, 1508, 1509 show local servers with free running clocks. The other curves 1503, 1504, 1505, 1506 show remote servers synchronized to a GPS. The curves with a slope of -1 represent white phase noise due to network jitter. While lines with a slope of +0.5 represent random-walk frequency noise due to clock oscillator wander. The intersection of the phase and the frequency noise lines is referred to as the Allan intercept. Generally, PPL is the better technique when Tc is below the Allan intercept, while FLL is the preferred technique when Tc is above the Allan intercept. Figures 16a and 16b are plots comparing the FLL weight, PLL weight and Hybrid modes of the poll intervals. Figure 16a shows a plot of the FLL 1604 and a plot of PLL 1603 weight 1601 verses the poll interval 1601. Figure 16b shows a plot of the standard error 1605 verses poll interval 1606 for the FLL 1609, PLL 1607 and Hybrid 1608 modes. It can be seen from these graphs, figures 16a, 16b, that FLL is preferred above 200 seconds, while PLL is preferred below 200 seconds. The Hybrid mode is preferred between 100 seconds and 1000 seconds and is close in its performance to FLL and PLL outside this range. PLL becomes unstable above 4000 seconds due to a loss of lock. Figure 17 is a plot showing the standard error 1701 vs. poll intervals 1702 for various network paths, including JEN 1706, USNO 1705, PEERS 1704, and BARN 1703. All of the servers of this graph are synchronized to a GPS and all processes of this invention are operative. The solid lines show the hybrid mode performance and the dashed lines the PLL mode, both over a ten-day period. This figure shows that the Hybrid mode is better than the PLL mode by up to a factor of 10 over an important range of poll intervals. The local time server, BARN 1703, is better than 200 μ seconds standard error at a poll interval equal to 64 seconds. All non-local time servers are better than 2 ms at poll interval <= 1024 seconds. The standard error of all non-local time servers (including the best USNO 1705) is better than any server separately.

Figure 18 is a plot of the measured performance, clock offset 1801 verses time (days) 1802, of the synchronization of stratum-2 servers synchronized to remote primary servers using this invention. Except for PPS 1803, which uses simulated phase noise, all use actual network noise measured in real time. Frequency noise is simulated with a curve to fit to PPS data. The PPS 1803 shown has a mean poll interval of 64 seconds, a mean error of 0 ms, an RMS error of 0.006 ms, and a maximum error of 0.026 ms. The BARN 1804 shown has a mean poll interval of 1024 seconds, a mean error of 0.186 ms, an RMS error of 1.019 ms, and a maximum error of 2.949 ms. The USNO 1805 shown has a mean poll interval of 1020 seconds, a mean error of 1.541 ms, an RMS error of 5.234 ms, and a maximum error of 18.66 ms. The -EN 1806 shown has a mean poll interval of 1024 seconds, a mean error of 20.36 ms, and RMS error of 38.00 ms, and a maximum error of 98.20 ms. The foregoing description is of a preferred embodiment of the invention and has been presented for the purposes of illustration and description of the best mode of the invention currently known to the inventors. This description is not intended to be exhaustive or to limit the invention to the precise form, connections or choice of the components disclosed. Obvious modifications or variations are possible and foreseeable in light of the above teachings. This embodiment of the invention was chosen and described to provide the best illustration of the principles of the invention and its practical application to thereby enable one of ordinary skill in the art to utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated by the inventors. All such modifications and variations are intended to be within scope of the invention as determined by the appended claims when they are interpreted in accordance with the breadth to which they are fairly, legally and equitably entitled.

Claims

ClaimsWe claim:
1. A system for synchronizing clocks on networked computers, comprising: (A) a peer processor; (B) a filter processor in data communication with said peer processor;
(C) an intersection processor in data communication with said filter processor;
(D) a clustering processor in data communication with said intersection processor; (E) a combining processor in data communication with said clustering processor; and (F) a clock adjustment processor in data communication with said combining processor.
2. A system for synchronizing clocks on networked computers, as recited in claim 1, wherein said clock adjustment processor further comprises a loop filter.
3. A system for synchronizing clocks on networked computers, as recited in claim 1, wherein said clock adjustment processor further comprises a local oscillator.
4. A system for synchronizing clocks on networked computers, as reciting in claim 1, wherein said intersection processor further comprises a means for determining the correctness interval of clock intervals.
5. A system for synchronizing clocks on networked computers, as recited in claim 1, wherein said clustering processor further comprises a means for picking a subset of peers and discarding outliers.
6. A system for synchronizing clocks on networked computers, as recited in claim 1, wherein said filter processor further comprises a means for selecting a sample from a window of clock offset samples.
7. A system for synchronizing clocks on networked computers, as recited in claim 1, wherein said combining processor further comprises a means for computing the weighted average offset of a clock.
8. A system for synchronizing clocks on networked computers, as recited in claim 1, wherein said clock adjustment processor further comprises a means for implementing a hybrid phase/frequency lock feedback loop.
9. A system for synchronizing clocks on networked computers, as recited in claim 1, further comprising a primary server, a campus server, a department server and a workstation.
10. A system for synchronizing clocks on networked computers, as recited in claim 1, further comprising a network selected from the group consisting of local area networks, wide area networks, an Intranet and active devices that use a TCP/IP protocol computer containing a UTC layer.
11. A method for synchronizing clocks on networked computers, comprising: (A) processing a peer clock data on a remote server to produce clock offset data; (B) filtering said clock offset data to produce filtered clock data;
(C) intersecting said filtered clock data to select data within an appropriate interval; (D) clustering said selected data to produce a weighted average of said filtered and selected data;
(E) combining said weighted average data to produce a weighted average of clock offsets; and (F) phase/frequency locking said clock offsets to produce a synchronized clock signal.
12. A method for synchronizing clocks on networked computers, as recited in claim 11, wherein said processing a peer clock data, further comprises calculating server variables and updating said server variables.
13. A method for synchronizing clocks on networked computers, as recited in claim 11, wherein said filtering said clock offset data further comprises selecting a clock offset sample.
14. A method for synchronizing clocks on networked computers, as recited in claim 11, wherein said intersecting said filtered clock data further comprises identifying truechimers and falsetickers.
15. A method for synchronizing clocks on networked computers, as recited in claim 11, wherein said clustering said intersected data further comprises producing a weighted average to be used in the final offset adjustment.
16. A method for synchronizing clocks on networked computers, as recited in claim 11, wherein said phase/frequency locking further comprises adjusting a clock to produce a synchronized clock signal.
PCT/US2001/030918 2000-10-03 2001-10-02 Method and system for synchronizing clocks on networked computers WO2002029529A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US68012700 true 2000-10-03 2000-10-03
US09/680,127 2000-10-03

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
AU1138602A AU1138602A (en) 2000-10-03 2001-10-02 Method and system for synchronizing clocks on networked computers

Publications (1)

Publication Number Publication Date
WO2002029529A1 true true WO2002029529A1 (en) 2002-04-11

Family

ID=24729777

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2001/030918 WO2002029529A1 (en) 2000-10-03 2001-10-02 Method and system for synchronizing clocks on networked computers

Country Status (1)

Country Link
WO (1) WO2002029529A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004100411A1 (en) * 2003-05-06 2004-11-18 Philips Intellectual Property & Standards Gmbh Method for temporal synchronization of clocks
US8731003B2 (en) 2009-01-16 2014-05-20 Tellabs Oy Method and arrangement for adjustment of a clock signal
CN104396180A (en) * 2012-06-19 2015-03-04 日本电气株式会社 Clock synchronization system, clock synchronization method, and storage medium whereupon clock synchronization program is stored

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5535217A (en) * 1994-12-20 1996-07-09 International Business Machines Corporation Method and apparatus for probabilistic clock synchronization with interval arithmetic
US5689688A (en) * 1993-11-16 1997-11-18 International Business Machines Corporation Probabilistic anonymous clock synchronization method and apparatus for synchronizing a local time scale with a reference time scale
US5958011A (en) * 1997-03-31 1999-09-28 International Business Machines Corporation System utilizing mastering and snooping circuitry that operate in response to clock signals having different frequencies generated by the communication controller
US6157957A (en) * 1998-01-22 2000-12-05 Cisco Technology, Inc. Clock synchronization system and method using a continuous conversion function for a communication network

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5689688A (en) * 1993-11-16 1997-11-18 International Business Machines Corporation Probabilistic anonymous clock synchronization method and apparatus for synchronizing a local time scale with a reference time scale
US5535217A (en) * 1994-12-20 1996-07-09 International Business Machines Corporation Method and apparatus for probabilistic clock synchronization with interval arithmetic
US5958011A (en) * 1997-03-31 1999-09-28 International Business Machines Corporation System utilizing mastering and snooping circuitry that operate in response to clock signals having different frequencies generated by the communication controller
US6157957A (en) * 1998-01-22 2000-12-05 Cisco Technology, Inc. Clock synchronization system and method using a continuous conversion function for a communication network

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004100411A1 (en) * 2003-05-06 2004-11-18 Philips Intellectual Property & Standards Gmbh Method for temporal synchronization of clocks
US8731003B2 (en) 2009-01-16 2014-05-20 Tellabs Oy Method and arrangement for adjustment of a clock signal
CN104396180A (en) * 2012-06-19 2015-03-04 日本电气株式会社 Clock synchronization system, clock synchronization method, and storage medium whereupon clock synchronization program is stored
EP2863576A4 (en) * 2012-06-19 2015-12-30 Nec Corp Clock synchronization system, clock synchronization method, and storage medium whereupon clock synchronization program is stored
US9634782B2 (en) 2012-06-19 2017-04-25 Nec Corporation Clock synchronization system, clock synchronization method, and storage medium whereupon clock synchronization program is stored

Similar Documents

Publication Publication Date Title
Arvind Probabilistic clock synchronization in distributed systems
US5666358A (en) Method and apparatus for supporting TDMA operating over hybrid fiber coaxial (HFC) or other channels
US7035281B1 (en) Wireless provisioning device
US7372846B2 (en) Method for transmitting time information via a data packet network
US6714563B1 (en) Network clock synchronization scheme
US7649912B2 (en) Time synchronization, deterministic data delivery and redundancy for cascaded nodes on full duplex ethernet networks
US20030137997A1 (en) Method and apparatus for frequency and timing distribution through a packet-based network
US6718476B1 (en) Method of synchronizing each local clock to a master clock in a data bus system
US5577075A (en) Distributed clocking system
US5790608A (en) Apparatus and method for synchronizing clock signals for digital links in a packet switching mode
US20050041691A1 (en) Apparatus and methods for providing synchronous digital data transfer over an ethernet
US7676679B2 (en) Method for self-synchronizing time between communicating networked systems using timestamps
US5177739A (en) Multiport - multipoint digital data service
US20050013394A1 (en) Clock synchronization in a distributed system
US20080273521A1 (en) Method and system for accurate clock synchronization for communication networks
US20050207387A1 (en) Method and apparatus for aligning time references when separated by an unreliable data packet network
US4894846A (en) Method for maintaining a correct time in a distributed processing system
US5128945A (en) Packet framing using cyclic redundancy checking
US20060262734A1 (en) Transport protocol connection synchronization
US7283568B2 (en) Methods, systems and computer program products for synchronizing clocks of nodes on a computer network
US20080049743A1 (en) Enhanced clock control in packet networks
US20100103781A1 (en) Time synchronization in cluster systems
US5274680A (en) Device for the transmission of synchronous information by an asynchronous network, notably an ATM network
Mills Internet time synchronization: the network time protocol
Jasperneite et al. Enhancements to the time synchronization standard IEEE-1588 for a system of cascaded bridges

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
122 Ep: pct application non-entry in european phase
32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: COMMUNICATION UNDER RULE 69 EPC (EPO FORM 1205A DATED 16.01.2004)

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase in:

Ref country code: JP