JP3600533B2 - Method and apparatus for synchronizing time of day (TOD) events between nodes - Google Patents

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present inventionTo synchronize time of day (TOD) events between nodessingleofTODIncrementerNeed to,Distributed parallel processing systemOfMethod and apparatus.
[0002]
[Prior art]
Occurs on a distributed processing systemTODEvents exactlySyncTwo differentnodeBetween TODs read simultaneously onDifferenceBut bothnodeIt must be less than the minimum latency for messages in between and must be continually increasing. Today this has a single central TOD repository or allNoーThe valueSingle clock transmitted(System transmitter)OrAllThis is done by running the system. CenterTODRepositoryMethodThenallAll nodes need to query this single node, and consequently all these queries are serialized, reducing the speedups intended to be achieved by parallelizing the application, thus creating a processing bottleneck. There are disadvantages. System wideIts value isSingle transmittedofUse system transmittermethodTo,Specific betweenLink activationIs required, otherwise the system will fail. On large distributed systems, theseLinkThere is a real concern, since it needs to be a cable that can be accidentally pulled out.
[0003]
[Problems to be solved by the invention]
The present inventionHardware and software implementations overcome the shortcomings of the known art. This realization isMode isOwnLocal transmitterSelfIncrements the TOD in minutes, UpdateNewPeriodically send TOD broadcast packets to performBy doing themTODCan be kept within the minimum waiting timeYou.In essence, the present inventionGreat differenceHave orIs TODSynchronouslyClock to operateThe value of theDistributeClothToNeeded thatIn contrast to the prior art solution, the clockThe value of theDistributeClothNeed toButNoGreat differenceThe entire distributed system so that is less than the minimum latencyEach nodeProvide a TOD incrementer. By using the present invention, allNoModeTo TODBroadcast packetIs arrangedClothThis eliminates the need for a single centralized TOD source.Also,According to the inventionIf,(singleofSystem transmitterMethod usingCertain cables)OperationMust be insideMonaIt becomes. "TOD MassChip ”(hereinafter referred to as“ TOD master ”)FromEach neighborAs long as there is a path to the node, the TOD is kept within tolerance.
[0004]
[Means for Solving the Problems]
The present inventionBeliefTODWhile each node updates its TOD accurately based on the broadcast packet,Tod broadcast packets to all neighborsDistribute to nodesHardware and softwareIn the form ofTake advantage of the switch design used. The method and apparatus of the present invention are described in "TOD Mass"AsDesignationFunction to calculate the delay of the cable connected to the connected switch, andAnd TOD broadcast packet receivedFaithAnd calculated cable delayAdjust the TOD value based onIncludes the ability to The present inventionOf each nodeTOD is received without decrementing the value.BeliefTOD broadcast packetInsideofTODFunction to adjust the value and the TOD broadcast packetEach nodeAll neighbors ofnodeSent toTrustFunction. The last mentioned feature isnodeIs specificTOD broadcast packetAlready receivedBeliefIs used to detect ifBeliefWasTOD BroadcastThe packet is discarded (ending the feedback loop).
[0005]
The method and apparatus of the present inventionBeliefWasTOD in TOD broadcast packetvalueTOD incrementer value of each nodeGreater than, TODofAdjustments areBeliefWasTODbroadcast・ TOD in packetIt can be a simple load of a value. However, TOD incrementer valueIs receivedBeliefWasTODbroadcast・ TOD value in packetIf greater,TOD incrementer valueIsThe differenceMust be held or incremented at a slower rate until there is no more. (In the design according to the present inventionOf the TOD incrementerThe value is the value incremented at normal speed.BeliefWasTODUntil the value is matched, it is incremented at a fractional value, eg, half the speed. )
[0006]
From a system point of view, the present invention relates to “TOD mass"becomeShouldRuns with the ability to select a single switch or nodePresentIs done. This "TOD mass"Is all my neighborsnodeToOn the other hand,TOD broadcast packetPeriodicallySendingI believe.The TOD broadcast packet contains the current TOD value and (used to detect feedback loops)Sequence IDToHold.Also,The TOD broadcast packet isThis isFlag indicating that the packet is a TOD broadcast packet instead of a normal data packet.And thepacketInsideChecksum for detecting errorsAlso keepbe able to. For the reliability of the system,"To provide a backup that takes over its function if the chip fails.
[0007]
BEST MODE FOR CARRYING OUT THE INVENTION
<Distributed computing environment>
Turning now to the drawings in more detail, in whichHow manyLike numbers refer to like parts throughout the figures.You.FIG. 1 is a block diagram of a distributed computing environment 100 that includes a plurality of nodes 102 coupled to one another via a plurality of network adapters 104. Each node 102 is an independent computer having its own operating system image 108, memory 110 on a system memory bus 118, and processor 106, and a system input / output bus 116 The adapter 104 is connected. Each network adapter104Is,Via network switch 120Each otherToLinkIs done.
[0008]
In one example, the distributed computing environment 100 includes onethat's allN nodes 102 having a number of processors 106.EachMode102Is, AIX(IBM version of the UNIX operating system)RISC / 6000 computer that runsIt can be.Node 102How manySome or all may include different types of computers and / or different operating systems 108You.
[0009]
FIG.Is, According to the present invention, of the distributed computing environment 100 of FIG.How manyTsukaNoFIG. In one embodiment, a group service 20 incorporating the features of the present invention is provided.6Plurality of distributed computing environment 100NoCode 10Distributed between the twoIt is.oneIn the example,OneMore thanCode 10Inside 2The group service daemon (Not shown) Is arranged. The group service daemon is collectively referred to as group service 206.
[0010]
Group services206Is used in various situations, for example, to facilitate communication and synchronization between multiple processes in a process group, including, for example, providing a distributed recovery synchronization mechanism.You.I want to use the group services 206 mechanismapplication·Process 202 is a group service demo.ToBe combined. For details,This application process 202Group services206By linking at least part of the code (eg, library code) associated with,Loop service206Is combined with In one embodimentSubordinateFor example, the application process 202 includes an application programming interface(API)The mechanism of the present invention is used via 204. For details,API 204Provides an interface for the application process 202 to use the mechanisms of the present invention.
[0011]
FIG.Is,The distributed computing environment 100 of FIG.128 waysofProcessorWhen consisting ofOne embodiment of distributed system 300State ofA schematic diagram is shown. The distributed system 300 includes the nodes 302(Workstation in this embodiment) and are connected via a network switch 120. The network switch 120typeThe switch, the sandChiSwitch board (NSB) 304 andMiddleSwitch board (ISB) 306Consists of NSB304 and ISB306The difference between ISB306NSB 304 is to connect both switches and nodes, whereas NS only connects switches. NSB304 and ISB306Are the same, just different componentsToJust connect.
[0012]
<Switch board and TOD hardware>
Figure 4 shows a single switch board304Is shown. The switch board 304Each otherConnected toandLabeled 1 to 8,Consists of eight switch chips 402. Figure 5 shows a normal switch・ Chip 402FIG. 5 illustrates the main data paths 502-504 of FIG.HowConnection or logic circuit. FIG.add toShows connections and blocks.Such additionsInclude a TOD logic circuit 602 and a service logic circuit 604. The TOD logic 602 is shown in FIG. 7 and will be described in more detail below. The service logic circuit 604 supplies the TODInsideTOD register702From any logic needed to initializeSuccessYou.Also,FIG.allReceiver 606To TOD logic circuit 602Connect to, And from TOD logic 602allConnection to all senders 612Also shownYou. These connections are available via TOD broadcast·packetReceivingTrustReceiver606But thatTODbroadcast·packetTo the TOD logic 602, and the TOD logic 602ThatTOD broadcast·packetToallSent from all senders 612FaithNeeded to be able to do so. In other embodiments, the sender 612 is the receiver 606To TOD broadcast·packetDirectlyFaithNote that you can.
[0013]
At power on, the system administratoris thereSwitch chip 402 or node 302Is selected as the “TOD master”.Also,The system administrator is "TOD master"InsideTOD value ofToinitialTransformYou. Then,"TOD master" broadcasts TOD for each "metacycle"·packetIs automatically generated. In the systemRemainingSwitch chip 402 is,These TODsbroadcast·packetReceivingTrustWhen,As described beloweachUpdate the TOD value.
[0014]
In FIG., BasisBasic TOD theoryReasonRoad 602TwoMainRegister: Consists of TOD register 702 and TOD broadcast register 706. TOD broadcast register 706, TODMasterNeeded to ensure that TOD is never decremented when updating TOD.That is, according to the requirements,TOD must be constantly increasing.MostThroatin the case of, TOD broadcast register 706 is used to load TOD register 702 every 75 MHz cycle. ActualToIsUsing the incrementer 716TOD Broadcast Register 706Increase the value ofMinuteTo be doneYou. However,TOD register 702Is, ItselfFeedback to (718),Half speedsoincrementalTo do (717),TOD setting packetOrTODAny of the mastersLoad fromThat (724 or 726)Must be possible. Likewise,TOD broadcast register 706The aforementionedlike, IncreaseMinuteDoing (730), ItselfToFeedbackDoing (728)Be able to take, update and loadPossibleThere must be.
[0015]
Broadcast and TOD setting detection logic 708, TODMasterfromTODbroadcast·packetOr of the TOD setting packetwhatReceived fromFaithTODvalueDetermine if there is.TODbroadcast·packetReceivingIn ShinchuIfThatTODvalueIs connected to the receiver 60 via the SVC bus 722.6come from. TOD settingWhile receiving a packetIfThatTODvalueComes from the SVC ring 712.
[0016]
The 75 MHz edge detection logic circuit 704 isPredeterminedFurther described below with reference to operating at the TOD frequency, in the given example,, This frequency is75 MHz and is used to gate the TOD increment.
[0017]
The master broadcast generation logic 710 looks at the TOD register 702 and, as described further below,TODFor each cycleIn theSu 729 of the TOD broadcast packet onLaunch transmissionI do.In the example given, this metacycle (when bit 16 flips)of) Equal to 65536 TOD cycle. Details of the transmission have already been given. thisIs TOD massOn theOnly done.
[0018]
<TOD process flow>
ThisKnown to the traderBut, Each node 30Inside 2ThenBusMultiple switch chips 402 or adaptersThe phaseConnect with each other.FigureReferring to the flowcharts of FIGS.Entity(Switch initialization code), step S1Then, One switch chip 402 or adapter"TODMaster".TODMasterIs selected, (steps S2 and S4)so, TOD setting communicated from the central controllerConstantWait for TOD to be setWork. Step S6Then, TODMasterBut,allThe TOD is automatically transmitted to all the switch chips 402 and the adapter.
[0019]
During the transmission phase,The hardware isFor delays in cables and intermediate switches, TAdjust OD automatically.SubordinateWhat,At the end of TOD transmission,allAll TODs should be within a few cycles of each other (accuracy should be about 3 cycles per switch board 304).Also, allTo handle asynchronous clocks on all switch boards 304The "TOD Master"Step S22so,Every X TOD cycles, hereinafter referred to as “metacycle”, for example, every 65536 TOD cycle corresponding to 872 microseconds, TIt should communicate the OD. The TOD cycle operates at a Y frequency, for example, 75 MHz corresponding to 13.3 ns per cycle. In the example given, this means that transmission occurs each time bit 16 (bit 0, least significant bit) of the TOD flips. The transmission of TOD based on this “metacycle” reduces the tolerance of the transmitter.AdjustmentAnd keep the TOD close to each other.Also,"TODMaster"Is, Receiving of TOD setting packetFaithThen, immediately transmit your TOD. Until the metacycle is completed, flow loops back to step S4 and no transmission occurs. The software is implemented on some computer readable media, such as RAM, ROM or 3.5 inch diskette, CD ROM, which is loaded into TOD logic 602 using methods known to those skilled in the art.
[0020]
Step S1SoSoftware"TODMasterIs added to the initialization packet by one bit. This bit is the initialization packetInsideWhen set in"TOD Master"Until the most significant bit of TOD is set, ThatofInitialization packetDo not act on The most significant bit of TOD isEffectivenessDefined as bits.is thereChipsIs TODMaster, Its TOD must be set. Step S2so,This is done with a TOD configuration packet.
[0021]
Software uses only one chip"TODMaster"Whendo itMust be defined. Multiple chipsTOD master, The two chips fight each other to update the remaining chips, and as a result,Will result in different TODs throughout the system.if,During transmission of TOD, A TOD massIsDifferent from the one I sent most recentlyRushiTOD with sequence IDBroadcast packetReceivingIf you believe, The chipInvalidIt should flag a TOD transmission error.
[0022]
Step S2so,The software isTODSet packet"TODMasterSent toTrustThus, the most significant bit of the TOD in that is set.is thereOn the switch chip 402,"TODMaster"SaidThe bit is reset whenever a power-on reset occurs.Also,softwareIsSends a new initialization packet with that bit off to the chip.TrustBy doing so, the bit can be reset. Step S4As evident from, The transmission of TOD, TODMasterOf TODEffectivenessOnly happens if marked. Step S6As shown, The transmission of TOD,The aforementionedIn the example given,65535 occur every TOD cycle (872 us) and when that happens,"TODMasterIs theallBy notifying all senders 612, for example, eight senders 612, to transmit the TOD, the TOD is transmitted to the entire system.C612For such noticeWhen the signal is sent,In step S7,SoTheseIs the sender,Capture TOD, interrupt data flow, step S8so,TOD control word followed by TODHoldTransmit two words, then continue normal operation. However,PreviousAs described above, transmission of TOD is performed in step S2.TODMasterOf TODEffectivenessOnly happens if marked.
[0023]
The TOD control word contains a 1-byte, 1-byte sequence ID, an update control bit, and an 8-bit ECC to indicate that it is a TOD control word.keeping. The sequence ID isRespectivelyIs incremented after the TOD of theTODbroadcast·packetUsed to eliminate the feedback loop of The update control bit, even if two TODsDifferenceEven if there isFaithUsed to indicate that the chip TOD should be updated to this value. The 8-bit ECC is the ECC 64 bits above the TOD value (the same ECC algorithm used on the 64-bit multicast node mask). If ECC is invalid, this TOD broadcast·packetIs ignored.
[0024]
Step S9, "TODMasterNext to their receivers 606One, for example eightFaithTOD on portcontrolReceive characterTrustYou. Receiver 606Port receives the TOD control characterTrustThenIn step S10,A signal is sent to the TOD logic circuit 602, and a sequence ID is given to the logic circuit. Step S11so,The TOD logic circuit 602 includes a comparator 713To compare the sequence ID with the final sequence ID register. Step S12In theseSequence ID is the sameIs determined asIn this case, the TOD control character is ignored (this chip has already received this TOD update).Beliefing).On the other hand,Step S12so,theseSequence ID ofIs differentIs determined to beIf so, step S14so,The new sequence ID is stored in the final sequence ID register, and step S15so,This new TOD reduces cable delayAdjustmentUpdated in order to execute step S19.so,The updated TOD is used to change the TOD register 702, and step S20so,Send TOD after updateallSendingFaithA signal is sent to the port.
[0025]
In step S16, the update control bitIs setSetIs determined to beOr step S17In, receiveTOD value is greater than current TOD valueIs determined asIn this case, the TOD register 702 is updated by writing the updated TOD to the TOD register 702.On the other hand,ReceivingBeliefIs smaller than the current TOD value,andUpdate control bit offIsIf so, step S18And the differenceButAdjustmentサ イ ク ル cycle incrementer 717UsingEvery otherThe current TOD value is incremented every 2 TOD cycles (26.6 ns). This feature guarantees that the TOD value will continually increase (never decrease). In step S21, the TOD register 702 sets the TOD broadcast register706Is determined.
[0026]
SendingFaithThe logic circuit receives the portFaithFrom the sideBeliefThe bestEndSequence IDTrustYou. Central TOD logic circuit 602InsideIs the last sequence ID register ofTheIf the port is different from the last sequence ID received, the sender 612 sends only the TOD control character when the signal is sent from the TOD logic 602.TrustYou. This restriction is necessary to reduce the number of TOD control characters in the system as quickly as possible. However,Receiver 606Discards redundant TOD control characters, so even if the sender612Receive the same TOD control characterBeliefSend TOD control character whileBeliefHowever, this design still works.
[0027]
TODMasterReceives the TOD setting packetBeliefWhen the TOD transmission is performed after the execution, the TOD load instruction is executed (the update bit is set). In this case, the TOD is greater than the current TOD valueIsNot tested. The transmitted TOD is loaded regardless of its value.
[0028]
The present inventionThe cable delay between switches is calculated using the initialization state (Ping state). That is,Send Ping control characterBefore receiving the Pong character responseTOD cycleofnumber(In this example, the TOD frequency is 75 MHz)Designed to counting.Then,Divide this delay by two,Get the cable delay (this delay includes the cable and all chip latency). Division by 2 moves the count to the right by 1 bitJust shiftIs achieved by doingSubordinateWhat,Accuracy is approximately plus or minus one cycle). Based on this, the worst of this designUnder circumstancesThe inaccuracy of TOD is"TODMaster”Plus or minus three cycles each time the switch board is moved.
[0029]
TOD has two reasons: cable delay and sender.612Collision with EDC cycleforAdjusted during transmission. Sender612And receiver606Because there are various delays in the cable between606Will be corrected when you arrive. Cable delay added to TOD, resulting in,When the TOD arrives at the service logic 604HarukaOrPositiveBe certain. TODButtransferToEDC sent when requestingInsideTo send612Cannot send TOD,612Means that TOD is delayedIt must beIncrement TOD for every cycle. This improves the accuracy of the TOD when it arrives at service logic 604.
[0030]
All switch boards 304 have two 75 MHz oscillators (switch chip 402ofInternally, thisDivided by 237.5MHzofCreate a transmitter). This transmitter is connected to the switch chip 402Against, To signal when their TOD registers 702 are incremented. Each switch chip 402 looks for a transition on the resulting 37.5 MHz clock. The inputs are two latches (Not shownThrough)RunYou. When a transition is seen on the output, the sender612, Receiver 606,andCenterTOD logic circuit 602ofTo allIs, Are signaled to increment their TOD values. Receiver 606IsUsing this signal,Gate control of the counter when adjusting the link timing in the ping stateWith,TheThe receiver writes chunks to the central queueTo be busyIncrement delayed TODYou.The sender 612Using this signal,EDC sentInsideToLateExtendTIncrement ODYou.
[0031]
The switch chip 402TheTwo 75 MHz clocks are provided to avoid any single point of failure on the board.If there isThe switch chip 402primary"75MHz transmitterToggleIf not, the switch chip switches to the backup input.SubordinateWhat,For the board to fail, both 75MHz oscillators must fail.theseThe transmitter is routed through two different chips for re-excitation,TheHalf of the board uses one transmitter as its primary,TheThe other half of the board uses the other oscillator.
[0032]
The most significant bit of the higher order TOD word isEffectivenessUsed as a bit. TODEffectivenessThe bit must be on to be considered.SubordinateWhat,TODEffectivenessTo be a TOD settingpacketButHighThe most significant bitYesMust.If TODMasterIn TODEffectivenessDenaif, The TOD is not transmitted. TODInvalidationThe reason for,It is as follows.
1. There is a parity error on TOD register 702.
a) TOD broadcast if it is a parity error only on broadcast・ Register 706And both TOD register 702InvalidationIs done.
b) If it is a parity error only on TOD register 702, TOD register 702InvalidationBut TOD broadcast register706 is invalidatedNot done. Rather, it loads the TOD register 702,SubordinateIt can be used to recover from TOD parity errors.
2.8 consecutive"Me"Tacycle"To,sushiーTOD chip receives broadcast updateBeliefIf not,The TOD is invalidated. BecauseCan't guarantee that the TOD is in sync with the rest of the systemIs from. "Metacycle" is based on a certain number of cyclesBecome.In the example given,This fixed number is the 65535 TOD cycle. The TOD cycle for the given example is 13.3 ns (75 MHz).
[Brief description of the drawings]
FIG.Distributed coComputing environmentIt is a block diagram.
FIG. 2Figure1 of distributed computing environmentHow manyTsukaNoEnlarged view of modeIs.
FIG. 3The distributed computing environment in Fig. 1128 waysofProcessorWhen consisting ofDistributed systemFruitShapingShow conditionIt is a schematic diagram.
FIG. 4Figure3 is a schematic diagram of a switch board of the distributed system No. 3;
FIG. 5 is a schematic diagram of a switch chip (not including service logic) according to the present invention.
FIG. 6 is a schematic diagram of a switch chip showing the connection of the novel TOD logic and service logic according to the present invention.
FIG. 7: TOD massOf5 is a schematic diagram showing the highest level control logic.
FIG. 8 is a flowchart illustrating essential aspects of the method and apparatus of the present invention.
FIG. 9 is a flowchart illustrating essential aspects of the method and apparatus of the present invention.
[Explanation of symbols]
102 Node 1
104 Network Adapter
106 processor
108 AIX Overall Image
110 memory
112 I / O adapter
114 channel controller
116 System I / O bus
120 Network Switch

Claims (10)

A method of synchronizing a time-of-day (TOD) events between a plurality of adjacent to that node which made knitting so as to form a distributed parallel processing system, each node includes a TOD incrementer,
The TOD incrementer in (a) each node, the method comprising topically oscillator or found locally Incremented of the in each node acting and coupled to said TOD incrementer at a predetermined frequency,
(B) transmitting a TOD setting packet from the central controller having a communication link to the selected one of the plurality of nodes to the one node via the communication link, and setting an initial TOD value in the packet; by setting the one node, a step of specifying the one node as a master node,
(C) responding to the register in the master node counting a predetermined number of cycles of the predetermined frequency , transmitting the TOD broadcast packet generated in the master node to the master node; a step of based on said TOD incrementer, to broadcast to nodes adjacent to the master node through the transmission path,
(D) in each of the neighboring nodes, in response to the said master node receives the TOD broadcast packet, the TOD value for the update in the TOD broadcast packets the reception of the in each node Compared with the value of the TOD incrementer, if the updated TOD value is greater than the value of the TOD incrementer, the value of the TOD incrementer is updated to be equal to the updated TOD value, and If the TOD value is less than the value of the TOD incrementer, incrementing the TOD incrementer in cycles of a frequency lower than the predetermined frequency until the value of the TOD incrementer reaches the updating TOD value. And a method comprising: Wherein each of the node you adjacent, further comprising the step of retransmitting the TOD broadcast the received packet to at least one other node from the nodes, the method according to claim 1. Wherein step (d), the delay of transmission paths connecting the respective node the you adjacent said master node calculates, based on the calculated delay, TOD broadcast packets the received 2. The method of claim 1, further comprising adjusting the TOD value for the update within . The method of claim 1, wherein step (c) further comprises inserting a sequence ID number into the TOD broadcast packet . Wherein step (c) further comprises the step of inserting a checksum to detect errors in the broadcast to the TOD broadcast packet The method of claim 1. An apparatus for synchronizing a plurality of adjacent to that node between in time-of-day (TOD) event was made knitting so as to form a distributed parallel processing system, each node includes a TOD incrementer,
Respectively provided to each of the plurality of nodes, at least one local oscillator, wherein operating in the TOD incrementer to be bonded to said TOD incrementer to locally incrementally and predetermined frequency in each node ,
By transmitting a TOD setting packet to the one node via a communication link with a selected one of the plurality of nodes, and setting an initial TOD value in the packet to the one node , one node and a central controller be configured as master node,
Said master node,
Means for generating a TOD broadcast packet for TOD updates should broadcast next adjacent node to the master node,
A register for counting a predetermined number of cycles of said predetermined frequency,
In response to the register, the TOD broadcast packet, based on the TOD incrementer of the master in the node, and means for broadcasting to the adjacent node via the transmission path,
Each of the adjacent nodes
Means for receiving the TOD broadcast packet from said master node,
Means for comparing the updated TOD value in the received TOD broadcast packet with the value of the TOD incrementer in each node;
Means for updating the value of the TOD incrementer equal to the value of the updating TOD, if the updating TOD value is greater than the value of the TOD incrementer;
If the updating TOD value is smaller than the value of the TOD incrementer, the TOD incrementer is cycled at a frequency lower than the predetermined frequency until the value of the TOD incrementer reaches the updating TOD value. Means for incrementing by: Wherein each of adjacent nodes, the TOD broadcast the received packet that having a means for re-transmitting at least one other node from the nodes, according to claim 6. Wherein each of adjacent nodes, the delay of transmission paths for connecting the with the respective node the master node calculates, based on the calculated delay, for the updating of the received the TOD in broadcast packet that having a means for adjusting the TOD value apparatus according to claim 6. The apparatus of claim 6 , wherein the TOD broadcast packet includes a sequence ID number. The TOD broadcast packet includes a checksum to detect errors in the broadcast apparatus according to claim 6.

Images