GB2456403A - Multiprocessor system with synchronization of error recovery to prevent errors spreading - Google Patents

Multiprocessor system with synchronization of error recovery to prevent errors spreading Download PDF

Info

Publication number
GB2456403A
GB2456403A GB0822313A GB0822313A GB2456403A GB 2456403 A GB2456403 A GB 2456403A GB 0822313 A GB0822313 A GB 0822313A GB 0822313 A GB0822313 A GB 0822313A GB 2456403 A GB2456403 A GB 2456403A
Authority
GB
Grant status
Application
Patent type
Prior art keywords
chip
error
chips
clockstop
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
GB0822313A
Other versions
GB0822313D0 (en )
Inventor
Andreas Koenig
Thomas Schlipf
Matthias Klein
Manfred Walz
Rolf Fritz
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/004Error avoidance
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0706Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
    • G06F11/0721Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment within a central processing unit [CPU]
    • G06F11/0724Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment within a central processing unit [CPU] in a multiprocessor or a multi-core unit
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0793Remedial or corrective actions

Abstract

A method of operating self-testing logic in a tree-like multi-chip processor cluster which generates an infrastructure signal 430, such as a clockstop or tracestop signal, used for error management and recovery. The operation intercepts 440 the infrastructure signal of a processor of the cluster then extracts error information from the infrastructure signal. Using the error information a pre-defined inter-chip error synchronisation scheme is selected 450 including clock-stop and/or trace-stop information for a respective one of the processors of the cluster. Notification signals are distributed 490 to chips of the cluster using dedicated wires or a low-level standard interface for chip-to-chip communication to prepare and execute error related internal operations for chips. On receipt of one of the notification signals a chip performs at least one of (i) performing a trace-stop command or (ii) performing a clock-stop command 495 on a respective one of the chips as derived from the synchronisation scheme. The synchronisation scheme may comprise a configurable delay adjustable according to the location of the failure within the chip.

Description

2456403

- 1 -

DESCRIPTION Communication And Propagation Of Infrastructure Signals 1. BACKGROUND OF THE INVENTION

1.1 FIELD OF THE INVENTION

The present invention relates to the area of computer chip technology, and particularly the field of chip error and trace data management. In particular, it relates to a method and respective circuit and system for operating self-checking logic in a computer processing chip being applicable in a multiple chip, tree-like processor cluster, wherein an infrastructure signal, such as clockstop and trace stop signals, is generated by said self-checking logic which is used for error management and recovery.

1.2 DESCRIPTION AND DISADVANTAGES OF PRIOR ART

The present invention is applicable preferably in a multiple-chip processor cluster, particularly in an I/O subsystem of high performing server computers and in particular in mainframe computers. I/O subsystem is usually understood as a term describing system hardware that connects main memory and CPUs to controllers of peripheral devices over various interfaces, preferably standardized interfaces.

Most of today's computer chips contain self-checking logic and/or logic observing the behaviour of the chip as well as it's correct handling and processing of commands and data.

Figure 1 illustrates the most basic structural components of a prior art hardware and software environment used for a prior art self-checking logic in a computer chip.

The shown computer chip represents a state of the art I/O chip in a tree structure of mainframe computers. Unit 14 represents link unit using a standardized interface like Infiniband (tt.tt-Infiniband Link Unit) to a next higher hierarchy of the I/O tree, while units 30, 32, 34 and 36 represent link units that provide the interfaces to the I/O chips on the next lower leve of the I/O tree. They are exemplarily implemented as STI Link Units, abbreviated as SLUs. STI is an abbreviation for "Self-Timed Interface" and denotes a prior art IBM proprietary Interface for connecting an I/O Hub to I/O Bridge chips in IBM mainframe computers.

The 1:4 fan-out depicted in the drawing is typical for an I/O chip in an I/O tree structure.

The functional computation on the packets streaming thru the shown I/O chip is done in various functional units between the north (ILU 14) and south interfaces (SLUs 30..36).

Functional units (FU) 20, 22, 24 and 26 represent an exemplary arrangement of those functional units.

Distributed trace facilities that are located in a subset or even in all of the described link and functional units, are controlled by a centralized trace control unit abbreviated as TU, and represented by box 18.

Errors that are detected in the various functional units and link units by internal error detection mechanisms are reported to a central error handling unit (EHU) 28 that analyses the occurred error situation and triggers the required recovery and/or error isolation steps.

Step 110 represents an error condition that is detected by the internal error detection mechanisms in the functional unit 26. This error condition is reported to the error handling unit 28 using dedicated wires as shown in step 120.

If the reported error condition is too severe to be handled within the functional unit 26 or by any other recovery

- 3 -

mechanisms within the chip, a clockstop request is issued to the central clock control unit (CCU) 16. This Clockstop request is shown in step 130.

Keeping in mind this basic error management configuration on a prior art chip as depicted in figure 1, a person skilled in the art may appreciate that if in such a chip an error or false behaviour is detected that can not be fixed by hardware or software mechanisms or might result in an unpredictable behaviour and corruption of data or actions, these checking mechanisms are usually able to shut down the chip by stopping it's internal clocks that are driving the functional logic and therefore prevent the failure from spreading within a system.

An article "IBM S/390 Parallel Enterprise Server G5 fault tolerance: A Historical Perspective", by Spainhower, L. et al, published in IBM Journal of Research and Development, Vol. 43, No. 5/6, September / November 1999, pp 863 to 873 summarizes chip error management in a historical overview and also describes the prior art management of fault tolerances of I/O subsystems.

A further article "Enhanced I/O subsystem recovery and availability on the IBM System z9", by Oakes, K.J. et al, published in IBM Journal of Research and Development, Vol. 51, No. 1/2, Jan/ March 2007, pp 131 to 144, focuses prior art problems of error recovery implemented by firmware. In a respective error recovery strategy a sequence of steps is performed including error detection, data capture, scheduling and executing recovery actions, software notifications, middleware notification, etc..

The main goal in this context is to preserve the integrity of customer data in order to avoid business critical situations. Severe hardware failures or firmware errors require to stop the

- 4 -

clock in order to avoid the propagation of errors from the nest (core) of the CPUs across the periphery, i.e., across the I/O subsystem.

In such prior art implementations, such a clockstop is executed unconditionally and immediately when such a critical situation ic hohorfoh hpvi a cvn i f- t*ho r-'h "i r\ ic ncnal 1 hpfprt vv\/ t~ vi p

— ~ ^ «- - —- ««..«. — £* ~ ^ — u

system, when it has become inaccessible, and when the interfaces to the stopped chip are no longer responding to respective requests. This kind of chip error management is in many hardware/software environments and in many business-critical situations not tolerable due to the tremendous impact of the not foreseen loss of components within the system. The time and actions that have to be performed within the system to analyse and to react to such an unexpected outage can lead to a temporary significant decrease of performance or furthermore create a situation that leads to a total fail of the whole system. Controlled actions to limit the impact of the outage can not be executed by the system in advance as well as chip internal mechanisms that could collect data to provide appropriate information about the root cause of the problem can not act anymore when the clockstop is executed.

With particular focus now to figure 2 the schematic scheme represents schematically an example of an I/O chip hierarchy in a mainframe computer, showing a multiple chip system including multiple cooperating chips, and illustrating the danger of error propagation between them.

Herein, different chips are given a two-dimensional index,

having exemplary functions as follows:

The Hub chips 1, 2,..., n represent the first level I/O chips covering I/O hub chip functionality like complex I/O mechanisms, protocol and address conversions as well as a first level of fan-out. On the second level of the hierarchy I/O bridge chips -shown in 11, 12, ... - offer additional I/O functions and further

- 5 -

fan-out connectivity including fail over paths between neighbouring bridge chips. Represented by A121, A122, A123 and A124 are four I/O Attachments of any type representing all kinds n"F ^icV_s, 0tli02rnst 2.cl02Ts , fibsir chsr.ns 1.s ,

Bridge 12 is assumed now to be clockstopped due to an internal failure. The clockstop is delayed and is corruuunicatc:'^ ex_l neighbouring chips visible in figure 2.

Hubl reports the forthcoming clockstop to the Processing Unit depicted above. All chips 121 to 124 below of Hubl can get clockstopped synchronously if enabled - e.g. to stop a detected failure from spreading across the whole system, or according to a preferred feature of the invention to allow the leverage of a consistent set of debug data from various chips.

In such multiple chips hardware environment the infrastructure signaling like control signals for distributed trace units or clock controlling signals are either handled locally within one single chip by dedicated hardware mechanisms or globally by a central system unit which either can be a hardware component or a firmware function.

While the mechanisms which are realized within a single chip are capable to react to certain trigger conditions that occur within the chip without any delay, centralized system mechanisms require a significant amount of time due to communication, analysis and distribution of control signals to react to triggers observed in one of the systems components like the mentioned chip.

On the other hand the local mechanisms on a single chip are not able to influence neighbouring components as a centralized system component could do and therefore a consistent picture of the error situation is not available in prior art.

- 6 -

1.3 OBJECTIVES OF THE INVENTION

The objective of the present invention is thus to provide an improved method and system for operating sel f-rhork-i ng and trace controlling logic in a multiple chip environment.

2. SUMMARY AND ADVANTAGES OF THE INVENTION

This objective of the invention is achieved by the features stated in enclosed independent claims. Further advantageous arrangements and embodiments of the invention are set forth in the respective subclaims. Reference should now be made to the appended claims.

The main idea of the present invention is to add a new logic to the local infrastructure signal control logic of each single chip that is using either dedicated wires or a dedicated low level communication as described further below and which uses standard interfaces to the neighbouring components to distribute and/or propagate the internal infrastructure signals such as clockstop and trace stop information within a subset of or even within the whole system. This enables for a quick error management across chip borders, which is not known from prior art. So, errors can be quickly limited before they spread across chip borders. Thus a chip error's business impact is reduced significantly compared to prior art.

Combined with some logic that is able to delay the chip-internal infrastructure signalling requests like e.g. clockstop requests, a synchronized handling of infrastructure signalling across different components/chips can be achieved.

According to the broadest aspect of the invention a method for operating self-checking logic in a computer processing chip is disclosed, applicable in a multiple chip, tree-like processor

- 7 -

cluster, particularly in a I/O subsystem, wherein an infrastructure signal, e.g. a tracestop signal or a clockstop signal, is generated by the self-checking logic which is used for error management and recovery, which method is rharflrt-pTi by the steps of:

a) intercepting the infrastructure signal of a processing unit on one chip of said multiple chip environment, in case of a clockstop signal preferably on the clockstop request path to the clock controlling unit,

b) extracting error information from the infrastructure signal,

c) using said error information for selecting a pre-defined inter-chip error synchronization scheme including clockstop information and/ or tracestop information for a respective one of said plurality of chips,

d) using dedicated wires or a low-level standard interface for said chip-to-chip communication for distributing notification signals to said plurality of chips,

e) performing the preparation and execution of error related, chip-internal actions,

f) on receipt of one of said notification signals doing at least one of:

g) performing respective tracestop commands on a respective one of said plurality of chips as derivable by said synchronization scheme,

h) performing a clockstop command on said plurality of chips as derivable by said synchronization scheme.

The term "infrastructure signal" thus includes either one of or more of:

clockstop signals, tracestop signals, and similar control signals, as far as implemented on a chip.

So, basically the inventive error management actions can be triggered by a trace unit in case of an intercepted trace stop

- 8 -

signal, or by the error handling and detection unit in case of an intercepted clock stop signal.

So, when in a multiple processor cluster re^ponsi^e to an intercepted clockstop signal a step of communicating the notification signal as a warning message to neighbor-processors is performed, then it is possible to advantageously set up a synchronization of error management between said multiple processor units. So, the error management of a processor cluster can be improved, and uncontrolled error spreading along the cluster is avoided.

The method of communicating the forthcoming clockstop situation and therefore the coming outage of a component within the system allows the centralized system wide control mechanisms to act proactively to the new situation. Time-consuming investigations to analyse the unexpected outage are no longer required since the information about failing components is available before the outage gets visible to the system. This also allows a system reconfiguration to bypass the failing components and/or to limit the impact on the whole system.

When the delay is made configurable according to at least one of the following criteria, then it is possible to provide a customizable error management adjustable to the respective individual needs in any respective hardware/ software environment:

- Depending on the location of the failure within the chip -relative to the boundaries of the effected chip - the delay needs to be short enough to make sure that the error does not spread across additional components,

Preferably, che following step is further performed after step c) above:

« '

- 9 -

- defining and using, or using a pre-defined respectively, delay time, preferably a number of cycles during which a chip-to-chip communication concerning recovery and error handling nrppa-re.tion actions, during which a chip-to-chip communication concerning recovery and error handling preparation actions are allowed to be processed.

The following step d) is then performed conforming to said error synchronization scheme.

- The delay should - under respect to above mentioned criteria -be at least as long as needed to guaranty the communication of the forthcoming Clockstop to neighbouring components/chips or to a centralized system mechanism.

- The delay should - under respect to above mentioned criteria -be at least as long as chip internal mechanisms require to perform any desirable action within the chip like e.g.

collecting and storing of debug data.

So, by means of the above features it is possible to communicate the forthcoming clockstop to multiple chips of the whole system or a subset of it. Trace information created at each of said multiple chips can be preserved in a state which is old enough to contain the information which stands in close relation to the status of the error - detected chip. So, in contrast to prior art the debug information of the neighbouring chips is correctly selected and properly saved by the inventive method.

Further, it is possible to delay the clockstop locally on the other chips of the system. Thus, it is possible to prepare the clockstop situation by executing or completing certain preparation actions before the clocks are stopped.

Furthermore,

- 10 -

According to a preferred embodiment an inventive apparatus is provided ready to be implemented on any given chip that intercepts clockstop requests before they reach the clock control logic of the chip. It is able to dpi ay i-he request to stop the clocks either unconditionally or depending on pre-definable conditions for a certain amount of time. The time won by the delay of the clockstop can be used by the inventive apparatus to execute certain required actions within the chip to e.g. capture and store debug data.

Furthermore, the apparatus can use special communication mechanisms like e.g. Infiniband-special flow control packets as described in the following subsection 2.1 to communicate the forthcoming clockstop to neighbouring components like attached chips via its interfaces or dedicated wires. This communication of the clockstop can solve certain problems of today's systems.

2.1 Special communication mechanisms:

It is disclosed to provide advantageously a communication protocol for inter-chip communication, wherein the protocol is based on a standard interface protocol, which is adapted to incorporate control, configuration and/or recovery information for computer chips, and the information is encapsulated within communication packets of a communication layer above the physical layer of the interface protocol.

One essential point of the new control traffic dedicated communication protocol according to the invention is that such protocol allows a reliable communication requiring only basically initialized connection of a main communication path. This communication bypasses all critical macros since the chip related information, for example control, configuration and/or recovery information, is encapsulated in a low layer of a standard communication protocol. Such communication enables

- 11 -

error recovery, which is typically a deadlock, and reestablishment of the traffic. However, the new protocol may also be used during hardware initialization, in order to go around not yet sufficiently initialized hardware rnmnnnpnt?

error recovery. Furthermore, during hardware initialization, the new protocol may be used to regularly/initially set up or configure non-initialized or not completely initialized hardware components. In any case an additional interface becomes superfluous, ie extra pins and wires are saved.

According to one preferred embodiment, the communication packets are manufacturer specific flow control packets defined by Opcodes (Operation Codes), which are not used by the standard interface protocol. That is, the basic structure of the standard communication protocol must not be changed. Proprietary enhancements may be introduced using open resources, which is easy and cost effective.

A variety of error cases may be handled if the Opcodes defining the communication packets each indicate different kinds of information. This extends the protocol to cover any failure occurring in control, configuration and/or recovery of chips, and therefore being extremely reliable. Furthermore, using different Opcodes, the information carried by the protocol is not restricted to mere failure management. For example, recovery of a system may require initialization of components. However, such mechanism may also be employed in regular control of a system, e.g., for initially preparing macros in routing, credits etc. before they are able to take up operation.

The amount of information to be transferred may be increased if such information is split up into several data packets having a header and a sequence number field. This allows restoring the full message of manufacturer specific flow control packets extending a defined length.

« »

- 12 -

Preferably, the inventive enhancement of a standard interface protocol is made to an InfiniBand protocol, preferably of Version 1 2. InfiniBand (also n8.rned. IB in the follou,inu) is a switched fabric communications link primarily used in high performance computing. Its features include quality of service

3 rl t 1 r* ▼ r a v" "s v% -i ♦— ■! n rvo ^ ♦» ^ V> <-*\ r" n 1 "< "K 1 rpV> "7" D

J- «-*-». J. w cx / j. j. o uc o u. y 11 ^ vu uv ooux J. . J. ii.cz J_

architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. Since this standard is widely used, application of present invention may be easily implemented.

In particular, the networking layer of the InfiniBand protocol may be used as the communication layer for transferring the chip related information. Such layer allows definition of a manufacturer specific subtype of flow control packets specified by the IB standard, which are also transferred on a very low layer of the IB communication protocol.

The IB standard defines a 32-bit flow control packet used to control the traffic flow on the link level. These packets contain a 4-bit OpCode field. However, only OpCode 0x0 and 0x1 are used by the IB standard. If Opcodes defining the communication packets are different from 0x0 and 0x1, the content of the remaining 28 bits is not defined by the IB specification, i.e. open for proprietary enhancement according to the invention.

The communication protocol is implemented by a method for interchip communication, wherein a communication protocol (CP) based on a standard interface protocol (SIP) is used, the method comprising the steps of:

- 13 -

- determining chip related information comprising data relevant for at least one of the following: control, configuration, recovery;

- encapsulating the information within cnmrnnniraHon packets of a communication layer above the physical layer of the interface protocol;

- inserting the communication packets into a regular traffic flow of the sending chip;

- extracting the information from an incoming data stream on the receiving chip.

One essential point of the communication method according to the invention is that the advantages of both current access methods for control, configuration and recovery functions, i.e.

dedicated interfaces and special command types, are combined. The encapsulated low-level communication can transfer all kinds of required messages and commands in both directions. It further allows a pretty reliable and direct access for nearly no additional costs, using the existing pins and wires. Moreover, it is not exposed to any kind of communication problem on the main data path, due to the fact that - besides the link protocol engine and the physical layer - no additional logical units involved in the main data communication are used, like routing, translation, buffering, checking etc.

The preferred embodiment of the invention is a processing unit for inter-chip communication, wherein on the one hand the unit is connected to a link protocol engine of a main interface to a neighbouring chip, and on the other hand the unit is connected to control and configuration mechanisms of the own chip.

One essential point of the processing unit according to the invention is that architectured manufacturer specific flow control packets or any other comparable low level communication packets of the used interface protocol can be employed for the

- 14 -

mentioned kind of control, configuration and recovery communication. This solution does neither need separate pins or wiring nor is it exposed to most of the misconfiguration,

failure or traffic backing prnhle^s in the main data path. Preferably, in order to save space and costs, the unit is integrally formed with a processing unit and/or a control unit of a computer chip.

According to the present invention system-wide recovery mechanisms mostly realized in firmware that get informed of a forthcoming clockstop for a certain chip do not need to execute long-lasting and "surgical" investigations when a chip becomes inaccessible anymore. This allows the system recovery mechanisms to react quicker for an error situation which is now -under usage of the invention - known in advance.

In cases the failure is detected too late on a chip, and if such failure has already spread across neighbouring chips, these chips can also be stopped immediately by use of the inventive method according to the invention, when the problem initially has been detected, just by immediately sending a respective Signal from one chip to another one. A further spread of the problem can therefore be limited to those chips that were already affected before the problem was discovered and that might not have detected the spreading failure yet.

The communication of a clockstop situation in a chip, which is part of a network like e.g. a tree hierarchy of chips as they typically occur in I/O subsystems of mainframe computers, can be used for debug purposes and to stop the whole network or predefined areas thereof.

When using the inventive method by intercepting and evaluating a tracestop signal, then the traces can be stopped synchronously across chip borders, and the debug information that now can be leveraged will contain more accurate data since all these chips

are now stopped nearly synchronously as soon as the failure is detected. There is no need for any centrally provided inter-chip monitoring and communication mechanism as a part of the overall system to stop the desired components which might take more time and does not guaranty to stop the components synchronously, and with a configurable strategy. If tracestop processing in this sense is managed basically without influencing the clock, i.e. , without a clockstop to be performed, then the logic 40, which is referred to later below as clockstop/tracestop preparation logic (CTPL) is rather a tracestop preparation logic (TSPL). This makes sense in particular in those cases in which a trace stop is triggered not by error-related causes, but instead by any desired dataflow evaluation using debug instrumentation according to prior art motivated by whatever reason, e.g. data verification on program variables, load measurements, etc..

3. BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and is not limited by the shape of the figures of the drawings in which:

Figure 1 illustrates the most basic structural components of a prior art hardware and software environment used for a prior art method,

Figure 2 illustrates schematically an example of an I/O chip hierarchy in a mainframe computer,

Figure 3 illustrates the most basic structural components of a inventive hardware and software environment used for a preferred embodiment of the inventive method,

Figure 4 illustrates the control flow of the most important steps of a part of the inventive method,

- 16 -

Figure 5 illustrates the control flow of the most important steps of a preferred embodiment of the inventive method,

Figure 6 illustrates the counterpart control flow of the most important steps of a preferred embodiment of the inventive momnnh t.tvi Qn o -i Tri t*< rv a i.ra >-■»-> ■> nA vv/-\ v nri en /"tv> al f v

• * " v V_ X V VA »« U. J. 11,1.11^ ^ 4 W X ^ J. V d o Ja W O X ^ A X .*» X. Vlll a neighbour chip according to figure 5.

4. detailed description of the preferred embodiment

Basically, figures 3 and 4 are used to illustrate the aspects of the inventive method which are performed and related to the error-affected chip alone, whereas figures 2, 5 and 6 are used to describe the inventive aspects of interchip communication and synchronized error management.

With reference to figure 3 illustrating the most basic structural components of an inventive hardware and software environment used for a preferred embodiment of the inventive method, a combined Clockstop/Tracestop Preparation Logic 40 is implemented in a set of i/o chips that are connected in a tree hierarchy as shown in figure 2.

This CTPL 40 intercepts clockstop requests issued by the central error handling mechanism implemented in the error handling unit 28 (EHU) of these chips on it's way to the clock control logic see step 210 in figure 3. A counter within this logic 40 is started as soon as the clockstop request is detected. When the clocks are stopped, this counter contains the number of cycles occurred between the clockstop request and the point in time the clocks are really stopped by the clock control unit 16 (CCU).

- 17 -

If enabled, the CTPL 40 allows thus to delay the clockstop request for a pre-definable amount of cycles, or until a desired action has completed, see step 240 in figure 3.

Within this delay, a catalogue of small tasks can be completed as follows:

a) CTPT, 40 will invoke the Trace Control Logic 18, in order to make it collect debug data from several buffer locations of the chip,

b) CTPL 40 will order the collected data according to the needs of a debug user according to any pre-configured scheme, and c) CTPL 40 will store the data at any pre-configured storage location preferably outside the defected chip, in order to enable a quick debug process.

Those actions will be done within the available delay time, sufficiently early before the clock will be stopped and no actions to access respective chip components are possible anymore.

Dependent of the respective prevailing hardware and software environment and the business situation, further emergency tasks can be advantageously performed, such as properly shutting down interfaces of the affected chip to neighbouring chips or adjacently connected storage devices, select and activate a certain, pre-configured emergency plan, by which pre-selected resources residing on different, not affected systems can be allocated for such emergency use, and sending an error management report to any pre-configured storage in order to provide full, complete and clear information about the error to respective error management tools or error management users. The skilled reader will appreciate, that further measures, here referred to as "emergency actions" can be taken in order to track and save that data which is known to be definitely

- 18 -

required to the respective business situation. All these tasks or actions are symbolically represented by step 470 in figure 4.

So the loop is run multiple time^.. anrl respective actions can bs completed. The end of the loop is reached by the counter reaching the predefined maximum value. Then in a step 490 the CTPL 40 releases its interception to pause the forwarding of the clock stop requests. Consequently, this request will be sent to the CCU 18 which in turn stops the clock, step 495.

In an alternate embodiment of the invention, the loop is exited only after a confirmatory message has been received by the EHU 2 8 and forwarded to the CTPL 40, saying that the - configurably defined most important - actions have successfully been completed.

In a preferred variation of the inventive method, each functional unit 20, 24, 26, but optionally also the management units 16, 18, 14, 28 stores an individual delay value which is reported together with an error code when the error is initially detected. This value is then used to determine the delay relevant for the loop condition 460. In this way, any desired error handling can be focused and individually adapted to the actual need of the business in which the chip is used.

With respect to the communication and propagation of errors to other chips the control logic 40 in figure 3 is enriched by further logic:

Figure 3 shows an example of a given I/O chip containing several components like a set of functional units (FU) handling the data processing, STI link units (SLUs) representing the interface units to neighbouring chips on the next lower level of the 10 chip tree, an Infinband link unit (ILU) representing the interface component to the neighbouring chip on the next higher

- 19 -

level of an I/O tree as depicted in figure 2, a chip-level centralized trace unit (TU), a clock control unit (CCU) and an error detection and handling unit (EHU). The inventive circuit is denoted ctpt. 40.

If in such a chip a tracestop or even worse a clockstop condition is detected by or reported to the error detection anu handling unit 28, the clockstop request is now routed thru the CTPL 40 on it's way to the clock control unit 16 where the clockstop would be executed, as tracestop requests are routed thru the same inventive logic 40 on it's way to the centralized trace unit 18.

With further reference to figure 2, 5, 6, and with respect to the inventive aspect of interchip error management synchronization, by using special flow control packets on it's infinband links connecting to the chip on the next higher and the same level of the tree hierarchy as well as using so called global packets on the sti links connecting to the neighbouring chips on the next lower level of the tree hierarchy, the ctpl 40 depicted in figure 3 will inform all the neighbored chips about the forthcoming clockstop.

In each of the chips shown in figure 2, it can be configured, whether this incoming clockstop information is only used to inform firmware about this fact and to store internal tracing information in a tracing unit 18, see step 230 in figure 3, or whether the clockstop information is also used to stop the own clocks - see step 240 in figure 3. Independently, the clockstop information and the trace stop information can be propagated by these chips to their further, neighbouring components. Each of the chips depicted in figure 2 can basically be built up as described above with reference to figure 3, except the additional functionality to send and receive and evaluate the

- 20 -

error relevant information to and from respectively, other chips.

If in such a chin a tracestop or even worse a clockstop condition is detected by or reported to the error detection and handling unit 28, the clockstop request is now routed thru the CTPL 40 on it's way to the clock control unit 16 where the clockstop would be executed, as tracestop requests are routed thru the same inventive logic 40 on it's way to the centralized trace unit 18.

Now, figure 5 will be referred to next below.

According to a preferred aspect of the invention the CTPL 40 is implemented in an extended form compared to the basic form as described above with reference to figures 3 and 4. Extended form means that this logic also may perform all the steps described above in figure 4, and some more functionality as described next below.

Some control logic is added which performs an interchip communication which is preferably implemented to work in a bidirectional form. This is, the logic 40 is able to send in a step 510, possibly subsequent to step 47 0 described further above, clockstop preparation information to other chips.

Further, it is also able to receive - see step 610 in figure 6 -such information from any other chip, also created by a respective control logic located on a different chip. This includes information on clockstops and particularly on trace stops.

A preferred way for implementing this bidirectional communication is when this logic uses a fast dedicated low level communication as described above for exchanging control information via ILU 14 in figure 3 or with respect to the before mentioned inter-chip error synchronization scheme.

- 21 -

A possible implementation for such scheme is a register implemented on each chip, in which the stored data define, if the clockstop of the "own" chip is to be notified to the chips in the same and/ or in the neighbnrpd level (see figure 2), and if Yes, to which one(s), or to all. Further, it is defined how to handle incoming notifications issued by other chips. Handling alternatives are: Perform also clockstop, ignore notifica'cion, forward notification to other neighboured chips of the I/O level tree, forward such information to a Processing unit (PU) in order to inform the firmware respectively. Also combinations of these alternatives can be implemented.

Any handling alternative may also include similar or identical rules for handling trace stop requests. This is preferably implemented such that these criteria can be configured freely and separately from those of clockstop handling by a skilled user.

More particularly, above mentioned control information contains the following details:

In case of using Infiniband Special Flowcontrol Packets, these packets might contain not only the information that a Clockstop or tracestop has been requested, but also additional data like:

a.) Whether the original source of this request is located at the sending chip (the chip that created this packet) or is just a forwarded information that was received from an other chip attached to one of the interfaces of the chip that created this packet.

b.) The information how long the receiving chip should delay a clockstop or tracestop in case it decides to take the same action that was taken by the sending chip and reported in this packet, to execute this action synchronously on both chips.

- 22 -

When using an implementation that does not make use of a certain type of low level communication but handles this communication by simple wires which e.g. transmit simple state changes indicating a forthcoming trar.estop or Clockstop, none or just a subset of the mentioned information might be implemented.

The nature of control data and the speed in which the data is transferred between respective same units located on different chips allows to distribute and propagate infrastructure signals - preferably clockstop and tracestop requests - quickly enough across chip borders such that error propagation is limited and trace information can be "frozen" at a point in time which is in close timely correlation with the event which caused the error.

After step 510, in a step 520 the CTPL 40 signals the trace stop and clockstop information to the chip-internal trace unit and clock control unit respectively. This step can also be performed before the interchip communication of step 510.

An initial configuration procedure is performed for determining whether at all or under which particular circumstances internal requests are communicated to neighbouring chips and whether incoming requests from neighbouring chips are executed and/or propagated to further neighbouring chips.

Control parameters used for this purpose are as follows: Bus addresses, where required and not pre-defined by a port ID, such as ILU 14, SLU 30, 32, 34 und 3 6 for example, / Chip IDs of the chips to be informed;

Error severity degree -chip ID pairs, telling beyond which severity degree which chips are to be informed, etc.

Severity degree values may encode and define various different error types ( e.g. recoverable or not) or error situations, e.g. error is locally limited, or is menacing the whole chip, or is menacing selected functions only, or is menacing a broad variety

- 23 -

of functions, or error is located on a certain location or component of a chip only, location, or component is rated important, less important, etc., tends more or less to cross chip borders due to local vicinity to the chip border, etc.

Error codes may be encoded according to any predefined encoding scheme, for example by a 5 bit long flag allowing differentiating 32 different error specifications.

Using this inventive mechanism, a synchronized clockstop and/or trace stop across a configurable subset of the I/O tree within a mainframe can be achieved without any performance penalty due to possible communication and computation overhead as one would expect in a technical alternative including a centralized approach for error management including firmware as the general error manager.

Therefore a consistent set of debug data from multiple chips that might be of interest for the debug of a certain error scenario can be collected as close as possible to the triggering condition.

Furthermore, clockstops triggered due to a detected error condition at a chip boundary can be propagated according to the invention to the neighbouring component that already might have received the malformed data.

Figure 6 depicts the counterpart control steps of the inventive method when receiving tracestop and/ or clockstop information from a different chip's 10' respective CTPL 40' generated according to the description of figure 4.

Here, after having received this information in a step 610, basically the same steps are performed as described before with reference to figure 4. The numerals are thus simply obtained by adding a value of "200". But in step 675, the trace is definitely stopped in order not to loose the trace information

- 24 -

closely related in time with the error condition at the other chip.

In an optional step the trace information may further forwarded to another chip, if nprps<;ary. This step may be built in in nearly each position in the workflow.

The invention can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

The circuit as described above is part of the design for an integrated circuit chip. The chip design is created in a graphical computer programming language, and stored in a computer storage medium (such as a disk, tape, physical hard drive, or virtual hard drive such as in a storage access network). If the designer does not fabricate chips or the photolithographic masks used to fabricate chips, the designer transmits the resulting design by physical means (e.g., by providing a copy of the storage medium storing the design) or electronically (e.g., through the Internet) to such entities, directly or indirectly. The stored design is then converted into the appropriate format (e.g., GDSII) for the fabrication of photolithographic masks, which typically include multiple copies of the chip design in question that are to be formed on a wafer. The photolithographic masks are utilized to define areas of the wafer (and/or the layers thereon) to be etched or otherwise processed.

- 25 -

Claims (8)

1. A method for operating seif-checking logic in a computer processing chip being applicable in a multiple chip, tree-like processor cluster, wherein an infrastructure signal is generated by said self-checking logic which is used for error management and recovery, which method is characterized by the steps of:
a) intercepting (440) said infrastructure signal of a processing unit on one chip (10) of said multiple chip environment,
b) extracting error information from said infrastructure signal,
c) using said error information for selecting a pre-defined interchip error synchronization scheme including clockstop information and/ or tracestop information for a respective one of said plurality of chips,
d) using dedicated wires or a low-level standard interface for said chip-to-chip communication for distributing notification signals to said plurality of chips,
e) performing the preparation and execution of error related, chip-internal actions,
f) on receipt of one of said notification signals doing at least one of:
g) performing respective tracestop commands on a respective one of said plurality of chips as derivable by said synchronization scheme,
h) performing a clockstop command on a respective one of said plurality of chips as derivable by said synchronization scheme.
2. The method according to claim 1, wherein said infrastructure signal comprises one or more members of the group of :
a) a clockstop signal,
b) a tracestop signal,
c) a further processor-related control signal.
- 26 -
3. The method according to claim 1, further comprising the step of:
defining and using, or using a pre-defined respectively, delay time durinq which a chin-t-n-rhi n communication ccnccrning recovery and error handling preparation actions are allowed to be processed, wherein said chip-to-chip communication is performed conforming to said error synchronization scheme.
4. The method according to claim 1, using the multiple chips of an I/O subsystem of a mainframe computer.
5. The method according to claim 3, wherein said delay is determined by a predetermined number of clock cycles.
6. The method according to claim 1, wherein in a multiple processor cluster a further step of communicating a warning message to neighbor-processors is performed in order to set up a synchronization of error management between said multiple processor units.
7. An electronic data processing system having processor means for operating self-checking logic in a computer processing chip of a multiple chip, tree-like processor cluster, wherein an infrastructure signal is generated by said self-checking logic which is used for error management and recovery, characterized by a logic circuit means (40) for:
a) intercepting (440) said infrastructure signal of a processing unit on one chip (10) of said multiple chip environment,
b) extracting error information from said infrastructure signal,
c) using said error information for selecting a pre-defined interchip error synchronization scheme including clockstop information and/ or tracestop information for a respective one of said plurality of chips,
u) using dedicated wires or a low-level standard interface for said chip-to-chip communication for distributing notification
- 27 -
signals to said plurality of chips,
e) performing the preparation and execution of error related, chip-internal actions,
f) on receipt of one of sairl notification signals doing at least one of:
g) performing respective tracestop commands on a respective one of said plurality of chips as derivable by said synchronization scheme,
h) performing a clockstop command on a respective one of said plurality of chips as derivable by said synchronization scheme.
8. A computer program product comprising a computer useable medium including a computer readable program, wherein the computer readable program includes a functional component (40) for operating self-checking logic in a computer processing chip of a multiple chip, tree-like processor cluster, wherein an infrastructure signal is generated by said self-checking logic which is used for error management and recovery, that when executed on a computer causes the computer to perform the steps of:
a) intercepting (440) said infrastructure signal of a processing unit on one chip (10) of said multiple chip environment,
b) extracting error information from said infrastructure signal,
c) using said error information for selecting a pre-defined interchip error synchronization scheme including clockstop information and/ or tracestop information for a respective one of said plurality of chips,
d) using dedicated wires or a low-level standard interface for said chip-to-chip communication for distributing notification signals to said plurality of chips,
e) performing the preparation and execution of error related, chip-internal actions,
f) on receipt of one of said notification signals doing at least one of:
g) performing respective tracestop commands on a respective one
- 28 -
of said plurality of chips as derivable by said synchronization scheme,
h) performing a clockstop command on a respective one of said plurality of chips ac derivable by said synchroni^aLiuix scheme.
GB0822313A 2008-01-15 2008-12-08 Communication and propagation of infrastructure signals Pending GB0822313D0 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP08100471 2008-01-15

Publications (2)

Publication Number Publication Date
GB0822313D0 GB0822313D0 (en) 2009-01-14
GB2456403A true true GB2456403A (en) 2009-07-22

Family

ID=40289625

Family Applications (1)

Application Number Title Priority Date Filing Date
GB0822313A Pending GB0822313D0 (en) 2008-01-15 2008-12-08 Communication and propagation of infrastructure signals

Country Status (1)

Country Link
GB (1) GB0822313D0 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6175931B1 (en) * 1997-01-31 2001-01-16 Hewlett-Packard Company Global hard error distribution using the SCI interconnect
US20040230865A1 (en) * 2003-05-12 2004-11-18 Internationalbusiness Machines Corporation System and method for providing processor recovery in a multi-core system
US20050149802A1 (en) * 2003-12-16 2005-07-07 International Business Machines Corporation Multi nodal computer system and method for handling check stops in the multi nodal computer system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6175931B1 (en) * 1997-01-31 2001-01-16 Hewlett-Packard Company Global hard error distribution using the SCI interconnect
US20040230865A1 (en) * 2003-05-12 2004-11-18 Internationalbusiness Machines Corporation System and method for providing processor recovery in a multi-core system
US20050149802A1 (en) * 2003-12-16 2005-07-07 International Business Machines Corporation Multi nodal computer system and method for handling check stops in the multi nodal computer system

Also Published As

Publication number Publication date Type
GB0822313D0 (en) 2009-01-14 grant

Similar Documents

Publication Publication Date Title
Mauch et al. High performance cloud computing
US7398380B1 (en) Dynamic hardware partitioning of symmetric multiprocessing systems
US7334086B2 (en) Advanced processor with system on a chip interconnect technology
US7346757B2 (en) Advanced processor translation lookaside buffer management in a multithreaded system
US5978870A (en) On-chip parallel-serial data packet converter to interconnect parallel bus of integrated circuit chip with external device
US20060230208A1 (en) System and method for presenting interrupts
US20050027793A1 (en) Advanced processor with mechanism for packet distribution at high line rate
US6125416A (en) Method and device for communicating across a chip boundary including a serial-parallel data packet converter having flow control logic
US7200776B2 (en) System and method for generating trace data in a computing system
US6378064B1 (en) Microcomputer
US8417774B2 (en) Apparatus, system, and method for a reconfigurable baseboard management controller
US7467243B2 (en) Advanced processor with scheme for optimal packet flow in a multi-processor system on a chip
US20050041666A1 (en) Advanced processor with mechanism for enforcing ordering between information sent on two independent networks
US20050055540A1 (en) Advanced processor scheduling in a multithreaded system
US20090125574A1 (en) Software Pipelining On a Network On Chip
US20090265501A1 (en) Computer system and method for monitoring an access path
US20090055496A1 (en) Advanced processor with credit based scheme for optimal packet flow in a multi-processor system on a chip
US6430727B1 (en) Diagnostic procedures in an integrated circuit device
US20050041651A1 (en) Advanced processor with mechanism for fast packet queuing operations
US20100082848A1 (en) Increasing available fifo space to prevent messaging queue deadlocks in a dma environment
US20080062927A1 (en) Delegating Network Processor Operations to Star Topology Serial Bus Interfaces
US6675284B1 (en) Integrated circuit with multiple processing cores
US20060230209A1 (en) Event queue structure and method
EP0959411A1 (en) Packet distribution in a microcomputer
US20070088974A1 (en) Method and apparatus to detect/manage faults in a system