US20050223275A1 - Performance data access - Google Patents

Performance data access Download PDF

Info

Publication number
US20050223275A1
US20050223275A1 US11/071,944 US7194405A US2005223275A1 US 20050223275 A1 US20050223275 A1 US 20050223275A1 US 7194405 A US7194405 A US 7194405A US 2005223275 A1 US2005223275 A1 US 2005223275A1
Authority
US
United States
Prior art keywords
processor
performance data
synchronized
non
elements
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US11/071,944
Inventor
Robert Jardine
James Smullen
Graham Stott
John Friedenbach
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hewlett Packard Development Co LP
Original Assignee
Hewlett Packard Development Co LP
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US55781204P priority Critical
Application filed by Hewlett Packard Development Co LP filed Critical Hewlett Packard Development Co LP
Priority to US11/071,944 priority patent/US20050223275A1/en
Assigned to HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. reassignment HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: FRIEDENBACH, JOHN W., JARDINE, ROBERT L., SMULLEN, JAMES R., STOTT, GRAHAM B.
Publication of US20050223275A1 publication Critical patent/US20050223275A1/en
Application status is Abandoned legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/1658Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/1629Error detection by comparing the output of redundant processing systems
    • G06F11/165Error detection by comparing the output of redundant processing systems with continued operation after detection of the error
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/1675Temporal synchronisation or re-synchronisation of redundant processing components
    • G06F11/1687Temporal synchronisation or re-synchronisation of redundant processing components at event level, e.g. by interrupt or result of polling
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3404Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for parallel or distributed programming
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466Performance evaluation by tracing or monitoring
    • G06F11/3476Data logging
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466Performance evaluation by tracing or monitoring
    • G06F11/3495Performance evaluation by tracing or monitoring for systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software
    • G06F11/362Software debugging
    • G06F11/3636Software debugging by tracing the execution of the program
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software
    • G06F11/362Software debugging
    • G06F11/366Software debugging using diagnostics
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/52Program synchronisation; Mutual exclusion, e.g. by means of semaphores
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/1629Error detection by comparing the output of redundant processing systems
    • G06F11/1641Error detection by comparing the output of redundant processing systems where the comparison is not performed by the redundant processing components
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/1629Error detection by comparing the output of redundant processing systems
    • G06F11/1641Error detection by comparing the output of redundant processing systems where the comparison is not performed by the redundant processing components
    • G06F11/1645Error detection by comparing the output of redundant processing systems where the comparison is not performed by the redundant processing components and the comparison itself uses redundant hardware
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/1675Temporal synchronisation or re-synchronisation of redundant processing components
    • G06F11/1683Temporal synchronisation or re-synchronisation of redundant processing components at instruction level
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/18Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits
    • G06F11/183Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits by voting, the voting not being performed by the redundant components
    • G06F11/184Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits by voting, the voting not being performed by the redundant components where the redundant components implement processing functionality
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/18Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits
    • G06F11/183Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits by voting, the voting not being performed by the redundant components
    • G06F11/184Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits by voting, the voting not being performed by the redundant components where the redundant components implement processing functionality
    • G06F11/185Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits by voting, the voting not being performed by the redundant components where the redundant components implement processing functionality and the voting is itself performed redundantly
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/88Monitoring involving counting

Abstract

Performance data access is described. In an embodiment, events are processed with non-synchronized processor elements of a logical processor in a redundant processor system. Performance data associated with execution of the processor events is stored in one or more accumulators corresponding to a respective processor element. The performance data from each of the non-synchronized processor elements is exchanged via a logical synchronization unit such that each processor element includes the performance data from each of the processor elements. Each processor element then conforms the performance data to generate synchronized performance data which is then communicated to a performance monitoring application that requests the performance data from the logical processor.

Description

    RELATED APPLICATION
  • This application claims priority to U.S. Provisional Application Ser. No. 60/557,812 filed Mar. 30, 2004, entitled “Nonstop Advanced Architecture”, the disclosure of which is incorporated by reference herein.
  • TECHNICAL FIELD
  • This invention relates to performance data access.
  • BACKGROUND
  • Multiple redundant processor systems are implemented as fault-tolerant systems to prevent downtime, system outages, and to avoid data corruption. A multiple redundant processor system provides continuous application availability and maintains data integrity such as for stock exchange systems, credit and debit card systems, electronic funds transfers systems, travel reservation systems, and the like. In these systems, data processing computations can be performed on multiple, independent processing elements of a processor system.
  • Processors in a multiple redundant processor system can be loosely synchronized in a loose lock-step implementation such that processor instructions are executed at slightly different times. This loosely synchronized implementation provides that each of the processors can execute the same instruction set faster than a typical tight lock-step configuration because the processors are not restricted to synchronized code execution. The performance of a multiple redundant processor system can be monitored to determine optimizations for software processing and for hardware configurations, such as for cache management and configuration to optimize cache hit rates.
  • When performance data is requested, such as the processing time for a processor event, the loosely-synchronized processor elements all execute the same instruction set in response to the request, but may all return a different performance response because the performance data is likely asymmetric (e.g., different in each of the multiple processor elements). The different data responses will appear as an error to the performance monitoring application that has requested the data.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The same numbers are used throughout the drawings to reference like features and components:
  • FIG. 1 illustrates an exemplary redundant processor system in which an embodiment of performance data access can be implemented.
  • FIG. 2 further illustrates various components of the exemplary redundant processor system shown in FIG. 1.
  • FIG. 3 illustrates various components of an exemplary redundant processor system in which an embodiment of performance data access can be implemented.
  • FIG. 4 illustrates various components of an exemplary redundant processor system in which an embodiment of performance data access can be implemented.
  • FIG. 5 is a flow diagram that illustrates an embodiment of a method for performance data access.
  • DETAILED DESCRIPTION
  • The following describes embodiments of performance data access. Performance monitoring is implemented to obtain system performance data from loosely-synchronized processor elements. Examples of performance data for a redundant processor system include time intervals for performing instruction sequences and counts of various processor events.
  • Although embodiments of performance data access may be implemented in various redundant processor systems, performance data access is described with reference to the following processing environment.
  • FIG. 1 illustrates an example of a redundant processor system 100 in which embodiment(s) of performance data access can be implemented. The redundant processor system 100 includes a processor complex 102 which has processor groups 104(1-3). Each processor group 104 includes any number of processor elements which are each a microprocessor that executes, or processes, computer executable instructions. For example, Processor group 104(1) includes processor elements 106(1-N), processor group 104(2) includes processor elements 108(1-N), and processor group 104(3) includes processor elements 110(1-N).
  • Processor elements, one each from the processor groups 104(1-3), are implemented together as a logical processor 112(1-N). For example, a first logical processor 112(1) includes processor element 106(1) from processor group 104(1), processor element 108(1) from processor group 104(2), and processor element 110(1) from processor group 104(3). Similarly, logical processor 112(2) includes processor elements 106(2), 108(2), and 110(2), while logical processor 112(3) includes processor elements 106(3), 108(3), and 110(3). In an alternate embodiment, a logical processor 112 may be implemented to include only two processor elements 106. For example, a processor complex may be implemented with two processor groups such that each logical processor includes two processor elements, one from each of the two processor groups.
  • In the example shown in FIG. 1, the three processor elements combine to implement a logical processor 112 and cooperate to perform the computations of the logical processor 112. Logical computations for an input/output operation or an interprocessor communication are executed separately three times in a logical processor 112, once each in the three processor elements of the logical processor 112. Additionally, the three processor elements in a logical processor 112 can coordinate and synchronize with each other to exchange data, replicate input data, and vote on input/output operations and communication outputs.
  • Each processor group 104(1-3) has an associated memory component 114(1-3), respectively. A memory component 114 can be implemented as any one or more memory components, examples of which include random access memory (RAM), DRAM, SRAM, a disk drive, and the like. Although the memory components 114(1-3) are illustrated as independent components, each processor group 104 can include a respective memory component 114 as an integrated component in an alternate embodiment.
  • In this example, processor complex 102 is a triplex redundant processor system having triple modular redundancy in that each logical processor 112 includes three redundant processor elements. To maintain data integrity, a faulty processor element can be replaced and reintegrated into the system while the redundant processor system 100 remains on-line without a loss of processing capability. Similarly, in an alternate embodiment, a duplex redundant processor system has dual modular redundancy in that each logical processor includes two redundant processor elements.
  • The processor elements of a logical processor 112 are loosely synchronized in a loose lock-step implementation such that instructions may be executed, or processed, in each of the processor elements at a slightly different time. This implementation provides that the logical processors can execute instructions faster than a typical tight lock-step configuration because the processor elements and logical processors 112 are not restricted to synchronized code execution. This implementation also provides for non-deterministic execution among the processor elements in a logical processor, such as non-deterministic branch prediction, cache replacement algorithms, and the like. The individual processor elements can also perform independent error recovery without losing synchronization with the other processor elements.
  • FIG. 2 further illustrates various components 200 of the redundant processor system 100 shown in FIG. 1. The processor elements 106(1-N) of processor group 104(1) are shown, one each of a respective logical processor 112(1-N). Each processor element 106(1-N) is associated with a respective memory region 202(1-N) of the memory component 114(1) for data storage. The memory component 114(1) associated with processor group 104(1) is partitioned among the processor elements 106(1-N) of the processor group 104(1). In an alternate embodiment, each memory region 202(1-N) can be implemented as an independent, separate memory for data storage. Although not shown, the processor elements 108(1-N) of processor group 104(2) are each associated with a respective partitioned memory region of the memory component 114(2). Similarly, the processor elements 110(1-N) of processor group 104(3) are each associated with a respective partitioned memory region of the memory component 114(3).
  • Each of the logical processors 112(1-N) correspond to one or more respective logical synchronization units 204(1-N). A logical synchronization unit 204 performs various rendezvous operations for an associated logical processor 112 to achieve agreements on data synchronization between the processor elements that cooperate to form a logical processor 112. For example, input/output operations and/or interprocessor communications can be communicated from each processor element of a logical processor 112 to an associated logical synchronization unit 204 to compare and vote on the input/output operations and/or interprocessor communications generated by the processor elements. Logical synchronization units and rendezvous operations are described in greater detail in U.S. patent application Ser. No. ______, which is Attorney Docket No. 200316143-1 entitled “Method and System of Executing User Programs on Non-Deterministic Processors” filed Jan. 25, 2005, to Bernick et al., the disclosure of which is incorporated by reference herein for the purpose of implementing performance data access.
  • A rendezvous operation may further be implemented by a logical synchronization unit 204 to exchange state information and/or data among the processor elements of a logical processor 112 to synchronize operations and responses of the processor elements. For example, a rendezvous operation may be implemented such that the processor elements deterministically respond to incoming asynchronous interrupts, to accommodate varying processing rates of the processor elements, to exchange software state information when performing operations that are distributed across the processor elements, and the like.
  • FIG. 3 illustrates various components of an exemplary redundant processor system 300 in which an embodiment of performance data access can be implemented. The redundant processor system 300 includes multiple logical processors and associated logical synchronization units as described with reference to the redundant processor system 100 shown in FIGS. 1 and 2. For illustration, however, only one logical processor 302 and one associated logical synchronization unit 304 is shown in FIG. 3. The logical synchronization unit 304 may be implemented as described with reference to the logical synchronization units 204 shown in FIG. 2.
  • In this example, logical processor 302 includes processor elements 306(1-3) which are each a microprocessor that executes, or processes, computer executable instructions. The redundant processor system 300 includes the memory components 114(1-3) that are each associated with a respective processor group 104(1-3) as shown in FIG. 1. Each of the processor elements 306(1-3) are one of the processor elements in a respective processor group, and each processor element 306 is associated with a partitioned memory region 308 in a respective memory component 114(1-3). For example, processor element 306(1) corresponds to memory region 308(1) in memory component 114(1), processor element 306(2) corresponds to memory region 308(2) in memory component 114(2), and processor element 306(3) corresponds to memory region 308(3) in memory component 114(3).
  • The memory regions 308(1-3) form a logical memory 310 that corresponds to logical processor 302. The processor elements 306(1-3) of the logical processor 302 each correspond to a respective partitioned memory region 308(1-3) of the logical memory 310. In practice, a logical processor 302 can communicate with a corresponding logical memory 310 via an input/output bridge memory controller (not shown).
  • The memory components 114(1-3) each include an instantiation of performance monitoring logic 312(1-3) that corresponds to a respective processor element 306(1-3) of the logical processor 302. Each of the processor elements 306(1-3) can execute the performance monitoring logic 312 to implement performance data access. In this example, the performance monitoring logic 312(1-3) is maintained by the memory components 114(1-3) as a software application.
  • As used herein, the term “logic” (e.g., the performance monitoring logic 312) can also refer to hardware, firmware, software, or any combination thereof that may be implemented to perform the logical operations associated with performance data access. Logic may also include any supporting circuitry utilized to complete a given task including supportive non-logical operations. For example, logic may also include analog circuitry, memory components, input/output (I/O) circuitry, interface circuitry, power providing/regulating circuitry, and the like.
  • Each of the processor elements 306(1-3) of logical processor 302 include a high-frequency clock 314, a cache memory 316, and one or more accumulators 318, respectively. For illustration, only the clock 314, cache memory 316, and accumulator(s) 318 for processor element 306(1) are shown. The description of the processor element components, however, applies to each processor element 306(1-3). The one or more accumulators 318 of a processor element 306 can be implemented as memory to store, update, and/or maintain performance data corresponding to a respective processor element 306.
  • The performance monitoring logic 312(1-3) implements performance data access such that system performance data can be obtained from the non-synchronized processor elements 306(1-3) of the logical processor 302. The performance of the processor elements 306(1-3) can be monitored for time durations to execute processor events, such as a procedure, and for any number of other operational features, such as cache hit rates, interrupt handling, and the like. While the non-synchronized processor elements 306(1-3) all execute the same instruction set (e.g., a processor event or procedure), each may return a different performance response and the corresponding performance data is likely asymmetric (e.g., different in each of the multiple processor elements 306).
  • The different performance data responses from each of the processor elements 306(1-3) may appear as an error when the data is compared by the logical synchronization unit 304, such as when an output operation of the performance data response is performed. The different performance data responses may also appear as an error if the performance monitoring logic 312 makes a decision based on that data and branches two (or three) different directions causing different action sequences that can be detected by the logical synchronization unit 304.
  • In an embodiment of performance data access, the performance data requested by the performance monitoring logic 312 can be exchanged via a rendezvous operation with the logical synchronization unit 304 such that the performance monitoring logic 312 receives consistent data from the processor elements 306(1-3). For example, a procedure may take 6.3 microseconds for processor element 306(1) to execute, 6.4 microseconds for processor element 306(2) to execute, and 5.9 microseconds for processor element 306(3) to execute. The time duration for each processor element 306 to execute the procedure can be stored in an accumulator 318 for each respective processor element 306(1-3).
  • When the performance data for each of the processor elements 306(1-3) is requested by the performance monitoring logic 312, the logical synchronization unit 304 exchanges the performance data of each of the processor elements such that each processor element has a copy of all three processor elements' individual performance measurement. For example, processor element 306(1) will have the 6.3 microseconds to execute the procedure, the 6.4 microseconds for processor element 306(2) to execute the procedure, and the 5.9 microseconds for processor element 306(3) to execute the procedure.
  • Each of the processor elements 306(1-3) then conform, or synchronize, the performance data. In this example, the 6.3 microseconds, 6.4 microseconds, and 5.9 microseconds can be averaged as 6.2 microseconds to execute the procedure. The averaging operation is deterministic, and all three processor elements 306(1-3) will arrive at the same answer of 6.2 microseconds. The average 6.2 microseconds is then returned to the performance monitoring logic 312 as the synchronized performance data.
  • Other conforming operations or algorithms can be implemented to synchronize the performance data from the multiple processor elements 306(1-3). For example, the processor elements 306(1-3) can select a performance measurement from any one of the processor elements 306(1-3), such as the minimum performance measurement, the middle performance measurement, or the maximum performance measurement corresponding to a particular processor element 306. Alternatively, the processor elements 306(1-3) can discard the performance data value that is the farthest from the other two, and then average the two remaining performance data values (e.g., for a system with triple modular redundancy), or any other form of a deterministic algorithm can be implemented.
  • Alternatively, each processor element 306(1-3) can replicate the performance measurements from the other processor elements 306. For example, prior to the logical synchronization unit 304 exchange of data, processor element 306(1) will have value A, processor element 306(2) will have value B, and processor element 306(3) will have value C. After the data exchange, each processor element 306(1-3) will have all three values A, B, and C which are replicated as if each processor element generated the performance data three times rather than just the one time.
  • In an implementation, the time duration of a processor event can be determined by obtaining a first time from a clock 314 of the respective processor element 306 at the beginning of a processor event, and subtracting the first time from an accumulator 318 of the processor element 306. A second time can be obtained from the clock 314 after the processor event has been executed by the processor element. The second time is then added to the accumulator 318 such that a time difference between the first time and the second time is the time duration of the processor event. The time duration is maintained in the accumulator 318 as the performance data.
  • For multiple performance data requests, alternate embodiments of performance data access can be implemented if it is not practicable to conform each individual performance data measurement of the processor elements 306(1-3). For example, the processor time required to accomplish each individual exchange and conforming operation may not be available within the implementation constraints of a redundant processor system.
  • In another embodiment of performance data access, the performance data is accumulated, or aggregated, in the accumulators 318 for the respective processor elements 306(1-3). For example, time durations for multiple executions of a repeated processor event can be stored and updated as the performance data in the accumulators 318 of each respective processor element 306(1-3). A procedure may be executed as a processor event multiple times by each of the processor elements 306(1-3). For a procedure that is executed ten-thousand times, and which takes on average 3 microseconds to execute, the accumulated time duration would be approximately 30 milliseconds. An accumulator 318 for processor element 306(1) can have stored performance data of 31.5 milliseconds, an accumulator 318 for processor element 306(2) can have stored performance data of 32.3 milliseconds, and an accumulator 318 for processor element 306(3) can have stored performance data of 29.7 milliseconds.
  • When the performance data for each of the processor elements 306(1-3) is requested by the performance monitoring logic 312, the logical synchronization unit 304 exchanges the data and an average (or other conforming operation) of the performance data for each processor element 306(1-3) is synchronized. In this example, an average 3.15 microseconds for processor element 306(1), an average 3.23 microseconds for processor element 306(2), and an average 2.97 microseconds for processor element 306(3) can be averaged, or conformed, to approximately 3.12 microseconds to execute the procedure each time. The approximate 3.12 microseconds is then returned to the performance monitoring logic 312 as the synchronized performance data.
  • This embodiment of performance data access avoids the extensive processing overhead of exchanging and conforming the performance data for each individual measurement, and provides performance data obtained for multiple processor events over a duration of time. The asymmetric performance data is maintained by the accumulators 318 in each respective processor element 306 such that the performance monitoring logic 312 can not directly access the performance data. Rather, the performance monitoring logic interfaces with the processor elements 306(1-3) of the logical processor 302 via application program interfaces (APIs) for performance data access.
  • In an implementation of performance data access, code (e.g., software) executing in each of the processor elements 306(1-3) interfaces with an array of the accumulators 318. The performance monitoring logic 312 calls the code via APIs to register and have accumulator(s) allocated, and to request the performance data stored in the accumulator(s). The code communicates the requested performance data to the logical synchronization unit 304, and the performance data is conformed, or synchronized. In an embodiment, the code can be implemented as millicode which is software running as the lowest-level software in the operating system.
  • FIG. 4 illustrates various components of an exemplary redundant processor system 400 in which an alternate embodiment of performance data access can be implemented. As described above with reference to the exemplary redundant processor system 300 shown in FIG. 3, logical processor 302 includes processor elements 306(1-3) which are each a microprocessor that executes processor events as computer executable instructions. The redundant processor system 400 includes the memory components 114(1-3) that are each associated with a respective processor group 104(1-3) as shown in FIG. 1. Further, each processor element 306 is associated with a partitioned memory region 308 in a respective memory component 114(1-3).
  • Each of the processor elements 306(1-3) of logical processor 302 include a high-frequency clock 314, a cache memory 316, and one or more accumulators 318, respectively. For illustration, only the clock 314, cache memory 316, and accumulator(s) 318 for processor element 306(1) are shown. The description of the processor element components, however, applies to each processor element 306(1-3). The one or more accumulators 318 of a processor element 306 can be implemented as memory to store, update, and/or maintain performance data corresponding to the respective processor element 306.
  • The exemplary redundant processor system 400 includes a remote computing device 402 configured for communication with components of the redundant processor system via a communication network 404. The remote computing device 402 includes a performance monitoring application 406 which implements performance data access as described above with reference to FIG. 3. Performance data can be requested by the performance monitoring application 406 and obtained from the non-synchronized processor elements 306(1-3) of the logical processor 302.
  • The performance of the processor elements 306(1-3) can be monitored for time durations to execute processor events, such as a procedure, and for any number of other operational features, such as cache hit rates, interrupt handling, and the like. While the non-synchronized processor elements 306(1-3) all execute the same instruction set (e.g., a processor event), each may return a different performance response and the corresponding performance data is likely asymmetric (e.g., different in each of the multiple processor elements 306). The different performance data responses from each of the processor elements 306(1-3) may appear as an error to the performance monitoring application 406 when the performance data responses are compared (or “voted”) by the logical synchronization unit 304.
  • The performance data requested by the performance monitoring application 406 can be exchanged via a rendezvous operation with the logical synchronization unit 304 and synchronized in each of the processor elements 306(1-3) such that the performance monitoring application 406 receives consistent data from each of the processor elements 306(1-3). The performance monitoring application 406 calls code (e.g., software) executed by each of the processor elements 306(1-3) via APIs to register and have accumulator(s) allocated, and to request that the performance data be stored in the accumulator(s). The code communicates the requested performance data to the logical synchronization unit 304 which exchanges the performance data. The performance data is conformed, or synchronized, in the processor elements 306(1-3) before being returned to the remote computing device 402 and to the performance monitoring application 406 via the communication network 404.
  • Methods for performance data access, such as exemplary method 500 described with reference to FIG. 5, may be described in the general context of computer executable instructions. Generally, computer executable instructions include routines, programs, objects, components, data structures, procedures, modules, functions, and the like that perform particular functions or implement particular abstract data types. The methods may also be practiced in a distributed computing environment where functions are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, computer executable instructions may be located in both local and remote computer storage media, including memory storage devices.
  • FIG. 5 illustrates an embodiment of a method 500 for performance data access. The order in which the method is described is not intended to be construed as a limitation, and any number of the described method blocks may be combined in any order to implement the method. Furthermore, the method can be implemented in any suitable hardware, software, firmware, or combination thereof.
  • At block 502, processor events are processed with non-synchronized processor elements of a logical processor in a redundant processor system. For example, each processor element 306(1-3) of logical processor 302 (FIG. 3) executes the same set of computer executable instructions, such as for a procedure or processor event. At block 504, time duration(s) of processor events are determined. For example, time durations for multiple executions of a repeated processor event can be determined.
  • In an embodiment of performance data access to determine a time duration of a processor event, a first time is obtained from a clock of a processor element at block 504(A). For example, a time is obtained from clock 314 of processor element 306(1) at the beginning of a processor event. At block 504(B), the first time is subtracted from a time stored in an accumulator of the processor element. For example, the time obtained from clock 314 is subtracted from accumulator 318 for the respective processor element 306(1).
  • If the time stored in the accumulator is initially zero, then the time obtained from clock 314 will be subtracted from zero and the accumulator will initially have a negative time. At block 504(C), a second time is obtained from the clock of the processor element after the processor event has been executed. At block 504(D), the second time is added to the accumulator such that a time difference between the first time and the second time is the time duration of the processor event. To accumulate multiple time durations for multiple executions of a repeated processor event or procedure, the method blocks 504(A-D) can be repeated to accumulate the performance data of processor elements 306(1-3). Each beginning time of a processor event is subtracted from the accumulator at block 504(B) and each time after the processor event has executed is added to the accumulator at block 504(D) such that a sum of all the time differences is accumulated.
  • At block 506, performance data associated with execution of the processor event(s) is stored in one or more accumulators corresponding to a respective processor element. For example, each processor element 306(1-3) includes one or more accumulators 318 to store, update, and maintain performance data associated with a respective processor element 306. Storing the performance data includes storing time duration(s) of a processor event as the performance data. For example, processor element 306(1) stores a first time duration of a processor event in an accumulator 318 of the processor element 306(1), processor element 306(2) stores a second time duration of the processor event in an accumulator 318 of the processor element 306(2), and processor element 306(3) stores a third time duration of the processor event in an accumulator 318 of the processor element 306(3). Performance data may also include counts of a repeated processor event, such as cache hits or misses, for example.
  • At block 508, the performance data from each of the non-synchronized processor elements is conformed as synchronized performance data. Conforming the performance data includes conforming an average of the time durations from each of the non-synchronized processor elements to generate the synchronized performance data. The logical synchronization unit 304 exchanges the performance data from each of the processor elements 306(1-3), and each of the processor elements conform the performance data to generate the synchronized performance data.
  • At block 510, the synchronized performance data is communicated to a performance monitoring application or logic that requests the performance data from the logical processor (e.g., the performance data stored in the one or more accumulators of the non-synchronized processor elements). For example, the logical synchronization unit 304 communicates the synchronized performance data to the performance monitoring logic 312 (FIG. 3) and/or to a performance monitoring application 406 in a remote computing device 402 (FIG. 4).
  • Although embodiments of performance data access have been described in language specific to structural features and/or methods, it is to be understood that the subject of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as exemplary implementations of performance data access.

Claims (25)

1. A redundant processor system, comprising:
non-synchronized processor elements of a logical processor, each processor element configured to process events and store performance data associated with execution of the processor events, each processor element including one or more accumulators configured to maintain the performance data corresponding to a respective processor element;
performance monitoring logic configured to request the performance data from the logical processor; and
a logical synchronization unit configured to exchange the performance data from each of the non-synchronized processor elements and return synchronized performance data to the performance monitoring logic, the synchronized performance data being generated by the processor elements.
2. A redundant processor system as recited in claim 1, wherein each of the processor elements are further configured to conform the performance data exchanged from each of the processor elements to generate the synchronized performance data.
3. A redundant processor system as recited in claim 2, wherein each of the processor elements are further configured to average the performance data exchanged from each of the processor elements to generate the synchronized performance data.
4. A redundant processor system as recited in claim 2, wherein each of the processor elements are further configured to conform the performance data exchanged from each of the processor elements based on a deterministic algorithm to generate the synchronized performance data.
5. A redundant processor system as recited in claim 2, wherein each of the processor elements are further configured to select the performance data from a particular processor element to generate the synchronized performance data, the selected performance data being at least one of a minimum, a middle, or a maximum of the performance data exchanged from each of the processor elements.
6. A redundant processor system as recited in claim 1, wherein:
a first time duration of a processor event is stored as performance data in a first accumulator of a first processor element;
a second time duration of the processor event is stored as performance data in a second accumulator of a second processor element;
a third time duration of the processor event is stored as performance data in a third accumulator of a third processor element; and
the logical synchronization unit is further configured to receive the first time duration, the second time duration, and the third time duration, and exchange the time durations with each of the processor elements.
7. Non-synchronized processors of a multiple redundant processor system each configured to maintain and update performance data associated with executing processor events, the performance data stored in one or more accumulators of a respective non-synchronized processor, and the performance data from each of the non-synchronized processors being conformed as synchronized performance data after being exchanged via a logical synchronization unit in response to a request for the performance data from a performance monitoring application.
8. Non-synchronized processors as recited in claim 7, wherein each of the non-synchronized processors are further configured to conform the performance data from each of the non-synchronized processors after the performance data is exchanged via the logical synchronization unit.
9. Non-synchronized processors as recited in claim 7, wherein each of the non-synchronized processors are further configured to average the performance data from each of the non-synchronized processors after the performance data is exchanged via the logical synchronization unit.
10. Non-synchronized processors as recited in claim 7, wherein a time duration of a processor event is stored as the performance data in an accumulator of the respective non-synchronized processor.
11. Non-synchronized processors as recited in claim 7, wherein counts for multiple executions of a repeated processor event are stored as the performance data in an accumulator of the respective non-synchronized processor.
12. Non-synchronized processors as recited in claim 7, wherein time durations for multiple executions of a repeated processor event are stored and updated as the performance data in an accumulator of the respective non-synchronized processor.
13. Non-synchronized processors as recited in claim 7, wherein each non-synchronized processor includes a clock, and wherein:
a first time is obtained from the clock at a beginning of a processor event, and the first time is subtracted from an initial time stored in an accumulator of the respective non-synchronized processor;
a second time is obtained from the clock after the processor event has been executed by the non-synchronized processor; and
the second time is added to the accumulator such that a time difference between the first time and the second time is a time duration of the processor event that is maintained as the performance data in the accumulator of the respective non-synchronized processor.
14. A method, comprising:
processing events with non-synchronized processor elements of a logical processor in a redundant processor system;
storing performance data associated with execution of the processor events in one or more accumulators corresponding to a respective processor element;
exchanging the performance data such that each of the processor elements includes the performance data from each of the other non-synchronized processor elements;
conforming the performance data from each of the non-synchronized processor elements to generate synchronized performance data; and
communicating the synchronized performance data to a performance monitoring application that requests the performance data from the logical processor.
15. A method as recited in claim 14, wherein each of the processor elements conform the performance data exchanged from each of the processor elements to generate the synchronized performance data.
16. A method as recited in claim 14, wherein conforming the performance data includes each of the processor elements averaging the performance data to generate the synchronized performance data.
17. A method as recited in claim 14, wherein conforming the performance data includes each of the processor elements using a deterministic algorithm to conform the performance data to generate the synchronized performance data.
18. A method as recited in claim 14, wherein conforming the performance data includes each of the processor elements selecting the performance data from a particular processor element, the selected performance data being at least one of a minimum, a middle, or a maximum of the performance data exchanged from each of the processor elements.
19. A method as recited in claim 14, further comprising determining a time duration of a processor event, and wherein storing the performance data includes storing the time duration of the processor event as the performance data.
20. A method as recited in claim 14, further comprising accumulating counts for multiple executions of a repeated processor event, and wherein storing the performance data includes storing the counts of the repeated processor event as the performance data.
21. A method as recited in claim 14, wherein storing the performance data includes:
storing a first time duration of a processor event in a first accumulator of a first processor element;
storing a second time duration of the processor event in a second accumulator of a second processor element;
storing a third time duration of the processor event in a third accumulator of a third processor element; and
wherein conforming the performance data includes conforming the first time duration, the second time duration, and the third time duration to generate the synchronized performance data.
22. A method as recited in claim 14, wherein communicating the synchronized performance data includes communicating the synchronized performance data to the performance monitoring application in a remote computing device configured for communication with the redundant processor system.
23. One or more computer readable media comprising computer executable instructions that, when executed, direct a performance data access system to:
process events with non-synchronized processor elements of a logical processor in a redundant processor system;
store performance data associated with execution of the processor events in one or more accumulators corresponding to a respective processor element;
conform the performance data from each of the non-synchronized processor elements to generate synchronized performance data; and
communicate the synchronized performance data to a performance monitoring application that requests the performance data from the logical processor.
24. One or more computer readable media as recited in claim 23, further comprising computer executable instructions that, when executed, direct the performance data access system to exchange the performance data such that each of the non-synchronized processor elements includes the performance data from each of the other non-synchronized processor elements.
25. One or more computer-readable media as recited in claim 23, further comprising computer executable instructions that, when executed, direct the performance data access system to:
store a first time duration of a processor event in a first accumulator of a first processor element;
store a second time duration of the processor event in a second accumulator of a second processor element;
store a third time duration of the processor event in a third accumulator of a third processor element; and
conform the first time duration, the second time duration, and the third time duration to generate the synchronized performance data.
US11/071,944 2004-03-30 2005-03-04 Performance data access Abandoned US20050223275A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US55781204P true 2004-03-30 2004-03-30
US11/071,944 US20050223275A1 (en) 2004-03-30 2005-03-04 Performance data access

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US11/071,944 US20050223275A1 (en) 2004-03-30 2005-03-04 Performance data access

Publications (1)

Publication Number Publication Date
US20050223275A1 true US20050223275A1 (en) 2005-10-06

Family

ID=35346428

Family Applications (5)

Application Number Title Priority Date Filing Date
US10/953,242 Abandoned US20050240806A1 (en) 2004-03-30 2004-09-28 Diagnostic memory dump method in a redundant processor
US10/990,151 Active 2025-09-20 US7890706B2 (en) 2004-03-30 2004-11-16 Delegated write for race avoidance in a processor
US11/042,981 Expired - Fee Related US7434098B2 (en) 2004-03-30 2005-01-25 Method and system of determining whether a user program has made a system level call
US11/045,401 Abandoned US20050246581A1 (en) 2004-03-30 2005-01-27 Error handling system in a redundant processor
US11/071,944 Abandoned US20050223275A1 (en) 2004-03-30 2005-03-04 Performance data access

Family Applications Before (4)

Application Number Title Priority Date Filing Date
US10/953,242 Abandoned US20050240806A1 (en) 2004-03-30 2004-09-28 Diagnostic memory dump method in a redundant processor
US10/990,151 Active 2025-09-20 US7890706B2 (en) 2004-03-30 2004-11-16 Delegated write for race avoidance in a processor
US11/042,981 Expired - Fee Related US7434098B2 (en) 2004-03-30 2005-01-25 Method and system of determining whether a user program has made a system level call
US11/045,401 Abandoned US20050246581A1 (en) 2004-03-30 2005-01-27 Error handling system in a redundant processor

Country Status (2)

Country Link
US (5) US20050240806A1 (en)
CN (2) CN100472456C (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080270653A1 (en) * 2007-04-26 2008-10-30 Balle Susanne M Intelligent resource management in multiprocessor computer systems
US20140172941A1 (en) * 2005-08-19 2014-06-19 Riverbed Technology, Inc. Capture, analysis, and visualization of concurrent system and network behavior of an application
US9411363B2 (en) * 2014-12-10 2016-08-09 Intel Corporation Synchronization in a computing device

Families Citing this family (72)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060020852A1 (en) * 2004-03-30 2006-01-26 Bernick David L Method and system of servicing asynchronous interrupts in multiple processors executing a user program
US20050240806A1 (en) * 2004-03-30 2005-10-27 Hewlett-Packard Development Company, L.P. Diagnostic memory dump method in a redundant processor
US7308605B2 (en) * 2004-07-20 2007-12-11 Hewlett-Packard Development Company, L.P. Latent error detection
US7412545B2 (en) * 2004-07-22 2008-08-12 International Business Machines Corporation Apparatus and method for updating I/O capability of a logically-partitioned computer system
US7047440B1 (en) * 2004-07-27 2006-05-16 Freydel Lev R Dual/triple redundant computer system
DE102004038590A1 (en) * 2004-08-06 2006-03-16 Robert Bosch Gmbh Method for delaying access to data and / or commands of a dual-computer system and corresponding delay unit
WO2006045807A2 (en) * 2004-10-25 2006-05-04 Robert Bosch Gmbh Data processing system with a variable clock speed
US7624302B2 (en) * 2004-10-25 2009-11-24 Hewlett-Packard Development Company, L.P. System and method for switching the role of boot processor to a spare processor responsive to detection of loss of lockstep in a boot processor
US7818614B2 (en) * 2004-10-25 2010-10-19 Hewlett-Packard Development Company, L.P. System and method for reintroducing a processor module to an operating system after lockstep recovery
US7627781B2 (en) * 2004-10-25 2009-12-01 Hewlett-Packard Development Company, L.P. System and method for establishing a spare processor for recovering from loss of lockstep in a boot processor
US7502958B2 (en) * 2004-10-25 2009-03-10 Hewlett-Packard Development Company, L.P. System and method for providing firmware recoverable lockstep protection
US7516359B2 (en) * 2004-10-25 2009-04-07 Hewlett-Packard Development Company, L.P. System and method for using information relating to a detected loss of lockstep for determining a responsive action
US7383471B2 (en) * 2004-12-28 2008-06-03 Hewlett-Packard Development Company, L.P. Diagnostic memory dumping
JP4528144B2 (en) * 2005-01-26 2010-08-18 富士通株式会社 Memory dump program boot method, mechanism and program
US20060190700A1 (en) * 2005-02-22 2006-08-24 International Business Machines Corporation Handling permanent and transient errors using a SIMD unit
US20080195836A1 (en) * 2005-02-23 2008-08-14 Hewlett-Packard Development Company, L.P. Method or Apparatus for Storing Data in a Computer System
JP4555713B2 (en) * 2005-03-17 2010-10-06 富士通株式会社 Error notification method and information processing apparatus
US7590885B2 (en) * 2005-04-26 2009-09-15 Hewlett-Packard Development Company, L.P. Method and system of copying memory from a source processor to a target processor by duplicating memory writes
DE102005037246A1 (en) * 2005-08-08 2007-02-15 Robert Bosch Gmbh Method and device for controlling a computer system having at least two execution units and a comparison unit
JP4645837B2 (en) * 2005-10-31 2011-03-09 日本電気株式会社 Memory dump method, computer system, and program
US7627584B2 (en) * 2005-11-30 2009-12-01 Oracle International Corporation Database system configured for automatic failover with no data loss
US20070124522A1 (en) * 2005-11-30 2007-05-31 Ellison Brandon J Node detach in multi-node system
US7668879B2 (en) * 2005-11-30 2010-02-23 Oracle International Corporation Database system configured for automatic failover with no data loss
US7516358B2 (en) * 2005-12-20 2009-04-07 Hewlett-Packard Development Company, L.P. Tuning core voltages of processors
US7496786B2 (en) * 2006-01-10 2009-02-24 Stratus Technologies Bermuda Ltd. Systems and methods for maintaining lock step operation
US20070220369A1 (en) * 2006-02-21 2007-09-20 International Business Machines Corporation Fault isolation and availability mechanism for multi-processor system
JP5087884B2 (en) * 2006-08-11 2012-12-05 富士通セミコンダクター株式会社 Data processing unit and data processing apparatus using the same
US8127099B2 (en) * 2006-12-26 2012-02-28 International Business Machines Corporation Resource recovery using borrowed blocks of memory
US20080165521A1 (en) * 2007-01-09 2008-07-10 Kerry Bernstein Three-dimensional architecture for self-checking and self-repairing integrated circuits
US7743285B1 (en) * 2007-04-17 2010-06-22 Hewlett-Packard Development Company, L.P. Chip multiprocessor with configurable fault isolation
US20080263391A1 (en) * 2007-04-20 2008-10-23 International Business Machines Corporation Apparatus, System, and Method For Adapter Card Failover
JP4838226B2 (en) * 2007-11-20 2011-12-14 富士通株式会社 Network logging processing program, information processing system, and network logging information automatic saving method
DE102007062974B4 (en) 2007-12-21 2010-04-08 Phoenix Contact Gmbh & Co. Kg Signal processing device
JP5309703B2 (en) * 2008-03-07 2013-10-09 日本電気株式会社 Shared memory control circuit, control method, and control program
US7991933B2 (en) * 2008-06-25 2011-08-02 Dell Products L.P. Synchronizing processors when entering system management mode
JP5015351B2 (en) * 2008-08-08 2012-08-29 アマゾン テクノロジーズ インコーポレイテッド Realization of reliable access to non-local block data storage by executing programs
JP5507830B2 (en) * 2008-11-04 2014-05-28 ルネサスエレクトロニクス株式会社 Microcontroller and automobile control device
US8429633B2 (en) * 2008-11-21 2013-04-23 International Business Machines Corporation Managing memory to support large-scale interprocedural static analysis for security problems
CN101782862B (en) * 2009-01-16 2013-03-13 鸿富锦精密工业(深圳)有限公司 Processor distribution control system and control method thereof
US8631208B2 (en) * 2009-01-27 2014-01-14 Intel Corporation Providing address range coherency capability to a device
US8875142B2 (en) * 2009-02-11 2014-10-28 Hewlett-Packard Development Company, L.P. Job scheduling on a multiprocessing system based on reliability and performance rankings of processors and weighted effect of detected errors
TWI448847B (en) * 2009-02-27 2014-08-11 Foxnum Technology Co Ltd Processor distribution control system and control method
CN101840390B (en) 2009-03-18 2012-05-23 中国科学院微电子研究所 Hardware synchronous circuit structure suitable for multiprocessor system and implement method thereof
US7979746B2 (en) * 2009-04-27 2011-07-12 Honeywell International Inc. Dual-dual lockstep processor assemblies and modules
US8364862B2 (en) * 2009-06-11 2013-01-29 Intel Corporation Delegating a poll operation to another device
JP5099090B2 (en) * 2009-08-19 2012-12-12 日本電気株式会社 Multi-core system, multi-core system control method, and multi-processor
DE102009054637A1 (en) * 2009-12-15 2011-06-16 Robert Bosch Gmbh Method for operating a computing unit
EP2550598A1 (en) * 2010-03-23 2013-01-30 Continental Teves AG & Co. oHG Redundant two-processor controller and control method
US8479042B1 (en) * 2010-11-01 2013-07-02 Xilinx, Inc. Transaction-level lockstep
TWI447574B (en) * 2010-12-27 2014-08-01 Ibm Method,computer readable medium, appliance,and system for recording and prevevting crash in an appliance
US8930752B2 (en) 2011-02-15 2015-01-06 International Business Machines Corporation Scheduler for multiprocessor system switch with selective pairing
US8635492B2 (en) * 2011-02-15 2014-01-21 International Business Machines Corporation State recovery and lockstep execution restart in a system with multiprocessor pairing
US8671311B2 (en) 2011-02-15 2014-03-11 International Business Machines Corporation Multiprocessor switch with selective pairing
EP2701063A4 (en) * 2011-04-22 2014-05-07 Fujitsu Ltd Information processing device and information processing device processing method
US8554726B2 (en) * 2011-06-01 2013-10-08 Clustrix, Inc. Systems and methods for reslicing data in a relational database
US8924780B2 (en) * 2011-11-10 2014-12-30 Ge Aviation Systems Llc Method of providing high integrity processing
DE102012010143B3 (en) 2012-05-24 2013-11-14 Phoenix Contact Gmbh & Co. Kg Analog signal input circuit with a number of analog signal acquisition channels
JP5601353B2 (en) * 2012-06-29 2014-10-08 横河電機株式会社 Network management system
GB2508344A (en) 2012-11-28 2014-06-04 Ibm Creating an operating system dump
US9367372B2 (en) 2013-06-18 2016-06-14 Advanced Micro Devices, Inc. Software only intra-compute unit redundant multithreading for GPUs
JP6175958B2 (en) 2013-07-26 2017-08-09 富士通株式会社 Memory dump method, program, and information processing device
US9251014B2 (en) * 2013-08-08 2016-02-02 International Business Machines Corporation Redundant transactions for detection of timing sensitive errors
JP6221702B2 (en) * 2013-12-05 2017-11-01 富士通株式会社 Information processing apparatus, information processing method, and information processing program
US10102052B2 (en) 2014-01-29 2018-10-16 Hewlett Packard Enterprise Development Lp Dumping resources
US9710273B2 (en) * 2014-11-21 2017-07-18 Oracle International Corporation Method for migrating CPU state from an inoperable core to a spare core
CN104699550B (en) * 2014-12-05 2017-09-12 中国航空工业集团公司第六三一研究所 A kind of error recovery method based on lockstep frameworks
JP2016170463A (en) * 2015-03-11 2016-09-23 富士通株式会社 Information processing device, kernel dump method, and kernel dump program
DE102015218898A1 (en) * 2015-09-30 2017-03-30 Robert Bosch Gmbh Method for the redundant processing of data
US10067763B2 (en) * 2015-12-11 2018-09-04 International Business Machines Corporation Handling unaligned load operations in a multi-slice computer processor
US9971650B2 (en) 2016-06-06 2018-05-15 International Business Machines Corporation Parallel data collection and recovery for failing virtual computer processing system
WO2018048720A1 (en) * 2016-09-09 2018-03-15 The Charles Stark Draper Laboratory, Inc. Voting circuits and methods for trusted fault tolerance of a system of untrusted subsystems
US10474549B2 (en) * 2017-07-31 2019-11-12 Oracle International Corporation System recovery using a failover processor

Citations (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4293921A (en) * 1979-06-15 1981-10-06 Martin Marietta Corporation Method and signal processor for frequency analysis of time domain signals
US4733353A (en) * 1985-12-13 1988-03-22 General Electric Company Frame synchronization of multiply redundant computers
US5016249A (en) * 1987-12-22 1991-05-14 Lucas Industries Public Limited Company Dual computer cross-checking system
US5193175A (en) * 1988-12-09 1993-03-09 Tandem Computers Incorporated Fault-tolerant computer with three independently clocked processors asynchronously executing identical code that are synchronized upon each voted access to two memory modules
US5226152A (en) * 1990-12-07 1993-07-06 Motorola, Inc. Functional lockstep arrangement for redundant processors
US5239641A (en) * 1987-11-09 1993-08-24 Tandem Computers Incorporated Method and apparatus for synchronizing a plurality of processors
US5255367A (en) * 1987-09-04 1993-10-19 Digital Equipment Corporation Fault tolerant, synchronized twin computer system with error checking of I/O communication
US5317726A (en) * 1987-11-09 1994-05-31 Tandem Computers Incorporated Multiple-processor computer system with asynchronous execution of identical code streams
US5339404A (en) * 1991-05-28 1994-08-16 International Business Machines Corporation Asynchronous TMR processing system
US5397365A (en) * 1990-06-01 1995-03-14 E. I. Du Pont De Nemours And Company Composite orthopedic implant with modulus variations
US5452443A (en) * 1991-10-14 1995-09-19 Mitsubishi Denki Kabushiki Kaisha Multi-processor system with fault detection
US5504859A (en) * 1993-11-09 1996-04-02 International Business Machines Corporation Data processor with enhanced error recovery
US5572620A (en) * 1993-07-29 1996-11-05 Honeywell Inc. Fault-tolerant voter system for output data from a plurality of non-synchronized redundant processors
US5600784A (en) * 1993-12-01 1997-02-04 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US5613127A (en) * 1992-08-17 1997-03-18 Honeywell Inc. Separately clocked processor synchronization improvement
US5751955A (en) * 1992-12-17 1998-05-12 Tandem Computers Incorporated Method of synchronizing a pair of central processor units for duplex, lock-step operation by copying data into a corresponding locations of another memory
US5790397A (en) * 1996-09-17 1998-08-04 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US5796939A (en) * 1997-03-10 1998-08-18 Digital Equipment Corporation High frequency sampling of processor performance counters
US5896523A (en) * 1997-06-04 1999-04-20 Marathon Technologies Corporation Loosely-coupled, synchronized execution
US5991900A (en) * 1998-06-15 1999-11-23 Sun Microsystems, Inc. Bus controller
US6141769A (en) * 1996-05-16 2000-10-31 Resilience Corporation Triple modular redundant computer system and associated method
US6223304B1 (en) * 1998-06-18 2001-04-24 Telefonaktiebolaget Lm Ericsson (Publ) Synchronization of processors in a fault tolerant multi-processor system
US6279119B1 (en) * 1997-11-14 2001-08-21 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US6327668B1 (en) * 1998-06-30 2001-12-04 Sun Microsystems, Inc. Determinism in a multiprocessor computer system and monitor and processor therefor
US6393582B1 (en) * 1998-12-10 2002-05-21 Compaq Computer Corporation Error self-checking and recovery using lock-step processor pair architecture
US20020133751A1 (en) * 2001-02-28 2002-09-19 Ravi Nair Method and apparatus for fault-tolerance via dual thread crosschecking
US20020152418A1 (en) * 2001-04-11 2002-10-17 Gerry Griffin Apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep
US20020152419A1 (en) * 2001-04-11 2002-10-17 Mcloughlin Michael Apparatus and method for accessing a mass storage device in a fault-tolerant server
US6604177B1 (en) * 2000-09-29 2003-08-05 Hewlett-Packard Development Company, L.P. Communication of dissimilar data between lock-stepped processors
US20030188221A1 (en) * 1998-12-18 2003-10-02 Rasmussen David C. Method and apparatus for processing control using a multiple redundant processor control system related applications
US20030191983A1 (en) * 1995-10-24 2003-10-09 Seachange International, Inc., A Delaware Corporation Loosely coupled mass storage computer cluster
US6658654B1 (en) * 2000-07-06 2003-12-02 International Business Machines Corporation Method and system for low-overhead measurement of per-thread performance information in a multithreaded environment
US20050155019A1 (en) * 2004-01-14 2005-07-14 International Business Machines Corporation Method and apparatus for maintaining performance monitoring structures in a page table for use in monitoring performance of a computer program
US20050201288A1 (en) * 2004-03-15 2005-09-15 Sharp Kabushiki Kaisha Performance evaluation apparatus, performance evaluation method, program and computer readable recording medium
US6948092B2 (en) * 1998-12-10 2005-09-20 Hewlett-Packard Development Company, L.P. System recovery from errors for processor and associated components
US20050240806A1 (en) * 2004-03-30 2005-10-27 Hewlett-Packard Development Company, L.P. Diagnostic memory dump method in a redundant processor
US7076397B2 (en) * 2002-10-17 2006-07-11 Bmc Software, Inc. System and method for statistical performance monitoring
US7194671B2 (en) * 2001-12-31 2007-03-20 Intel Corporation Mechanism handling race conditions in FRC-enabled processors
US7308605B2 (en) * 2004-07-20 2007-12-11 Hewlett-Packard Development Company, L.P. Latent error detection

Family Cites Families (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3665404A (en) 1970-04-09 1972-05-23 Burroughs Corp Multi-processor processing system having interprocessor interrupt apparatus
US4228496A (en) 1976-09-07 1980-10-14 Tandem Computers Incorporated Multiprocessor system
US4481578A (en) * 1982-05-21 1984-11-06 Pitney Bowes Inc. Direct memory access data transfer system for use with plural processors
JPS61253572A (en) 1985-05-02 1986-11-11 Hitachi Ltd Load distributing system for loose coupling multi-processor system
JP2695157B2 (en) 1986-12-29 1997-12-24 松下電器産業株式会社 Variable pipeline processor
JP2644780B2 (en) 1987-11-18 1997-08-25 株式会社日立製作所 Parallel computer having a processing request function
JPH0797328B2 (en) 1988-10-25 1995-10-18 インターナシヨナル・ビジネス・マシーンズ・コーポレーシヨン Fuoruto-tolerant synchronization system
US5369767A (en) 1989-05-17 1994-11-29 International Business Machines Corp. Servicing interrupt requests in a data processing system without using the services of an operating system
US5317752A (en) * 1989-12-22 1994-05-31 Tandem Computers Incorporated Fault-tolerant computer system with auto-restart after power-fall
US5295258A (en) * 1989-12-22 1994-03-15 Tandem Computers Incorporated Fault-tolerant computer system with online recovery and reintegration of redundant components
US5291608A (en) 1990-02-13 1994-03-01 International Business Machines Corporation Display adapter event handler with rendering context manager
US5111384A (en) * 1990-02-16 1992-05-05 Bull Hn Information Systems Inc. System for performing dump analysis
US5295259A (en) * 1991-02-05 1994-03-15 Advanced Micro Devices, Inc. Data cache and method for handling memory errors during copy-back
CA2068048A1 (en) * 1991-05-06 1992-11-07 Douglas D. Cheung Fault tolerant processing section with dynamically reconfigurable voting
US5535397A (en) 1993-06-30 1996-07-09 Intel Corporation Method and apparatus for providing a context switch in response to an interrupt in a computer process
JP2669363B2 (en) 1994-10-14 1997-10-27 日本電気株式会社 Process switching control method and apparatus due to the system call cluster coupled multiprocessor system
JP3481737B2 (en) * 1995-08-07 2003-12-22 富士通株式会社 Dump device and the dump method
US5999933A (en) * 1995-12-14 1999-12-07 Compaq Computer Corporation Process and apparatus for collecting a data structure of a memory dump into a logical table
US5850555A (en) 1995-12-19 1998-12-15 Advanced Micro Devices, Inc. System and method for validating interrupts before presentation to a CPU
CN1154749A (en) 1996-04-16 1997-07-16 美国电报电话Ipm公司 Method for monitoring a digital multiprocessor
GB9617033D0 (en) * 1996-08-14 1996-09-25 Int Computers Ltd Diagnostic memory access
US5903717A (en) * 1997-04-02 1999-05-11 General Dynamics Information Systems, Inc. Fault tolerant computer system
US6173356B1 (en) * 1998-02-20 2001-01-09 Silicon Aquarius, Inc. Multi-port DRAM with integrated SRAM and systems and methods using the same
US6141635A (en) * 1998-06-12 2000-10-31 Unisys Corporation Method of diagnosing faults in an emulated computer system via a heterogeneous diagnostic program
US6199171B1 (en) * 1998-06-26 2001-03-06 International Business Machines Corporation Time-lag duplexing techniques
US6314501B1 (en) * 1998-07-23 2001-11-06 Unisys Corporation Computer system and method for operating multiple operating systems in different partitions of the computer system and for allowing the different partitions to communicate with one another through shared memory
US6754763B2 (en) * 2001-07-30 2004-06-22 Axis Systems, Inc. Multi-board connection system for use in electronic design automation
US6195715B1 (en) 1998-11-13 2001-02-27 Creative Technology Ltd. Interrupt control for multiple programs communicating with a common interrupt by associating programs to GP registers, defining interrupt register, polling GP registers, and invoking callback routine associated with defined interrupt register
US6263373B1 (en) * 1998-12-04 2001-07-17 International Business Machines Corporation Data processing system and method for remotely controlling execution of a processor utilizing a test access port
US6543010B1 (en) * 1999-02-24 2003-04-01 Hewlett-Packard Development Company, L.P. Method and apparatus for accelerating a memory dump
US6397365B1 (en) 1999-05-18 2002-05-28 Hewlett-Packard Company Memory error correction using redundant sliced memory and standard ECC mechanisms
US6304645B1 (en) 2000-03-04 2001-10-16 Intel Corporation Call processing system with resources on multiple platforms
US6820213B1 (en) * 2000-04-13 2004-11-16 Stratus Technologies Bermuda, Ltd. Fault-tolerant computer system with voter delay buffer
EP1213650A3 (en) * 2000-08-21 2006-08-30 Texas Instruments France Priority arbitration based on current task and MMU
US6604717B2 (en) * 2000-11-15 2003-08-12 Stanfield Mccoy J. Bag holder
US6704887B2 (en) * 2001-03-08 2004-03-09 The United States Of America As Represented By The Secretary Of The Air Force Method and apparatus for improved security in distributed-environment voting
US7065672B2 (en) * 2001-03-28 2006-06-20 Stratus Technologies Bermuda Ltd. Apparatus and methods for fault-tolerant computing using a switching fabric
KR20040002388A (en) 2001-04-26 2004-01-07 코닌클리케 필립스 일렉트로닉스 엔.브이. Turntable for a disc-shaped information carrier, and player with such a turntable
US7207041B2 (en) * 2001-06-28 2007-04-17 Tranzeo Wireless Technologies, Inc. Open platform architecture for shared resource access management
US7076510B2 (en) * 2001-07-12 2006-07-11 Brown William P Software raid methods and apparatuses including server usage based write delegation
US6859866B2 (en) * 2001-10-01 2005-02-22 International Business Machines Corporation Synchronizing processing of commands invoked against duplexed coupling facility structures
US6687799B2 (en) * 2002-01-31 2004-02-03 Hewlett-Packard Development Company, L.P. Expedited memory dumping and reloading of computer processors
CN100411380C (en) 2002-08-19 2008-08-13 万达信息股份有限公司 Conversion method between synchronization and asynchronization in network data exchange
US6983337B2 (en) 2002-12-18 2006-01-03 Intel Corporation Method, system, and program for handling device interrupts
US7231543B2 (en) * 2004-01-14 2007-06-12 Hewlett-Packard Development Company, L.P. Systems and methods for fault-tolerant processing with processor regrouping based on connectivity conditions
US7162666B2 (en) 2004-03-26 2007-01-09 Emc Corporation Multi-processor system having a watchdog for interrupting the multiple processors and deferring preemption until release of spinlocks
US7380171B2 (en) * 2004-12-06 2008-05-27 Microsoft Corporation Controlling software failure data reporting and responses
US7328331B2 (en) 2005-01-25 2008-02-05 Hewlett-Packard Development Company, L.P. Method and system of aligning execution point of duplicate copies of a user program by copying memory stores

Patent Citations (47)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4293921A (en) * 1979-06-15 1981-10-06 Martin Marietta Corporation Method and signal processor for frequency analysis of time domain signals
US4733353A (en) * 1985-12-13 1988-03-22 General Electric Company Frame synchronization of multiply redundant computers
US5255367A (en) * 1987-09-04 1993-10-19 Digital Equipment Corporation Fault tolerant, synchronized twin computer system with error checking of I/O communication
US5384906A (en) * 1987-11-09 1995-01-24 Tandem Computers Incorporated Method and apparatus for synchronizing a plurality of processors
US5239641A (en) * 1987-11-09 1993-08-24 Tandem Computers Incorporated Method and apparatus for synchronizing a plurality of processors
US5317726A (en) * 1987-11-09 1994-05-31 Tandem Computers Incorporated Multiple-processor computer system with asynchronous execution of identical code streams
US5016249A (en) * 1987-12-22 1991-05-14 Lucas Industries Public Limited Company Dual computer cross-checking system
US5193175A (en) * 1988-12-09 1993-03-09 Tandem Computers Incorporated Fault-tolerant computer with three independently clocked processors asynchronously executing identical code that are synchronized upon each voted access to two memory modules
US5397365A (en) * 1990-06-01 1995-03-14 E. I. Du Pont De Nemours And Company Composite orthopedic implant with modulus variations
US5226152A (en) * 1990-12-07 1993-07-06 Motorola, Inc. Functional lockstep arrangement for redundant processors
US5339404A (en) * 1991-05-28 1994-08-16 International Business Machines Corporation Asynchronous TMR processing system
US5452443A (en) * 1991-10-14 1995-09-19 Mitsubishi Denki Kabushiki Kaisha Multi-processor system with fault detection
US5613127A (en) * 1992-08-17 1997-03-18 Honeywell Inc. Separately clocked processor synchronization improvement
US5751955A (en) * 1992-12-17 1998-05-12 Tandem Computers Incorporated Method of synchronizing a pair of central processor units for duplex, lock-step operation by copying data into a corresponding locations of another memory
US5572620A (en) * 1993-07-29 1996-11-05 Honeywell Inc. Fault-tolerant voter system for output data from a plurality of non-synchronized redundant processors
US5504859A (en) * 1993-11-09 1996-04-02 International Business Machines Corporation Data processor with enhanced error recovery
US5600784A (en) * 1993-12-01 1997-02-04 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US5615403A (en) * 1993-12-01 1997-03-25 Marathon Technologies Corporation Method for executing I/O request by I/O processor after receiving trapped memory address directed to I/O device from all processors concurrently executing same program
US6038685A (en) * 1993-12-01 2000-03-14 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US5956474A (en) * 1993-12-01 1999-09-21 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US20030191983A1 (en) * 1995-10-24 2003-10-09 Seachange International, Inc., A Delaware Corporation Loosely coupled mass storage computer cluster
US6240526B1 (en) * 1996-05-16 2001-05-29 Resilience Corporation Triple modular redundant computer system
US6141769A (en) * 1996-05-16 2000-10-31 Resilience Corporation Triple modular redundant computer system and associated method
US5790397A (en) * 1996-09-17 1998-08-04 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US5796939A (en) * 1997-03-10 1998-08-18 Digital Equipment Corporation High frequency sampling of processor performance counters
US5896523A (en) * 1997-06-04 1999-04-20 Marathon Technologies Corporation Loosely-coupled, synchronized execution
US6279119B1 (en) * 1997-11-14 2001-08-21 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US6473869B2 (en) * 1997-11-14 2002-10-29 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US20020026604A1 (en) * 1997-11-14 2002-02-28 Marathon Technologies Corporation, A Delaware Corporation Fault resilient/fault tolerant computing
US5991900A (en) * 1998-06-15 1999-11-23 Sun Microsystems, Inc. Bus controller
US6223304B1 (en) * 1998-06-18 2001-04-24 Telefonaktiebolaget Lm Ericsson (Publ) Synchronization of processors in a fault tolerant multi-processor system
US6327668B1 (en) * 1998-06-30 2001-12-04 Sun Microsystems, Inc. Determinism in a multiprocessor computer system and monitor and processor therefor
US6393582B1 (en) * 1998-12-10 2002-05-21 Compaq Computer Corporation Error self-checking and recovery using lock-step processor pair architecture
US6948092B2 (en) * 1998-12-10 2005-09-20 Hewlett-Packard Development Company, L.P. System recovery from errors for processor and associated components
US20030188221A1 (en) * 1998-12-18 2003-10-02 Rasmussen David C. Method and apparatus for processing control using a multiple redundant processor control system related applications
US6658654B1 (en) * 2000-07-06 2003-12-02 International Business Machines Corporation Method and system for low-overhead measurement of per-thread performance information in a multithreaded environment
US6604177B1 (en) * 2000-09-29 2003-08-05 Hewlett-Packard Development Company, L.P. Communication of dissimilar data between lock-stepped processors
US20020133751A1 (en) * 2001-02-28 2002-09-19 Ravi Nair Method and apparatus for fault-tolerance via dual thread crosschecking
US20020152419A1 (en) * 2001-04-11 2002-10-17 Mcloughlin Michael Apparatus and method for accessing a mass storage device in a fault-tolerant server
US6928583B2 (en) * 2001-04-11 2005-08-09 Stratus Technologies Bermuda Ltd. Apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep
US20020152418A1 (en) * 2001-04-11 2002-10-17 Gerry Griffin Apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep
US7194671B2 (en) * 2001-12-31 2007-03-20 Intel Corporation Mechanism handling race conditions in FRC-enabled processors
US7076397B2 (en) * 2002-10-17 2006-07-11 Bmc Software, Inc. System and method for statistical performance monitoring
US20050155019A1 (en) * 2004-01-14 2005-07-14 International Business Machines Corporation Method and apparatus for maintaining performance monitoring structures in a page table for use in monitoring performance of a computer program
US20050201288A1 (en) * 2004-03-15 2005-09-15 Sharp Kabushiki Kaisha Performance evaluation apparatus, performance evaluation method, program and computer readable recording medium
US20050240806A1 (en) * 2004-03-30 2005-10-27 Hewlett-Packard Development Company, L.P. Diagnostic memory dump method in a redundant processor
US7308605B2 (en) * 2004-07-20 2007-12-11 Hewlett-Packard Development Company, L.P. Latent error detection

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140172941A1 (en) * 2005-08-19 2014-06-19 Riverbed Technology, Inc. Capture, analysis, and visualization of concurrent system and network behavior of an application
US10075509B2 (en) * 2005-08-19 2018-09-11 Riverbed Technology, Inc. Capture, analysis, and visualization of concurrent system and network behavior of an application
US20080270653A1 (en) * 2007-04-26 2008-10-30 Balle Susanne M Intelligent resource management in multiprocessor computer systems
US9411363B2 (en) * 2014-12-10 2016-08-09 Intel Corporation Synchronization in a computing device

Also Published As

Publication number Publication date
US7890706B2 (en) 2011-02-15
US20050246587A1 (en) 2005-11-03
US20050223178A1 (en) 2005-10-06
CN1690970A (en) 2005-11-02
US7434098B2 (en) 2008-10-07
US20050240806A1 (en) 2005-10-27
CN100472456C (en) 2009-03-25
CN1696903A (en) 2005-11-16
US20050246581A1 (en) 2005-11-03

Similar Documents

Publication Publication Date Title
Denning The working set model for program behavior
US5640584A (en) Virtual processor method and apparatus for enhancing parallelism and availability in computer systems
AU753120B2 (en) Fault resilient/fault tolerant computing
US5157663A (en) Fault tolerant computer system
EP0772136B1 (en) Method of commitment in a distributed database transaction
US5005174A (en) Dual zone, fault tolerant computer system with error checking in I/O writes
US4914657A (en) Operations controller for a fault tolerant multiple node processing system
Schiper et al. P-store: Genuine partial replication in wide area networks
US4907228A (en) Dual-rail processor with error checking at single rail interfaces
US5896523A (en) Loosely-coupled, synchronized execution
US5249187A (en) Dual rail processors with error checking on I/O reads
EP0306252B1 (en) Fault tolerant computer system input/output interface
US7962458B2 (en) Method for replicating explicit locks in a data replication engine
US8140523B2 (en) Decision based system for managing distributed resources and modeling the global optimization problem
CN101523355B (en) Failover system and method
US7523344B2 (en) Method and apparatus for facilitating process migration
EP3118743A1 (en) Fault tolerance and failover using active copy-cat
EP0306244A2 (en) Fault tolerant computer system with fault isolation
US4941087A (en) System for bumpless changeover between active units and backup units by establishing rollback points and logging write and read operations
US20040034807A1 (en) Roving servers in a clustered telecommunication distributed computer system
US6031987A (en) Optimistic distributed simulation based on transitive dependency tracking
EP2049997B1 (en) Match server for a financial exchange having fault tolerant operation
Kim et al. Distributed execution of recovery blocks: An approach for uniform treatment of hardware and software faults in real-time applications
Hanmer Patterns for fault tolerant software
US8301593B2 (en) Mixed mode synchronous and asynchronous replication system

Legal Events

Date Code Title Description
AS Assignment

Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P., TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:JARDINE, ROBERT L.;SMULLEN, JAMES R.;STOTT, GRAHAM B.;AND OTHERS;REEL/FRAME:016365/0274

Effective date: 20050301

STCB Information on status: application discontinuation

Free format text: ABANDONED -- AFTER EXAMINER'S ANSWER OR BOARD OF APPEALS DECISION