JPH01298459A - Common memory multiprocessor system - Google Patents

Abstract

PURPOSE: To efficiently control a mutual connection mechanism by dynamically detecting a hot spot so as to turn a message. CONSTITUTION: This system mutually connects an optional processor PO to PN to an optional module MO to MQ and is provided with a first low-waiting network 16 to lead the memory request of the low possibility of generating the competition of memories and a second network 18 to lead the memory request of the high possibility of generating the competition of memories. When a specific address is repeated enough times for regarding as a hot spot within the frame of a prescribed time, each of hot spot detectors 12 informs a turning device 10 of this to shore. When present memory access is at the hot spot, each turning device 10 transfers memory access to a memory system through the second mutual connection network 18. Thereby, the combination network of satisfactorily low waiting times is constituted.

Description

[Detailed description of the invention] The present invention will be explained in the following order.

A. Industrial field of application B. Prior art C0 Problems to be solved by the invention Means for solving the problems E. Example El. Introduction E2. - Boat fishing explanation E3. PREFERRED EMBODIMENTS (FIGS. 1-6) F0 EFFECTS OF THE INVENTION A. INDUSTRIAL APPLICATION FIELD OF THE INVENTION The present invention relates to data processor memory systems, and more particularly to data processor memory systems and, more particularly, to data processor memory systems. - Concerning such memory system controllers for use in large multiprocessor computer systems with modules. More specifically, the present invention relates to techniques for controlling the flow of data between individual processors and memory modules over complex interconnect networks.

B. Prior art According to recent research, the memory capacity of large-scale parallel computers
The system has been found to have drawbacks in that performance may be degraded. When the majority of memory references are directed to a single memory address, the overall system performance is limited by this single memory capacity. When this memory is accessed through a multi-stage interconnection network, it is called a "tree blockade".
e) The phenomenon known as J gives rise to further conflicts;
This results in significant delays for all users of the system.

This issue and related performance issues are discussed in the paper "Hot Spot Contention and Combination in Multi-Stage Interconnection Networks" listed below.

One solution to this problem is the “combinatorial network” proposed during the NYtJ supercomputer design (
rNYtJ Super Computer... Design of MIMD, Shared Memory Parallel Machine J IEEE Proceedings on Computers February 19B3, pp. 175-189 ("T
he N Y U Ultracomputer-De
signing a M I M D% 5hared
I'remory Parallel Machine
, “I E E E Transactions
onComputers, February 19
83, pp. 175-189). With this combinatorial network, references to hot spots in memory are combined before going to memory, reducing contention for hot spots, but combinatory networks require complex and expensive hardware designs. . It is currently difficult to construct a combinational network with sufficiently low latency to support all memory references in large multiprocessor systems. Another method is to form two networks, one for low latency and the other for high contention traffic handling, and send messages that are expected to create hot spots to the second network. It is something to be directed toward. However, problems remain in effectively selecting hot spot references from general memory traffic. It is of course possible to avoid hot spot problems in such systems by relying on software or message type scheduling to indicate what references are being made to hot spots. . Hot spots can therefore be identified in advance. However, such a solution is clearly impractical in large multi-user systems.

This is because the performance of the entire system depends on correct and efficient programming by all users, and consistency in indicating what types of references are being made to hot spots must be implied. It is.

Therefore, such multiprocessor, multimemory,
The concept of a module interconnection system has been proposed,
Low-latency networks and potential high-latency hot
Spot combinatorial networks have been proposed, but hot
There is no known efficient means or mechanism to dynamically identify messages indicating spots and subsequently control the interconnection mechanism.

U.S. Patent Application No. 04 filed May 12, 1987
No. 8982 discloses a two-network interconnection system for use in a user multiprocessor system with multiple processors and multiple memories, which must be interconnected via a suitable interconnection network. In this system, the concept of a delta network is combined with a simple crossroads network, with messages being sent alternately to either network under certain circumstances. However, this patent application has a hot
A method for continuously monitoring spots on a real-time basis and directing subsequent memory references to these hot spots onto the appropriate combinatorial network and directing other regular memory references onto the low-latency network. No disclosures.

The four papers listed below describe the attributes of an experimental multiprocessor computing system with large shared memory, known as RP3, published August 2, 1985.
Proceedings of 1985 International Conference on Parallel Processing held on 0-23
85 International Conference
ce on Parallel Processing
, August 20-23, 1985).

(1) Prototype of parallel processor for rlBM research (RP
3): Introduction and Architecture (“ThelBMRe
search Parallel Processor
Prototype (RP3): Intr
duction and architecture
”) J, pp. 764-771.

(2) V, A, Norton et al.
How to predict multiprocessor performance A
Methodology for Predict
ionMultiprocessor Perform
ance”), pp. 772-781. This paper discloses a method for predicting the performance of a given multiprocessor and shows how to predict various types of memory closures that can significantly impair overall system performance. .

(3) K, P, McAuliffe
"RP3 Processor/Mesori Element ("The
RP 3Processor/Memory Elem
ent') J pages 782-789.

(4) “Hot-5pot Contention and Combination in Multistage Interconnection Networks” by G., F., Pfister et al.
Combining in Multistage I
connection networks”
)” pp. 790-797.

(5) U.S. Patent No. 3,956,737.

(6) U.S. Patent Nos. 4,484,262 and 470,778
No. 1.

PROBLEM SOLVED BY THE C0 INVENTION It is an object of the present invention to provide an interconnection network management architecture for shared memory multiprocessor systems.

In accordance with the present invention, an interconnection network management architecture is provided for a system having at least one low latency communication network and at least one other interconnection network.

According to the invention, an interconnection network architecture is provided having means for diverting memory accesses to one or the other of the two networks based on memory addressing criteria.

According to the invention, means for redirecting said memory accesses are provided in an interconnection network architecture that uses a hot spot memory criterion as the redirection criterion.

According to the invention, hot
An interconnect network architecture is provided that dynamically detects and removes spots and uses the hot spots to divert the messages when detected.

SUMMARY OF THE INVENTION In accordance with the present invention, there is provided a plurality of processors and a plurality of individually addressable memory modules, each of which is connected to any one of said processors via a memory interconnection network. An interconnection network architecture for use in large shared memory multiprocessor computer systems that can be accessed by computers is provided. The system consists of two parallel interconnected memory networks, each capable of interconnecting any processor to any module. The two networks are comprised of a first low-latency network, where memory requests with a low probability of memory contention are directed, and a second network, where memory requests with a high probability of memory contention are directed.

A hot spot detection device is associated with each memory module to detect when a particular address in that module becomes a hot spot.

A diverting device is associated with each processor and memory network for selectively directing memory requests to the first or second memory network.

A hot spot notification device is provided for interconnecting all of the hot spot detection devices with all of the diversion devices. Each of the above-mentioned turning devices includes all hot spots detected by any of the above-mentioned detection devices.
A memory device is provided for storing the spots. Each diversion device includes a device that determines whether the current memory access is in a hot spot listed within the memory system and routes the memory access, if any, through a second interconnect network. An apparatus is provided for transferring to a memory system.

Each of the above detection devices determines whether, within a predetermined time frame, a particular address is repeated enough times to be considered a hot spot, and if so, all the turns associated with each processor are This fact to the device,
It is notified and stored along with the address of the new hot spot, including the device for later use in each diversion device.

E., EXAMPLE EL, INTRODUCTION The present invention provides an interconnect architecture for large shared memory multiprocessor systems having hardware techniques for detecting hot spots in network traffic and redirecting memory references through a combinatorial network. Concerning connection network management architecture. The term "hot spot" refers to the same memory location that is referenced repeatedly within a predetermined time frame. hot·
It will be clear that the spot designation may be varied freely by the system designer to suit different system applications. In other words, not only the time frame but also the total number of references to the same memory can be changed. However, the time frame is irrelevant to the underlying architecture that provides this level of control.

The present invention provides many advantages in managing memory traffic. This management can be easily automated so that programmer error or carelessness (other than inefficient user performance) does not degrade the overall system.

When traffic is diverted from one network to another, there can be inconsistencies due to changes in the order of accesses to variables accessed by the same processor. For example, load and store instructions directed to the same address may not appear in the order in which they occur when they travel on different networks. Maintaining order consistency in software is difficult, requires compiler assistance, or requires significant execution time margins. Such consistency is absolutely critical to maintaining correct execution of programs within the system. The present invention provides a hardware procedure that guarantees order consistency even when network traffic to the same memory sometimes flows through different networks (eg, combinational or non-combinational networks). Memory references issued from any one processor to any one memory occur in the order in which they occur, regardless of the network through which these memory references pass.

Since only the processor's hot spot references are diverted through the high contention network, such a network is likely to encounter the worst-case scenario, with a minimal Can be configured by cost.

If, for example, all synchronous traffic is diverted through the combinational network, most of which do not result in hot spots, then the combinational network is considerably more complex than the non-combined network and therefore requires more logic levels, checks and The fact that the comparison operation has to go through increases the latency.

The cost of the interconnection control architecture of the present invention is that all memory accesses go through the combinatorial network with sufficient bandwidth to provide adequate memory efficiency in large systems. It is considered cheaper than the cost of configuring the .

Furthermore, the required hardware and procedures of the present invention can be overlapped with functions such as cache-lookup and translation functions, and the total time for memory references using a single ultra-large, low-latency network is The time will not increase significantly. As mentioned above, the cost of such a low-latency network is several times lower than the cost of a combinational network.
It is.

Another advantage of the interconnection network management architecture of the present invention is that hot
Information or data indicating the spot may become available during execution time. This information can then be reported to users of the system to correct software errors when hot spots are undesirable in the application. Therefore, change the data organization etc. with this information,
Data can be better distributed over interleaved memory systems to eliminate or at least minimize hot spots.

The architecture of the present invention is intended for use in systems of multiple (eg, 10 or more) independent processing units. Two communication networks are required.

One is the New York University Supercomputer Project (New
York UniversityUltracomp
ter Project) and proposed by A.

The paper rNY by Gottlieb et al.
U Ultracomputer...MIND Shared Memory Parallel Machine ('The NYU Ultracomputer M
``emory Parallel Machine'')'',
IEEE Transactions on Computers, February 1983
The combinatorial network described in detail on pages 175-185 of ``Computers'', pages 175-185, is capable of handling high contention traffic. Other networks are of sufficient low latency to efficiently transfer all non-hot spot references to memory. Both networks are provided internally in hardware that can maintain order consistency in the sense that the order of two messages to the same destination address arriving at any input is preserved until the output of the network. This means that there should be no message skipping to any one destination. In order to prevent such jumps, the present invention uses a "fencing" feature and primitive memory accesses described below within the memory itself, allowing one access to complete before another is guaranteed to start.

A memory system has one physical address space such that one address can determine a memory module as well as the location within the module of a particular word or addressable element. It may also be desirable to have other features that reduce the likelihood of memory blockage caused by factors beyond the scope of the present invention. These features include:

(1) Memory interpeening: This operation must be performed to minimize memory contention that occurs when many processors attempt to access different addresses within a page.

(2) Address hashing or skew inter:
This operation must be done to minimize memory contention resulting from contention due to stride accesses to memory. For such hashing methods, see, for example, US Pat. No. 07/114,909, filed Oct. 29, 1987.

(3) Caching of read-only data: This prevents shared code and constants from creating hot spots since at most one access per processor occurs, at least within the current time frame. Of course, such data is not specified as cacheable, but
The management architecture of the present invention prevents the creation of hot spots.

The three functions described above are incorporated into the large memory multiprocessor structure known as RP3 mentioned above.

E2. General Description The interconnection network management architecture of the present invention will now be described in general functional terms. The hardware required for the basic architecture can be summarized as follows.

Hot Spot Detection Device: This device detects certain memory
Monitor all memory references in a module. Each memory module is provided with such a detection device. The device has a counter that lists the most recently accessed physical address in memory. This detection device requires associative memory (with or without set associativity), such as is commonly used in cache directories. The total number of accesses is tabulated by the detection device over a time window of memory cycles, for example 1000-10000 cycles. However, this value can be changed as specified by the system designer. References to any one address are sufficiently frequent and hot
When a spot is formed, the hot spot detection device activates the hot spot notification bus to notify all diversion devices, as described below.

As mentioned above, the particular darkness value at which a hot spot is considered to exist can vary and is determined empirically for a given system. However, the particular darkness value does not limit the overall architecture of the present invention. This is because the dark value is essentially a count given to the system against which the hot spot test is repeated.

After gaining control of the hot spot notification bus, the detection device simultaneously reports its address to all diverting devices associated with each processor. These turning devices have a memory device designated as a hot spot lookaside buffer, preferably an associative memory. Memory locations already designated as hot spots are repeatedly re-designated (remained) as hot spots until intensive accesses cease. The darkness value for re-designating a previously designated hot spot as such may be lower than the originally determined darkness value. This is because when traffic is diverted to a combinational network, there are fewer memory hot spot accesses due to the combination of messages in the network. However, since whether or not to re-specify is a matter of choice of the system designer, the architecture for re-specifying the dark value for a particular hot spot that has been specified by − degrees is not specifically described in this example. will not be disclosed. However, this functional group can easily be achieved by using a two-level darkness value and using a second, lower darkness value for the previously designated hot spot. This functionality can be accomplished, for example, by providing additional bits in the hot spot memory associated with each detection device.

The next important functional element of the system is the hot spot notification bus or hot spot notification device. This bus provides a means for all hot spot detection devices to communicate with all diverting devices, and thus with the hot spot index buffers in the diverting devices. All hot spot detection devices can forward messages onto the bus, and all hot spot detection devices on each
The spot index buffer is controlled by messages sent over the bus. Access to this bus is determined in a fair mechanism, such as a round robin priority mechanism, with each hot spot detection device periodically gaining control of its own allocation period on the bus. Bus bandwidth must be sufficient to handle all such messages. If bus cycles are comparable to memory cycle times, such serial communication bandwidth may be suitable for systems containing thousands of independent memory modules.

The bus must be wide enough to transfer one physical address and the information necessary to resolve control and error detection. Other means of communication from the hot spot detection device to the diversion device may also be used. It is not necessary that all messages reach the diverter at the same time. For example, a token passing ring would be suitable for this communication. It will be clear that this means of communication is a design choice, determined by considerations of how much one wants to invest in this particular piece of hardware and the speed of the other parts of the system.

The last major functional element of the invention is the diversion devices, each having a hot spot lookup buffer.

Diverting devices exist between the various processors of the system and the input ports of the network. The translation device needs access to the translated (physical) addresses of the system and is therefore located after the memory mapping function. The hot spot index buffer contains an index table of all physical addresses currently designated as hot spots in the system. All of the Hot Spot index buffers maintain exactly the same list and operate entirely under the control of Hot Spot Notification. Once a hot spot address is specified, this address is compared to the current contents of the buffer. If this address has already been specified, the time stamp is updated to give an up-to-date indication of the hot spot status.

As will become clear later, this time stamp is used in the hot spot index buffer replacement algorithm.

If this address is not the one previously specified, then
The item with the oldest time stamp is deleted and replaced by the current address.

The previous discussion has assumed that the hot spot index buffer consists of a fully associative memory.

Alternatively, if a double or quadruple associative index is used, the oldest entry within the same congruence class is replaced by the new hot spot address. Since each hot spot lookup buffer uses the same replacement algorithm, they maintain the same list of unresolved addresses. At the same time that the index buffer is updated, a "fencing" operation is initiated at the processor's node, delaying the next memory reference from this node until all outstanding memory requests are resolved. If the system is designed so that there is a cache on each processor node, then fences only apply to non-cacheable references. Fencing is only necessary when addresses are added to or subtracted from the index buffer. If hot
There is no need for fencing when the spot address has already been specified, and subsequent references to the hot spot do not need fencing. In many architectures, fencing is not required because only one reference is allowed to be pending at any given time. The function of the hot spot look-up buffer within each conversion device is to compare each translation address generated by the memory mapping device with the address in the buffer. All matching addresses are diverted to high contention networks. Other references are forwarded to the low latency network. The index proceeds in parallel with the time stamp update, but normal indexing operations of the buffer must be suspended when entries in the hot spot index buffer are being replaced. There are many variables in the present invention that must be set after considering the system design and experimenting with different set values. Improper set values result in excessive bus contention or inefficient selection of hot spots. These variables include:

(1) the time window used to detect the hot spot by the hot spot detection device; (2) the dark value or number of references needed to designate it as a hot spot; and (3) the hot spot index buffer, also hot·
Dimensions of the spot detection device's address buffer, e.g. number of entries.

E3. Preferred Embodiment FIG. 2 shows the overall layout of a prior art multiprocessor system having a plurality of processors Po-Pn selectively connectable to a plurality of memory units Mo-Mq by a suitable interconnection network. As mentioned above, this interconnection network consists of a simple intersection switch
It may be either a network or a complex combinatorial network. Both of these are already generally known. The number of processors and the number of individually addressable memory modules may or may not be the same, ie n≠q. The drawback of such interconnection networks is clearly the complexity or sophistication of the network itself.

FIG. 3 is a drawing similar to FIG. 2, but showing two possible interconnection networks for selectively connecting individual processors Po-Pn to individual memory modules Mo-Mn. This network is an interconnection network of the type described in the above-mentioned article "Hot Spot Contention and Combination in Multi-Stage Interconnection Networks". Network 1 here is, for example, a low-latency network, and network 2 is, for example, a much more complex combinatorial network.

The block diagram of FIG. 3 does not specifically show a mechanism for determining whether an individual memory access is to be performed via network 1 or network 2. This entire network diagram applies equally to prior art networks or to the present invention.

A significant feature of the present invention is, of course, that the decision to route a message to network 1 or network 2 is not made on a predetermined, fixed, static basis (e.g., by a software method implemented in a compiler). It is in a specific architecture that is done dynamically. The present invention provides a dynamic interconnection network management architecture that bases network routing on dynamic factors of specific problems encountered in the system. As mentioned above, switching simply by varying the time window for the hot spot detection cycle, the hot spot reference value or the actual dimensions of the hot spot index buffer and address buffer of the hot spot detection device. Changing the number of darkness values is relatively easy.

The overall organization of the interconnection network management architecture of the present invention is shown in FIG. FIG. 1 clearly shows the three main basic functional units of the invention. These units include a turning device 10 and a hot spot detection device 12.
and a hot spot notification device or notification bus 14. Although two networks 16 and 18 are shown schematically, being a low latency network and a combinatorial network respectively, it should be noted that the functionality of these networks can easily be reversed by appropriate changes in the control of the diverter. It should be obvious. It will also be clear that the n+1 processors have separate turning devices 10. Same (q+
Each memory module has its own dedicated detection device 12. Of course, a single notification bus is used to interconnect all sensing devices and all diverting devices.

All deflection and detection devices can also be selectively connected to the two networks 16 and 18. As shown in FIG. 1, the details of the detection device 12 are shown in FIG. 4(a).
The details of the turning device 10 are shown in FIG. 6(a).

Briefly repeating the overall operation of the system of the present invention with reference to FIG. 1, the function of the detection device is to continuously monitor each new memory address received by the detection device so that Determine if it is a hot spot. When the hardware in the detection device determines that a hot spot exists according to the criteria used, a message is sent via the notification bus to all turning devices whose hot spot index buffers are new. Updated appropriately with hot spot information. Once the information is received by the diverters and the hot spot index buffer is updated appropriately, one of the diverters acts as a communication device for the diverters (because the diverters are the same and contain the same information). ), all of the detection devices are notified that a particular hot spot is being removed. As will be made clear below, this operation is done based on a suitable replacement algorithm such that the particular address being removed from the hot spot has set up the hot spot via the notification bus. Returned to the particular detection device, this hot spot is effectively removed from the hot spot memory array in the detection device. Obviously, since a specific address is involved, only one of the detection devices will be affected by this address. That is, the detection device refers to the memory block or module pointed to by this address.

Table 4(A) is a mid-level functional block diagram of one of the detection devices 12 shown in FIG. All detection devices are the same in terms of hardware components;
Of course, the content of the data is different. The main functional elements of the detection device are the address comparison block 20, the hot
The locations of the spot address memory 22, the dark value count comparison block 24, and the hot spot address memory are all determined for states H=1 (hot spot) and C=0 (notification not completed). A comparison block 26 is periodically scanned. Blocks 20.24 and 26
It will be clear that all of the comparison functions performed by are performed with respect to the contents of hot spot address memory 22.

In FIG. 4A, only data and control lines are shown according to the functions performed and the general area of the hot spot address memory with which each of these functional elements is associated. The address comparison block 20 therefore serves to compare the currently received address with the entire contents of the hot spot address memory and thus examine the memory address in such an associative memory.

If a successful match is found in the address comparison block 20, the dark value count comparison block 24 counts (CT)
Examine the field to determine if a hot spot darkness value has been reached. The dashed line between blocks 20 and 24 is
It is shown that block 24 is only activated when there is an address match in the hot spot address memory, as will be seen below. Obviously, the actual value used for a particular comparison is fixed and determined by the system designer, while the particular address being compared in block 20 is the current address being requested in memory Mj. They are different according to.

Block 26, on the other hand, operates under its own separate system clock and operates under conditions H=1, C
H in the hot spot address memory to find = 0
and C field are continuously scanned. Assuming that the hot spot address memory is fully associative across all fields, this scanning operation is also done on an associative basis and when the search criteria are satisfied, the hot spot address memory is fully associative across all fields.
A particular row in the hot spot address memory containing the spot address is accessed, read from the address memory, and placed on the notification bus.

Specific arguments for hot spot address memory and address buffers are not shown because such memory is well known and can take many forms. Similarly block 20.
Although the specific gate circuits and registers for 24 and 26 are not shown, it is likely that the hardware circuitry for embodying these functions will be obvious to the computer system designer and the fine level of detail will be appreciated. This is because it would confuse the description of the present invention.

The control mechanism used to properly order the operations of the hardware of FIG. 4A is typically one or more read-only memories with associated decoders and logic circuitry; etc., as shown in Figure 4 (B), which will be explained below.
, (C) and (D).

Table 1 shows typical formats of typical hot spot address memories suitable for use with the present invention.

Table 1 Hot Spot Address Each access to memory is determined by the address stored in the address field. Other fields include a reference count, a hot spot field (H), and a communication completion field (C).

An optional time stamp field is also shown. The functional control element for this field is not shown since it is optional for the system designer to include this functionality. The ability to compare the current set value of all time stamps to some current value is simple.

If a particular item in hot spot address memory is not incremented within a scheduled period of time, its reference count is reset to zero.

Therefore, if the reference count does not reach hot stock status for a significant period of time, this may improve the performance of the system under certain circumstances. This is because only currently activated hot spots are placed in the converter's hot spot list.

The time stamp is also used by the detector's replacement algorithm as an additional criterion to use when a particular hot spot address must be removed to make room for a new address. The address field within this address memory is large enough to identify the address of the memory. The reference count has as many bits as needed by the system designer to represent the highest count magnitude encountered before resetting the counter. The time stamp, like the reference count, has as many bits as necessary to represent the current setting of the system's time clock. The hot spot field (I()) and notification completion field (C), of course, only require one bit in the current embodiment.

FIG. 4B shows a flowchart of the operations that occur during a normal memory address procedure in which an address is presented to a particular memory module. The address must first be passed to a detection device, specifically a functional block 20 which stores the current memory address in the appropriate register, and then this request must be passed to memory. This operation occurs in block 10.

At block 12, the current address is currently hot.
A determination is made as to whether there is a match with any address in spot memory 22. This function is performed in compare block 20, which must sequentially match the address argument against all entries in associative hot spot address memory 22. If a match is found, the procedure continues at block 14. If no match is found, the procedure branches to block 22.

Assuming a match is found, block 14 increments the reference count (CT) associated with the matched address in the hot spot address memory;
The procedure continues at block 16.

In block 16, the current reference count magnitude is compared to the dark value in comparison block 24 to determine if the hot spot dark value has not yet been reached. If not, the procedure is complete and nothing further needs to be done. If the reference count is greater than the dark value, the sequence proceeds to block 18 which determines whether the current hot spot H set value is equal to one. H
Since the -1 means that this particular address is already a designated hot spot, no further action is taken and the procedure ends via the YES output line. If the H field is not currently equal to l, the procedure continues at block 20 with setting the appropriate H field to one and ensuring that the notification complete field (C) is set to O. Once these operations are complete, the procedure ends.

Assuming that the test in block 12 resulted in a branch to block 22, this means that a new address must be entered into the hot spot address memory.

A replacement procedure is initiated by block 22, whereby one of the currently used rows in the hot spot address memory is removed and replaced by a new address. Any suitable replacement algorithm can be used for this operation. For example, criteria for replacement include the oldest used, the one with the lowest reference count, etc. Such replacement addresses are generally known in the art and will not be described in detail. Also, the overall concept of the invention is not limited by the particular replacement algorithm used.

If a slot is found in the hot spot address memory, the system controller in block 24 fills in the address, sets the reference count field CT to zero, sets the H and C fields to zero, Sets the time stamp (if used) to the current set value of the system clock. The access procedure ends here.

The procedure in Figure 4(B) is the actual hot
It will be clear that the spot scanning procedure and the termination procedure of FIG. 4(D) operate independently. In summary, the fourth
The procedure in Figure (B) keeps a record of requested addresses and how often an address is requested and whether it is hot or
By determining whether the memory
Maintain statistical data on access or hot spots.

How this statistical data is used to inform the diverter is shown in the hot spot scanning procedure of FIG. 4(C). Block 10 implies the beginning of this procedure, and it is noted that this procedure occurs every memory cycle. This procedure is actually independent of the memory access procedure, but since its operation continuously scans the hot spot address memory and examines the H and C fields, this argument is a basic associative It is more convenient to add it to the search. However, the address matching does not interfere with the continuity of the hot spot address memory access procedure, and the H and C field matching does not interfere with the hot spot address memory access procedure.
The control method is set up so that the spot scanning procedure is carried out continuously. However, although functionally the scan procedure can be driven by a separate clock, the scan must occur when the memory access procedure is not in effect. However, this independence is a design choice of the system designer and does not significantly affect the principles of the invention.

The explanation of FIG. 4(C) will be continued. The procedure begins at block 10. The first operation begins at block 12, where comparator 26 of FIG. 4A scans the content addressable memory for the first entry having fields H=1 and C=0.

Once the search operation is completed in block 12 and the first detected but unannounced hot spot is found, the procedure continues to block 14. At block 14, a notification function is initiated on the notification bus and the newly detected hot spot address is transferred to all diverting devices on the next available bus cycle. At this point, the hot spot scanning procedure proceeds to a waiting state and remains in this state until a response is received from the particular diverter. This particular turning device is the one designated to notify all detection devices about a particular hot spot that has been removed from the hot spot index buffer.

The redirecting device sends a reply via the notification bus to the detecting device that sent the new hot spot address. This reply is received in block 16 of the hot spot scanning procedure and sets the C field of the new hot spot's address to one.

This means that the notification function for new hot spots is complete. Other functions that must be performed as a result of the termination of the notification function are shown in FIG. 4(D), which are sent to all of the detection devices to provide feedback.

The "termination procedure" in FIG. 4(E) is called when a termination message is received from the notification function, specifically from the notification conversion device. The sequence is shown starting at block 10.

Referring again to FIG. 4A, the termination message is shown as being received via the bottom line of the notification bus in the figure. The termination message causes the detection device to assign the address received via the termination message to the hot spot.
Checks all entries in address memory for a match. If no match is found, the procedure ends at the end of block 12. If a match is found, the procedure proceeds to block 14, where the C and H fields for this particular address are set to O. This action effectively eliminates the particular address that was currently a hot spot. If an address is not found in the hot spot address memory, this means that this address has been removed by the detector's own replacement algorithm when a new address is operated on the system. Conversely, if the address still exists, the system indicates that this address is no longer a hot spot because the hot spot has been replaced by a more recent hot spot.
You must know that it is not designated as a spot. In order for this state to be re-established as a hot spot, a predetermined number of accesses must be made until the predetermined hot spot darkness value is reached.

Referring to FIG. 5A, a very high level organizational chart of the notification bus is shown. The bus connects all of the n+1 turning devices associated with each of the n+1 processors P and the q+ bus associated with each of the q+1 memory modules M.
All connected to one detection device. The bus is simply a two-way bus that only needs to transfer addresses and simple control fields from the sensing device to the diverting device. Of course,
The address specifies the new hot spot being designated by the system, and conversely, the address forwarded from the specified redirecting device to all detecting devices indicates that a particular hot spot address Indicates that it has been removed from the buffer. message,
For example, an address may be given to all sensing devices, but only the particular sensing device associated with the memory module containing this address will respond to (and be affected by) this message. As mentioned above, the architecture of the notification bus can take many possible forms, including Token Ring, Round Robin, and the like. The only condition for a notification bus is that the detection devices have access to the bus on an orderly and organized basis and that hot spots are not left unmarked for long periods of time. The architecture of the bus also requires that a particular turning device access the bus during the cycle immediately following the transfer of a new hot spot message, which must respond to new hot spot notifications throughout the bus cycle. It is necessary to be able to do it. Obviously, this mechanism must be coupled to the architecture of the deflection device.

FIG. 5(B) is a very simple flowchart that repeats the above. This flowchart is labeled the notification bus procedure and consists of only two blocks 10 and 12.

As shown in block 10, when a - degree hot spot is found in one of the detection devices, the detection device takes control of the notification bus on a scheduled basis and detects a new hot spot.
Spot addresses must be sent to all diverters.

Once this transmission is made, on the very next bus cycle the designated diverting device must send a confirmation message back to the particular detecting device whose current hot spot address is being removed from the system. As is clear from the previous description of the operation of the detection device, hot spots are effectively eliminated by resetting the H and C fields to O (block 12). Of course, this operation assumes that this address is still in the hot spot address memory of the particular detection device. If this address has already been removed by a local replacement operation, no further action is required. This completes the operation and functionality of the notification bus.

Next, the operation of each turning device will be explained with reference to FIGS. 6(A) to 6(D) and Table 2. Referring first to FIG. 6A, a mid-level functional block diagram for a turning device is shown. As in FIG. 4A, control signals for gating various addresses and various data items from, e.g., the Hot Spot Lookaside Buffer (H3LB), the Hot Spot Address Compare Block, etc., and the various registers. It is not necessary to show details of the control circuit for transferring the . These detailed parts are shown in the entire block diagram in Figure 6(A) and in Figure 6(B).
This can be easily guessed from the flowcharts from (D) to (D).

-i-wise, the function of the conversion device is to determine whether a new address that is being presented to the memory system is actually a hot spot, and if it is, it is The goal is to route memory access requests onto a combinatorial network. Additionally, when a hot spot is found, a mechanism must be provided to write it into the hot spot index buffer and to send a notification to the detection device indicating that the current hot spot has been removed from the system.

Furthermore, once a new hot spot is entered into the turning device, a mechanism is provided to ensure that all current memory requests by the associated processor are completed before the new memory request is granted. There must be. This allows later messages to be serviced sooner than earlier messages, preventing the possibility of data non-uniformity that would cause data to be replaced in system memory or returned to the processor out of order. .

Referring to FIG. 6A, the message return bus is shown at the top of the drawing and the message output bus is connected to the diverter 10.
It comes out from the right side. Obviously, these buses are both associated with network 1 and network 2, but they may or may not be bidirectional. Hot spot address comparison operations are performed in block 30 by appropriate registers. Here the current address is the hot spot index buffer (
H3LB) 32, and when a match occurs,
A signal indicating this match controls switch 34 to divert the current memory access towards combinational network (NET) 2. If a match does not occur, this indicates that this particular address has not yet been determined as a hot spot, and switch 34 directs the message onto low latency network 1.

Holding block 26 is a flip-flop that is set to 1 as soon as a notification is received by the diversion device that a new hot spot has been found. Counter F38 is for recording the outstanding memory requests active in each particular conversion device at a given time. Counter F38 is incremented whenever a new memory address is received and counter F3B is decremented when a message is returned via the message return bus. It is clear that the counter F is set to O or to some finite number depending on the number of memory accesses that are allowed to accumulate in the system at a given time. The block 40 labeled "wait when hold = 1" is simply a switch or interlock that causes processor Pj to hold a memory request whenever it is determined that hold block 36 is currently set to l. . This feature is well known in the art and requires the processor to wait until it can make a memory request. Also provided is a system controller associated with holding block 36 such that as soon as holding block 36 is set to 1, counter F is immediately interrogated and if counter F is currently zero, holding block 36 is interrogated. As soon as the counter F returns to the set value of O, the holding block 36 is reset to zero.

Referring to the two-man inputs to the underside of diverter 10 marked Hot Spot Found and Hot Spot Exit, these inputs represent two message functions performed via the notification bus. It is clear that all of the diverting devices respond to the hot spot discovery message, but only one designated diverting device need respond to the hot spot termination message.

Thus, in summary, all transfer devices of all processors are approximately the same with respect to the basic functional blocks and contents of the hot spot index buffer at any given time.

Each counter F contains different values. The communicating turning devices must also have a control sequence that notifies the detection device that the hot spot address is being deleted from the hot spot index buffers of all turning devices.

Also referring to FIG. 6A, the hot:spot discovery bus is connected to a hot bus lookaside buffer (H3LB) 32, and new hot spots are stored in the buffer 32 vacated by the replacement algorithm. It is now possible to do so. The notification bus also transfers a signal that sets holding block 36 to one.

Referring now to the flow diagrams of FIGS. 6(B), (C) and (D), the various operations within the converting device may be carried out by, for example, appropriate ROMs, conventional decoders, multiplexers and other sequential circuits. Describe the sequence of controlled operations.

The following table 2 shows the individual hot spot index buffers 32.
The format is shown below.

Table 2 Each buffer of each transfer device is identical and contains the same data at any given time. Each entry in the buffer contains a given hot spot address and a time stamp indicating when the hot spot was introduced into the system. The time stamp is for use in the buffer replacement algorithm and is used to determine which hot spots are removed when new hot spots are discovered. This is a simple replacement algorithm, but
Other more complex algorithms can be used if desired. Of course, regardless of which replacement algorithm is used, all index buffers and all conversion devices must use the same replacement algorithm to ensure that the same address is removed from all elements.

Referring to the flowchart in Figure 6(B), the hot bot l
- The sequence of events that must occur when (HS) is discovered by one of the detection devices and this is notified to all diverting devices with appropriate data, e.g. the address of the hot spot is shown. . In block 10 the procedure in the turning device begins and in block 12
Then, the holding block 36 is set to 1. The procedure then continues at block 14.

During block 14, the replacement algorithm within each turning device is called to make a determination as to which "current" hot spot must be removed from the hot spot index buffer as described above. A pointer points to the location in the hot spot index buffer where the new item is inserted. The procedure then continues at block 16.

Block 16 is not an actual step in the procedure, in the sense that all diverters, except the one selected for the notification function, proceed directly from block 14 to block 20. Must be introduced on the 18th.

The address of the item to be removed in the hot spot index buffer is then sent through the notification buffer to the detection device to update the hot spot address memory appropriately.

Items are removed in block 20 and new items are inserted in block 22. In practice, as described above, the new item is simply written over the old item in the hot spot index buffer. Next, in block 24, the time stamp for the new item is set.

Again, the time stamp is only set if this is used in the replacement algorithm, thus giving the time stamp field.

In block 26, a timeout or wait operation takes place, and the set value of counter F is continuously sampled;
It is determined whether it is set to the value 0, meaning that there are no current memory requests outstanding. Once it is determined that counter F has been decremented to O (or a new hot spot entry has been placed in H3LB and set to O), the procedure continues at block 28.

At block 2, the retention flip-flop is reset to 0, allowing recovery of the memory operations of the particular turning device and its associated processor.

This ends the procedure.

Referring to FIG. 6C, the holding flip-flop 36 is interrogated at block 10, where normal memory accesses from the processor are performed by the turning device. If this is set to 1, the system will wait until this is reset to 0. This action is often referred to as a "fence" action. One such specific prior art description is provided in the above-mentioned article "The RP 3 Processor/Memory Device".
See Chapter 7.3 on page 787 of ``Soy/Memory Element''). If the controller determines that the hold flip-flop is equal to O, the procedure continues at block 12.

At block 12, the current address presented by the processor to the diversion device is used as an argument to H3LB, such as within comparison block 30, to determine whether the current address is a hot spot. Incidentally, in architectures where the H3LB is an associative memory, the comparison is done within the H3LB itself, as shown in Figure 6 (
Note that the comparison block 30 in A) is just a holding register. If no match is found, the procedure continues at block 16 and the memory access request is sent over the low latency network 1.

If there is a match, the procedure continues at block 14, where the switch 34 sends the request over a more complex network 2, such as a combinational network. Once the steps of block 14 or 16 have been completed, the procedure continues with block 18.

During block 18, a counter F is incremented by one to record the current number of outstanding requests from conversion devices associated with the processor, as described above.

FIG. 6D is a one-step flowchart that illustrates the only action that takes place within the converter when a message is returned from the memory system indicating that a memory access operation is complete. For example, when the request is a load request, the data is accompanied by an "operation complete" signal, otherwise only this signal is returned. As mentioned above, counter F maintains a count of outstanding memory requests from the processor. Given all of the controls and sequences described above, this count is merely a statistic since all that is required of the system is to maintain the necessary data consistency.

This concludes the description of the present invention.

Effects of F8 Invention According to the present invention, large shared memory, multiprocessor,
An interconnection network management architecture for the system is provided.

[Brief explanation of the drawing]

FIG. 1 relates to a typical processor and memory module of a multiprocessor system. 1 is a high level functional block diagram illustrating the main functional units of the present invention. FIG. FIG. 2 is an organizational chart of a conventional large multiprocessor system, large shared main memory, and interconnection network. FIG. 3 is an organizational diagram of an applied multiprocessor memory interconnect system for use with the present invention that incorporates two separate interconnect networks with different transmission characteristics. FIG. 4A is a functional schematic diagram of one of the detection devices associated with each memory module of FIG. FIG. 4(B) is a flowchart of a memory access procedure performed in the detection device of FIG. 4(A). FIG. 4C is a flowchart of the independent procedures for performing hot spot detection within each detection device. FIG. 4(D) is a flowchart of the termination procedure performed within each detection device. FIG. 5A is a schematic diagram of the notification bus of the present invention. FIG. 5(B) is a flowchart showing the protocol of the notification bus. FIG. 6A is a functional schematic diagram of one of the turning devices of FIG. FIG. 6B is a flowchart of the procedure in the diverter when a new hot spot is discovered. FIG. 6C is a flow diagram of the procedure in the converter when receiving a standard memory access from the processor. FIG. 6D is a one-step flowchart of the operations that occur within the turning device when the memory reference is completed. 10... Turning device, 12... Detection device, 14.
...Notification device, 16...Network 1.18.
...Network 2゜Applicant International Business Machines Corporation Agent Patent Attorney Hitoshi Yamamoto ωj (1 other person) A in Figure 2, Figure 3, Rokuguchi. Memory access hand movement rooster honotos C throat run j 對手 J end figure 5 (A) '''7 tube 5 figure (B)

Claims (1)

Claims: In a shared memory multiprocessor system having a plurality of processors and a plurality of individually addressable memory modules each accessible by any of the processors via a memory interconnect network. , (a) comprising a first low-latency network to which memory requests that are unlikely to conflict are forwarded and a second network to which memory requests that are likely to conflict are forwarded;
at least two parallel interconnection networks; (b) a hot spot connected to each memory module for detecting when a particular address becomes a hot spot based on a scheduled frequency of access criteria; (c) a diversion device connected between the output of each processor and the interconnection network for selectively forwarding memory requests to either said first or second network; (d) provided between all said diverting devices and all said hot spot detection devices to notify all said diverting devices when a hot spot is encountered and to indicate that a current memory request is currently being directed; When being made to a hot spot, the memory request is
A shared memory multiprocessor with a hot spot notification device for transmission over a network of
system.