JP4361910B2 - Cache coherence device - Google Patents

Description

The present invention relates to a cache coherence apparatus for a multiprocessor system in which a plurality of processor modules are connected via a system bus, and more particularly, to a multiprocessor system in which a plurality of processor elements with caches are connected to each other via a main memory and an internal shared bus. This invention relates to a cache coherence apparatus.

  In recent years, since it can be expected that the processing performance of a computer apparatus is dramatically improved, development of a multiprocessor system in which a plurality of processors are connected via a shared bus has been advanced. In a multiprocessor system, by adopting a sparse scalar type or a VLIW (Very Long Instruction Word) type as a processor, the processing performance of a single processor is improved, and at the same time, the adoption of a cache mechanism greatly contributes to the performance improvement.

  As a cache mechanism of such a processor, a primary cache is built in the processor, and at the same time, a secondary cache is provided between the main cache and an external main memory. To improve performance by reducing access.

  Further, as a shared memory for a plurality of processors, a single memory unit is not provided, a local memory functioning as a main memory is distributed in a predetermined number of processors, and a shared memory area is distributed in this local memory. The shared memory can be flexibly constructed by the number of local memory units corresponding to the scale.

  FIG. 65 shows a typical conventional multiprocessor system. This system has, for example, two processor modules 1000-1 and 1000-2. The processor modules 1000-1 and 1000-2 have the same configuration. For example, when the processor module 1000-1 is taken as an example, the processor elements 1100-1 and 1100-2, cache units 1200-1 and 1200-2, and a common bus 1300 are used. -1 and local storage unit 1400-1.

  The processor elements 1100-1 and 1100-2 incorporate a primary cache, and cache units 1200-1 and 1200-2 as secondary caches are provided outside. The local storage unit 1400-1 functions as a main memory by assigning a unique physical space to the processor elements 1100-1, 1100-2, and at the same time, all the processor elements of the processor modules 1000-1, 1000-2. A shared memory space shared by 1100-1 to 1100-4 is allocated.

  Similarly, the processor module 1000-2 side includes processor elements 1100-3 and 1100-4, cache units 1200-3 and 1200-4, a common bus 1300-2, and a local storage unit 1400-2. The common buses 1300-1 and 1300-2 of the processor modules 1000-1 and 1000-2 operate as one bus connected via the back panel of the module housing.

  The cache control of the cache units 1200-1 to 1200-4 provided in the plurality of processors 1100-1 to 1100-4 is read access by a certain processor element. Responds by transferring a copy of the address to the cache.

  Further, when the cache is miss-hit by the write access of the processor, the corresponding local storage copy value is transferred to the cache and overwritten. At this time, since the value in the main memory is different from the copy value in the overwritten cache, cache coherence is realized by copy back in which the latest value on the cache is transferred to the main memory and the old value is updated.

If a local storage value is copied to multiple caches, the copy value on the cache other than the cache that was accessed for write access is invalidated, and then the copy value on the access source cache is invalidated. Is updated to the latest value, and then copied back to the local storage to realize cache coherence.
JP-A-6-259384 Japanese Patent Laid-Open No. 5-100952

  However, the multiprocessor system shown in FIG. 65 has the following problem because all processor elements, cache units, and local storage are connected to one bus.

  First, in order to realize cache coherence, instructions and data are transferred between a cache unit of a processor element that has generated a read or write access and a local storage unit having an access address. For this reason, bus transfer requests for realizing coherence among a plurality of processor elements compete with each other, the load on the common bus increases according to the number of processors, and high-speed processing cannot be performed.

  Of course, by providing a plurality of common buses, the bus load can be reduced. However, when the number of processors increases to 10 or 20, it cannot be handled.

  The processor modules 1000-1 and 1000-2 are constructed in module housing units. For this reason, the common buses 1300-1 and 1300-2 are cable-connected by connectors via the back panel of the housing. For this reason, the line length of the bus becomes long, and the clock frequency of the bus cannot be increased due to electrical characteristics.

  For example, if only the bus in the module is used, the clock frequency can be set to 60 MHz, but if the bus connection sharing the back panel is used, the clock frequency is reduced to 40 MHz. The present invention has been made in view of such a conventional problem, and a cache of a plurality of processors built in each processor module is provided by a two-level common bus structure in which a common bus is provided inside and outside the processor module. An object of the present invention is to provide a cache coherence device that efficiently realizes cache coherence between local storages provided in each module and improves processing performance.

  In a multiprocessor system, the system bus has a great influence on the performance. The system bus must increase the data transfer rate and the bus use efficiency, which is the ratio of cycles used for data transfer in the bus cycle. Otherwise, the system bus becomes a bottleneck, and performance will not improve even if the number of processors is increased.

  In order to improve the data transfer speed, it is common to increase the bus operating frequency by increasing the width of the data bus or adopting an interface with a small signal amplitude. In order to increase the bus use efficiency, a split type transfer method in which the use right is transferred to another bus unit from when a request such as memory access is issued until a response is received can be employed.

Accordingly, an object of the present invention is to provide a common bus of a cache coherence device that efficiently realizes cache coherence and does not impair the expandability of the system in a distributed shared memory multiprocessor system.

  FIG. 1 is a diagram illustrating the principle of the present invention.

  First, the present invention has a processor group including a plurality of processors 16. As shown in FIG. 1A, this processor group is divided into, for example, three processor modules 10-1 to 10-3 configured by at least one processor. A main memory shared by the processor group is provided. The main memory is divided into a plurality of local storage units 28, and each local storage unit 8 is arranged on the processor modules 10-1 to 10-3. In the following description, the main memory is labeled with a local memory.

  Each of the processors 16 is provided with a cache unit 18 that speeds up access to the main memory 28. The registration state of data in the main memory 28 in the cache unit 18 is stored in the directory storage unit 30 in units of cache lines. The main memory 28 and the cache unit 18 are connected by a first common bus 22, and the processor modules 10-1 to 10-3 are connected by a second common bus 12. Cache coherence between the main memory 28 and the cache unit 18 is realized by an internal coherence processing unit by the snoop unit 20. Further, cache coherence between the plurality of processor modules 10-1 to 10-3 is realized by an external coherence processing unit by the cache conversion unit 80.

  The cache unit 18 manages the cache line by dividing it into a plurality of sublines, and the directory storage unit 30 is in a dirty state where the data on the cache unit 18 is the latest value and the data on the main memory 28 is the old value. Are stored in units of sublines, and a share state in which the data on the cache unit and the main memory have the same value is stored in units of cache lines.

  Further, the cache state is stored in the directory storage unit 30 in units of processor modules. Specifically, each of the processor modules 10-1 to 10-3 has a directory control unit that stores a corresponding state of the plurality of processor modules 10-1 to 10-3 in the directory storage unit 30, and this directory control unit The registration to the directory storage unit 30 is designated for a plurality of processor modules.

  Each of the plurality of processor modules 10-1 to 10-3 includes a plurality of processors 16, a cache unit 18 provided for each processor 16, a local memory 28 used as a main memory, a main memory 28, and a directory. A memory management unit 26 that manages the storage unit 30, a first common bus 22 that connects the plurality of cache units 18 and the memory management unit 26, and a second common that connects the plurality of processor modules 10-1 to 10-3. The bus 12, the protocol management unit 24 that manages the protocol conversion of the first common bus 22 and the first common bus 12, and the module connection unit 32 that couples the processor modules 10-1 to 10-3 with the second common bus 12. The cache unit 18 is synchronized with a specific machine cycle on the first common bus 22 Coherence is realized between the processor modules 10-1 to 10-3 based on the reference of the directory storage unit 30 provided in the memory management unit 26 and the second common bus 12 provided in the memory management unit 26 and the cache coherence by knoop. A cache conversion unit 80 is provided.

  The cache unit 18 manages, for example, a 256-byte cache line by dividing it into 64-byte sublines. Correspondingly, a dirty state in which the data on the cache is the latest value and the data on the main memory is the old value is registered in the directory memory 30 in units of sublines. A share state in which the data on the cache and the main memory have the same value is registered in units of cache lines.

  As shown in FIG. 1B, the directory entry 104 registered in the directory storage unit 30 for each cache line includes, for example, a plurality of subline bits D3 to D0 indicating the presence / absence of a dirty state, and a shared state for each of a plurality of processor modules. The shared map bits S7 to S0 shown in FIG. Further, a configuration register for registering the correspondence between the bit positions of the shared map bits S7 to S0 and the processor module 10 is provided, and the shared state of a plurality of processor modules can be registered in the same map bit by this configuration register.

  Further, when the subline bits D3 to D0 of the directory entry 104 are set, information (PM-ID) of the processor module 10 in which the cache unit 18 having the latest value corresponding to this bit set exists is stored in the main memory 28. Store in a specific area. When the subline bits D3 to D0 are reset, the information (PM-ID) of the processor module 10 having the latest value is stored in a specific area of the main memory 28 corresponding to the reset bit.

  The processor 16 performs read access or write access for each subline of the cache line. The system bus 12 as the second common bus and the snoop bus 22 as the first common bus transfer information in a split format in which command transfer and response transfer are separated.

  The cache coherence for the read access of the processor has the following read modes 1 to 3.

[Read mode 1]
This read mode 1 is a miss hit in its own cache unit 18 and all other cache units 18 connected by the snoop bus 22 for read access of any processor 16 in the first processor module 10-1, and The read address is in the main memory in the first processor module 10-1, and the data of the read address exists in the dirty state having the latest value in the cache unit 18 in the second processor module 10-2 to be remote. Is the case.

  In this case, the first processor module 10-1 that has generated the read access issues a command to the system bus 12 to access the second processor module 10-2, and enters the dirty state via its own snoop bus 22. After obtaining the latest value from a certain cache unit 18, the latest value is returned to the first processor module 10-1 via the system bus 12. The first processor module 10-1 realizes cache coherence by storing the latest value obtained by this response in the cache unit 18 of the access source and responding to the read access, and simultaneously writing the latest value in the main memory 28. To do.

  Here, when the latest value is written in the first processor module 10-1 and the main memory 28, the directory entry 104 of the directory storage unit 30 is updated to the shared state, and the access-source cache unit 18 is updated to the tag. Update the memory invalidation state to the shared state.

[Read mode 2]
This read mode 2 is a miss hit in its own cache unit 18 and all other cache units 18 connected by the snoop bus 22 for read access of any processor 16 in the first processor module 10-1, and When the read address is in the main memory 28 in the second processor module 10-2 and the data of the read address exists in the dirty state having the latest value in the cache unit 18 in the third processor module 10-3. is there.

  In this case, the first processor module 10-1 issues a command to the system bus 12 to access the second processor module 10-2, and the second processor module 10-2 receives the command received from the system bus 12. Based on the above, a command is issued to the system bus 12 to access the third processor module 10-3. After the third processor module 10-3 obtains the latest value from the cache unit 18 in the dirty state by its own snoop bus 22, the third processor module 10-3 obtains the latest value on the system bus 12 by the first and second processor modules 10-1, 10 -2.

  The second processor module 10-2 writes the latest value obtained by reception of the response to the main memory 28, and the first processor module 10-1 stores the latest value obtained by reception of the response of the access source. Stored in the cache unit 18 to respond to read access.

  Here, the second processor module 10-2 updates the directory entry 104 of the directory storage unit 30 to the shared state when the latest value is written in the main memory 28. The cache unit 18 that is the access source of the first processor module 10-1 updates the invalidation state of the tag memory to the share state.

[Read mode 3]
This read mode 3 realizes cache coherence between caches in a certain processor module. That is, when the cache unit 14 of the own processor has a miss-hit for the read access of the arbitrary processor 14 in the arbitrary processor module 10-1, the latest value exists in the cache unit of another processor connected by the snoop bus 22. It is.

  In this case, the latest value is transferred between the cache units via the snoop bus 22 and stored in the access source cache unit 14 to respond to the read access. In this case, transfer of the latest value using the snoop bus 22 is executed in units of cache sublines.

  Next, the cache coherence for the write access of the processor includes the following write modes 1 to 5.

[Write mode 1]
In this write mode 1, write access is granted to the cache unit 14 of the first processor module 10-1 and the other cache unit 14 connected by the snoop bus 22 with respect to the write access of the arbitrary processor 16. The copy value is not stored in the possessed state, resulting in a miss-hit, the write address is in the main memory 28 in the first processor module 10-1, and the copy value of the write address in the main memory 28 Is in a shared state in the cache units 18 of the plurality of processor modules 10-2 and 10-3 other than the first processor module 10-1.

  In this case, the first processor module 10-1 issues an invalidation command to all the processor modules 10-2 and 10-3 in the share state using the system bus 12. All the processor modules 10-2 and 10-3 in the shared state simultaneously receive the invalidation command, and respond to the first processor module independently using the system bus 12 when the invalidation is successful. .

  The first processor module 10-1 recognizes the success of the invalidation by receiving all the responses from the system bus 12, and sends the copy value of the write address of the main memory 28 to the cache unit 18 of the access source. Store and overwrite by write access.

  Here, when the first processor module 10-1 recognizes the success of the invalidation and copies back to the access source cache unit 18, the first processor module 10-1 updates the share state in the directory entry of the directory storage unit 30 to the dirty state. .

[Write mode 2]
In this write mode 2, the copy value is written to the cache unit 18 of the first processor module 10-1 and the other cache unit 18 connected by the snoop bus 22 for the write access of the arbitrary processor 16. The right is not registered in the state where the right is registered and a miss occurs, the write address is the main memory 28 in the first processor module 10-1, and the copy value of the write address in the main memory 28 is the first value. This is a case where the cache units of the plurality of processor modules 10-1 to 10-3 including one processor module 10-1 exist in a shared state.

  In this case, the first processor module 10-1 issues an invalidation command to all the processor modules 10-2 and 10-3 in the share state using the system bus 12 in parallel. An invalidation command is issued to the cache unit in the first processor module 10-1 using the snoop bus 22.

  All the processor modules 10-2 and 10-3 in the shared state simultaneously accept the invalidation command and respond to the first processor module 10-1 independently using the system bus 12 to the success of the invalidation. In parallel, the cache unit 18 in the first processor module 10-1 responds with a successful invalidation using the snoop bus 22.

  The first processor module 10-1 recognizes the success of invalidation by receiving all responses from the system bus 12 and responses from its own snoop bus 22, and sets the write address of the main memory 28. The copy value is stored in the cache unit 18 and overwritten by the write access of the processor 16.

  Also in this case, as in the write mode 1, the first processor module 10-1 recognizes the success of the invalidation and copies it to the access source cache unit. Is updated to a dirty state.

[Write mode 3]
In this write mode 3, the copy value is written to the other cache unit 18 connected via the own cache unit 18 and the snoop bus 22 for the write access of the processor 16 in the first processor module 10-1. The right is not registered in the state where the right is registered and a miss occurs, the write address is in the main memory 28 in the second processor module 10-2, and the copy value of the write address in the main memory 28 is the second. This is a case where the cache modules of the plurality of processor modules 10-3 and 10-4 other than the processor module 10-2 exist in a shared state.

  In this case, the first processor module 10-1 uses the system bus 12 to issue a write ownership request command (home command) to the second processor module 10-2, thereby requesting the write ownership. The second processor module 10-2 that has received the command issues an invalidation command (purge command) to the other processor modules 10-3 and 10-4 in the shared state using the system bus 12. To do.

  All the processor modules 10-3 and 10-4 in the shared state receive the invalidation command at the same time, and when succeeding in the invalidation, use the system bus 12 to be independent of the second processor module 10-2. Respond to.

  When the second processor module 10-2 recognizes the success of the invalidation by receiving all the invalidation responses, the second processor module 10-2 uses the system bus 12 for the first processor module 10-1 to determine the main processor module 10-1. A write ownership transfer including the copy value of the access address of the storage 28 is responded.

  The first processor module 10-1 stores the copy value obtained by receiving the write ownership response in the access source cache unit 18 and overwrites it with the write access of the processor 16.

  Here, when the second processor module 10-1 recognizes the success of the invalidation and moves the write ownership, the second processor module 10-1 updates the share state in the directory entry of its own directory storage unit 30 to the dirty state.

[Write mode 4]
In this write mode 4, a copy value is written to the other cache unit 14 connected via the own cache unit 18 and the snoop bus 22 for the write access of any processor in the first processor module 10-1. The registered address is not registered in the state of possessing the right to write, and a miss occurs, the write address is in the main memory 28 in the second processor module 10-2, and a copy value of the write address in the main memory 28 Is in a shared state in the cache units 18 of the plurality of processor modules 10-3 and 10-4 including the second processor module 10-2.

  In this case, the first processor module 10-1 issues a write ownership request command (home command) to the second processor module 10-2. The second processor module 10-2 that has received the write ownership request command uses the system bus 12 to invalidate the command (purge) to all the processor modules 10-3 and 10-4 in the share state. In parallel with issuing the command (command), an invalidation command is issued using the snoop bus 22 to the cache unit 18 in its own share state.

  All the processor modules 10-3 and 10-4 in the shared state accept the invalidation command at the same time, and secondly, the success of invalidation is responded independently to the processor module 10-2 using the system bus 12. In parallel with this, the cache unit 18 of the second processor module 10-2 responds to the invalidation success using the snoop bus 22.

  The second processor module 10-2 recognizes the success of invalidation by receiving all the invalidation success responses from the system bus 12 and the invalidation success response from the snoop bus 22, and receives the first invalidation response. The system bus 12 is used to respond to the processor module 10-1 about the transfer of write ownership including the copy value of the write address of its own main memory 28.

  The first processor module 10-1 stores the copy value obtained by receiving the write ownership response in the access source cache unit 18 and overwrites it with the write access of the processor 16.

  Here, when the second processor module 10-2 recognizes the success of the invalidation and responds to the transfer of write ownership, the second processor module 10-2 updates the share state in the directory entry of its own directory storage unit 30 to the dirty state. To do.

  The invalidation command in the write mode is issued for each cache subline. Further, at the time of response to the invalidation command, the state of the other subline of the cache line including the subline commanded for invalidation is returned.

[Write mode 5]
In this write mode 5, the copy value is written to the cache unit 18 of the first processor module 10-1 and the other cache unit 18 connected by the snoop bus 22 for the write access of the arbitrary processor 16. The right is not registered in the state of possessing the right and a miss occurs, the write address is in the main memory 28 of the first processor module 10-1, and the latest value of the write address in the main memory 28 is the second value. This is a case where the cache unit 18 of the processor module 10-2 exists in a dirty state having the write right.

  In this case, the first processor module 10-1 issues a write ownership request command (remote command) to the second processor module 10-2 in the dirty state using the system bus 12. The second memory module 10-2 accepts the write ownership request command and uses the system bus 12 to transfer the write ownership including the latest value of the cache unit 18 to the first processor module 10. Responds to -1.

  The first processor module 10-1 stores the latest value obtained as a result of the write ownership response in the cache unit 18 that is the access source, and overwrites it with the write access by the processor 16.

  Here, the cache unit 18 of the second processor module 10-2 updates the dirty state of the tag memory to the invalid state when the write ownership is transferred, and the first processor module 10-1 When the ownership is received, the dirty state in the directory entry of its own directory storage unit 30 is updated to the dirty state changed to the information of the memory module 10-1 having the latest value on the cache.

[Write mode 6]
In this write mode 6, the copy value is written to the other cache unit 18 connected via the own cache unit 18 and the snoop bus 22 for the write access of the processor 16 in the first processor module 10-1. The right address is not registered and a miss occurs, and the write address is in the main memory 28 in the second processor module 10-2, and the latest value of the write address in the main memory 28 is the first value. This is a case where the cache unit 18 of the third processor module 10-3 exists in a dirty state having the write right.

  In this case, the first processor module 10-1 uses the system bus 12 to issue a write ownership request command to the second processor module 10-2. The second processor module 10-2 that has received the write ownership request command issues a write ownership request command to the third processor module 10-3 using the system bus 12.

  The third processor module 10-3 having received the write ownership request command uses the system bus 12 to transfer the write ownership including the latest value of the cache unit 18 to the first and second processor modules 10-. Responds to 1,10-2.

  The second processor module 10-2 receives the write ownership response and recognizes the write ownership transfer. The first processor module 10-1 stores the latest value obtained by receiving the write ownership in the access source cache unit 18 and overwrites it with the write access of the processor 16.

  Here, the cache unit 18 of the third processor module 10-3 updates the dirty state of the tag memory to the invalidated state when the write ownership is transferred. Further, when the second processor module 10-2 recognizes the transfer of the write ownership, the second processor module 10-2 changes the dirty state in the directory entry of its own directory storage unit 30 to the information of the processor module 10-1 having the latest value. Update to the changed dirty state.

  Further, the access source cache unit 18 of the first processor module 10-1 updates the invalidation state of the tag memory to the dirty state where the write right exists when the latest value is stored.

[Write mode 7]
This write mode 7 is a case where access is made between a plurality of caches in the processor module. That is, the cache unit 18 of the first processor module 10-1 has a miss-hit for the write access of any processor 16, but the write address is transferred to the other cache unit 14 connected via the snoop bus 22. Is the case where the latest value exists in a dirty state where the write right is owned.

  In this case, data transfer and transfer of write ownership are performed between the cache units 18 via the snoop bus 22 and stored in the access source cache unit 18, and then overwritten by the write access of the processor 16. Here, the cache unit 18 to be accessed updates the dirty state of the tag memory to the invalidated state after transferring the write ownership. Further, after acquiring the write right, the access source cache unit 18 updates the tag memory from the invalidated state to the dirty state where the write ownership exists.

[Cache unit access conflict processing]
According to the present invention, when the access from the inside of the module and the access from the outside of the module compete for the cache unit of the processor module, the following contention processing is performed. First, the first module is made to acquire a cache unit, and the second module is instructed to retry. Furthermore, the competing partner module and the access address are stored in the starting module, and when an access command is received from a partner module other than the competing partner module after completion of its own access, a retry is instructed, and the competing partner is retried Accept access commands preferentially.

  Specifically, for the cache unit 18 of the processor module 10-2, an access from a processor 16 inside the module and an access from an external processor module 10-2 compete, and the processor 16 starts the cache unit 18 first. Is obtained.

  In this case, the processor 16 instructs the processor module 10-1 to retry, further stores the competing processor module 10-1 and the access address, and from other than the competing processor module 10-1 after its own access ends. When the access command is received, a retry is instructed, and the access command by the retry of the competing second processor module 10-1 is preferentially received.

  As a result, the external processor module 10-1 repeats retries due to access conflicts with the plurality of processors 16 built in the processor module 10-2, resulting in a time error of the system bus 12. Therefore, the efficiency and high reliability of the cache coherence via the system bus 12 can be improved.

  In addition, if the module that has acquired the cache unit 18 does not access from the competitor module for a predetermined time after the end of its own access, the memory contents of the competitor and the access address are initialized, Cancel priority acceptance.

[Copyback write ownership to main memory]
When there is no latest data in the main memory 28 and the cache modules 18 exist in the plurality of cache units 18, the processor module of the present invention has write ownership for one of the plurality of cache units to copy the latest data back to the main memory 28. is doing. In this state, when the latest data is replaced by the processor 16 in the cache unit having write ownership, the write ownership is transferred to the cache unit holding the latest data, and the cache unit holding the latest data Until the number becomes one, copy back of the latest data to the main memory 28 is suppressed and is not performed.

  As a result, the burden on the snoop bus 22 connecting the plurality of cache units 18 and the memory management unit (directory side) 26 on the main memory 28 side is reduced.

  Since the plurality of cache units have unique identification numbers # 0 to # 3, the write ownership transfer is performed according to a predetermined order based on the identification number of the cache unit in which the latest data is replaced. Delegate.

  The memory management unit 26 that manages the main memory 28 has the write ownership indicating that the latest data exists in the cache unit 18 and the data in the main memory 28 is in a dirty state. Even if the ownership transfer is performed, the write ownership of the memory management unit 26 is maintained without being transferred to another processor module.

[Shared system bus]
Further, according to the present invention, there is provided a common system bus of a cache coherence device that efficiently realizes cache coherency and does not impair the expandability of the system in a distributed shared memory multiprocessor system. Furthermore, a highly reliable shared system bus is provided in consideration of use in a backbone system that is expected to use a large-scale multiprocessor. Furthermore, in consideration of connection with different buses, a shared bus that enables effective use of existing bus assets is provided.

  The second shared bus 12 functioning as a shared system bus of the present invention combines a bus connection unit 32 of a plurality of processor modules and a bus arbiter (bus arbitration unit), and performs split packet transfer between the plurality of processor modules. The bus command is transferred, and a source field indicating an access request source (source ID), a first destination field indicating a first access request destination (first destination ID), and a second access request destination ( Three second destination fields indicating a second destination ID) are provided.

  The bus command of the present invention having three of a source field, a first destination field, and a second destination field is used as a read bus command. The first processor module 10-1 that is the read request source specifies its own unit ID # 1 in the source field when the target data of its own read request exists in the main memory 28 of the second processor module 10-2. The first bus command specifying the unit ID # 2 of the second processor module in the first destination field is transmitted. In this case, the second destination field is not used.

  The second processor module 10-2 that has received the first bus command, if the latest data of the read request does not exist in the main memory 28 but exists in the cache unit 18 of the third processor module 10-3, Unit ID # 2 of the third processor module 10-3 having the latest data in the first destination field, and the unit ID # 3 of the first processor module 10 of the access request source in the second destination field. A second bus command designating unit ID # 1 is transmitted.

  Receiving the second bus command, the third processor module 10-3 designates its own unit ID # 3 in the source field, and unit ID # 2 of the second processor module 10-2 in the main memory 28 in the first destination field. And a reply command in which the unit ID # 1 of the first processor module 10-1 as the read request source is specified in the second destination field, and the first and second processor modules 10-1, 10-2 are transmitted. Simultaneously with read data.

  In this case, the first processor module 10-1 takes longer than normal bus transfer when the second bus command is sent from the second processor module 10-2 to the third processor module 10-3. The set time of the timer for monitoring the bus transfer timeout is changed from the first time to the longer second time.

  Further, the bus command used in the second shared bus 12 indicates a source field, a first destination field, and a second destination field, a bus identifier (bus ID) indicating the second shared bus, and a unit such as a processor module. It is expressed by two unit identifiers (unit IDs). As a result, when a plurality of second shared buses are connected to each other via the inter-bus connection unit, a unit in another second shared bus can be designated.

  Further, in the case of an extended system in which a plurality of subsystems connected by a plurality of processor modules via the second shared bus 18 are coupled via a subsystem connection unit, a subsystem extension identifier indicating transfer between subsystems is provided in the bus command. . When this subsystem extension identifier is valid, it has a command field for each subsystem of the bus command, and designates each of the source field, the first destination field, and the second destination field, and a unit of a different subsystem. Can be specified.

  Further, when a subsystem having a different system bus is coupled to the second shared bus 18 to which a plurality of processor modules are connected via a different bus connection unit, the bus command emulates a different system bus. A command is provided in a part of the command field so that units on different system buses can be specified.

  When the first processor module 10-1 as the bus master detects an abnormality in the second processor module 10-2 while transferring the bus command to the second processor module 10-2 as the bus slave, the second processor module 10 -2 sends an error reply bus command to notify the first processor module 10-1 of an error.

  Also, the bus arbiter monitors the bus abnormality until the second processor module 10-2 is determined as a bus slave by bus transfer of the bus command from the first processor module 10-1 as the bus master to the second processor module 10-2. When the bus arbiter detects an abnormality, it sends a reply bus command to notify the first processor module 10-1 of an error.

  This error reply bus command has a classification field indicating the cause of the detected abnormality, a field indicating in which access path an abnormality has occurred when there are a plurality of access paths, and a field indicating the type of detected abnormality.

  The bus command used in the second shared bus 12 is provided with an access identifier field for storing the operation identifier (access ID) of the source unit in each of the first destination field and the second destination field, and the operation of this access identifier field. The bus command is modified by the identifier, and the operation mode of the bus command is changed.

  For example, for cache coherence, a first bus command is sent to the second processor module 10-2 having the main memory 28 of the access destination from the first processor module 10-1 as the access request source, and the second processor module 10 -2 to send the second bus command to the third processor module 10-3 having the latest data in the cache unit 18 and finally send the reply bus command from the third processor module 10-3, the access identifier is Specify as follows.

  First, the first processor module 10-1 designates its own operation identifier in the access identifier field of the first destination field in the second bus command. The second processor module 10-2 designates the same operation identifier as that of the second bus command in each of the access identifier fields of the first and second destination fields in the second bus command.

  The third processor module 10-3 designates the same operation identifier as that of the second bus command in each of the access identifier fields of the first and second destination fields in the reply bus command. Each of the first and second processor modules 10-1 and 10-2 determines its own match between the operation identifier of its own destination field of the reply bus command and the operation identifier stored and held when the bus command is transmitted. It is determined that the reply is for the bus command.

  For this reason, the first processor module 10-1 as the access request source sends a reply to the second bus command as a result of the transmission of the second bus command by the second processor module 10-2 by the first bus command sent by itself. The command is sent from the third processor module, but the first processor module is not aware of the second bus command and can receive it as a reply to its own first bus command.

  The hardware of the second shared bus 12 includes a data bus line that transfers data and parity in parallel, and a tag information line that defines the operation of the bus sequence independently of the data bus line. In this way, a plurality of bus cycles are continued with one bus acquisition. By specifying the retry by the tag information line, it is possible to instruct invalidation of the bus sequence being executed at an arbitrary timing of the bus cycle.

  Further, the bus connection unit 32 and the bus arbiter of the plurality of processor modules connected to each other by the tag control line, when none of the plurality of bus connection units starts a bus cycle, from the bus arbiter to each bus connection unit. A specific bus connection unit that receives a bus use permission signal (bus grant signal) that is output individually and a bus use start instruction signal (tag bus signal) that is output in common to all bus connection units from the bus arbiter simultaneously performs a bus cycle. Start.

  If another bus connection unit has already started the bus sequence, the bus connection unit that has received the bus use permission signal detects the bus cycle end signal output by the other bus connection unit and executes the bus cycle. Start.

  The bus arbiter suspends subsequent bus use request processing until all bus use requests from the bus connection units received at the specific timing are processed.

  The tag information line has a first bus use request line used for a normal bus use request signal and a second bus use request line used for an emergency bus use request signal such as an abnormality report. The bus arbiter gives the bus use permission in preference to the bus connection unit that outputs the emergency request signal among the bus use request signals received at the specific timing.

  Furthermore, a combination of signals on the first and second bus use request lines is used to notify a normal bus use request, an emergency bus use request, no bus use request, and a non-operating state of the bus connection unit itself.

[Second directory storage unit]
In another embodiment of the present invention, in addition to the directory storage unit 30 of the memory management unit 26, the bus connection unit 32 is provided with a second directory storage unit. In this second directory storage unit, a state of waiting for a response to an access request to another processor module is registered in units of cache lines, and the same cache is accessed by referring to the storage unit of the second directory for accesses from other processor modules. If it recognizes line access, it responds busy.

  Specifically, the second processor module 10-2 as the home that has received write access from the first processor module 10-1 as the remote invalidates the same cache line with respect to the remote third processor module 10-3. Is registered in the second directory storage unit, and a busy response is sent to the second shared bus 12 in response to an access request for the same cache line from another processor module. .

  This busy response is transmitted to the cache unit in the requesting processor module to cause a retry. If the buffer connection of the path connection unit 32 is full due to an access request from another processor module to the home second processor module 10-2 while waiting for the completion of invalidation, the bus connection unit of the processor module that is the access request source Is sent to the second shared bus 32 without being transmitted to the cache unit.

  By such a second directory storage unit on the second shared bus side, it is not necessary to transmit an access request of another processor module to the memory management unit 26 side, and the Vichy response can be speeded up.

  According to such a cache coherent device of the present invention, the following operation is obtained. First, a plurality of processors with caches provided in a plurality of processor modules are connected to each other via the internal snoop bus (first common bus) and to the main memory. Furthermore, each processor module connects the main memory side via a system bus (second common bus). With this two-layer common bus configuration, it is possible to separate the cache coherence bus operation within the processor module and the cache coherence bus operation between the processor modules. That is, the internal snoop bus may be a bus load obtained by adding one processor to be accessed from the outside to the number of processors in the module. In addition, the external system bus requires a bus load corresponding to the number of processor modules. Therefore, the load on each of the internal and external common buses can be suppressed even when the number of processors increases, and high-speed cache coherence can be realized to improve processing performance.

  Also, since the internal snoop bus does not require connection to the outside of the processor module, the line length can be shortened to maintain good electrical characteristics, thereby increasing the clock frequency of the bus and providing it internally. Cache coherence between processor caches can be realized with high-speed processing.

  In cache coherence between processor modules using an external system bus, the latest value is transferred to the access source cache at the time of write access, but to the main memory having the write address. No update (copy back) is performed, and only the cache state of the directory storage unit is updated to the dirty state (the state where the latest value exists on the cache and the old value exists in the main memory), and the process ends. For this reason, cache coherence at the time of write access can be completed at a high speed because the owner (copy back) to the main memory is not performed.

The main memory that is in a dirty state by this write access is in a dirty state at the time of transfer from the processor module of the cache having the latest value to the processor module in which the access source cache exists in the read access of the same address. Transfer owner (copy back) to a certain main memory is performed, and read access and main memory update by copy back are performed in parallel, so that cache coherence of higher speed read access can be realized.

  As described above, according to the cache coherence device of the present invention, a plurality of processor elements with cache units are connected to each other by an internal common bus in each of the plurality of processor modules, and a memory module having a main memory is provided. In addition, a two-layer common bus configuration in which the memory module side is mutually connected to another processor module by an external common bus is adopted, so that the cache operation of the cache coherence in the processor module and the cache between the processor modules are performed. The coherence bus operation can be separated, and the processing performance can be greatly improved.

  Further, the internal common bus may be a bus load corresponding to an access source including a memory module that accepts external access to the number of processors in the module, and the processing load between the processors can be improved by reducing the bus load. Also, the bus load of the external system bus is determined by the number of processor modules connected to each other and does not depend on the number of processor elements provided in the processor module. Can improve processing performance.

  In addition, since the internal common bus does not require connection to the outside of the processor module, the line length can be shortened to maintain good electrical characteristics, thereby increasing the bus clock frequency and increasing the internal frequency. The processing performance of cache coherence between connected processors can be further increased.

  Furthermore, in cache coherence performed between processor modules using an external common bus, in the case of write access, the latest data is transferred to the access source cache, but it has a write address. The main data is not owned by the latest data, but the latest data is stored on the cache side, the dirty state is set and the process is terminated, so that the latest data is not owned by the main memory. Access can be completed quickly.

Further, the main memory that is in a dirty state by the write access receives the reply data from the external common bus and writes it to the main memory at the time of read access of the same address, so that it can be simultaneously stored in the access source cache. Since the main memory can be updated to the latest data and the transfer to the main memory owner and the read access source can be performed simultaneously, cache coherence for the read access can be realized at higher speed.

<Contents>
1. System configuration 2. Cache coherence at CPU read access 3. Cache coherence at CPU write access 4. Cache unit access contention processing 5. Copyback write ownership System common bus
(1) Bus configuration
(2) Bus arbitration operation
(3) Bus transfer sequence
(4) Transfer between buses
(5) Bus command Second directory memory 8. Other

1. System Configuration FIG. 2 is a block diagram of a multiprocessor system to which the cache coherence device of the present invention is applied. This system includes, for example, five processor modules 10-1 to 10-5 as a processor group having at least one processor element, and the processor modules 10-1 to 10-5 are connected to two system buses (second common buses). ) 12-1 and 12-2 are connected to each other.

  The processor modules 10-1 to 10-5 include, for example, four processor elements 14-1 to 14-4 as shown in the representative of the processor module 10-1 in FIG. 3, and are connected to each other via the snoop bus 22. Connected to. Each of the processor elements 14-1 to 14-4 includes processors 16-1 to 16-4, cache units 18-1 to 18-4, and snoop units 20-1 to 20-4.

  The memory module 25 is connected to the snoop bus 22 and simultaneously connected to the system buses 12-1 and 12-2 from other ports. The memory module 25 includes a protocol management unit 24, a memory management unit 26, and a module connection unit 32. The protocol management unit 24 performs protocol conversion between the system buses 12-1 and 12-2 and the internal snoop bus 22. The bus connection unit 32 performs bus access between the memory module 25 and the system buses 12-1 and 12-2.

  The memory management unit 26 is connected to a local storage 28 that functions as a main memory and a directory memory 30 that is used to realize cache coherence between processor modules.

  The snoop units 22-1 to 22-4 are synchronized with the local storage 28 as the main memory and the cache according to the snoop protocol that synchronizes the cache units 18-1 to 18-4 with the snoop bus 22 in synchronization with a specific machine cycle. The cache coherence between the units 18-1 to 18-4 is realized. That is, the snoop units 22-1 to 22-4 function as an internal coherence processing unit.

  The function as an external coherence processing unit for realizing cache coherence with other processor modules is performed by a cache conversion unit 80 provided in the memory management unit 26. The cache conversion unit 80 realizes cache coherence between the processor modules based on the reference of the directory memory 30 through exchanges with other processor modules using the system buses 12-1 and 12-2.

  As represented by the processor element 14-1 in FIG. 4, the processor elements 14-1 to 14-4 are provided with a CPU 34 having a primary cache 36 in the CPU circuit 16-1, and are external to the CPU 34. A secondary cache 38 is connected. A secondary cache control module 35 is provided as a control unit for the secondary cache 38, and a tag memory 40 is connected to the secondary cache control module 35. The primary cache 36 of the CPU 34 is controlled by write through, and the secondary cache 38 is controlled by copy back.

  The secondary cache control module 35 registers the data address of the secondary cache 38 and information indicating the cache state of the entry in the tag memory 40. Here, as shown in FIG. 5, the secondary cache 38 registers the cache data as data of 256-byte other mark. Therefore, the cache control module 35 manages cache registration in units of 256-byte addresses. The 256-byte data on the secondary cache 38 is referred to as a cache line 100.

  Therefore, the tag memory 40 holds the address of the cache line 100 in units of 256 bytes. Further, in the present invention, the 256-byte cache line 100 is divided into 64-byte units of sublines 102-1 to 102-4, and the cache state is held for each of the sublines 102-1 to 102-4.

  The secondary cache 38 is always controlled to include the primary cache 36 of the CPU 34. In order to guarantee the coherence of data between the cache units 18-1 to 18-4 in the four processor elements 14-1 to 14-4 shown in FIG. 3, for example, connected by the snoop bus 22, the snoop bus 22 The secondary cache 38 is snooped.

  The snoop bus 22 is a bus based on a packet protocol, and has a snoop transmission / reception unit 42, a snoop buffer 44, a copy back buffer 46, and a processor access unit 48 as a mechanism for snooping the secondary cache 38. Various snoop commands are prepared for the snoop bus 22. Since the snoop bus is a packet protocol, multiple accesses can proceed simultaneously.

  The snoop of the secondary cache 38 is simultaneously performed by the secondary cache control module 35 of each processor element 14-1 to 14-4 connected to the snoop bus 22, and the result is simultaneously assigned to the processor element on the snoop bus 22. This is reflected in the cache status signal. This cache status signal is referred to by all the processor elements 14-1 to 14-4 and the memory module 25 connected to the snoop bus.

  As units for the snoop bus 22 on the memory module 25 side, a space identification unit 50, a snoop transmission / reception unit 52, and a snoop bus transmission unit 60 are provided. Similarly to the case where the secondary cache control module 35 of the processor element 14-1 displays the cache status by the cache status signal, the memory management unit 26 provided in the memory module 25 also indicates the status of the local storage 28 in which the main memory is divided and arranged. It has a memory status signal for display. This memory status signal is also supplied to all the secondary cache control modules 35 in the snoop bus 22.

  From the cache status signal related to the secondary cache 38 on the snoop bus 22 and the contents of the memory status signal of the local storage 28 connected to the memory module 25, the secondary cache provided in the processor elements 14-1 to 14-4 is provided. Each of the control modules 35 determines the processor element that responds to the access of the CPU 34, determines the so-called owner having the latest value of the cache data to be accessed in the possession state of the write right, and various snoop commands Determine success or failure.

  Further, the cache state of the secondary cache 38 held in the tag memory 40 is changed as necessary. Further, the primary cache 36 may be invalidated and a data response from the secondary cache 38 may be involved. A series of operations by such a snoop bus 22 is controlled by a pipeline.

  The display by the cache status signal of the secondary cache 38 includes busy, miss, clean, dirty, and error. The busy display is displayed when the received command cannot be processed due to the exhaustion of processing resources or when the protocol causes a contradiction. Miss, clean and dirty indications are snoop results of the tag memory 40 of the secondary cache 38.

  That is, the miss display is displayed when a miss hit is detected in which data at the address is not held in the secondary cache 38 as a result of snooping the secondary cache 38 with the address of the snoop command. In the clean display, as a result of snooping the secondary cache 38, the same data as the contents of the local storage 28 is held in the secondary cache 38, and the data in which the contents of the local storage 28 are changed are 2 This is a case where it is held in the next cache 38.

  Further, the dirty display is a case where the latest data in which the content of the local storage 28 is changed is retained in the secondary cache 38 as a result of snooping the secondary cache 38, and is changed by the owner of the local storage 28. This state is called the owner who has the write right because it is responsible for the reflection. Thus, cache coherence is maintained between the processor elements 14-1 to 14-4 by the snoop method using the snoop bus 22.

  Next, the memory module 25 side will be described. A memory management unit 26 is provided in the memory module 25, and a local storage 28 is connected to the memory management unit 26 via a memory bus 66, and a directory memory 30 is connected to the memory module 25 via a dedicated line. The local storage 28 is connected to the memory access units 68-1 to 68-4 and the SRAMs 70-1 to 70-4 as main memory elements for every four memory buses 66.

  The memory bus 66 is a 32-bit width bus, and by accessing the two buses simultaneously, one word of data corresponding to the 64-bit width of the sub-line in the cache line is accessed. The local storage 28 is divided into four groups by four memory buses 66. For example, two groups of memory access units 68-1 and 68-2 access one word, and the memory access units 68-3 and 68- 4 enables access by an interleave method in which one word is accessed.

  The memory module 25 can map the memory space of the local storage 28 connected by the memory unit 26 to a physical space that can be referred to by all the PMs 10-1 to 10-5 by the space identification unit 50. FIG. 6 shows the mapping function of the space identification unit 50 of FIG. First, the CPU physical address space 86 of the CPU 34 provided in the processor element 14-1 has, for example, an address space of 64 GB (gigabytes).

  Of these, the first 0 to 4 GB area is allocated to the control register space 88 used by the CPU. The last 60 to 64 GB area is allocated to the ROM space 92 of the CPU. The remaining 4 to 60 GB 56 GB area is a shared space 90 accessible by all the CPUs 34 provided in the PMs 10-1 to 10-5. When a CPU access (read access or write access) designating an arbitrary 36-bit physical address 94 occurs in the CPU, the 36-bit physical address 94 is sent to the space of the memory control module 25 via the snoop bus 22 by a snoop command. It is sent to the identification unit 50.

  Of course, what is sent to the space identification unit 50 via the snoop bus 22 is the cache of the physical address 94 to be accessed on the cache of the processor elements 14-1 to 14-4 connected by the snoop bus 22. This is the case when the line does not exist.

  The space identification unit 50 holds a control table 95. The control table 95 is provided with entries from 4 GB to 60 GB in the shared space 90, and an ID area is provided corresponding to the entry by this address. In the corresponding ID area, a unit ID is registered as information indicating the processor module to which the address of the shared space 90 is assigned. This unit ID physically indicates a slot number in which a processor module can be connected to the system buses 12-1 and 12-2 in FIG.

  In this example, unit IDs # 00 to # 07 are registered corresponding to addresses from 4 GB to 24 GB. In the corresponding ID area, information indicating mounting and non-mounting of the processor module is registered in addition to the unit ID. In this example, it can be seen that the processor module of unit ID = # 02 having a physical address of 12 to 16 GB is not mounted. In addition, an address area having a 4 GB width is assigned to four unit ID = # 00 to # 03, but an address having a 1 GB width is assigned to a processor module to four unit ID = # 04 to # 03. Yes.

  Further, own home information is also stored in the corresponding ID area of the control table 95. The own home information means that the 36-bit physical address 94 issued by the CPU access exists in the address space of the local storage built in the processor module itself.

  By accessing the space identification unit 50 from the snoop bus 22 using the 36-bit physical address 94, home information 96, a unit ID 98, and a 36-bit physical address 94 are generated as generation information. The home information 96 indicates whether or not the accessed physical address 94 is its own home, that is, whether or not the physical address 94 exists in the processor module itself that caused the access.

  The unit ID 98 indicates a processor module in which the physical address 94 exists. A specific example will be described as follows. As a 36-bit physical address 94 by CPU access, X'4. It is assumed that D00000.000 ′ has occurred. In this case, the unit ID = # 03 is generated from the corresponding ID area in the entry of the control table 95 of the space identification unit 50, and as indicated by the broken line on the right side, the home information 96 indicates that the PM itself is not home. not Home ", unit ID = # 03, physical address 94 X'4. D0000.0000 ′ is generated.

  Referring again to the memory module 25 of FIG. 4, the memory module 25 is accessed from the CPU 34 of the processor element 14-1 through the snoop bus 22, and other processor modules through the system bus 12-1 (including 12-2). It is also accessed from the CPU 34 provided in 10-2 to 10-5.

  While cache coherence between the caches of the processor elements 14-1 to 14-4 provided in the processor module 10-1 is realized by using the snoop bus 22, there is no cache between the processor modules 10-1 to 10-5. In other words, cache coherence is maintained in a directory system using the system buses 12-1 and 12-2. In order to realize the cache coherence by this directory system, the directory memory 32 stores one entry for each 256-byte cache line.

  This directory entry is selected by a 36-bit physical address generated by CPU access, and displays the state of the cache line. The state display of the cache line in this directory entry includes a dirty state, a share state, and an invalid state.

  Dirty means that the cache line on a certain secondary cache 38 is rewritten to become the latest data, and there is only old data on the local storage 28 from which the cache line is copied, and the rewritten processor element is the latest. Means you own the data.

  The share indicates that the data on the cache line is referred to by the processor module, and indicates the possibility that the same data as the data on the local storage 28 exists in the secondary cache 38 at the same time.

  Further, “invalid” indicates a state in which the cache line is not referred to by any processor module and is not rewritten.

  Further, when a certain cache line is dirty, identification information of the processor module having the rewritten latest data is held in the memory management unit 26. Even when the cache line is shared, the identification information of the processor module having a copy of the data in the local storage 28 is held in the memory management unit 26.

  FIG. 7 shows details of the memory management unit 26 provided in the memory module 25 of FIG. 7, the memory management unit 26 includes a bus arbiter 74, a memory bus control unit 76 connected to the local storage 28 via the memory bus 60, and a directory control unit 78 connected to the directory memory 30 via a dedicated line.

  In addition, the memory management unit 26 is provided with a cache conversion unit 80 for maintaining cache coherence between the processor modules using a directory system, and includes a local state machine 81, a home state machine 82, and a remote state machine 83. .

  In the control of cache coherence between processor modules using the directory system of the present invention, the processor module that has started access is called local, and the processor module that has the accessed address in local storage is called home. In addition, the address of the accessed local storage is in a dirty state, that is, the processor module to which the processor element that has the oldest data in the cache of other processor modules and the processor element that owns the latest data in the cache belongs is remote. Call it.

  Such local, home, and remote states of the processor module are realized by processing operations of the local state machine 80, the home state machine 82, and the remote state machine 84 provided in the cache conversion unit 80.

  Corresponding to these three state machines 81, 82, and 83, the memory module 25 of FIG. 4 has a buffer queue for holding commands and data to be processed by each state machine via the bus arbiter 74. Is provided. That is, a local access buffer queue 54 is provided corresponding to the local state machine 80.

  The local access buffer queue 54 stores home information Home, unit ID, and 36-bit physical address, which are generation information of FIG. 6 generated by the space identification unit 50. Further, a response interrupt INT is given to the local access buffer queue 54 from the snoop transmission / reception unit 52. When the local access buffer queue 54 is full, information Full indicating the full state is returned to the snoop transmission / reception unit 52.

  In correspondence with the home state machine 82, a home access buffer queue 56 is provided in the memory module 25. The home access buffer queue 56 stores an access command issued from the local processor module using the system buses 12-1 and 12-2, a so-called home command. Corresponding to the remote state machine 84, a remote access buffer queue 58 is provided. The remote access buffer queue 58 stores an access command issued from the processor module serving as a home using the system buses 12-1 and 12-2, a so-called remote command.

  With respect to the local commands stored in the local access buffer queue 54 and the home commands stored in the home access buffer queue 56, the local state machine 80 and the home state machine 82 provided in the memory management unit 26 each access the local storage 28. However, since the remote command of the remote access buffer queue 58 is an access intended for the cache line on the secondary cache 38 of the processor element 14-1, it is directly connected to the snoop bus 22 by the snoop bus transmission unit 60. Yes.

  Next, the contents of the directory control unit 78 provided in the memory management unit 26 of FIG. 7 and the contents of the directory memory 30 connected by a dedicated line will be described.

  FIG. 8 shows the contents of the directory entry register 104 provided in the directory control unit 78 provided in the memory management unit 26, and the contents of the directory entry register 104 are read out for each cache line stored in the directory memory 30. It is a result. Therefore, the contents of the directory entry register 104 explain the directory entry data itself stored in the directory memory 30.

  The directory entry register 104 assigns the shared map bit area 106 to the lower 8 bits. The shared map bit area 106 is a bit indicating which processor module exists in the secondary cache by referring to the data corresponding to the cache line of the local storage 28 which is the main memory in the processor module.

  The shared map bit area 106 is combined with the shared map bit corresponding register 112 shown in FIG. 9, and the shared map bit corresponding register 112 is combined with the processor module configuration register 114 shown in FIG.

  The shared map bit area 106 of the directory entry register 104 is bits S0 to S7. By setting these bits S0 to S7 to 1, the cache line data is shared on the processor module indicated by the shared map bit corresponding register 112. Indicates that it exists in a state.

  The shared map bit correspondence register 112 has a map bit correspondence area corresponding to unit ID = # 00 to # 15 of 16 processor modules, for example. On the other hand, the shared map bit area 106 has only half of 8 bits of S0 to S7. Therefore, by storing the shared map bit in the shared map bit area 106 in the shared map bit corresponding register 112 in duplicate, the shared map bit can be expressed for a plurality of processor modules, and the bit of the directory entry register 104 is reduced. Is realized.

  Further, the shared map bits S0 to S7 stored in the shared map bit area 106 are held for each 64-byte cache subline obtained by dividing the 256-byte cache line into four on the secondary cache. On the other hand, since the shared map bit has only one bit, the shared state of the four cache sublines is represented by OR.

  In the expression of the shared map bit by OR of these four sublines, since the cache coherence of the secondary cache is realized by the snoop method, it can be accessed without being aware of the processor element in which the cache line in the shared state exists. Sub-line OR expression is possible.

  The processor module configuration register 114 of FIG. 1010 provided corresponding to the shared map bit correspondence register 112 of FIG. 9 is provided with a processor module mounting bit area 116 and a bus ID bit area 118. The bus ID bit area 118 stores bus IDs divided into system buses 12-1 and 12-2. Each system bus 12-1, 12-2 has 16 slots. Therefore, a maximum of 16 processor modules can be mounted.

  The processor module mounting bit area 116 has a bit area corresponding to the same slot number # 00 to # 15 as the unit ID, and bit 1 is set in the mounted state and reset to bit 0 in the unmounted state. .

  FIG. 11 shows a specific correspondence relationship between the directory entry register 104, the shared map bit correspondence register 112, and the processor module configuration register 114. First, the processor module configuration register 114 indicates only the processor module mounting bit area 116, and indicates a processor module in which the unit ID set in bit 1 is mounted.

  In this example, processor modules are mounted on unit IDs = # 00, # 02, # 05, # 06, # 08, # 10, # 11, and # 13, and the others are not mounted. Each bit of the processor module configuration register 114 corresponds to the shared map bit correspondence register 112 on a one-to-one basis.

  When a CPU access of a certain processor module refers to a cache line of the local storage that is the main storage of a certain processor module, and a shared state occurs in which a copy exists in the cache, shared directory entry data of that cache line is shared The empty bit in the map bit area 106 is set to 1, and at the same time, the index information indicating the register area of the processor module in which the copy data of the shared map bit corresponding register 112 exists in the shared state and the bit 1 of the shared map bit area 106 is stored. The

  For example, for shared map bit S6, S6 is stored in the areas corresponding to unit ID = # 06 and # 10 of shared map bit corresponding register 112, respectively. As a result, the shared map bit S6 is referred to by the two processor modules of unit ID = # 06 and # 10, and indicates that each is in a shared state on the cache.

  Referring to the directory entry register 104 in FIG. 8 again, a dirty subline bit area 108 is provided after the shared map bit area 106. In the dirty subline bit area 108, bits D0 to D3 are allocated corresponding to four sublines obtained by dividing a 256-byte cache line.

  When these dirty subline bits D0 to D3 are set to bit 1, the latest data rewritten with respect to the old data in the local storage 28 as the main memory is stored in the cache of any processor module in the system. It shows that it exists. When the dirty subline bit area 108 is reset to bit 0, it means a non-dirty state.

  In this case, it means that the latest data exists in the local storage 28 as the main memory, and the copy data exists in a shared state on any of the processor modules specified by the shared map bits S0 to S7. The identification information of the processor module regarding the data subline bits D0 to D3 is stored in a specific area on the local storage 28 as the main memory, for example, the control register space 88 in the CPU physical address space 86 of FIG.

  FIG. 12 shows specific contents of the directory entry register 104. First, the shared map bit area is divided into shared map bit areas 106-1 and 106-2 corresponding to the system buses 12-1 and 12-2. Here, when the processor module is mounted only on the system bus 12-1 side and the processor module is not mounted on the system bus 12-2 side, the shared map bit area 106-1 side is mounted processor module. Used for.

  In shared map bit area 106-1, bit 1 is set in shared map bits S3 and S4. In this case, the shared map bit correspondence register 112 shown in FIG. 9 corresponds to, for example, the shared map bit S3 means OR of the share information of all the sublines of the cache line data of the processor module of unit ID = # 03. Similarly, the shared map bit S4 is designated by the shared map bit corresponding register 112 in FIG. 9, and for example, the OR of the shared state of all sublines on the cache of a certain processor element on the processor module of unit ID = # 04 Become.

  On the other hand, for the dirty subline bit area 108, for example, dirty subline bits D1 and D4 are set to bit 1 and D2 and D3 are reset to bit 0. Corresponding to the dirty subline bits D1 to D4, the local storage 28 is provided with processor module information storage areas 120, 122, 124, and 126.

  For example, the processor module information area 120 of the dirty subline bit D1 indicates that the latest data exists in the dirty state on the cache existing in the processor module with the unit ID = # 01. Similarly, the dirty subline bit D4 set to bit 1 indicates that the latest data exists in the processor module information area 126 as a dirty state in the cache existing in the processor module of unit ID = # 02.

  As for the processor module information areas 122 and 124 of the dirty subline bits D2 and D3 reset to bit 0, the latest data is stored in the local storage as the main memory where the physical address of the cache line exists, and any of the processor modules Indicates that the copy data exists in the shared state by referring to the latest data in the main memory.

  The unit ID of the processor module in which the copy data exists in the shared state is obtained by referring to the shared map bit corresponding register 112 in FIG. 9 with respect to the shared map bits S3 and S4 set to bit 1 of the shared map bit area 106-1. For example, information indicating the possibility that copy data exists in a shared state is stored on the processor modules of unit ID = # 03 and # 04.

  Further, the contents of the processor module information areas 122 and 124 of the dirty subline bits D2 and D3 set to bit 0 may be stored as non-shared in addition to shared. In this case, the local memory as the corresponding main memory may be stored. Indicates that the latest data exists only in the storage.

  Next, the packet of the snoop bus 22 will be described. Packets are classified into start commands and response commands. The start command and the response command are basically a pair. A packet consists of a command phase and a data phase. The command phase is 1 clock. The data phase can take a length from 0 to 8 clocks depending on the command and size.

  The command phase of the start command includes a destination field indicating a packet destination, a flag field including an auxiliary instruction for packet processing, a size field indicating a data size to be processed, a type field indicating a command type, and a command processing target It consists of an address field indicating an address. The distinction between the start command and the response command is identified by the type field.

  On the other hand, the response command also includes a command phase and a data phase. The command phase is 1 clock, and the data phase can take 0 to 8 clocks depending on the command and size. The command phase of the response command includes a destination field indicating the packet destination, a success / failure of access, a reply code field indicating an error factor and a retry instruction, a reply field indicating a reply type, and a command field indicating a command type (reply Is stored in the address field indicating the address of the command processing target.

  Also for the system buses 12-1 and 12-2 connecting the plurality of processor modules, the packet is basically the same as the snoop bus 22, but the bus clock is different. That is, the snoop bus is short because the bus line length is sufficient in the module housing, and has good electrical characteristics. For example, the clock frequency can be 60 MHz.

  On the other hand, since the system buses 12-1 and 12-2 are connected between a plurality of processor modules via the back panel, the line length is long, and the clock frequency is suppressed to, for example, 40 MHz due to restrictions on electrical characteristics. . The timing due to the difference in clock frequency between the snoop bus 22 and the system buses 12-1 and 12-2 is adjusted by the protocol conversion function by the protocol management unit 24 shown in the memory module 25 of FIG. Yes.

  FIG. 13 shows the types of commands used in the snoop bus 22 separately for the processor 16, the snoop unit 20, and the memory management unit 26.

  Next, the transition of the cache state on the cache control module 35 side provided in the processor element 14-1 of FIG. 4 will be described. First, cache states held in the tag memory 40 of the secondary cache control module 35 in units of cache sublines include the invalid state, shared clean state, shared modify state, exclusive dirty state, and shared dirty state shown in FIG. There is one.

  The invalid state INV is a non-cache state in which no data exists on the secondary cache 38. The shared clean state SH & C is a state in which data having the same content as the local storage that is the main memory of the processor module serving as the home is held. In the shared modified state SH & M, the data on the main memory that is the home is old data that is not the latest data, the copy data exists in its own secondary cache, and other data in its own processor module This is a state in which only the latest data exists in the system due to the exclusive dirty state EX & D on the secondary cache.

  The exclusive dirty state EX & D is a state in which only the latest data is held in the system. In this case, the exclusive dirty state EX & D is a so-called owner who owns the write right. Further, in the shared dirty state SH & D, there is no latest data on the main memory as a home, the latest data is held in the secondary cache group in the own processor module, and the secondary cache in the own processor module is further stored. This is a case where the group has a shared modified state SH & M having a copy of home main storage data, and the cache control module itself holding the latest data owns the write right and becomes the owner.

  FIG. 15 shows the transition of the cache state in the secondary cache control module 35 of FIG. First, the initial state is the invalid state INV of the block 130. In this state, when the primary cache 36 becomes a miss hit due to the read access of the processor, an access request is issued to the secondary cache state module 35. At this time, if the corresponding cache subline of the secondary cache 38 is in the invalid state INV, a command requesting data is sent to the snoop bus 22.

  In this case, there are two transition states: Case 1 in which the block 150 transitions to the shared modify state SH & M and Case 2 in which the block 140 transitions to the shared clean state SH & C. In case 1, the shared modified state (SH & M) of the block 150 is a case where another processor element is in the exclusive dirty state EX & D or the shared dirty state SH & D and owns the accessed subline on the cache. Dirty is indicated by the cache status signal of any processor element.

  Therefore, the cache control module 35 of the processor element displaying the dirty accesses the secondary cache 38 and issues a data response to the snoop bus 22. At this time, in the processor element which has made a response, when the cache state is the exclusive dirty state EX & D of the block 160, the state transits to the shared dirty state SH & D of the block 170. The processor element that has received the response transitions the cache state to the shared modify state SH & M in block 150.

  On the other hand, in the case 2, the other processor element is in the shared clean state SH & C in block 140 or the invalid state INV in block 130 in response to an access request for a miss hit in the primary cache of the read access of the processor. Yes, the cache status signal and the memory status signal on the memory module 25 side indicate clean or miss. The cache control module 35 that has performed the clean display refers to the cache status signal and determines whether or not a response should be created.

  Similarly, the memory module 25 that refers to the memory status signal, that is, the memory management unit 26 provided in the memory module 25 also refers to the memory status signal and determines whether or not a response should be created. If the module that has detected that it should respond is the secondary cache control module 35, it accesses the secondary cache memory 38 and issues a data response to the snoop bus 22. If the module that has detected that it should respond is the memory management unit 26 of the memory module 25, it activates access to the local storage 28 as the main memory and issues response data to the snoop bus 22.

The access source secondary cache control module 35 that has received the response data from the snoop bus 22 in this way makes a transition from the invalid state INV of the block 130 to the shared clean state SH & C of the block 140.

2. Cache Coherence at CPU Read Access Next, cache coherence processing when read access occurs in any of the processor modules 10-1 to 10-5 shown in FIG. I will explain.

(1) Read mode 1
FIG. 16 shows a cache coherence protocol in read mode 1.

  In the read mode 1, any other CPU read access P-RD in the processor module 10-1 misses in its own cache unit in the cache unit group 18 and is connected by the snoop bus 22. This is a case where a miss hit occurs in the cache unit 18.

  In this case, the command T-CR is issued as a remote command for accessing the local storage 28 which is the main memory to the memory management unit 26. Here, since the read address is in the local storage 28 of the processor module 10-1, the processor module 10-1 becomes home, and any of the cache unit groups in the processor module 10-2 in which the data of the local storage 28 is remote. Suppose that it existed in a dirty state with the latest data.

  In such a case, the memory management unit 26 of the processor module 10-1 that has generated the read access issues a remote command 220 to the system bus 12 to access the processor module 10-2. The memory management unit 26 of the processor module 10-2 that has received this remote command 220 acquires the latest data from the cache unit in the dirty state D via the snoop bus 22, and then uses the system bus 12 to acquire the latest data. The reply data 230 is returned to the local and home processor module 10-1.

  The processor module 10-1 stores the reply data 230 obtained as a response of the system bus 12 in the access source cache unit via the snoop bus 22, and makes it respond as read data for the read access of the CPU. At the same time, cache coherence is realized by writing the latest data to the local storage 28 as the main memory. Here, in the cache unit of the processor module 10-2 that has received the remote command 220 and responded to the latest data, the dirty state D is updated to the clean state C. Further, for the directory memory that manages the cache data of the local storage 28 of the access source processor module 10-1, the dirty state D of the cache line that is the access target is updated to the share state S. Further, the invalid state I is updated to the clean state C for the access source cache unit.

(2) Read mode 2
FIG. 17 shows a protocol for cache coherence in read mode 2. In this read mode 2, a read access P-RD of any CPU is missed in its own cache unit and missed in all other cache units connected by the snoop bus 22, and the read address is other than It exists in the local storage 28 which is the main memory of the processor module 10-2, and the data at the read address of the local storage 28 exists in a dirty state D having the latest data on any cache unit of another processor module 10-3. This is the case.

  In this case, the memory management unit 26 of the processor module 10-2 in which the read access has occurred operates as a local and issues a home command 240 to the system bus 12 to access the processor module 10-2. Based on the home command 240 received from the system bus 12, the processor module 10-2 recognizes that the latest data exists in the dirty state D in the cache of the processor module 10-3 by referring to the directory memory by the read address, A remote command 250 is issued to the system bus 12 to access the processor module 10-3.

  In the processor module 10-3, based on the remote command 250 received from the system 12-2, the memory management unit 26 operates as a remote and accesses the cache unit in the dirty state D by the snoop bus 22 and acquires the latest data. Thereafter, the system bus 12 is used to reply the reply data 260 to each of the processor module 10-2 as the home and the processor module 10-1 as the local. The processor module 10-2 as the home performs the owner of the corresponding data in the local storage 28 as the main memory by using the reply data 260 obtained by the response of the system bus 12, and updates it to the latest data.

  At the same time, in the processor module 10-1 as a local, the reply data 260 received as a response from the system bus 12 is transferred to the access source cache unit via the snoop bus 22 and stored in the secondary cache. Then, it responds as read data to the CPU read access.

  Here, the cache unit that has made the response of the latest data in the remote processor module 10-3 updates the cache state from the dirty state D to the clean state C. Further, in the processor module 10-2 that has become the home, the state of the cache line that is the access target is updated from the dirty state D to the share state S.

  Specifically, the corresponding subline bit in the dirty subline bit area 108 of the directory entry register 104 in FIG. 8 is reset from 1 to 0. In addition, since the processor module 10-3 as a remote and the processor module 10-1 as a local are in a shared state where the same data as the data of the local storage 28 in the home exists in the shared state, the correspondence of the shared map bit area 106 Set the shared map bit to 1

  Further, in the cache unit in which the access of the processor module 10-1 that has become local occurs, the invalid state I is transitioned to the clean state C.

(3) Read mode 3
FIG. 18 shows a cache coherence protocol in the read mode 3. In this mode, cache coherence using the snoop bus 22 is performed between the cache units inside the module in which the access has occurred. There are two cases of the reading mode, FIG. 18A and FIG.

  FIG. 18A shows that the cache unit 18-4 in which the CPU read access P-RD has occurred is in the invalid state INV and has become a miss hit, but the other cache unit 18-1 connected by the snoop bus 22 has This is a case where the latest data exists in the shared clean state SH & C. In this case, the cache unit 18-4 uses the snoop bus 22 to issue a command T-CR, the reply of the reply data 270 including the corresponding subline is performed from the cache unit 18-1, and the cache unit 18- 4 and responds to read access as read data.

  At this time, the access-source cache unit 18-4 transitions from the invalidation state INV to the shared clean state SH & C, and the access-destination cache unit 18-1 maintains the shared clean state SH & C.

  FIG. 18B shows that the latest data is in an exclusive dirty state EX− in the cache unit 18-1 connected by the snoop bus 22 in response to the read access P-RD from the CPU to the cache unit 18-4 in the invalidated state INV. This is a case where it exists in D, that is, a case where it exists in a state where the write right is owned. In response to the access T-CR from the cache unit 18-4 in this case, the reply data 280 including the subline of the latest data is similarly sent from the cache unit 18-1.

  In this case, the access-source cache unit 18-4 indicates that the latest data exists in the shared dirty state SH & M from the invalidation state INV, that is, in the shared dirty state in the other cache units in its own processor module. In the cache unit 18-1 to be accessed, the exclusive dirty state EX & D transitions to the shared dirty state SH & D.

(4) Local Processor Module Read Processing The flowchart of FIG. 19 is the read processing of the processor module that is local in the read modes 1 and 2 of FIGS. First, at step S1, the processor element receives a read access, and at step S2, it is determined whether or not there is a cache hit. If it is a cache hit, at step S14, data is returned to the processor element and the process is terminated.

  In the case of a cache miss hit, in step S3, the cache status is acquired from the tag memory 40, and in step S4, it is checked whether or not there is the latest data in the cache of another processor element. If there is the latest data on the cache of another processor element, the process proceeds to step S15, a command is sent to the snoop bus 22, the reply data is received at step S16, and it is stored in the cache at step S13 and responds to the processor element. To do. Steps S15 to S17 are processing in the reading mode 3 in FIG.

  On the other hand, if there is no latest data in the cache of another processor element in step S4, a local command is sent to the memory management unit 26 via the snoop bus 22 in step S5. On the memory management unit 26 side, first, in step S6, the space identification unit 50 shown in FIG. 4 determines the processor module space from the physical address of the command.

  If the read address is the main storage space of the local storage 28 of the processor module in step S7, the cache status is acquired from the directory entry in the directory memory 30 in step S8 with the read address as the home. The cache status is checked in step S9 to see if the subline is dirty. If it is dirty, a remote command is sent to the remote processor module via the system bus in step S10.

  If there is a response to the remote command in step S11, the reply data is written in the local storage 28 as the main memory in step S12, and the dirty state is updated to the shared state. In step S13, the response is made to the access source cache unit, and the process ends. If the subline is not dirty in step S9, in step S22, data is read from the local storage 28 as the main memory and responded to the access source cache unit, and in step S23, the local storage 28 as the main memory is read. Turn on the shared bit of the subline.

  If the accessed physical address is another processor module space in step S7, a home command is sent to the home processor module via the system bus in step S18. In response to the sending of the home command, there is a reply of the latest data via the system bus. When the reply is determined in step S19, the process ends in response to the cache for each access in step S20.

(5) Read Processing of Home Processor Module The flowchart of FIG. 20 shows the read processing of the processor module that has become home in the read modes 1 and 2 of FIGS. First, in step S1, a home command is received. The home command may be received via the system bus, or may be received by the home command generated by itself when it is local and home.

  When the home command is received, the status of the cache line is acquired from the directory memory 30 using itself as the home in step S2, and it is checked in step S3 whether the subline is dirty. If the subline is dirty, it is checked in step S4 whether or not there is the latest data on the cache of the processor element in the own processor module.

  If the latest data is on its own cache, a command is sent to the snoop bus 22 at step S5, a reply is waited at step S6, the reply data is written to the local storage 28 as the main memory at step S7, and a directory entry is entered. Update the dirty state to shared state. In step S8, reply data is sent to the local processor module using the system bus.

  If the latest dirty data exists in the cache of the processor element in another processor module in step S4, a remote command is sent to the remote processor module via the system bus in step S12. In step S13, the reply is waited, and in step S14, the latest data is written in the local storage 28 as the main memory to update the dirty state of the directory entry to the shared state.

  On the other hand, if the subline is not dirty in step S3, the latest data is read from the local storage 28 as the main memory, the reply data is sent to the local processor module via the system bus, and the process is terminated.

(6) Remote Processor Module Read Process The flowchart of FIG. 21 is a remote processor module read process in the read modes 1 and 2 of FIGS. First, in step S1, when a remote command is received via the system bus, in step S2, the command is sent to the cache unit via the snoop bus by making itself remote. When a reply is received from the cache unit of the corresponding processor element, the reply data is received from the snoop bus 22 in step S4, and the reply data is sent to the home processor module and the local processor module via the system bus in step S5. To do.

(7) Processor Element Read Process The flowchart of FIG. 22 is a read process for the processor element itself. First, when a CPU read request is generated in step S1, the presence or absence of a cache hit is determined in step S2, and if it is a cache hit, the cache data is responded to the read access in step S12.

  If a cache miss hit occurs, the cache control module 35 acquires the status of the cache line from the tag memory 40 in step S3, and in step S4, it is determined whether or not there is a hit in the cache of another processor element. In this case, if it is a miss hit, the process proceeds to step S13, and processing for another processor module is performed by accessing the memory module 20 side.

  If a cache hit occurs in step S4, it is checked in step S5 whether the cache state of the other processor element is a shared clean state SH & C. In the shared clean state SH & C, a command is sent to the snoop bus 22 in step S6, reply data is received and responded to the cache in step S7, and the status is changed from the invalidation state INV to the shared clean state in step S8. Update to SH & C.

On the other hand, if the other processor is in the exclusive dirty state EX & D, after sending a command to the snoop bus 22 in step S9, the reply data is received and responded to the cache in step S10, and in step S11, The status is updated to the shared modified state SH & M.

3. Cache coherence for CPU write access (1) Write mode 1
FIG. 23 shows a cache coherence protocol in the write mode 1. In this write mode 1, a write access P-WT of an arbitrary CPU of the processor module 10-1 misses in its own cache unit, and also misses in other cache units connected by the snoop bus 22. The write address is in the local storage 28 which is the main memory of the processor module 10-1, and the copy data of the write address in the local storage 28 is cached in the other processor modules 10-2 and 10-3. It is a case where it exists in a share state above.

  In this case, the memory management unit 26 of the processor module 10-1 that has received the write access T-CRI via the snoop bus 22 makes itself local and home, and is in a shared state by referring to the cache directory. A purge command 300 that is an invalidation command is issued to the processor modules 10-2 and 10-3 using the system bus 12.

  In response to the purge command 300, the processor modules 10-2 and 10-3 operate as remotes, and the respective memory management units 26 request invalidation of the memory units in the shared state C via the snoop bus 22. If the cache unit in the shared state is successfully invalidated, the processor modules 10-2 and 10-3 perform purge responses 310 and 320 independently using the system bus 12. The purge responses 310 and 320 include the cache status of the target cache line in the processor modules 10-2 and 10-3.

  The processor module 10-1 as the home recognizes the success of invalidation by receiving all the purge responses 310 and 320 from the system bus 12, and accesses the copy data of the write address of the local storage 28 as the main memory. Stored in the original cache unit and overwritten by write access. At this time, in the processor module 10-1 as the home, the share state S of the directory entry of the cache line to be accessed is updated to the dirty state D.

  As described above, in the write access, the same data as the local storage 28 as the main storage is changed by overwriting on the access source cache to become the latest data, so that the data in the local storage 28 becomes old data. In this state, the process ends, and the coherence of the old data in the local storage 28 with the latest data is performed through the read access described above.

(2) Write mode 2
FIG. 24 shows a cache coherence protocol according to the write mode 2. In the write mode 2, the cache unit of the processor module 10-1 in which the CPU access has occurred, the local storage 28 as the main storage in the share state C in which the other cache units connected by the snoop bus 22 do not have the write right This is a case where the same data exists, and the other states are the same as in mode 1 of FIG.

  In the write mode 2, the processor modules 10-2 and 10-3 as remote units using the system bus 12 from the memory management unit 26 operating as a home based on the write access P-WT of the CPU. At the same time, the purge command 325 is issued to the cache unit group 18 in the same processor module 10 at the same time.

  For this reason, the memory management unit 26 of the processor module 10-1 operating as a home has all the responses of the purge responses 310 and 320 from the processor modules 10-2 and 10-3 as remote via the system bus 12, Receiving the purge response from its own snoop bus 22 recognizes the success of the invalidation, stores the copy data of the latest data in the local storage 28 as the main memory in the cache unit of the access source, and writes the CPU Overwrite by access. Also in this case, the share state S of the directory entry is updated to the dirty state D as in the write mode 1.

(3) Write mode 3
FIG. 25 shows a cache coherence protocol according to the write mode 3. In the write mode 3, the copy data is copied to any of the cache group 18 including the cache unit connected via the snoop bus 22 and the own cache unit with respect to the write access P-WT of the CPU of the processor module 10-1. Is not registered in the state where the owner is owned, becomes a miss hit, the write address is in the local storage 28 which is the main memory of the processor module 10-2, and the copy data of the latest data in the local storage 28 is the processor. This is a case where a plurality of processor modules 10-3 and 10-4 other than the module 10-2 exist in the share state C.

  In this case, the memory management unit 26 of the processor module 10 as a local uses the system bus 12 to issue a write ownership request command, a so-called home command 330, to the processor module 10-2 as a home. The processor module 10-2 that has received the home command 330 that is a write ownership request command uses the system bus 12 to invalidate the other processor modules 10-3 and 10-4 that are in the shared state. The purge command 340 is issued.

  When the processor modules 10-3 and 10-4 receive the purge command 340, the processor modules 10-3 and 10-4 instruct the cache unit in the cache group 18 in the shared state via the snoop bus 22 and succeed in the invalidation. The purge responses 350 and 360 are independently returned to the processor module 10-2 as the home using the system bus 12.

  When the processor module 10-2 as the home recognizes the success of the invalidation by accepting all the purge responses 350 and 360, the processor module 10-2 as the home uses the system bus 12 itself to recognize the success of the invalidation. Response of transfer of write ownership including the copy data of the latest data in the local storage 28 which is the main memory of the main storage.

  The local processor module 10-1 stores the copy data obtained by receiving the reply data 370, which is a response to write ownership from the system bus 12, in the cache unit of the access source, and overwrites it by the write access of the CPU. . Here, the memory management unit of the processor module 10-2 as the home that recognized the success of the invalidation and moved the write ownership, updates the share state S of the directory entry to the dirty state D. Further, in the access source cache unit of the processor module 10-1, the cache state is updated to the dirty state D having the write right at the end of the write access.

(4) Write mode 4
FIG. 26 shows a cache coherence protocol according to the write mode 4. In the write mode 4, the cache access to the CPU in the processor module 10-1 operating as the local is P-WT, and the cache unit of the processor module 10-1 is in a miss-hit and connected to another cache unit connected via the snoop bus 22. However, the copy data is not registered in the state of possessing the write right and becomes a miss hit, the write address is in the local storage 28 which is the main memory of the processor module 10-2, and the latest data in the local storage 28 is also stored. This is a case where copy data exists in the cache state of the plurality of processor modules 10-3 and 10-4 including the processor module 10-2 in the share state C.

  In this case, the memory management unit 26 of the processor module 10-1 as the local where the write access has occurred uses the system bus 12 as a home to issue a request command for write ownership to the processor module 10-2. A certain home command 370 is issued. The memory management unit 26 of the processor module 10-2 that has received the home command 370 serving as the write ownership request command uses the system bus 12 for all the processor modules 10-3 and 10-4 in the shared state. Then, a purge command 380 as an invalidation command is issued.

  In parallel with the issuance of the purge command 380, the memory management unit 26 uses the snoop bus 22 to issue a purge command 410 as an invalidation command to the cache units in the cache group 18 in its own share state. Issue.

  All the processor modules 10-3 and 10-4 in the shared state C on the cache operate as remote, accept the purge command 340 at the same time, and command invalidation to the cache group 18 side via the snoop bus 22. If the invalidation is successful, the system bus 12 is used to independently return purge responses 350 and 360 to the processor module 10-2. In parallel with this, from the cache unit in the share state C of the processor module 10-2, success of invalidation is responded using the snoop bus 22.

  The home processor module 10-2 recognizes the success of invalidation by receiving all the invalidation success purge responses 350 and 360 from the system bus 12 and the invalidation success response from the snoop bus 22. Then, using the system bus 12, the write ownership transfer including the copy data of the latest data in the local storage 28, which is its own main memory, is returned as reply data 430 to the local processor module 10-1.

  The processor module 10-1 stores the copy data obtained by receiving the write ownership response from the system bus 12 in the cache unit of the access source, and overwrites it by the write access of the CPU. Here, the processor module 10-2 updates the share state S of its own directory entry to the dirty state D when it recognizes the success of the invalidation and responds to the transfer of the write ownership.

(5) Write mode 5
FIG. 27 shows a cache coherence protocol according to the write mode 5. In this write mode 5, the copy data has the right to write to the cache group 18 of the other cache unit connected by the snoop bus 22 and the own cache unit with respect to the write access P-WT of the CPU of the processor module 10-1. It is not registered in the possessed state and becomes a miss hit, the write address is in the local storage 28 which is the main memory of the processor module 10-1, and the latest data of the write address in the local storage 28 is further stored in the processor module 10. -2 in the dirty state D in which the write right is owned by the cache unit.

  In this case, the local and home processor module 10-1 issues a remote command 440 serving as a write ownership request command to the processor module 10-2 in the dirty state D using the system bus 12. To do. The processor module 10-2 receives a remote command 440 that is a write ownership request command from the system bus 12, and receives reply data indicating the transfer of write ownership including the latest data in the share state D on the cache unit. 450 responds to the processor module 10-1 using the system bus 12.

  The processor module 10-1 stores the latest data obtained in response to the write ownership by the reply data 450 in the access source cache unit, and overwrites it with the write access by the CPU.

  Here, in the cache unit to which the write ownership of the processor module 10-2 has been transferred, the dirty state D is updated to the invalidated state I. Further, when the memory management unit 26 of the processor module 10-1 receives the write ownership, it updates the dirty state D of the directory entry to the dirty state D ′ in which the latest data exists in its own cache.

(6) Write mode 6
FIG. 28 shows a cache coherence protocol according to the write mode 6. In this write mode 6, copy data is written to the cache group 18 including its own cache unit and other cache units connected by the snoop bus 22 with respect to the write access P-WT by the CPU of the processor module 10-1. The right address is not registered and a miss hit occurs, the write address is in the local storage 28 which is the main memory of the processor module 10-2, and the latest data of the write address in the local storage 28 is the processor. This is a case where the cache unit of the module 10-3 exists in the dirty state D having the write right.

  In this case, the memory management unit 26 of the processor module 10-1 as a local uses the system bus 12 to issue a home command 460 serving as a write ownership request command. The memory management unit 26 of the processor module 10-2 serving as the home that has received the home command 460 uses the system bus 12 to issue a request for writing ownership to the processor module 10-3 serving as the remote. A remote command 470 is issued.

  The processor module 10-3 that has received the remote command 470 uses the system bus 12 as a reply data 480 to give home ownership of the write ownership including the latest data of the cache unit in the dirty state D to the processor module 10- 2 and respond to the local processor module 10-1.

  The home processor module 10-2 receives the reply data 480 as a response to the write ownership, and recognizes the transfer of the write ownership. The local processor module 10-1 stores the latest data obtained by receiving the write ownership by the reply data 480 in the access source cache unit, and overwrites it by the CPU write access.

  Here, in the cache unit of the remote processor module 10-3, the dirty state D is updated to the invalidation state I when the write ownership is transferred. Further, when the home processor module 10-2 confirms the transfer of the write ownership, the dirty state D of the directory entry is changed to the dirty state D ′ which is changed to the information of the processor module 10-1 in which the latest data exists. Update.

  Further, in the access source cache unit of the local processor module 10-1, after the latest data is stored, the cache unit is updated to the dirty state D having the write right.

(7) Write mode 7
FIG. 29 shows a cache coherence protocol according to the write mode 7. This write mode 7 is a case where access is made between a plurality of cache units in the processor module 10-1. That is, for the write access P-WT of the CPU in the processor module 10-1, the own cache unit 18-4 has a miss-hit, but the write is written to another cache unit 18-1 connected via the snoop bus 22. This is a case where the latest data of the embedded address exists in the dirty state in which the write right is owned, that is, in the exclusive dirty state EX & D.

  In this case, the access-source cache unit 18-4 issues a command T-CRI via the snoop bus 22, and in response to this, the cache unit 18-1 responds with reply data 490 including the latest cache subline data. After being stored in the cache unit 18-4, it is overwritten by write access.

  Here, in the cache unit 18-1 to be accessed, the write ownership is transferred and then updated to the invalidation state INV. Further, after acquiring the write right, the access source cache unit 18-4 is updated from the invalidation state INV to the exclusive dirty state EX & D which is a dirty state in which the write right is owned.

(8) Processor Write Processing for Write Access The flowchart in FIG. 30 is a write processing for the processor module that has become local due to cache coherence associated with write access. When a processor element write access occurs in step S1, a cache hit of the secondary cache 38 is determined in step S2, and if it is a cache hit, data is overwritten on the cache in step S14.

  If it is a cache miss hit, the cache status is acquired from the tag memory 40, and it is checked in step S4 whether there is the latest data in another cache unit acquired by the snoop bus 22. If there is the latest data in another cache, a command is sent to another processor element in step S15, a reply is waited in step S16, the latest reply data is overwritten in step S17, and the write access is terminated. To do.

  If there is no latest data in the cache of another processor element connected by the snoop bus 22 in step S14, a local command is sent to the memory management unit 26 via the snoop bus 22 in step S5, and step S6. Thus, the processor module space is determined from the space identification unit 50, and it is determined in step S7 whether or not it is its own processor module space.

  If it is the processor module space of its own, in step S8, the directory entry is acquired from the directory memory 30 to see the status of the cache line with itself as the home. In step S9, it is checked whether or not the subline of the directory entry is dirty. If it is dirty, a purge is requested to the remote processor module via the system bus 12 in step S10.

  If there is a response to this purge request in step S11, dirty is set in another subline of the directory entry, and the cache status is updated. In step S13, the reply data is overwritten on the access source cache of the processor element, and the invalidation state INV is updated to the dirty state EX & D with write ownership.

  On the other hand, if the subline of the directory entry is not dirty but a share in step S9, a purge is requested to the remote processor module in step S21, a reply is waited in step S22, and the directory entry is checked in step S23. The cache status is updated to update the shared state S to the dirty state D. Finally, in step S24, the cache data is overwritten to update from the invalidated state INV to the shared dirty state SH & D.

  Steps S8 to S13 and steps S21 to S24 are processes when the local processor module simultaneously becomes the home processor module.

  On the other hand, if the access space is not its own processor module space in step S7, a home command is sent to another processor module in step S18, and a reply of the latest data is waited in step S19. In step S20, the reply data is overwritten on the cache and updated from the invalidation state INV to the exclusive dirty state EX & D. This is accompanied by transfer of write ownership by the home command.

  The flowchart in FIG. 31 shows a write process when the home processor module is obtained by cache coherence by write access. When a home command is received in step S1, a directory entry as a cache status is acquired from the directory memory 30 with itself as a home in step S2.

  In step S3, it is checked whether it is dirty by referring to the subline of the directory entry. If it is dirty, it is checked in step S4 whether the latest data in the dirty state is on the cache of its own processor module. If it is on its own cache, a purge is requested via the snoop bus 22 in step S5, the new data is waited for a reply in step S6, and the cache is stored in the local storage 28 as the main memory in step S7. A new subline of a directory entry is set to a dirty state, and the latest data is sent as reply data to the local processor module in step S8.

  If the latest dirty data is present in the cache of another processor module in step S4, a purge is requested to the remote processor module in step S9, a reply is waited in step S10, and the process proceeds to step S11. Set the new subline's dirty in the directory entry.

  On the other hand, if the subline of the directory entry is not dirty but is in a shared state in step S3, it is checked in step S12 whether it is on the cache in its own processor module. If it is not in its own cache, it requests purge to the remote processor module in step S13, waits for a reply in step S14, and reads the reply data as the latest data from the local storage as the main memory in step S15. The cache directory is updated from the dirty state D to the share state S.

  In step S2, when there is a shared state on the cache in its own processor module, in step S16, a purge is requested via the snoop bus 22, and in step S17, a reply is waited, and the flow proceeds to step S18. Update the directory entry share status to dirty. In step S19, the latest data in the local storage 28 as the main memory is sent as reply data to the local processor module.

  The flowchart in FIG. 32 shows a write process of the processor module that is a remote controller in the write access. In the write processing of the remote processor module, when a remote command, that is, a write ownership request command is received from the system bus 12 in step S1, a command is sent to the snoop bus 22 in step S2, and step S3. Then, a reply from the snoop bus 22 is awaited, and in step S4, a command for responding to the transfer of write ownership and the latest data are sent as reply data to the local processor module and the home processor module.

  FIG. 33 is a flowchart of the write process of the processor element when a write access occurs. First, when a write request is generated in step S1, it is checked in step S2 whether or not the latest data having the write right exists in its own cache unit and other cache units connected by the snoop bus 22.

  If it exists in another cache, in step S9, it is transferred to the access source and the cache data is overwritten, and it is checked in step S2 whether or not a cache hit occurs in the cache unit at the time of reading. If it is a cache hit, the process proceeds to step S9, where the cache data is overwritten with the data by the write request, and its own cache state is transmitted to the exclusive dirty state EX & D ′ with the same write ownership.

  If the latest data does not exist while owning the write right in the own cache unit, the process proceeds to step S3 and the cache control module 35 refers to the tag memory 40 to acquire the cache line status. Then, it is checked whether or not the latest data exists in a state where the write right is owned on the cache of another processor element.

  If it exists, it is determined to be a hit, and in step S5, a command is sent to the snoop bus 22, and a write ownership request command is issued. If there is a reply in step S6, the reply data is stored and overwritten on the access source cache, and in step S8, the cache status is updated from the invalid state INV to the exclusive dirty state EX & D.

In step S4, when there is no latest data in the state where the write ownership exists in the other processor element, the process proceeds to step S10, and the access process to the other processor module is performed, which is the write of the local processor module in FIG. The process will be started.

4). Cache unit access contention processing In the cache coherence device of the present invention, a cache unit provided for each processor element of a plurality of processor modules is an access target of a processor element in the module and an external processor module. Therefore, access may concentrate on a specific cache unit.

  For example, when data is being exchanged between a processor element and a cache unit in a remote processor module, when a remote command from the processor module functioning as an external home appears on the snoop bus, the subsequent remote command is Retry is instructed by the cache control. This is because the response to the remote command cannot be determined because the cache state is in the transition transition period due to the first command.

  As described above, when data transmission / reception in the remote processor module continues indefinitely, the home processor module that has transmitted the remote command may repeat the retry. At this time, the bus connection unit for the system bus monitors the time of access response, and repetition of the retry triggers the bus timeout.

  In FIG. 34, the processor module 10-1 functions as a home because the main memory 28 has an address of data to be accessed. Since the data of the access address in the main memory 28 is not the latest data, the memory management unit 26 that manages the directory registers the dirty state D. The latest data of the access address data is held in the exclusive dirty state EX & D in the cache unit of the fourth processor element PE # 3 in the cache unit group 18 of the processor module 10-2.

  In this state, a write command P-WT is generated in the fourth processor element PE # 3 of the processor module 10-1, and a write command P-WT for the same address is generated in the first processor module PE # 0 of the processor module 10-2. Suppose that it occurred.

  In the processor module 10-2, based on the write access generated by the first processor element PE # 1, the coherent write & invalidate CRI command is issued from the cache unit 18-1 to the snoop bus 22 as shown in a. Is done. In this case, since the data of the access address is recorded in the exclusive dirty state EX & D in the cache unit 18-4 of the fourth processor element PE # 3, the data from the cache unit of the fourth processor element PE # 3 is Data is supplied to one processor element PE # 0.

  In parallel with the access in the processor module 10-2, the remote command 220 is sent to the system bus 12 based on the write access of the fourth processor element PE # 3 of the processor module 10-1, as shown in b. A remote command appears on the snoop bus 22 of the remote processor module 10-2.

  At this time, there is no problem if the access between the internal processor elements PE # 0 and # 3 shown in a and c is completed, but if a remote command appears during the access, the bus becomes busy and the processor module 10-1 In response, a reply command 230 for instructing a retry is transmitted via the system bus 12 as e.

  For this reason, the processor module 10-1 that has received the reply command 230 performs a retry operation for sending the remote command 220 to the processor module 10-2 again. The snoop bus is accessed by the internal access of the processor module 10-2 at this time. If 22 is busy, the retry command 230 will be retried again, and the retry may continue indefinitely.

  FIG. 35 is a time chart of processing of the processor module 10-2 in which a retry instruction for a remote command is repeated.

    In the time chart of FIG. 35, the horizontal axis represents the processing timing of the bus clock cycle, the vertical axis represents the processor element in use of the snoop bus 22, the busy period for each of the processor elements PE # 0 to PE # 3, and each Cache status signals CST0 to CST3 corresponding to the processor elements # 0 to # 3 are shown.

  First, as shown in a, with the issuance of the write command of the first processor element PE # 0, the snoop bus 22 is occupied by the processor element PE # 0, and a coherent and invalidated CRT command is transmitted. In accordance with this CRT command, the busy state of the access target processor element PE # 0 is set. If the CRI command of the remote command #R from the external processor element 10-1 appears during the busy period of the processor element PE # 0 as the access destination as shown in b, the processor element PE # 0 as the access destination is busy. Since it is in the state, it becomes a busy response, and a reply command of a busy response to the remote command #R is sent to the processor module 10-1 at the timing d after 4 cycles, for example.

  On the other hand, a data transfer DT command appears from the processor element PE # 3 after the CRI command of the remote command #R of b, and data transfer from the processor element PE # 0 receiving the CRI command to the requesting processor element PE # 3 is performed. Done. After the data transfer based on the DT command of c is completed, the CRI command of the processor element PE # 1 appears on the snoop bus 22 as shown in e, and the same processor as the remote command #R whose access request destination is the busy reply. If it is the element # 0, the processor element PE # 0 is again busy.

  Therefore, even if the retry of the remote command #R appears on the snoop bus from the processor module 10-1 based on the busy reply to the remote command of e as in f, the processor element # 0 that is the access request destination is busy. Since it is in the state, it becomes a busy reply to the remote command #R as in h.

  Then, the retry instruction due to the busy state of the processor element # 0 that is the access request destination for the remote command #R is repeated indefinitely. Eventually, the access from the processor module 10-1 is terminated due to a bus timeout. End up.

  In the present invention against such access conflicts in a specific cache unit, each processor element of the processor module stores information indicating that a remote command was received and a retry instruction was issued during its own access processing. And an address register for holding the access address at that time.

  In addition, during the period when the flag indicating that the retras instruction for the remote command has been issued is valid, the access request other than the same remote command as the access address set in the address register is not accepted and the retry is instructed. To do.

  For this reason, if access by a remote command conflicts during internal access on the snoop bus 22 of the processor module, a retry instruction is given once, but no other access is made for the remote command by the second retry. It has been rejected. Therefore, it is possible to perform processing that preferentially accepts access by the second remote command.

  FIG. 36 is a time chart of the processing operation according to the present invention when the processor element is provided with a flag and an address register for giving priority to remote command access.

  In FIG. 36, if a CRT command by write access of the processor element PE # 0 appears on the snoop bus 22 as shown in a, the processor element PE # 0 that issued this CRT command uses the processor element PE as shown in b. The address to be accessed on the # 3 side is stored in the register.

  When the CRI command of the remote command #R appears during the busy period of the processor element PE # 0 accompanying the CRI command of the processor element PE # 0, a retry instruction is issued to the remote command #R as shown by d Set a flag to indicate

  The first remote command #R is a retry instruction by busy reply at f as in FIG. Subsequently, after data transfer from the access request destination processor element PE # 3 is performed and the use of the snoop bus 22 is released, the same processor element PE # 3 is sent from another processor element PE # 1 as in f. Assume that a CRI command is issued to the same address.

  On the other hand, the processor element PE # 0 that has given the retry instruction to the first remote command #R stores the set state of the flag indicating that the retry instruction has been issued to the remote command and the access address at that time. Therefore, the bus is busy with g for the CRI command of the processor element PE # 1 of f, the retry is instructed by busy reply to the processor element PE # 1, and the access request is rejected.

  Subsequently, when a remote command #R by retry appears at i, the processor element PE # 0 of the access request destination is in an empty state at this time, so the processing by the CRI command of the remote command #R is accepted, and data for the remote side is received. Enter transfer. When this data transfer is started, as shown in (10), the flag and access address stored in the processor element # 0 are sequentially initialized and released at a later timing, and the remote command #R that has been retried. Priority acceptance for will be canceled.

The time chart of FIG. 36 shows an example of the conflict between the remote command from the external processor module and the access command of the internal processor element. The access command is retried once, but the second is preferentially accepted, and the access contention for cache coherency on the snoop bus 22 can be adjusted appropriately.

5. Copyback Write Ownership FIG. 37 shows local storage 28 as distributed main storage when the latest data held in any of the cache units of the processor module 10-1 is replaced with the latest data. It is explanatory drawing of the control procedure regarding the transfer of the write ownership for copying back to (1).

  Now, in the processor module 10-1, not the data of the local storage 28 as the distributed main memory or the latest data, but the cache units 18-1, 18-1 of the four cache units connected to the memory management unit 26 by the snoop bus 22; Suppose that it exists in 18-4. Further, it is assumed that the cache unit 18-1 among the cache units 18-1 and 18-4 sharing the same latest data has the write ownership for copying back the latest data to the local storage 28.

  The cache state at this time is represented by the shared dirty state EX & D in which the cache unit 18-1 having the latest data and the write ownership has the latest data but has no write ownership. 4 appears in the shared modify state SH & M. Further, the memory management unit 26 indicates that the target data in the local storage 28 is in the dirty state D. Here, the cache unit 18-1 having the write ownership for copy back to the local storage 28 is referred to as an owner.

  In the cache state in the processor module 10-1 of FIG. 37, it is assumed that the latest data of the cache unit 18-1 having the write ownership as the owner is replaced by data discard or the like by the operation of the processor. Along with this replacement, a replace command P-RP is generated on the snoop bus 22. Further, the cache unit 18-1 having the write ownership transitions to the invalid state INV.

  On the other hand, the cache unit 18-4 that holds the latest data but does not have the write ownership receives the transfer of the write ownership based on the replace command P-RP shown in the snoop bus 22. Becomes the owner and transitions to the shared dirty state EX & D. For this reason, even if the latest data is replaced in the memory unit 18-1, the write ownership is transferred to another cache unit 18-4, and the copy back of the latest data to the local storage 28 is not performed at this time. .

  If the replacement by the same processor operation occurs in the cache unit 18-4 after the write ownership is transferred, the write ownership cannot be transferred to another memory unit in this case. Data is copied back.

  Further, when the write ownership is transferred from the cache unit 18-1 to the cache unit 18-4, the memory management unit 26 holds the dirty state D for the data to which the write ownership is transferred. However, even if the write ownership is performed between the cache units 18-1 and 18-4, this dirty state is maintained as it is.

  The dirty state of the memory management unit 26 is moved only when the cache line having the latest data is moved to another processor module. When the latest data is copied back to the local storage 28, the dirty state D transitions to the clean state C.

  FIG. 37 shows an example in which the latest data of the local storage 28 is held in two of the four cache units 18-1 to 18-4, but the latest data is stored in three or four cache units. If it is held, there are a plurality of cache units to which the write ownership is delegated. In this case, for example, the write ownership is delegated according to a predetermined order shown in FIG.

  FIG. 38 shows the identification numbers of the cache units 18-1 to 18-4 as # 0 to # 3, and the write ownership to the remaining three cache units is indexed with the cache unit replaced with the write ownership. It shows the order in which rights are transferred.

For example, if cache unit # 0 has write ownership and the other three cache units # 1 to # 3 have the latest data, write ownership when cache unit # 0 is replaced The priorities for shifting are determined in the order of # 1, # 2, and # 3. For this reason, the write ownership is delegated to the cache unit # 1 having the highest priority. The next is cache unit # 2, and the last is cache unit # 3. Of course, the order of the cache units to which the write ownership is transferred can be determined as appropriate in addition to the unit identification numbers in FIG.

6). System Common Bus (1) Bus Configuration FIG. 39 is a block diagram of a subsystem based on the cache coherence device according to the present invention of FIG. In FIG. 39, the subsystem 500-1 connects, for example, five processor modules 10-1 to 10-5 as a basic system 501 by a system bus 12 as a second shared bus. The system bus 12 is provided with a bus arbiter 502 that performs bus arbitration.

  For example, an expansion system 506 in which three processor modules 510-1 to 510-3 are connected to a system bus 12-10 having a bus arbiter 503 is connected to the basic system 501. In contrast to the basic system 501, the extended system 506 uses the bus arbiter and the processor module having the same configuration. Further, the heterogeneous system 505 is connected to the system bus 12 of the basic system 501 via the heterogeneous bus expansion unit 540. The heterogeneous system 505 is connected to appropriate processing units 544-1 to 544-3 by a system bus 542. The heterogeneous subsystem 505 is a system that is completely different from the basic system 501 and the expansion system 506 that implement the cache coherence device according to the present invention, and constitutes, for example, an I / O system for an external storage device.

  For such a subsystem 500-1, another subsystem 500-2 having the same configuration can be expanded. The subsystem 500-1 can be connected to the subsystem 500-2 expanded by the inter-subsystem connection unit 512. Similarly to the subsystem 500-1 side, the subsystem 500-2 side is also connected to its own system bus via an inter-subsystem connection unit.

  Here, for example, up to 16 modules or units can be connected to the system bus 12 of the basic unit 501. In this embodiment, eight processor modules 10-1 to 10-5, inter-subsystem connection unit 512, heterogeneous bus expansion unit 540, and bus arbiter 502 are connected to the system bus.

  Unit IDs # 0 to # 15 are assigned to modules or units connectable to the system bus 12. For example, unit IDs (UID) # 0 to # 4 are assigned to the processor modules 10-1 to 10-5, a unit ID # 6 is assigned to the inter-subsystem connection unit 512, and a unit is assigned to the heterogeneous bus expansion unit 510. ID # 8 is assigned, and unit ID # 16 is assigned to the bus arbiter 502.

  The remaining unit IDs are empty. Note that the bus expansion unit 504 is merely an interface between the system buses 12, and it is not necessary to specify a unit ID.

  The bus arbiter 502 and the processor modules 510-1 to 510-3 connected to the system bus 12-10 of the expansion system 506 connected via the bus expansion unit 504 are also 16 units unique to the system bus 12-10. An ID is appropriately assigned. Further, the heterogeneous subsystems 505 are assigned unit IDs #A, #B, and #C unique to the heterogeneous systems that are completely different from the basic system 501 and the extended system 506.

  FIG. 40 shows a configuration of signal lines of the system bus 12 provided in the basic system 501 of FIG. In FIG. 4040, the processor modules 12-1 to 12-5 and the bus arbiter 502 are basically composed of a bus data line 507 and a tag control line 509.

  The bus data line 507 includes a common data bus line 514 and a data parity line 516. The common data bus line 514 is composed of, for example, 128 lines, and if 1 byte is 8 bits, 16 bytes of bit data can be transferred in parallel. The data parity line 516 is 16 lines and transfers 1-byte data parity.

  The tag control lines are bus request line 520, emergency bus request line 522, bus grant line 524, bus ID line 526, unit ID line 528, halt line 530, master right switching line 532, installation line 534, disconnect notification line 536. And a clock line 538.

  The configuration of the data bus 12 of FIG. 40 will be described in detail as follows. First, the common data bus line 514 and the data parity line 516 are connected in common to all the processor modules 12-1 to 12-5 and the bus arbiter 502, and parallel data 128 bits (16 bytes) and parity 16 bits (1 byte). Send and receive data through two-way transmission.

  The common tag bus line 518 is composed of four lines and is commonly connected to all the processor modules 12-1 to 12-5 and the bus arbiter 502. By referring to the signal of the common tag bus line 518, the type of data on the common data bus line 514 in the current bus cycle can be known. The types of data on the common data bus line 128 indicated by the common tag bus line 518 are as shown in FIG.

  In FIG. 41, a tag bus code (TB code) designates an operation in each bus clock cycle of the common data bus line 514 by a 3-bit code. The signal state of the common tag bus line 518 is referred to by the bus master serving as the transmission source and the bus slave serving as the destination in the processor modules 12-1 to 12-5, and further by the bus arbiter 502, and will be clarified in the following description. ), Bus cycle switching, and data reception control.

  The tag bus code No. 0 in FIG. 41 means the end of the command and the continuation of the bus use right. The tag bus code No. 1 means termination of the command and release of the bus use right. Tag bus code No. 2 means continuation of the command. The tag bus code No. 3 means retry (invalidation) and release of the bus use right. Tag bus code No. 4 means a single word command and continuation of the right to use the bus. The tag bus code No. 5 means a single word command and release of the bus use right. The tag bus code No. 6 means the start of a multiword command and, if necessary, the continuation of bus usage rights. The tag bus code No. 7 means an idle cycle.

  Here, the single word command is a command for ending packet transfer in one bus clock cycle. On the other hand, a multiword command is a command that requires packet transfer in a plurality of bus clock cycles. The release of the bus use right means that the bus use right is abandoned in the bus cycle of the tag bus code indicating this, and the bus is released to other bus use requesters.

  The continuation of the bus use right indicates that the bus is continuously used in the next cycle of the bus cycle indicated by the tag bus code. Further, the retry indicates that the bus cycle (bus command) indicated by the tag bus code is forcibly terminated to retry. At this time, the bus slave discards the bus command instructed to retry. Whether or not to execute a retry depends on the implementation of the system. Furthermore, the idle cycle indicates a state when no unit is using the bus. At this time, the bus slave side ignores the bus cycle which is an idle cycle.

  Next, the bus request line 520 and the emergency bus request line 522 in FIG. 40 will be described. The bus request line 520 and the emergency bus request line 522 are each composed of 15 signal lines, and are connected one-to-one between the processor modules 12-1 to 12-5 and the bus arbiter 502, and the processor modules 12-1 to 12-12. A bus transfer request is made to the bus arbiter 502 from the −5 side.

  For a normal transfer request, a transfer request is made only by the bus request line 520. In contrast, in an emergency such as abnormality detection, a transfer request using both the bus request line 520 and the emergency bus request line 522 is made.

  FIG. 42 shows requirements for the bus arbiter 502 by a combination of the bus request line 520 and the emergency bus request line 522. First, when both the bus request BRQ and the emergency bus request BRE are at the L level, it indicates that the unit of the slot with the unit ID #N is in an inoperative state.

  When only the next emergency bus request BRE becomes H level, there is no bus request, and the unit not currently making a bus request asserts only the emergency bus request.

  It is a normal bus request that only the third bus request BRQ becomes H level, and a normal bus request other than the emergency bus request. It is an emergency bus request that both the fourth bus request BRQ and the emergency bus request BRE become H level, and when a bus slave detects an error, this request is made to acquire a bus for error reply. .

  Referring to FIG. 40 again, the next bus grant line 524 is connected one-to-one between the bus arbiter 502 and each of the processor modules 12-1 to 12-5, and the bus arbiter 502 sends the signal of the bus grant line 524 to H. A level indicates that the use of the bus has been granted.

  The bus ID line 526 sets the number of the system bus 12 to which the processor modules 12-1 to 12-5 and the bus arbiter 502 are connected. The same value is set for all units connected to the same system bus. In the system configuration of FIG. 39, the bus ID # 1 is set to the system bus 102 of the basic system 501. On the other hand, the bus ID # 2 is set to the system bus 12-10 of the expansion system 506 connected via the bus expansion unit 504.

  Next, the unit ID line 528 in FIG. 40 will be described. The unit ID line 528 sets identification numbers on the bus of the bus arbiter 502 and the processor modules 12-1 to 12-5. In FIG. 39, unit IDs # 1 to # 5 are set for the processor modules 10-1 to 10-5, and unit ID # 16 is set for the bus arbiter 502.

  For this reason, the numbers of the processor modules 12-1 to 12-5 and the bus arbiter 502 in the entire system are uniquely determined by the combination of the bus ID and the unit ID by the bus ID line 526 and the unit ID line 528.

  Next, the halt line 530 and the master right switching line 532 in FIG. 40 will be described. The halt line 530 is connected from the bus arbiter 502 to all units including the processor modules 12-1 to 12-5, for example, in nine ways. By setting the halt line 530 to the H level, the bus arbiter 502 detects its own failure and stops. Indicates that it is in a state.

  The master right switching line 532 is commonly connected to all units including the processor modules 12-1 to 12-5 from the bus arbiter 5-2, and is used together with the halt line 530 for bus arbitration processing. The bus arbitration processing by the combination of the signals of the halt line 530 and the master right switching line 532 is as shown in FIG.

  In FIG. 43, when the halt HLT is at the H level and the master right switching MFC (MasterForce Change) is at the L level, the bus arbiter 502 is not mounted. When both the next halt HLT and the master right switching MFC are at the H level, an error occurs in the halt of the bus arbiter 502, that is, the self-diagnosis of the bus arbiter 502, and the bus arbiter 502 is notified that it is in a stopped state.

  When both the third halt HLT and the master right switching MFC are at the L level, the bus grant GBR is normally switched. When the bus arbiter 502 holds the bus request, the bus grant GBR is detected. It means the timing to switch. When the last halt HLT is L level and the bus right switch MFC is H level, the bus grant GBR is forcibly switched. The bus grant switch when the bus arbiter 502 holds a bus request and the bus is not used. Give timing.

  Referring again to FIG. 40, the installation line 534 is connected one-to-one between each unit including the processor modules 12-1 to 12-5 and the bus arbiter 502, and is set to the L level to unit the bus arbiter 502. Notify that is implemented.

The disconnect notification line 536 is connected to the other one between each unit including the processor modules 12-1 to 12-5 and the bus arbiter 502. By setting the disconnect notification line 536 to the L level, the bus arbiter 502 is notified that the unit is disconnected from the system bus 12.

(2) Bus Arbitration Operation Next, the bus arbitration operation of the system bus 12 composed of the bus data line 507 and the tag control line 509 in FIG. 40 will be described. The bus arbitration operation of the system bus according to the present invention includes two modes, a first mode and a safety mode. The priority bus arbitration is divided into a case where a bus slave requests priority bus arbitration and a case where a bus arbiter requests priority bus arbitration.

  FIG. 44 is a time chart of the arbitration in the first mode. The internal operation of the bus arbiter 502 and the interface signal on the system bus 12 in the vertical axis direction with respect to the bus clock cycles T1, T2, T3,. They are shown separately. The internal operation of the bus arbiter 502 is a bus request hold operation. As interface signals on the system bus 12, a bus request, a bus grant, a tag bus, and a data bus are shown.

  First, in cycle T1, it is assumed that one cycle of bus request signal is asserted as indicated by unit IDs # 0 and # 1 to a. At this time, since the data bus is in an idle state, the bus arbiter 502 monitors the rise of bus requests from all bus request sources.

  When two bus request signals of unit IDs # 0 and # 1 are asserted as in a, all request requests are held as shown in b in the next cycle T2. Here, when there are a plurality of bus requests, it is assumed that the lower unit ID number has a higher priority.

  For this reason, the bus arbiter 502 asserts the bus grant signal for the unit ID # 0 having a higher priority from the idle state together with the master right switching signal as shown in c. Subsequently, in the next cycle T3, the bus arbiter 502 discards the hold request for the bus request of the unit ID # 0 permitted to use the bus, and at the same time switches the bus grant signal if there is another hold request.

  At this time, since there is a hold request for unit ID # 1, the bus request signal is switched to unit ID # 1. As described above, the unit ID # 0 as the bus master that has been permitted by the bus grant signal and the master right switching signal from the bus arbiter 502 starts data transfer from the next cycle T3. From this cycle T3, the tag bus signal indicates one of the tag bus codes 010, 100 or 110 (CC, SC & C, MC & S) shown in FIG. In this case, since data is sent in a plurality of bus clock cycles, the tag bus code is the start of a 110 (MC & S) multiword command.

  During the data transfer from cycle T3, if the bus sequence indicates end END by the tag bus as shown in cycle T5, for example, as shown in d, the bus master is switched to unit ID # 1 according to the bus grant signal at this time. From the next cycle T6, data transfer from the unit ID # 1 that has become the bus master is started. Here, the tag bus code of the tag bus indicating the end of data transfer is one of 001, 011, 101 (CE & E, retry, SC & E) in FIG.

  On the other hand, when the bus arbiter 502 detects the rise of the bus request from the other unit IDs # 0 and # 2 during the processing of the bus request from the unit ID # 1 once held in the cycle T2, for example, the cycles T3 and T4. Each bus request is stored separately. Then, as shown in e, new bus requests of unit IDs # 0 and # 2 are held at the timing of completion notification by the tag bus when there is no hold request as in cycle T5.

  Further, the second bus request from the unit ID # 0 of the cycle T4 is a single command. Therefore, when the data transfer of the unit ID # 1 is completed in the cycle T8, the lower order issued in the cycle T3 is performed in the next cycle T9. Prior to the bus request of unit ID # 2, data transfer of a single command of unit ID # 0 is performed. Since the data transfer of the single command means the end of the transfer at the same time, the bus request held by the unit ID # 1 in cycle T6 is held as shown in f of cycle T10.

  In the fast mode bus arbitration shown in FIG. 44, as long as a bus request is held in the bus arbiter 502, the bus transfer is continuously performed without generating an empty cycle on the data bus, and no empty cycle is generated. Can increase the bus transfer speed.

  FIG. 45 is a time chart of the safety mode in the bus arbitration of the present invention. Even in the time chart of this safety mode, the case where a bus request by unit IDs # 0 to # 2 similar to the first mode of FIG. 44 is made is taken as an example.

  In the safety mode, after the unit ID as the bus master is permitted by asserting the bus grant signal and the master right switching signal from the bus arbiter 502, the data transfer is started after one idle cycle. Yes.

  That is, the bus request of the bus arbiter 502 is held for the bus request a in cycle T1 as in b in cycle T2, and the bus grant signal for the higher priority unit ID # 0 and further the master switching signal are set to c. In this way, after the next cycle T3 is set as an idle cycle, data transfer is started from cycle T4.

  The idle cycle of the cycle T3 is a period for stabilizing the transient state of the tag bus. In this idle cycle, all units ignore the value of the tag bus. In the cycle T6, the data transfer by the bus master of the unit ID # 0 is completed, and for the data transfer by the bus grant signal of the unit ID # 1 at that time, after providing an idle period of one cycle of the cycle T7, as shown in f Data transfer has started.

  The operations other than the one-cycle idle period between the time when data transfer permission is received from the bus arbiter 502 and the start of data transfer are the same as in the fast mode of FIG.

  FIG. 46 is a time chart of priority bus arbitration for error reply operation when the bus slave detects an error during data transfer. As shown in cycle T1, bus requests from three units of unit IDs # 0 to # 2 are simultaneously made in an idle state. When all bus requests are held in the bus arbiter 502 in the next cycle T2, priority is given. A bus grant signal is output from the bus arbiter 502 to the unit ID # 0 having the highest rank.

  At the same time, the master right switching signal is also asserted, whereby data transfer is started from unit ID # 0, which has become a bus master from cycle T3, to unit ID # 3 as a bus slave, for example. If an error is detected in unit ID # 3, which is a bus slave, in cycle T4 during this data transfer, both a bus request signal and an emergency bus request signal are output from unit ID # 3 to bus arbiter 502 as a. The

  The bus arbiter 502 that has received the bus request signal and the emergency bus request signal in cycle T6 holds the bus request with the highest priority, and at the same time, as in b, a unit that performs error detection for the bus grant to switch the bus early. Switch to ID # 3. Therefore, when the data transfer is completed in cycle T8, the unit ID # 3 as the bus slave that has detected the error for which the bus grant signal is valid at this time is switched to the bus master, and in cycle T9, as shown in c, an error occurs. The error reply is performed from the unit ID # 3 of the bus master that detected the error.

  At this time, the highest priority hold request in the bus arbiter 502 is discarded, and normal arbitration is continued from the next cycle T10. Here, errors detected on the bus slave side include block detection such as physical buffer full. When error detection is performed on the bus slave side during bus transfer in this way, the bus slave that forcibly detected the error before switching to the next bus transfer can switch to the bus master and perform error reply. It is possible to quickly detect an error during transfer and take a countermeasure such as retry.

  FIG. 47 is a time chart of priority bus arbitration for error reply when the bus arbiter 502 detects an error that has occurred during bus transfer before the bus slave side determines. That is, in the bus transfer of the present invention, split-type packet transfer is adopted, and the command word is transferred as the first word of the packet. Therefore, if the decoding of the destination field of the command word is not completed, the bus slave is determined. do not do.

  Until the bus slave is determined, the bus arbiter 502 monitors a bus transfer error, and when an error is detected, the bus arbiter 502 performs error reply instead of the slave.

  In FIG. 47, when there is a bus request from the unit IDs # 0 to # 2 of the unit T1 in the cycle T1, the bus arbiter 502 is all held in the cycle T2. For the unit ID # 0 having the highest priority, the bus grant signal is validated and the master right switching signal is asserted, and data transfer is started from the next cycle T3.

  In this cycle T2, the bus arbiter 502 has a bus master flag of unit ID # 0 indicating the bus master that has given permission to use the bus, as in a, and this bus master flag is used for the bus cycle period extending from cycle T3 to T8 of unit ID # 0. Hold inside.

  When the arbiter 502 detects an error as in b in cycle T5 during data transfer from the unit ID # 0 as the bus master, for example, when detecting a parity error in the packet header, the bus arbiter 502 holds the error content and at the same time. Set its own emergency bus request flag as shown in c. The cycle T5 in which the error detection is performed is an example in which the unit ID # 01 as the bus master detects an error while transferring a plurality of continuous bus commands.

  Subsequently, the bus arbiter 502 masks other currently held bus requests by the emergency request bus flag c set by itself, and does not assert the bus grant for the bus request being held. When data transfer with the bus master as unit ID # 0 is detected in cycle T8, the bus arbiter 502 itself becomes the bus master at this bus master switching timing, and an error reply is transferred in cycle T9 as in d.

That is, due to the arbitration within the bus arbiter 502 that does not appear on the bus, the bus arbiter 502 itself becomes the bus master and performs error reply. When such an error reply of the bus arbiter 502 is completed, the emergency bus request flag and the bus master flag set in the bus arbiter 502 are discarded at the same time, and the switching to the unit ID # 1 as the next bus master is performed by the bus grant. Normal arbitration is executed from cycle T10.

(3) Bus Transfer Sequence Next, a bus command and data transfer sequence on the system bus 12 in FIG. 40 will be described. FIG. 48 shows a transfer sequence when a memory read request is made from unit # 1 to unit # 2. Here, the read request source unit # 1 is a source unit, and the read request destination unit # 2 is a destination unit (destination unit).

  First, the request source unit # 1 transmits, as a bus command, a packet header specifying the command contents, the request source, the request destination, the number of transfer bytes, and the access address. This packet header constitutes the first word of the bus command. Specifically, unit # 1 designates memory read in the command field of the packet header, designates its own unit ID # 1 as the request source in the source field, and designates unit ID # 2 of the request destination in the destination field. specify.

  Further, 16 bytes are designated in the side field indicating the number of transfer bytes, and a packet header in which the access address is designated in the address field is generated and transmitted. In the bus transfer according to the present invention, the first word of the bus command transmitted as the packet header includes the second field indicating the second request destination in addition to the source field indicating the request source and the destination field indicating the request destination. A destination field is provided.

  Here, the normally used destination field is hereinafter referred to as a first destination field. Note that the first destination field may be simply referred to as a destination field when the second destination field is not designated. Details of the bus command having the source field, the first destination field, and the second destination field of the first word of the bus command transmitted as the packet header will be made clear in the following description.

  In the memory read transfer sequence of FIG. 48, since there is only one destination, the packet header transmitted from unit # 1 only specifies the read request destination unit ID # 2 in the first destination field. It is. The system bus is released when transmission of the packet header from unit # 1 is completed, and the right to use the bus is transferred to the next assigned unit.

  In unit # 2, which is the read request destination, the transfer packet on the system bus is received and the ID of the destination field is identified. If the packet itself is designated, it is received and received. The process designated by the packet, that is, the memory read process designated by the packet header from unit # 1 is started. When the requested read data is ready in unit # 2, the reply bus command and data shown in b are created and returned.

  The first word of this reply bus command specifies its own unit ID # 2 in the source field, specifies the requesting unit ID # 1 in the destination field, specifies that the command field is a reply command, Furthermore, status information is added as necessary and transmitted. When the first word, which is the packet header of the reply bus command, has been sent, the data for the designated number of bytes of 16 bytes is returned. In this embodiment, since the data bus width is 16 bytes, a 2-word reply bus command completes transfer in two bus cycles.

  FIG. 49 shows a memory write transfer sequence in the system bus of the present invention. First, the write request source unit # 1 creates and transfers a packet header as the first word of the bus command to the write request destination unit # 2, and writes, for example, 32-byte data following the packet header. In this case, a 2-word data packet is transferred in units of 16 bytes.

  In the packet header of the first word, the source field is the unit ID # 1 of itself, the destination field is the unit ID # 2 of the write partner, the memory write is specified in the command field, and the address field is 32. Specify byte address and send to bus.

  In the write request destination unit # 2, if the packet specifies its own unit ID # 2 in the destination field, the packet is received and the writing process specified in the command field is started. When the write process is completed, a reply bus command for the write request source unit # 1 is created and the process status is returned as shown in b.

    That is, in the reply bus command, the source field is its own unit ID # 2, the destination field is the unit ID # 1 of the write request source, the command field is reply, and further, for example, a status indicating normal writing is set in the status field. Will be returned.

  FIG. 50 shows a broadcast transfer sequence in which a plurality of units connected to the system bus are simultaneously specified to transfer data. In this example, unit # 1 transfers a cache invalidation bus command for cache coherent to units # 2, # 3, and # 4 as a broadcast command.

  First, the request unit # 1 of the cache invalidation specifies the requesting unit ID # 1 in the source field and the requesting units # 2, # 3, # in the broadcast field for the packet header of the broadcast bus command. Specify 3 of 4. Furthermore, the cache invalidate is designated in the command field, and the access address is designated and transmitted.

  Each of the units # 2, # 3, and # 4, which are cache invalidation request destinations, receives the broadcast bus command from the request source unit # 1, and receives it if the broadcast field has its own designation. Start processing the specified cache invalidate. When the cache invalidation processing is completed, reply bus commands are sent to the request source unit # 1 in the order of completion.

  In this example, the cache invalidation process is completed in the order of units # 2, # 4, and # 3, and a reply bus command is returned. For example, in the first unit # 2, the source field is its own unit ID # 2, the destination field is the requesting unit ID # 1, the reply is specified in the command field, and the cache invalidation is further specified in the status field. Set the processing result of and return it. The same applies to units # 4 and # 3.

  The request source unit # 1 receives the reply bus command from all the request destination units # 2, # 3, and # 4, and recognizes the success of the cache invalidation from the status information, and proceeds to the next processing. FIG. 51 shows a transfer sequence when bus transfer is performed between the unit of the basic system 501 and the unit of the extended system 506 in FIG. FIG. 51 shows a transfer sequence when a read request is made from the unit # 1 of the system bus 1 in the basic system to the unit # 3 of the system bus # 2 of the expansion system via the bus expansion unit 504.

  First, the unit # 1 of the system bus # 1 creates and sends a bus command to which the unit # 3 of the system bus # 2 is a request destination as shown in a. The bus command from the unit # 1 specifies the bus ID # 1 indicating the request source system bus and the unit ID # 1 indicating the unit # 1 in the source field. In the destination field, a bus ID # 2 indicating the destination system bus and a unit ID # 3 indicating the destination unit are specified.

  Except for this point, memory read is designated in the command field, and the number of transfer bytes is designated and transmitted in the same manner as a bus command on the same system bus. When the bus expansion unit 504 receives the bus command from the unit # 1 and recognizes that the bus ID # 2 in the destination field of the bus command is a bus that is connected and expanded, the bus expansion unit 504 receives the bus command. The same bus command is issued to the system bus # 2 as shown in b.

  When unit # 3 of system bus # 2 recognizes that the unit ID of the destination field is its own unit ID # 3, the bus command is received, read processing is performed, and read data is ready Then, as in c, a reply bus command having the command as the first word and the data as the second word is transmitted.

  Also for this reply bus command, the system field ID # 2 and own unit ID # 3 are specified in the source field, and the bus ID # 1 and unit ID # 1 of another system bus are specified in the destination field. Further, a reply is specified in the command field, and the status of the read result is further added and transmitted. When transmission of the first word of the reply bus command is completed, a packet of 16 bytes data of the next second word is transmitted.

  When the bus expansion unit 504 receives the reply command c and recognizes from the bus ID # 1 in the destination field that it is the system bus # 1 that is connected and expanded, it sends the same reply command to the system bus # 1. Forward. When unit # 1 of system bus # 1 recognizes its own destination unit ID # 1 from this reply command, it receives the reply bus command, receives the 16-byte read data of the next second word, and ends the processing. To do.

  FIG. 52 is a sequence of bus transfer via the inter-subsystem connection unit 512 between the subsystem 500-1 and the subsystem 500-2 in FIG.

  In FIG. 52, a read request is made from unit # 1 of subsystem # 1 to unit # 3 of another subsystem # 2. First, the unit # 1 of the subsystem # 1, which is the request source, sends out a bus command for a read request as indicated by a. This bus command is composed of a 2-word bus command using two packet headers.

  First, in the first word of the bus command, an identifier for designating access of the extended subsystem # 2, that is, a subsystem extension identifier EX is designated. Subsequently, the bus ID # 1 and the unit ID # 1 of the subsystem # 1 are set in the source field, and the unit ID # 6 of the inter-subsystem connection unit 512 for the subsystem # 2 is designated in the destination field.

  Further, memory read is designated in the command field, an access address is designated in the address field, and the number of transfer bytes is designated in a size field (not shown). The next second word is an extended bus command added by specifying the system bus extension identifier EX, and the requesting subsystem ID # 1, bus ID # 1, and unit ID # 1 are specified in the source field.

  In the destination field, the requested subsystem ID # 2, the requested bus ID # 2, and the requested unit ID # 3 are specified. When this two-word bus command is sent to the system bus, the inter-subsystem connection unit 512 recognizes the unit ID # 6 of the destination field of the first word, and the first and second words of the bus command. Receive.

  Then, the same bus command is transferred as it is to the inter-subsystem connection unit 513 provided on the subsystem # 2 side. This transfer control is determined by each ID of the second word of the bus command. The inter-subsystem connection unit 513 of the sub-system # 2 that has received the bus command from the inter-subsystem connection unit 512 of the sub-system # 1 sets its unit ID in the source field of the first word of the bus command as its own unit ID. While changing to designation of # 7, the destination field is changed to designation of unit ID # 3 decoded from the second word and sent to system bus # 2. The second word of the bus command is sent as it is.

  When unit # 3 of system bus # 2 recognizes that the unit ID of the destination field is its own unit ID # 3 upon reception of the first word of the bus command sent from inter-subsystem connection unit 513, This command is received, and the second word is also received. Then, the memory read designated by the command field of the first word is performed for the designated access address.

  When the read data is prepared, a reply bus command is sent as shown in d. In this reply bus command, the subsystem identifier EX is set in the source field of the first word, the bus ID # 2 and the unit ID # 3 are set, and the destination field is the unit ID # 7 of the inter-subsystem connection unit 513. Is specified. Of course, the command field designates reply and sets a status (not shown).

  The second word of the reply bus command designates subsystem ID # 2, system bus ID # 2 and unit ID # 3 in the source field, and subsystem ID # 1 and system bus ID as the request source for the destination field. Specify # 1 and unit ID # 1. The third word is 16-byte data serving as read data.

  When the inter-subsystem connection unit 513 recognizes its unit ID # 7 from the destination field of the first word of the reply bus command sent from the unit # 3, it receives this bus command, The first word, the second word, and the third word of the reply bus command are transferred to the inter-subsystem connection unit 512 of the sub-system # 1 through the inter-system interface.

  The inter-subsystem connection unit 512 of the subsystem # 1 changes the unit ID of the source field of the first word of the reply bus command to the designation of its own unit ID # 6, and changes the destination field to the requesting unit ID #. Change to 1 and send to system bus # 1.

  The unit ID of the source field and the destination field can be changed by referring to the second word of the received reply bus command. The second word of the reply bus command is sent as it is. The 16-byte data of the second word is also transferred as it is.

  When the unit # 1 of the subsystem # 1 recognizes its unit ID from the destination field of the first word of the reply bus command, it starts receiving the bus command, receives up to the third word, and ends the processing.

  FIG. 53 shows a transfer sequence when the bus width of the system bus of the present invention is an 8-byte bus. In the system bus of FIG. 40, 16 bytes are transferred by setting 128 common data bus lines 514. However, depending on the system, a half 8-byte bus may be used. When the 8-byte system bus is used in this way, the bus transfer in one clock cycle having a 16-byte width described so far is performed for two bus clocks.

  FIG. 53 shows a transfer sequence when a read request is made from unit # 1 to unit # 2 when an 8-byte bus is used. Unit # 1 divides the 16-byte bus command into a first word and a second word in 8-byte units and sends them to the system bus. A source field, a destination field, a command field, and a side field are provided in the first word of the 8-byte bus command. The second word is an address field.

  When such a bus command consisting of 2 words of 8 bytes is sent to the system bus, the requested unit # 2 recognizes the destination unit ID # 2 of the first word of the bus command and sends the bus command. The memory read operation is performed for the access address specified by the second word. When the read data is prepared, a reply bus command is transmitted as shown in b. The reply bus command is sent with the first word of 8 bytes serving as a packet header, followed by 2 words of 16 bytes in total divided by 8 bytes.

  FIG. 54 shows a bus transfer sequence between the basic system 501 and the heterogeneous subsystem 505 in the subsystem 500-1 of FIG.

  In FIG. 51, the system bus # 1 of the basic system is connected to the heterogeneous system bus # 2 via the heterogeneous bus expansion unit 540. The heterogeneous bus expansion unit 540 is provided with a command conversion function for converting a command on the system bus # 1 side into a command on the heterogeneous system bus # 2 side.

  Assume that an access request is made from unit # 1 of system bus # 1 to unit #B of heterogeneous system bus # 2. Unit # 1 sends a 2-word packet as a packet header as a bus command. In the first word of this bus command, the bus ID # 1 and the unit ID # 1 are specified in the source field, and the bus ID # 1 and the unit ID # 8 of the heterogeneous bus expansion unit 540 are specified in the destination field. .

  Further, an emulation identifier EM for conversion to the command system of the heterogeneous system bus # 2 is specified in the command field. The second word is a bus command suitable for the heterogeneous system bus # 2, the unit id # 1 of the heterogeneous bus expansion unit 540 in the heterogeneous system bus # 2 space is designated as the source field, and the heterogeneous system bus # is designated as the destination field. 2 unit ID uid # B is designated.

  The 2-word bus command sent from the unit # 1 of the system bus # 1 is received by the heterogeneous bus expansion unit 54 by recognizing its own unit ID # 8 from the destination field of the first word. When it is recognized that the heterogeneous bus expansion unit 540 itself is an emulation command for the heterogeneous system bus # 2 that is connected and expanded, only the second word that is a command dependent field is sent to the heterogeneous system bus # 2 as shown by b.

  The unit #B of the heterogeneous system bus # 2 recognizes and receives its own destination from the uid # B of the destination field, executes the processing specified in the command field, and when the processing is completed, the reply command shown in c Is sent out. The reply command from unit #B designates unit id #B in the source field of the first word, designates unit id # 1 of the heterogeneous bus expansion unit 540 in the destination field, and further, status and other necessary items in the second word Transfer data.

  When the heterogeneous bus expansion unit 540 recognizes that it is a reply command for the system bus # 1 to which it is connected from the first word of the reply bus command from unit #B, it accepts this reply command, A similar reply bus command is sent to the system bus # 1 side as shown in d.

  The first word of this reply bus command specifies the system bus ID # 1 and the unit ID # 8 of the heterogeneous bus expansion unit 540 by the source field, and specifies the unit ID # 1 of the system bus ID # 1 of the request source in the destination field. is doing. The bus command transmitted from the unit #B of the heterogeneous system bus # 2 is added as it is to the second word and the third word.

By such simple bus command conversion by the different bus expansion unit 540, it is possible to transfer data between units of different bus systems. Further, the unit ID on each bus used in the emulation bus command only needs to be unique on each bus, and the ID on the expanded heterogeneous system bus including the heterogeneous bus expansion unit 540 is also unique.

(5) Bus Command FIG. 55 shows a basic command format of a bus command used for bus transfer of the system bus 12 of the cache coherence device of the present invention.

  In FIG. 55, if one byte is 8 bits, the basic bus command is composed of 16 bytes. In FIG. 55, the command field is divided into 4 bytes (32 bits).

  The basic bus command has a source field 552, a first destination field 560, and a second destination field 564 in the first basic byte portion. In the source field 552, access request source information is designated. In the first destination field 560, information on the destination of the access request from the source field requester is designated. In the second destination field 564, in addition to the source field 552 and the first destination field 560, information on another destination to be accessed is designated.

  Specifically, for example, the processing operation in the read mode 2 in the cache coherence device of the present invention shown in FIG. 17 is as follows. In FIG. 17, a read request is issued from the local processor module 10-1 that is the access request source to the home processor module 10-2 that holds the main memory to be accessed. Is the process when the latest data exists in the cache unit of the remote processor module 10-3.

  In such a read process, when a read request is first made from the local processor module 10-1 to the home processor module 10-2, a local unit ID is designated in the source field 552 of FIG. The 1 destination field 560 specifies the home unit ID. At this time, the second destination field 564 is not used.

  Next, when a read request is made from the home processor module 10-2 to the remote processor module 10-3, the source field 552 is set as the home ID, the first destination field 560 is set as the remote ID, and the second destination field 564 is further set. The local ID which is the first access request source is specified in.

  Furthermore, when sending the reply data 260 from the remote processor module 10-3, the remote ID is specified in the source field 552 of FIG. 55, the home ID that is the request source is specified in the first destination field 560, and The second destination field 564 specifies the local ID that is the first request source. Specifically, the same information as the bus command received by the remote processor module 10-3 may be set as it is, and the reply data 260 as the reply command may be transmitted.

  As described above, in the read access that requires data transfer between the three processor modules that are local, home, and remote in the cache coherence device of the present invention, the source field 552, the first destination field 560, and the second destination in FIG. The command format in field 564 will be meaningful.

  In the basic bus command of FIG. 55, the source field 552 is actually divided into a source bus ID field and a source unit ID field. By having the source bus ID field, data transfer between different system buses becomes possible. Similarly, a destination bus ID field and a destination unit ID field are also provided for the first destination field 560 and the second destination field 564, respectively. Further, an access ID field is provided in the first destination field 560 and the second destination field 564.

  The access ID field of the first destination field 560 is a field for designating an operation identification ID number for performing a bus access multiplexing operation in split transfer, and is effective for a command or a command group. The use of the operation identification ID number by this access ID field is simultaneous operation of a plurality of bus access ports, pipeline operation of a single DMA port, etc., and up to 16 types of multiplex operations can be designated per unit.

  In a normal bus command in which the second destination field 564 is not specified, an arbitrary operation identification ID number of the source unit may be specified in the ID field of the first destination field 560. In this case, the source unit stores the same operation identification ID number as the access unit ID, and when the reply command is received, the access ID field of the first destination field 560 and the operation identification ID number stored in the source unit. If they match, it is determined that it is a reply command for the bus command issued by itself.

  On the other hand, in the read process in the read mode 2 of FIG. 17, the second destination field 564 is specified like the second bus command from the home processor module 10-2 and the reply command from the remote processor module 10-3. When the source unit itself sends the bus command, the operation identification ID number stored in the bus command is compared with the access ID field of the second destination field 564 in the reply command. It is determined that the reply to the command is the reply command specified in the first destination field 560.

  Specifically, in FIG. 17, the local processor module 10-1 is waiting for a reply command for the bus command sent to the home processor module 10-2. A reply command corresponding to the second bus command sent to the processor module 10-3 is returned from the remote processor module 10-3 instead of the home processor module 10-2.

  At this time, the access ID field of the second destination field 564 is compared with the operation identification ID number stored at the time of sending the bus command in the local processor module 10-1, and if they match, this corresponds to the home processor module 10-2. The reply command is received as if it were a reply command for the bus command sent by itself.

  As a result, the local processor module 10-1 which is the original request source does not exist in the main memory 28 of the home processor module 10-2 and is not aware of the remote processor module 10-3. The latest data existing in the cache unit of the processor module 10-3 can be received.

  At this time, in the local processor module 10-1, a time T1 (first time) considering the reply from the home processor module 10-2 is set as a bus timer for monitoring the reception time of reply data. Since the reply is made by sending the second bus command from the home processor module 10-2 to the remote processor module 10-3, the time of the bus timeout monitoring timer in the local processor module 10-1 is The bus transfer time-out is monitored by changing to a longer time T2 (second time) in consideration of the processor module 10-3.

  The time change of the bus time-out counter may be performed by monitoring the sending of the remote command 250 sent from the home to the remote by the local processor module 10-1. Of course, it is also possible to change the time using the tag information line 509 shown in FIG.

  Further, for the return of the reply command as reply data 260 from the remote processor module 10-3 in FIG. 17, the bus command as the remote command 250 from the home processor module 10-2 triggered by the reply command. The operation identification ID number of the access ID field in the first destination field 560 is designated as it is.

  The basic bus command in FIG. 55 has a subsystem expansion designation flag 550 at the head. This subsystem expansion designation flag 550 is set to 1 when data is transferred between units of different subsystems as shown in FIG. 52. As a result, in addition to the basic bus command of FIG. The second word of the bus command for subsystem expansion is created. The second word for subsystem expansion will be clarified later.

  In the basic bus command of FIG. 55, a broadcast designation flag 558 is provided at the boundary between the source field 552 and the first destination field 560, and is set to 1 when used as a broadcast bus command. A second destination designation flag 562 is provided at the head of the second destination field 564 and is set to 1 when the second destination field 564 is used.

  A command field 566, a size field 568, and a flag field 570 are provided in the 5th to 8th byte portions of the second line of the basic bus command. In the command field 566, command identification codes such as a read command, a write command, a reply command, and a broadcast command are set. A data transfer size is set in the next side field 568.

  In the read bus command, the data size returned by the reply command is specified, and in the write bus command, the data size of the write data following the bus command is specified. In the next flag field 570, a command modification flag and an attribute flag are provided as necessary. The last command parameter field 572 specifies, for example, an access address depending on the command type of the command field 566.

  FIG. 56 shows a command format of a broadcast bus command used in the present invention. In the broadcast bus command, for the first destination field 560 provided following the source field 552, a 16-bit wide broadcast destination map field 574 is provided instead of the destination unit ID in the basic bus command of FIG.

  Since a maximum of 16 units can be connected to the system bus 12 of the present invention, the broadcast destination map field 574 can specify the unit that is the counterpart of the broadcast command by bit correspondence. The broad course bus command has the same format as the broadcast bus command.

  FIG. 57 is a command format of basic reply and error reply bus commands used in the system bus of the present invention. This basic reply and error reply bus command is characterized in that fields of a reply type 574, an error level 576, an error type 578, and an error code 580 are provided in the second line portion. The other structure is the same as the basic bus command of FIG. The last command parameter 572 field is not used.

  In the reply type field 574, the command code of the bus command issued by the source unit that requested the reply is specified. The format of the additional information of the reply command and the presence / absence of data are determined by the command code in the reply type field 574.

  The next error level field 576 stores an error level when used as an error reply bus command. In the next error type field 578, an error type indicating the error classification at the time of error reply is notified. Further, the error code field 580 is notified of the detailed cause of the error when the error is returned.

  FIG. 59 is an example of the error type field 578 of FIG. 57, and software error, system configuration error, hardware error, and bus error classification information is set based on error detection. FIG. 60 shows the detailed classification of errors by the error level field 576 of FIG. 57 in combination with the error type.

  Here, “nnn” in FIG. 60R> 0 differs depending on the type of error. “Xx” represents a mirror unit of 00 as a source (local), 01 as a first destination (home), 10 as a second destination (remote), and 11 as a first destination (home). “R” is 0 for writing and 1 for reading.

  The broadcast bus command shown in FIG. 56 is for cache invalidation in the write mode 1 of FIG. 23, the write mode 2 of FIG. 24, the write mode 3 of FIG. 25, the write mode 4 of FIG. Used as a bus command.

  FIG. 58 is a command format of a cache status reply bus command used in the system bus of the present invention. In this cache status reply command, a cache subline status field 582-1 is provided following the source field 552 and the first destination field 560, and the cache subline status field 582 is also added to the last 4 bits of the second line. 2 is provided.

  In the cache coherence device of the present invention, the 64-byte cache line is divided into 16-byte cache sublines. Therefore, the cache sublines are represented by # 0 to # 3, and each subline is assigned 3 bits by a combination of the cache subline statuses 582-1 and 582-2, and the cache subline to be accessed by the reply bus command for the cache bus command is assigned. Displays the current state of.

  That is, bit 0 and bit 1 of subline IDs # 0 to # 3 are assigned to the cache subline status field 582-1, respectively, and the second cache subline status field 582-2 in the second row is assigned the second bit of bit 2. The third bit is assigned. The cache subline state by the cache subline status fields 582-1 and 582-2 is as shown in FIG.

  FIG. 62 shows a command format of the subsystem expansion bus command created as the second word of the bus command when the subsystem bus expansion identification flag 550 of FIG. 55 is set to 1. This subsystem expansion bus command provides a source system field 586 on the first line, and specifies the source system ID, source bus ID, and source unit ID fields in which the source unit is provided.

  For the second row, a subsystem first destination field 558 is provided, and each field of a destination system ID, a destination bus ID, and a destination unit ID is provided. Subsequently, a subsystem second destination field 590 is provided, and similarly, fields of a destination system ID, a destination bus ID, and a destination unit ID are provided. A second destination designation flag 592 is also set. 62 is combined as the second word with respect to the first word of the basic bus command of FIG. 55 to transfer data between the units of the plurality of subsystems shown in FIG. be able to.

  As described above, the system bus 12 as the second common bus used in the cache coherence apparatus of the present invention is a common bus for efficiently realizing cache coherence in a distributed shared memory multiprocessor system according to the directory system. It is possible to provide a common bus that does not impair the expandability of the directory system.

In addition, it is possible to provide a highly reliable common bus considering expansion to a large-scale multiprocessor system in which a plurality of subsystems constructed in units of system buses are combined. Furthermore, since consideration is given to connection with different buses, a common bus that enables effective use of existing bus assets can be provided.

7). Second Directory Memory FIG. 63 is another embodiment of the present invention and is characterized in that a second directory memory is provided in the bus connection unit for the system bus of the processor module.

  63, the processor module 10-1 issues a home command 600 to the main memory 28 of the home processor module 10-2 as a local to perform write access, and the processor modules 10-3 and 10 as remote from the home. -4 shows a case where a cache invalidate remote command 602 is issued.

  Each of the processor modules 10-1 to 10-4 is provided with a local storage 28 and a directory memory 30 as a distributed main memory for the memory management unit 26, which is the same as the embodiment of FIG. In addition to this, the second directory memory 31 is provided in the bus connection unit 32 in the embodiment of FIG. The directory memory 30 is called a first directory memory with respect to the second directory memory 31.

  The second directory memory 31 registers the access state of data in cache line units existing in its own local storage 28 by another bus command on the system bus. As this registered content, for example, there is a cache invalidation waiting by a remote command. The processor module 10-2 in FIG. 63 issues a remote command 602 of cache impari data to the processor modules 10-3 and 10-4 as a home, and is waiting for a command response. At this time, for the corresponding cache line in the second directory memory 31 of the processor module 10-2, waiting for a cache invalidation response from the remote is registered.

  In this state, for example, a write access home command 608 is issued for the same cache line from the second cache unit side of the processor module 10-3. In response to this home command 602, the bus connection unit 32 refers to the second directory memory 31 and immediately recognizes that it is waiting for completion of cache invalidation without sending it to the memory management unit 26, and returns a busy response as a reply command.

  In this case, since retrying on the system bus 12 is useless, the busy unit is notified of the busy and the retry is performed. As described above, the state inaccessible by the remote command can be determined by referring to the second directory memory 31. Therefore, the processing speed can be increased as compared with the case where the memory management unit 26 determines inaccessibility and responds with Vichy.

  When the cache invalidation is completed in the remote processor modules 10-3 and 10-4 and the reply command is returned, the registration of the second directory memory 31 is deleted, and the remote command 602 by the retry from the processor module 10-3 is issued. Accepted.

  In FIG. 64, a plurality of home commands 606 are issued from other processor modules to the processor module 10-2 as a home waiting for completion of the cache invalidation of the home in FIG. This is when it becomes full. If a remote command 602 is issued from the processor module 10-3 in this buffer full state, a buffer full busy response is immediately returned without referring to the second directory memory 31.

In this case, when a buffer full busy response is returned to the access request source cache unit, the snoop bus 22 and the cache unit are busy with the retry operation, so the remote command is retried on the system bus 12.

8). Others The secondary cache control module 35 provided in the processor element 14-1 of FIG. 4 manages the cache data of the secondary cache 38 with LRU, and when the secondary cache 38 overflows with the owner of new cache data, Cache data LRU eviction processing is performed. This LRU eviction process issues a copyback command to the memory module 25 via the snoop bus 22 when the cache line targeted for eviction is in an exclusive dirty state where the write right is owned, that is, the owner. Then, the evicted cache data is stored in the local storage 28 of the processor module serving as the home.

  On the other hand, when LRU eviction of a cache line in the exclusive dirty state EX & D in which write ownership exists, if cache data exists in a shared modified state on another cache connected via another snoop bus After deleting the evicted cache data, a reply command RP is sent to the other cache units to switch from the shared modify state SH & M to the exclusive dirty state EX & D and transfer the write ownership. In this case, the owner of the cache data evicted by the LRU to the local storage 28 is unnecessary.

In the above embodiment, the case where four processor elements are provided in one processor module is taken as an example, but it is sufficient that at least one processor element is provided in the processor module. Further, the number of processor elements can be appropriately provided as necessary. Furthermore, the embodiment is not limited to the embodiment of the processor module connected by the system bus, and any number of processor modules can be connected as long as it is two or more.

Principle explanatory diagram of the present invention Block diagram of multiprocessor module configuration of the present invention Block diagram of the internal configuration of the processor module of FIG. Detailed block diagram of internal configuration of processor module of FIG. Explanatory drawing of the cash line Functional explanatory diagram of the space identification unit of FIG. Block diagram of the memory management unit of FIG. Illustration of directory entry register Explanation of shared map bit compatible register Explanation of processor module configuration register Illustration of specific examples of directory entry register, shared map bit compatible register and processor module configuration register Illustration of a specific example of the directory entry register Command explanatory diagram for the snoop bus of FIG. Explanatory drawing of the cache state of the cache unit of FIG. Explanatory drawing of the cache transition state of the cache unit of FIG. Explanatory drawing of processing operation in read mode 1 of the present invention Explanatory drawing of processing operation in read mode 2 of the present invention Explanatory drawing of processing operation in read mode 3 of the present invention Flowchart of read processing of local processor module of the present invention Flow chart of read processing of home processor module of the present invention Flowchart of read processing of remote processor module of the present invention Flowchart of processor element read processing of the present invention Explanatory drawing of processing operation in writing mode 1 of the present invention Explanatory drawing of processing operation in writing mode 2 of the present invention Explanatory drawing of processing operation in write mode 3 of the present invention Explanatory drawing of processing operation in writing mode 4 of the present invention Explanatory drawing of processing operation in writing mode 5 of the present invention Explanatory drawing of processing operation in writing mode 6 of the present invention Explanatory drawing of processing operation in writing mode 7 of the present invention Flow chart of write processing of local processor module of the present invention Flow chart of write processing of home processor module of the present invention Flowchart of write processing of remote processor module of the present invention Flow chart of write processing of processor of the present invention Explanatory diagram of operations in which internal command for cache unit and external remote command conflict and repeat retry Time chart of specific example of retry retry in FIG. Time chart of processing of the present invention that preferentially accepts a retry instruction for a remote command as one time Explanatory drawing of delegation of write ownership (owner) between cache modules of processor modules Explanatory diagram of the order to transfer write ownership Explanatory diagram of a system configuration in which the cache coherence device of FIG. 39 is an explanatory diagram of the line configuration of the system bus in FIG. Explanatory drawing of tag bus code by tag bus line of FIG. Explanatory drawing of control contents by combination of request line and emergency request line of FIG. Explanatory drawing of control contents by combination of halt line and master right switching line in FIG. Time chart of bus arbitration in fast mode of system bus in FIG. Time chart of bus arbitration in safety mode of system bus in FIG. Time chart of priority bus arbitration when an error is detected in the bus slave of the system bus in FIG. Time chart of priority bus arbitration when an error is detected by the bus arbiter of the system bus in FIG. 39 is a time chart of the memory bus transfer sequence of the system bus in FIG. 39 is a time chart of the memory bus transfer sequence of the system bus in FIG. Time chart of system bus broadcast / broad call transfer sequence of FIG. Time chart of transfer sequence between system buses in FIG. Time chart of transfer sequence between subsystem buses in FIG. Time chart of transfer sequence when the system bus in FIG. 39 is an 8-byte bus Time chart of transfer sequence between system bus of FIG. 39 and different system bus 39 is a diagram illustrating the basic bus command format of the system bus in FIG. 39 is a format explanatory diagram of the broadcast bus command of the system bus in FIG. 39 is a format explanatory diagram of a basic reply / error bus command of the system bus in FIG. 39 is a format explanatory diagram of the cache status bus command of the system bus in FIG. Explanatory diagram of error type specified by error bus command in FIG. Explanatory drawing showing the error level specified with the error bus command in FIG. 57 together with the error type Explanatory drawing of the cache status specified by the cache status bus command of FIG. 39 is an explanatory diagram of the format of an extended bus command used in addition to the basic bus command during bus transfer between subsystem buses in FIG. Write access explanatory diagram of the embodiment in which the second directory memory is provided in the bus connection unit Explanation of access when home access request becomes buffer full waiting for completion of cache invalidation in FIG. Block diagram of a conventional multiprocessor system

Explanation of symbols

10, 10-1 to 10-5: Processor module (PM)
12, 12-1, 12-2: System bus (first common bus)
14, 14-1 to 14-4: Processor element (PE)
16, 16-1 to 16-4: CPU circuit 18, 18-1 to 18-4: Cache unit (cache)
20, 20-1 to 20-4: Snoop unit 22: Snoop bus (first common bus)
24: Protocol management unit 25: Memory module 26: Memory management unit 28: Local storage (main storage divided)
30: Directory memory 31: Second directory memory 32: Module connection unit 34: CPU
35: secondary cache control module 36: primary cache 38: secondary cache 40: tag memory 42, 52: snoop transmission / reception unit 44: snoop buffer 46: copyback buffer 48: processor access control unit 50: space control unit 54: Local access buffer queue 56: Home access buffer queue 58: Remote access buffer queue 62: Bus reception unit 64: Bus transmission unit 66: Memory bus 68-1 to 68-4: Memory access controller 70-1 to 70-4: SRAM
74: Bus arbiter 76: Memory bus control unit 78: Directory control unit 80: Cache conversion unit 81: Local state machine 82: Home state machine 83: Remote state machine 86: CPU physical address space 88: Control register space 90: ROM space 94 : 36-bit physical address 95: Control table 96: Home information 98: Unit ID
100: Cache lines 102-1 to 102-4: Cache subline (subline)
104: Directory entry register 106: Shared map bit correspondence register 108: Processor module configuration register 120, 122, 124, 128: Cache status storage area 500-1, 500-2: Subsystem 501: Basic system 502, 503: Bus arbiter 504: Bus expansion unit 505: heterogeneous system 506: expansion system 507: data bus line 509: tag control lines 510-1 to 510-3: processor module 512: inter-subsystem connection unit 514: common data bus line (DB)
516: Data validity line (DP)
518: Common tag bus line (TB)
520: Bus request line (BRQ)
522: Emergency bus request line (BRE)
524: Bus Grant Line (BGR)
526: Bus ID line (BID)
528: Unit ID line (UID)
530: Holt wire (HLT)
532: Master Force Change Line (MFC: Master Force Change)
534: Installation line (INST)
536: Disconnect line (DCON)
538: Clock line (CLK)
550: System bus extension designation flag 552: Source field 558: Broadcast designation flag 560: First destination field 562: Second destination designation flag 564: Second destination field 566: Command field 568: Size field 570: Flag field 572: Command Parameter field 574: Broadcast destination map field 575: Reply type field 576: Error level field 578: Error type field 580: Error code field 582-1, 582-2: Cache subsystem status field 586: Source system field 588: Subsystem First destination field 590: Subsystem second destination field 592: Second destination designation flag

Claims (4)

A plurality of processor modules, a second common bus connecting the plurality of processor modules, each of the processor modules,
Multiple processors,
A cache unit provided for each processor;
Local memory used as main memory;
A memory management unit for managing the main memory and the directory memory ;
A first common bus to connect multiple cache unit and memory management unit,
A protocol management unit that manages the pre SL protocol conversion of the first common bus and a second common bus,
A bus connection unit that couples the processor modules with the second common bus;
A snoop unit to implement cache coherence by the first Common bus Death Knoop,
A cache conversion unit that is provided in the memory management unit and realizes coherence between processor modules based on the reference of the directory storage unit by the second common bus;
A second directory storage unit is provided in the bus connection unit, and in the second directory storage unit, a state waiting for a response to an access request to another processor module is registered in units of cache lines, and access from other processor modules is performed. when recognizing access the same cache line in the reference of the second directory Symbol憶部hand, the cache coherence apparatus characterized by responding to busy.
When the second processor module as a home that has received write access from the first processor module as a local requests the remote third processor module to invalidate the same cache line, the remote processor waits for the completion of the invalidation request to the remote. 2. The cache coherence apparatus according to claim 1, wherein a state is registered in the second directory storage unit, and busy is responded to an access request for the same cache line from another processor module.
3. The cache coherence apparatus according to claim 2, wherein the busy response is transmitted to a cache unit in a requesting processor module to perform a retry.
When the buffer connection of the bus connection unit becomes full due to an access request of another processor module to the home second processor module, the bus connection unit of the processor module that is the access request source responds with a busy response, and the second common bus The cache coherence apparatus according to claim 2, wherein a retry is performed on the.

Images