Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
  • G06F13/38—Information transfer, e.g. on bus
  • G06F13/40—Bus structure
  • G06F13/4004—Coupling between buses
  • G06F13/4027—Coupling between buses using bus bridges
  • G06F13/405—Coupling between buses using bus bridges where the bridge performs a synchronising function
  • G06F13/4054—Coupling between buses using bus bridges where the bridge performs a synchronising function where the function is bus cycle extension, e.g. to meet the timing requirements of the target bus

Definitions

• This invention is related to communication between computer buses, and more particularly to bus-to-bus bridge devices.
• Busses may now be found both within an integrated circuit (IC) processing unit and connecting the processing units to other agents such as external memory and peripheral devices.
• IC integrated circuit
• agents including peripheral devices and processors
• Many modern applications of computer systems therefore rely on multiple-bus architectures having a number of physically separate buses to further expand their functionality.
• a bridge may include hardware (digital hardwired circuitry), software (high or low level commands and instructions to be executed by one or more processors), and firmware (software typically stored in different types of read-only memory) or combinations thereof, that monitor and control data traffic between at least two physically separate buses.
• the bridge interfaces one bus protocol to another to facilitate communication between agents on the different buses.
• the bridge that couples two buses is typically configured to be transparent, so that the physically separate buses may be treated by the agents and the system as one bus.
• an address space is shared by agents on both buses. Requests (a read or a write) bearing an address range within the shared address space are generated by an initiator agent on an initiating bus.
• the bridge recognizes the address range and can forward the request to a target agent on a target bus. The bridge may thus be said to automatically perform the request on the target bus on behalf of the initiator agent.
• PCI Peripheral Components Interconnect
• SIG PCI Special Interest Group
• the PCI SIG also maintains a bridge architecture described in PCI-to- PCI Bridge Architecture Specification, Revision 1.0, April 5, 1994.
• the PCI bridge is also often referred to in this disclosure to illustrate some of the principles behind and operation of the various embodiments of the invention. However, those principles may also be applied to other bridge designs.
• Transactions are defined here as complete transfers of data between an initiator and a target, where the initiator and target are on different physical buses coupled by a bridge.
• bridges When forwarding data from one bus to another, bridges typically implement a number of data queues to hide the delay associated with requesting and obtaining access to the target bus for obtaining or forwarding the data.
• Each transaction is typically assigned a logical queue which is released when the transaction is completed.
• the queue will typically be part of a memory or buffer that implements a First-In-First-Out (FIFO) data structure.
• the FIFO is a data structure from which items are taken out in the same order they were put in. It is also known as a "shelf", from the analogy with pushing items onto one end of a shelf so that they fall off the other.
• the FIFO may be written to and read from simultaneously.
• a FIFO in a bridge is useful for buffering a stream of data between an initiator and a target which are not synchronized, i.e., not sending and receiving at exactly the same rate.
• a transaction as defined herein involves a request from an initiator to read from or write to a given address range which is claimed by a target. If the request is accepted by the bridge, then the transaction begins and an appropriate access is started.
• An access typically includes an idle phase for setup, an address phase during which address information for the particular request is exchanged, and sometimes a data phase during which data is exchanged.
• the request may be denied by the bridge.
• the bridge issues a termination known as a retry signal to the initiator. This may occur if the assigned bridge queue is full or has no data to transfer.
• the new request may be denied if there are no free queues available to be assigned, where all the queues are being used for other pending transactions. If the request is denied, the initiator may repeat the request to complete an ongoing transaction or attempt to start a new one.
• the access may be prematurely terminated by either the initiator, the target, or the bridge, for various reasons. If this happens, the request may be repeated or a subsequent request may be issued by the initiator to complete the transaction and transfer all of the requested data.
• Splitting the transaction so that the desired data is transferred in multiple accesses introduces increased overhead in the form of additional accesses having additional idle and address phases.
• the increased overhead can reduce throughput, where throughput is the amount of data transferred across the bridge per unit time, averaged over a given period.
• latency is defined as the time needed to provide the initiator or target with the first data block of a multiple-block transaction.
• the write transaction is typically performed as a posted write transaction in the PCI model.
• the initiator transfers data into a queue in the bridge after the bridge accepts the initiator's request.
• the bridge requests control of the target bus and after receiving control forwards the data from the queue to the target.
• the transaction is completed on the initiating bus before being completed on the target bus.
• Write transactions include the typical memory write, and the memory write and invalidate (MWI).
• MWI memory write and invalidate
• the two write transactions differ in that MWI must be for an integer number of cache memory lines, whereas the plain memory write can be used to write smaller amounts of data.
• the buses on either side of the bridge will be kept busy for a much longer time with MWI transactions than with plain memory writes.
• This may unnecessarily tie up the target bus that has agents which are not configured to respond to MWI requests.
• the initiator may be a newer generation peripheral device that is plugged into an older multiple-bus computer system having older generation targets, where the initiator supports MWI but its target does not.
• the software in the initiator could be modified to not issue the MWI and instead use plain memory writes to perform transactions aimed at targets which don't support MWI.
• Such a change will need to be implemented on each device and may present a cumbersome task for the system operator as many new devices are added over the lifetime of the system. Therefore, it would be desirable to have a technique for handling MWI requests in a multiple-bus computer system without having to modify the software in each new device that may be added to the system over its lifetime.
• the exact amount of data that the bridge reads over the target bus is not specified, but rather may depend on the particular PCI command type and whether the memory address space to be read from is prefetchable or not. While the initiator knows the exact amount of data it needs to read, it cannot specify this amount under the PCI model.
• the bridge in response to a read request reads and stores data from the target up to a fixed and predetermined number of blocks, or until the assigned bridge queue is full. This speculative operation on behalf of the initiator is done in anticipation of any subsequent or repeated read requests from the initiator. Upon arrival of the repeated read request, the read data begins to flow to the initiator from the bridge queue, but can be stopped by the initiator at any time. Any data in the queue which the bridge had read from the target but which is not transferred to the initiator is then discarded.
• a drawback of this read prefetching scheme is that it wastes valuable bus time keeping the target bus busy.
• the initiating and /or target buses may be occupied or busy while data is transferred into or out of the queue in the bridge. While occupied, a bus normally cannot be used by other agents. If some of the prefetched data in a read access is subsequently discarded, the target bus was kept busy without resulting in any data transfer through the bridge. Therefore, an optimization scheme is desirable to help reduce the time during which the target bus is unnecessarily kept busy in this way.
• Another area of bridge performance optimization lies in controlling the rate of data transfer between the initiator and the target.
• the initiator and target agents on either side of a bridge are allowed to "throttle" the rate of data flow between them. Throttling occurs when either agent requests and obtains wait states from the bridge while data for a transaction is being stored in the bridge. The wait states reduce the rate at which data is accepted into or removed from the bridge.
• the invention is directed at a bridge for coupling a first computer bus to a second computer bus.
• the bridge has first and second bus interfaces, a queue for temporarily storing data to be transferred from one bus interface to the other as part of a transaction, and a control register that represents one or more storage conditions to be met by the queue while performing the transaction.
• Figure 1 illustrates part of a computer system having a bridge according to an embodiment of the invention.
• Figure 2 is a flow diagram of the steps performed by a bridge in implementing a queue non-full state according to another embodiment of the invention.
• Figure 3 is a flow diagram of the steps performed by a bridge in implementing write request control according to another embodiment of the invention.
• Figure 4 shows the steps performed by a bridge in implementing read completion control according to another embodiment of the invention.
• Figure 5 is a block diagram of an intelligent I/O subsystem featuring a performance-tunable bridge, according to another embodiment of the invention.
• Figure 1 illustrates an embodiment of the invention as a computer system having first and second buses 134 and 138 coupled by a bridge or bridge device 100.
• the bridge 100 includes a number of logical queues 110, 114, 118, and 122 that are used to buffer data transfers between the two buses. Each queue is used on a per transaction basis in that each new transaction when initiated is assigned a separate queue.
• the bridge 100 also has a first bus interface 104 and second bus interface 108 which couple the queues to the respective buses.
• a control register 130 is provided which identifies trigger points according to the different embodiments of the invention which are more fully described below.
• the first bus 134 is taken to be the initiating bus where a transaction is initiated by the initiator 140, while the second bus 138 is the target bus coupled to the target 144.
• a host processor 148 coupled to the first bus 134 may also be used to execute a program which configures the control register 130 with the appropriate values.
• This first embodiment of the invention is based on the idea that it is not desirable to completely fill a bridge queue during a write transaction if doing so prevents the bridge from accepting any data in subsequent write requests, or prevents accepting such data in a single access. If the bridge is forced to reject such data because its write queue for that particular transaction is full, then simultaneous flow into and out of the bridge is halted. This reduces throughput if, in addition, the bridge cannot move data already in the queue to the target, perhaps because the target bus or agent is busy Throughput is also reduced due to the overhead of multiple accesses needed on the initiating bus if the data from a subsequent request cannot be accepted in a single access because the queue had become full.
• the bridge 100 is configured with a "non- full" state of the write queue 110.
• the non-full state is a condition of the queue 110 between completely empty and completely full.
• the queue 110 must be in the non-full state, i.e. have a minimum amount of free space (one or more data blocks as specified in the control register) available before a write request may be accepted by the bridge.
• the state may be defined in the control register 130 in response to a user command or configuration program executed by the host processor 148 on either bus 134 or bus 138. For example, two bits may be used in the control register to encode up to four different non-full states, as shown in Figure 1.
• the bridge 100 is configured (in hardware and software, as described earlier) to perform the operations in Figure 2.
• a request for a write transaction from the initiator 140 is received over the initiating bus 134 and interface 104.
• the bridge then assigns a queue 110 to the transaction and begins an access to fill the queue with write data from the initiator.
• the initiator may terminate the access and resume the transaction at a later time by issuing a subsequent write request.
• Step 208 When a subsequent write request is received for the same transaction, operation continues with block 208 where the control register 130 is checked to identify the non-full state for the queue 110. Step 208 may be performed in response to detecting each write request, or may be performed any time earlier or after the transaction is started, depending on the particular implementation. In either case, operation continues with step 212.
• step 212 a comparison is made to determine if the queue 110 is in the non-full state.
• the non-full state is a storage condition of the queue, other than completely full or completely empty. If the write queue 110 has reached in the non-full state, the queue is deemed "full" and operation continues with step 220. There, the bridge signals a retry in response to the subsequent write request, and the request is not accepted. Meanwhile, the data in the queue may be transferred to the target 144 independent of requests by the initiator.
• step 216 If, however, the write queue 110 is emptier than the non-full state by the time the subsequent request is received, the queue is "non-full", and operation continues instead of step 220 with step 216. There, the bridge is allowed to accept the write request. If accepted, then a write access over the initiating bus may begin in step 220 and more data for the transaction may be transferred to the queue 110.
• the above-described subsequent requests may be repeated several times during a transaction, depending on the total amount of data to be written, the size of the queue, and the ability to transfer the data from the queue to the target. A benefit of operating the bridge in this way may be illustrated by the following performance tuning example.
• a computer such as a network server or a workstation may be designed with a bridge and the multiple-bus architecture of Figure 1. If there is an acceptable likelihood that, given the expected traffic patterns across the bridge and the data processing capabilities of the different agents on the buses, the target bus 138 will stay relatively busy and/or relatively large burst transactions (having a relatively large amount of data) are requested on the initiating bus 134, then the non-full state of the queue is set closer to empty. This means that the queue 110 must be close to empty before the bridge accepts a write request. In this way, there will be a greater likelihood that the queue will have sufficient space to store all of the write request data in a single data phase, thereby reducing overhead associated with packetizing or splitting the transaction into multiple accesses on the initiating bus.
• the non-full state In comparison, if the non-full state were set closer to full, then there will be greater likelihood that the queue will fill up prematurely (and thereby block or force the initiator to stop sending data) because of a busy target bus or a large amount of write data requested. On the other hand, if it is expected that the target bus will remain relatively quiet and /or that mostly small amounts of write data are requested, then the non-full state is set closer to full. In this case, the likelihood of the queue becoming completely full (and therefore blocking the initiator) is less (even though the non-full state is set closer to full), because either the target bus is fast enough to remove data from queue and /or only small amounts of data are to be transferred in each access on the initiating bus. At the same time, the initiator is allowed to use almost the entire capacity of the queue, thereby further reducing the likelihood of blocking the initiator by reason of a full queue.
• the predictions concerning the queue behavior may be determined based on bus traffic simulations or monitoring of actual traffic on the initiating and /or target buses, and the queue size. Indeed, simulation results have shown that varying the non-full state of the write queues in a bridge has a significant impact on throughput.
• This second embodiment of the invention focuses in part on how quickly the bridge 100 requests control of the target bus 138 in order to remove and forward write data from the queue 110 to the target 144.
• This embodiment is based on the idea that it is not always desirable to allow the bridge to request control of the target bus as soon as the bus becomes available.
• this embodiment of the invention may be useful for performance tuning of the latency of the bridge, in particular when the initiating and target buses have different bandwidth requirements, e.g., initiating bus is 64 bits wide while the target bus is only 32 bits wide.
• the embodiment in Figure 1 includes the bridge 100 having the queue 110 for temporarily storing data received from the initiator over the bus 134 and to be forwarded to the target over bus 138 as part of a write transaction.
• the control register 130 includes Memory Write Request Control bits which determine a trigger point when the target bus may be requested by the interface 108 for forwarding the data that was received from the initiating bus and had been stored in the queue 110.
• the bits may specify a non-zero amount of data (greater than one data block) that must be received and stored in the queue (as part of the write transaction) before the bridge 100 may request control of the target bus.
• two bits may be used to encode at least three different quantities of QWORDs in the queue, where each QWORD is a block of data having a width of 64 bits.
• the queue 110 may be the posted memory write queue in a PCI bridge.
• the write request control storage condition is set to a relatively small number of blocks of data. Data may begin to be transferred to the target from the queue as soon as the specified number of blocks have been received and stored.
• a lower storage condition may be beneficial is when the target bus is expected to be relatively busy, such that the queue might become full and block the initiator before the target bus can be acquired. Under those conditions, a lower storage condition will give the bridge a head start in emptying the queue, and may provide enough time for the bridge to acquire the target and begin unloading write data from the queue before the queue has become full.
• the write request control storage condition may be set to indicate a larger amount of data, an amount less than the full capacity of the queue. This will help ensure that a large amount of data is available to be forwarded to the target in a single access when the target bus is acquired, thereby improving throughput.
• Yet another situation in which it might be beneficial to set the write request trigger point to a larger amount is when the initiator 140 is likely to request delays during a transaction, for example in the form of wait states, from the bridge 100.
• FIG. 3 illustrates an exemplary method of performing a write transaction using the write request control mechanism in a bridge embodiment of the invention.
• the bridge 100 receives an initial write request from the initiator to begin a write transaction. Operation continues with step 308 where data begins to transfer into the queue 110 over the initiating bus 134.
• step 312 a determination is made as to whether the queue meets the write request control storage condition specified in the control register 130, i.e., sufficient data has been transferred to the queue. If yes, then in step 320, the target bus may be requested. When the target bus is subsequently acquired, the bridge begins to transfer the write data from the queue to the target. Meanwhile, fresh write data may continue to be received into the queue from the initiator.
• step 312 if the storage condition has not been met, then the bridge is not permitted to request the target bus, but may continue to receive fresh write data into the queue.
• the transfers into and out of the queue may otherwise occur independently of one another.
• the bridge 100 is configured to forward a MWI request on the initiating bus as a conventional memory write over the target bus if it is expected that the target bus couples agents, such as target 144 in Figure 1, which are not equipped to process the MWI request.
• agents include, for example, older generation memory controllers and bridges.
• the computer system operation may be optimized for the plain memory write transaction by configuring the control register 130 to enable the MWI conversion feature.
• logic circuitry in the bridge is directed to change the command portion of the MWI transaction into that of a plain memory write before performing the transaction on the target bus.
• a first embodiment of the invention dealing with read transactions permits the maximum size of data prefetched from the target during a read transaction to be adjusted by bits in the control register 130. This improvement may be particularly effective in improving bridge performance when the bridge is configured to discard an unrequested portion of the prefetched data that resides in a bridge read queue.
• the bridge After receiving an initial read request and signaling a retry on the initiating bus, the bridge determines the maximum data size to be prefetched by checking the appropriate bits in the control register 130. Only after the bridge obtains as much data as possible (up to the maximum amount stated in the control register 130) from the target and stores the data in the read queue 122 may the bridge accept a repeated read request on the initiating bus, and cause the transfer of the read data from the queue 122 to the initiator 140.
• the system operator or designer reduces the amount of unrequested data that may be subsequently discarded by adjusting the prefetch size in control register 130 to any number of data blocks.
• the bridge may thus be tuned to the particular application. For example, if the application is mass storage and the initiator is likely to request large burst read transactions from the target 144 being a magnetic disk storage controller, then the prefetch size may be set to a larger amount with some confidence that a relatively large amount of data will be subsequently requested by the initiator. The prefetch size should be adjusted to reduce the amount of unrequested data that will be discarded, thereby making more efficient use of the target bus, in view of the particular bridge application.
• the embodiment provides bits in the control register 130 that control the behavior of the initiating bus interface 104 for a delayed memory read transaction.
• the bits identify a read completion control storage condition for the queue 122 that determines when to allow the initiator 140 access to the read data stored in the queue 122.
• the control register 130 may be programmed to change the minimum number of data blocks that must be received from the target and stored in the queue 122 before the bridge may accept a repeated read request on the initiating bus 134 and return data from the queue 122. For example, two bits may be used to encode four different quantities of QWORDS (blocks of data 64 bits wide) that must be received and stored in the queue before a repeat request may be accepted.
• QWORDS blocks of data 64 bits wide
• the stored data may be delivered to the initiator regardless of whether or not the completion control storage condition specified in the control register 130 has been met by the queue.
• Figure 4 illustrates a series of steps to be performed by the bridge in implementing this embodiment of the invention.
• a first read request is received from the initiating bus 134 to begin a read transaction.
• the read transaction could be any one of memory read, memory read line (MRL) and memory read multiple (MRM).
• MRM memory read multiple
• the bridge After assigning a read queue 122 to the transaction (provided, of course, that a free read queue is available) and obtaining the read address information, the bridge then signals a retry on the initiating bus 134 in step 406.
• the bridge attempts to acquire the target bus 138. If successful, transfer of read data from the target 144 will start in step 408.
• the bridge may otherwise continue to attempt to acquire the target bus in order to begin the transaction on the target bus.
• step 412 When a repeated request is received on the initiating bus prior to completion of the read on the target bus, as in step 412, then a decision is made in block 416 as to whether enough read data has been received in the queue to meet the completion control storage condition specified in the control register 130. If not, then the initiator is signaled a retry in block 406. Meanwhile, data may continue to be transferred to the queue 122 from the target in block 408.
• the queue has received a sufficient amount of read data from the target to satisfy the storage condition, then the repeated request may be accepted and the read transaction may be completed on the initiating bus by attempting to transfer read data for the transaction from the queue to the initiator, preferably in a single read access.
• a user can tune bridge performance to better match the bridge to its surrounding environment. For example, consider a mass storage application such as data backup where the initiator is a disk controller seeking to read data from the target 144 which is a memory controller. The initiator in this case might not be concerned with latency, but rather would desire high throughput. Therefore, the user would configure the Memory Read Completion Control portion of the control register 130 to indicate a relatively large amount of data. In this way, more data may be transferred to the initiator in a single access to complete the transaction, at the expense of increasing latency.
• a performance-tunable bridge 100 featuring the control register 130 may be used as a component of an intelligent I/O subsystem 510 in Figure 5.
• the I/O subsystem 510 features a subsystem processor 512 and memory controller 514 on a third bus (local bus 516).
• the local bus is coupled to the primary 521 and secondary 522 PCI buses through respective address translation units (ATUs) 526 and 528.
• ATUs address translation units
• the I/O subsystem 510 may be implemented as a single IC and used as part of a system application such as a network server motherboard.
• the motherboard would include, in addition to the I/O subsystem 510 and the PCI buses 521 and 522, a host processor and memory coupled to the primary bus 521, and one or more network interface controllers coupled to the secondary bus 522.
• the network controllers may be depicted by several instances of the target 144 in Figure 1.
• the bridge 100 in the I/O subsystem 510 may be tuned as described above to optimize its latency in order to
• the embodiments of the invention described above are directed at a bridge and computer system, in which the performance of the bridge may be tuned in view of the depth of the data queues and the particular system application.
• the embodiments of the invention are, of course, subject to other variations in structure and implementation.
• the bits in the control register 130 may be programmable (read and write) or alternatively, one-time programmable (programmable read-only- memory), in-circuit programmable (such as in a flash memory), or even test bits (programmable during a test batch of the IC, but then fixed as read-only production).
• the control register 130 may be accessed by either the host processor 148 (see Figure 1) or by the subsystem processor 510 of the I/O subsystem 510 embodiment in Figure 5.
• control register 130 has two bits representing up to four different trigger point/storage conditions of the different data queues, a greater number of bits may alternatively be used, thus allowing finer granularity in the performance tuning of the bridge and the surrounding computer system.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Computer Hardware Design (AREA)
• Quality & Reliability (AREA)
• Bus Control (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (2)

Family

ID=21877567

Family Applications (1)

Country Status (6)

Families Citing this family (36)

Family Cites Families (15)

• 1998
  • 1998-03-04 US US09/034,624 patent/US6298407B1/en not_active Expired - Lifetime
• 1999
  • 1999-02-24 AU AU27935/99A patent/AU2793599A/en not_active Abandoned
  • 1999-02-26 KR KR1020007009689A patent/KR20010041519A/ko not_active Ceased
  • 1999-02-26 DE DE19983026T patent/DE19983026B4/de not_active Expired - Fee Related
  • 1999-02-26 WO PCT/US1999/004234 patent/WO1999045470A2/en not_active Ceased
  • 1999-02-26 JP JP2000534945A patent/JP2002506250A/ja active Pending

Also Published As

Similar Documents

Legal Events