KR20100098596A - Mesosynchronous data bus apparatus and method of data transmission - Google Patents

Abstract

The transfer time of data between memory modules describes a memory system in which the overall time delay between specific points in the memory system is managed to remain constant. Each lane of the multilane bus can be managed individually and the data frame can be evaluated in the destination module, without the need for deskew in the intermediate module. The time delay in the propagation of data through the module, which may have a switch to route the data, operates the data path through the module at one or more divisors of the bus serial data rate, and changes in time delay due to temperature changes or aging. Is reduced by selecting a sampling point of the received data to be accepted.

Description

Mesosynchronous data bus apparatus and method of data transmission

The present invention relates to the management of distributed clocks in a memory system.

In computer systems, a central processor (CPU) accesses program information and data stored in a memory system. There are layers of memory systems in terms of size, speed, and performance that computer system designers select during the design phase, which may include, for example, cache memory, main memory, and secondary memory. Cache memory is characterized by low latency, high bandwidth and high cost per bit, and can be integrated into the CPU. The cache memory may be a semiconductor device and may be, for example, static random access memory (SRAM). Main memory, also in semiconductor technology, typically in the form of dynamic random access memory (DRAM), is used for less frequently accessed data and program data. Currently, personal computers can have up to about 4 GB of DRAM, while high-end servers can have about 16 GB or more of DRAM. Strategies such as using multiple memory controllers and computer cores can provide access to larger amounts of such memory; However, most computer bus systems have practical upper limits due to propagation time, bus loading, power consumption, and the like. Larger amounts of data can be stored in mass storage. For example, a single disk may include terabytes (TB) of memory, and a FLASH memory disk magnetic disk (sometimes called a solid state drive (SSD)), and a copy of the disk may be used. The access time for data stored on the magnetic disk is sufficiently longer than the access time for data stored in the main memory.

Large amounts of other memory, such as DRAM or FLASH, can be provided for memory applications as described in US patent application Ser. No. 11 / 405,083, filed April 17, 2006. If a large memory array has a latency approaching that of conventional main memory, such a memory array can be considered similar to main memory and a large amount of data stored in other ways in mass storage, such as rotating disk media. Can provide quick access to

Data is moved between the central processor and other devices and memory on a path, known as a bus, which is, for example, parallel, serial, point-to-point, daisy chained, or multi- stub. It can have various structures, including (multi-stub).

The data bus can be operated in a synchronous manner if the clock frequencies for the transmission and reception of data are the same at all points in the system, and a known phase relationship is the data bits at each point at which data is sensed (eg received). Exists between. However, considering the transfer of data on a parallel bus between two adjacent nodes, with time-delay skew, the phase relationship of data bits on different lines changes. In low speed data transmissions, and for short bus lengths, this may be acceptable, but in high speed data transmissions, data bits may be received in a varying phase relationship to the system clock and delayed by more than one clock interval. This can result in errors or typically require de-skewing and phase alignment at each memory node.

This problem can be mitigated by sending clock and data on each line, and each at one node recovers the clock for the channel. This clock may vary in time delay for the system clock from line to line. In addition, when a clock is transmitted along with the data, a transfer of data, or at least an idle data pattern, may be required to keep the clock synchronized for each line.

Alternatively, the data accumulate data for each line in the buffer, determine the time delay adjustment required to compensate for data skew, and reconstruct the data received at each node prior to the action on the data. (Word data is understood to include in-band commands such as read or write as well as information, which includes instructions to be written to or read from memory.) To adjust skew-compensation, the amount of data that must be buffered can amount to the number of clock cycles of skew that can accumulate along the bus. The total delay in data transfer from one end of the bus to the other is the sum of each maximum skew of node-to-node skew values along the data path. An example of this type of bus is a fully-buffered DIMM (FB-DIMM), which is the subject of the JEDEC standard.

The bus may also operate as a multi-drop bus in which data is transmitted from a transmitting end (such as a memory controller) and received at a target memory module, eg, a third memory module along the line bus. The module may be a dual in line memory module (DIMM), as known in the art, and the maximum overall skew may be the same as that of a particular bus line with the longest transmission delay. The transfer delay results from the difference in trace lengths for the individual data lines, and the difference in transfer length may include traces on the motherboard as well as traces on the circuit card containing the memory modules.

As described in US application Ser. No. 11 / 405,083, for example, the effect of skew between lines of a bus operating in a serial point-to-point, or branched fashion is that skew between each of the communication nodes is experienced on the bus line. Depending on the amount of λ, it can be mitigated by properly exchanging the logical allocation of data lanes to specific bus lines so that at the node at the intended receiving end of the bus, skew is minimized, known or controlled. As such, the amount of correction required for skew compensation at the receiving node that may affect the data may be reduced, accompanied by a reduction in device complexity and power consumption.

Included in the context of the present invention.

An interconnect system is disclosed that includes a first node and a second node in communication with the first node. A first clock is provided to the first node and the second node; The second clock has a first integral relationship with respect to the first clock, is generated with reference to the first clock, and the transmission time of the bit between the output of the first node and the output of the second node is substantially constant. Have a time delay offset relative to the first clock, which is adjusted to remain stable.

A data transmission system is disclosed comprising at least two modules connectable by a data bus. The module has a transmitter for transmitting serial data and a receiver for receiving serial data. The module is supplied with a signal from a common clock and has a clock generator that generates an internal clock on each module derived from the common clock. The clock data recovery circuit generates a received data clock that keeps in sync with the bits of the serial data signal, and the alignment buffer is operable to establish and maintain synchronization between the serial data bits and the internal clock. The switch is operable to route the received serial data to either an external port or an internal port. The module may have a plurality of external and internal ports.

In another aspect, a memory system includes a plurality of modules connected by a link, wherein at least one module has a data memory. The system clock is divided into at least two modules, and the data clock rate of the data transferred between the modules is completely related to the system clock. The buffer on the module is operable to maintain synchronization between the bit position of the received data character and the previously set bit position.

In another aspect, the data interface includes a clock data recovery circuit; A clock phase alignment buffer, and a data transmission circuit. The clock data recovery circuit recovers a clock having the same frequency as the clock used in the data transmission circuit, and the phase alignment circuit compensates for the change in the time of arrival of the data.

In another aspect, the memory module has a data receiver, a data recovery circuit, a routing switch, a memory interface, and a data transmitter. The clock data recovery circuit recovers a clock having the same frequency as the clock used in the data transmission circuit, and the phase alignment buffer circuit compensates for the arrival time of the data.

In another aspect, a node of an interconnect system may comprise data receiving circuitry; Data transmission circuits; And a switch connecting the data receiving circuit and the data transmitting circuit. The circuitry is operable to maintain a sampling time in a fixed relationship with the first bit of the received character having a plurality of bits; The input data buffer is configured to resample the first bit such that the overall time delay measured between the resampled bit and the corresponding bit transmitted by another module remains substantially constant.

A method of transmitting data between modules is provided, the method comprising providing at least two modules, the module comprising a receiver, a transmitter, a clock data recovery circuit, a phase alignment buffer, and a routing switch. The module is connectable by line. The characters of the frame of data are assigned to the first line. The first line at its receiving end is initialized such that when the time delay between the transmitted character and the reception of the character changes, the first bit of the character is sampled to maintain the alignment of the sampling of the first bit of the character.

In another aspect, a method of managing an interconnect system is described, wherein the system comprises a plurality of nodes in communication with each other. The method includes transmitting a character of data comprising a plurality of bits from a first node to a second node; Receiving a text at a second node; Recovering a clock from the received data and aligning the sampling time with the first bit of the character; And resampling the sampled data by a divisor of the clock frequency and adjusting the phase or time delay of the resampling clock such that the overall time delay between the transmitted data and the resampled data is substantially constant.

Included in the context of the present invention.

1 is a block diagram of a computer system having a memory module;
2A, 2B illustrate a naming convention regarding the logical and physical aspects of an embodiment;
3 is a simple block diagram of a memory module;
4 is a more detailed block diagram of the memory module of FIG. 3;
5 shows an example of the logical arrangement of data bits at the input and output of the deserializer circuit;
6 shows an arrangement of functional elements in a switch of a memory module;
7A and 7B show timing schemes associated with intermediate synchronous operation.

Exemplary embodiments may be better understood with reference to the drawings, but these examples are not intended to have properties that limit the invention. Elements of the same number in the same or different figures perform equivalent functions. The elements may be marked or numbered by acronyms, both are possible, and the choices in the representation are made for clarity only, the elements indicated by numbers and the same elements indicated by acronyms or alphabetic markers. Not distinguished by basis.

Those skilled in the art will appreciate that the methods and apparatus depicted in the figures may be configured or implemented in machine-executable instructions, for example software, or hardware, or a combination thereof. Instructions may be used to cause digital processors or the like, programmed as instructions to perform the described operations. In the alternative, the operation may be programmed by a specific hardware component that includes hardwired logic and firmware instructions, or may include analog circuitry, or user specified programming to perform the described operation. (custom) may be performed by any combination of hardware components. For example, a microprocessor, a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC) may be used. Such circuitry may have integrated or associative memory to store any necessary instructions or data.

A computer program product that can include a machine-readable medium having stored instructions that can be used to program a computer (or other electronic device) to perform a method, wherein the method can be provided at least in part. For the purpose of this specification, the term “machine-readable medium” may store or encode a sequence of data or instructions for execution by a computing machine or special purpose hardware, and the machine or special purpose hardware may be used to perform the functions of the present invention. Or any medium to perform any one of the methodologies. The term “machine readable medium” thus includes, but is not limited to, solid state memory, optical disks and magnetic disks, magnetic memory, optical memory, and carrier signals. The term carrier signal is understood to include instructions and electronics necessary to generate or receive such a signal, whether an electrical signal having instructions or data introduced thereon is radiated or conducted.

For example, a machine-readable medium may include read only memory (ROM); Any type of random access memory (RAM) (eg, SRAM, DRAM); Programmable read only memory (PROM); Electronically alterable read only memory (EPROM); Magnetic random access memory; Magnetic disk storage media; Transmissions using flash memory and electrical, optical, acoustical or other forms of signals.

In addition, referring to software in one form or another (eg, a program, procedure, procedure, application, module, algorithm or logic), as it takes action or causes an effect, is common in the art. to be. This representation is merely common in that the execution of software by a computer or equivalent device causes the processor of the computer or equivalent device to perform an action or produce a result, as is known to those skilled in the art.

When describing specific examples, the examples may include specific features, structures, or properties, but not all examples necessarily include specific features, structures, or properties. This is a suggestion or suggestion that two or more examples of features, structures, or portions of a feature may or may not be combined, and should not be taken unless such combination is expressly excluded. Where specific features, structures, or features are described in connection with the examples, one of ordinary skill in the art, whether or not explicitly described, may implement such features, structures or features in connection with other examples.

In the interest of clarity, not all conventional features of the implementations disclosed herein are described. In the development of any such practical implementation, a number of implementation specific decisions must be made to achieve the developer's specific goals, such as compatibility with system and business constraints, and these goals are another in the implementation of the HA. It will of course be understood that this will change to implementation. Furthermore, while such development efforts can be complex and time consuming, it will nevertheless be understood that this is a common task of engineering for those skilled in the art, having the advantages of the present invention.

As described herein, a connector or connector interface, such as a memory module connector interface, is not limited to a physically separate interface where a male connector or interface engages with a female connector or interface. The connector interface may also include any type of physical interface or connection, such as a connection from a memory circuit, a solder ball or lead in which the lead is electrically connected to another memory circuit or circuit board. For example, in a stacked die approach, multiple integrated circuit dies (e.g., memory devices, or buffer devices, etc.) may be connected via a ball grid array or other type of connector interface to a memory module, memory controller, or the like. The substrates forming the base and the interface in the processor may be stacked on top of each other. As another example, the memory device and buffer device may be interconnected via an interface and a flexible tape interconnect to the memory controller through either a physically detachable socket type connector interface or a ball grid array type connector interface. The connection type may include an interface between the substrates, the substrates, the interconnect conductors on the printed circuit board, or the integrated circuit chip.

Although the examples are described in the context of a memory module in a computer system, it is not intended to limit the scope of the present application to such a local system. The apparatus and techniques described herein can be used in data communication systems in which the modules are physically separated and in which data transmission techniques using wireless techniques or the like can be used, in whole or in part. Similarly, a plurality of memory modules may be fabricated or assembled on a common substrate, with interconnects therebetween. The choice of physical implementation depends on engineering and economic considerations at the time the product is designed.

1 and 2 are provided to identify symbols and terms that may be used in the described examples. This is done for convenience, clarity, consistency and brevity, and those skilled in the art will recognize that various equivalent terms may be used. However, for the purposes of the description provided herein, the terminology described will be used except where specifically noted. It is not intended to exclude or distinguish between terms that may be used in the future that describe equivalent concepts, or terms that apply to the same.

1 illustrates an example of a system including a central processing unit (CPU), a memory controller (MC), and a memory module (MM) and a system clock (SYSC). Other aspects of the computer, such as external interfaces, mass storage, displays, and power supplies, which may also be provided, are not shown. In addition to cache memory, disk storage memory, tape memory, and other forms of storage, not shown in FIG. 1, the system includes RAM attached for quick access to program instructions and data for use or adjustment by the program; It may have volatile and nonvolatile memory such as FLASH. This memory may be connected to the CPU by the memory controller MC and may include a plurality of memory modules MM.

The memory modules may be connected to the memory controller and to each other by a plurality of electrical connections, printed wiring connections, or lines, which are collectively referred to as buses, lines, or channels. In this example, several memory modules MM are shown connected in series, such as MM1 through MM2, and some MM are connected in a branched arrangement, such as MM3 and MM4 through MM2.

In an intermediate synchronous system, the system clock SYSC is distributed to a module (e.g., MM) in the domain such that a plurality of modules of a group of modules can have access to a clock source of a common frequency. Various clock frequencies may be derived from SYSC, which may be a product of, or a divisor of, the SYSC clock frequency. As such, clocks across a system of a certain product or fraction of the SYSC clock frequency have the same frequency, but may differ in time or phase offset from each other. This offset can be described as a phase offset or as a fraction of clock periods, where 0, 90, 180, 270 degrees of phase correspond to 0.0, 0.25, 0.5, and 0.75 of the clock period. The offset can change very slowly over time due to temperature or circuit aging. Slow will be understood to mean that a number of characters may be transmitted before a sufficient change in offset occurs that may require a change in compensation or alignment.

Because the total clock rate of data is on the same module, or on the same clock, the padding character need not compensate for the difference between the clocks on the other modules. Similarly, the use of synchronous characters on a frame-by-frame basis can be avoided. In another aspect, because clock rates are the same in multiple modules, gaps in preamble or data transmission can be avoided.

The data may be adjusted and the calculation may be performed using various clocks for the timing of the operation, the operation may be performed, for example, on the rising edge of the clock signal or on both the rising and falling edges of the clock signal. It is known that it can. The latter type of operation is commonly referred to as dual data rate (DDR) clocking. For the sake of simplicity, discussion has been made in terms of rise-edge clock operation, and those skilled in the art will understand the application for a system in which dual data rate clocking or the like is performed. Other clocking schemes are known and can be used.

2 defines a convention used to describe a connection or bus between two devices, which may be, for example, two memory modules (MM). Data can be said to flow in an upstream or downstream direction, where upstream is typically defined as towards the end of a channel having a management interface, such as a memory controller (MC) or a CPU, where downstream is In the opposite direction. The bus may be reconfigured such that the direction of upstream is downstream. Other terms such as northbound and southbound may be used.

 A bus, link or channel connects two devices for high speed data, address and command transfers. While there are various configurations of channels, typically channels can have one group of lines, which can use unipolar, bipolar, or other signaling techniques. The line can be unidirectional or bidirectional. In this example, a differential line is used, which may include a pair of traces on the motherboard, ending in the memory module of the differential signaling electronic circuit, as known in the art. (Electrical differential lines are shown as single lines in the figures.) Currently, this type of connection is used for high speed interfaces in current product designs, but any type of connection, including optical or wireless technologies, may be used. Can be used.

In one aspect, the channel may have a unidirectional group of lines. In an example, ten unidirectional lines are provided in which nine lines can be used at any time, and the tenth line may be redundant. Other numbers of lines and spares may be used. Extra lines are not required. Data may be transmitted on a channel in the form of a character that may have a fixed length in clock cycles, and the data bits of the character may be associated with a clock cycle. In this example, we use a 20 bit character length, relative to 20 clock cycles. Characters on a plurality of lines can be grouped into logical clusters and called frames. That is, as shown in FIG. 2A, a character on each active line may be associated with a character on another line that is transmitted concurrently with other character (s). However, the characters on the individual lines need not be bit-synchronized with each other in time, can be offset from each other in time by one or more bit periods, and the clock offset can include fractional bit periods.

The transfer of data between the first module and the second module can be controlled at the transmitting module and the receiving module can accept the data at an intermediate synchronous clock data rate.

Similar channels may exist in the reverse direction. In this discussion, the direction of signal propagation will be described in the downstream direction; However, it can be understood that signal propagation in the upstream direction will have similar characteristics.

The flow of characters from an upstream device, such as an MC or an MM, to a downstream device, such as an MM, may be referred to as a hop. In this discussion, for the sake of clarity, the transfer of data is between two memory modules MM without losing generality. The term line or lines is commonly used when describing the physical properties of a connection between two modules in a hop, and these lines are used, for example, on a printed circuit board, auxiliary elements such as connectors, and transceiver interface electronics. May be associated with the The term lane is commonly used to refer to the logical assignment of data in characters in a frame. Each lane may be assigned (bound) to a line by an operation such as MC, or MM, and this binding may change from hop to hop, and may change over time. In this discussion, the binding is considered static once the system is configured, but may be different in each module. That is, the association of lines and lanes at each hop is established at a previous time of system configuration and remains assigned during operation or until reconfiguration is performed. Lanes can be bound to other lines on different hops to manage and possibly reduce or minimize skew between the arrival times of the characters of the frame in the destination module.

A channel or link between two modules may be considered to exist when the module is initialized, trained, and when the lanes are bound to the line and the process of frame alignment is complete. This process can be called construction.

The group of lines that exist from the module in the downstream direction, which is connected by hops to another module, is called an output sub-port and entry into another module of such a line is through the input sub-port. Another similar group of lines, configured as sub-ports, may exist in the upstream direction, and a combination of output and input sub-ports associated with a hop between two modules may be called a port.

Characters in a data frame may include steering data, which generally refers to the addressing information needed to route the data in the frame, which is the destination module, on the next frame or on another line on the current line. It may be another frame or the current frame. This may be adjustment data as described in US 11 / 405,083. Other means of providing addressing or routing of data are known to the person skilled in the art and can be used. The same lane or multiple lanes, which may not include a steering lane, may include characters for commands, addresses, or data.

2B is an example of a physical interface between two modules, which may be memory modules (MM). Trace pairs on the motherboard may connect between connectors on the motherboard. Single or multiple connectors can be used to allow the MM to be plugged into the motherboard and connected to printed circuit wiring. For example, other connections such as power, power management, clock and check may also exist, but are not shown.

3 shows a simple block diagram of a memory module showing only a downstream path. Input port P ₀ has n data lines, in this example 10. In the case of a module for combining in a binary tree, there may be two output ports P ₁ and P ₂ , each also having 10 data lines. Depending on the information provided in the coordination data or by another routing method, the input data at port P ₀ is routed to one of the memories at the MM or to port P ₁ or port P ₂ .

A memory that may be integrated with the MM or that is in communication with the MM and may be any type of data memory described above may be associated with the MM. The memory may be replaced or supplemented by an interface to another computer or communication equipment, or other external device, such that the memory address may bring in or output data, for example, from the MM to a display, a network interface, or the like.

Devices that perform routing and other related functions may be referred to as configurable switching elements (CSEs). 4 shows a portion of the CSE. In this example, the MM circuit can operate at several different clock rates. Multiple clock rates can facilitate electronic implementation of the system; However, if a sufficiently fast electronic device can be obtained, all of the operations of the CSE can be performed at a common clock speed, which can be at the speed of the bus data rate. The system clock SYSC may be distributed from the common clock SYSC to each of the modules in which the system clock SYSC is used. The system clock (SYSC) rate may differ from the bit clock rate of the serial bit data on the line of hops, and as is known, a clock-rate multiplier or divider may be used in the module or elsewhere to derive a local clock. . The serial bit rate clock may be a multiple, m, or a divisor of the system clock SYSC, and part of the MM may be at another clock rate, for example; It can operate at a switch clock rate (SWC). In this example, the data rate clock rate DC is 16 times SYSC and the switch clock rate SWC is 4 times SYSC. Other clock frequencies may also be used in the module. Some of the clocks used may not be integrally related to SYSC.

One of the lines connecting the module MM is described as representing the process used on the other line. The input line at port P ₀ may be accommodated by or interfaced with an analog circuit that may provide impedance matching, which may include explicit or intrinsic bandwidth filtering, which The differential signal on the line can be converted into an electrical signal suitable for further data processing or manipulation. As mentioned, this discussion assumes that the module has been previously configured with respect to the remainder of the system, including determination of clock phase offsets or alignment of clocks; As such, the configuration process, error detection, error recovery, and the like are not discussed in detail.

As shown in FIG. 4, the output signal from the receiver circuit RX is processed by a clock data recovery (CDR) circuit. As can be used for switch circuitry (SW), a deserializer (DES) can be used to convert data from a serial format into multiple data streams so that data can be processed simultaneously at a low clock rate (SWC). have. CDR circuits are known as DLLs (delay locked loops), PLLs (phase locked loops), etc., to establish a recovery data clock (RDC) having a fixed relationship to bit positions in the received data. May be one of the circuit types. The recovery data clock RDC can be used to sample the signal on the line at the point in time of the received bit for which the data signal is valid. That is, the sampling time of the received data signal is aligned so that data samples are not taken at the boundary between bits, and the effect of distortion that can occur in the transmission on the hop is minimal. The RDC can also be aligned with character boundaries at initialization time.

The recovery data-rate-clock RDC is due to the phase offset (or at least fractional bit time) which is the differential delay time in the module because of the different propagation path taken by the clock signal SYSC from the common system clock to another module, and It may be offset from the locally generated data clock DC by other signal propagation delays for different lines between adjacent modules.

The difference in propagation delay time between modules may be called skew, and the skew may have fixed and variable components. Additional propagation time differential delays handled by each path on the module are also relevant and, when skew is described, means the total differential time delay, unless specifically specified otherwise. For example, the propagation time between modules may be a speed greater than about 1/2 of the speed of light, the propagation time of the signal along the trace on the motherboard, the transmitter of the previous module (TX) and the transmitter of the current module. Propagation delay in a filter at (RX), other buffers, and the like. Real analog electronic circuits have finite bandwidth, and such circuits operating near bandwidth limits can introduce additional propagation delays in the signal being processed.

Propagation delay can be considered as either phase delay or time delay. The analog circuit may have a time delay of an order of clock period, and the delay may be a function of component value, which may depend on temperature. Some obsolescence may occur, but this is generally on a long time scale. Therefore, the phase or time relationship between the incoming first bit of the character and the corresponding module clock edge may not be known a priori. When a time range for a change in skew or propagation is described in a fixed circuit configuration, it is understood that long time means that the time range of change is greater than the lower multiple of the duratuin at the clock rate.

The clock and data can be aligned during the configuration process, and for each line, the learning character can be used to establish the relationship between the character or frame boundary, the signal valid window, and the RDC. Once established, the relationship is maintained by the DLL or PLL, and the RDC can also be used to clock the parallelizer (DES). For example, due to environmental factors (typically temperature), even if the propagation time delay of the hop changes, the DLL or PLL can be updated by the CDR circuitry so that the sampling point remains with the valid data window. That is, the association between a particular clock bit in the recovery data clock RDC and the 0 th bit of the character in the module for the line does not change even if the time delay on the hop changes. This change is slow compared to the frame or character rate, and the DDL or PLL may not be updated continuously. Depending on the system design, if data is not transmitted continuously on the line of hops, periodic synchronous transmission may be initiated to adjust the RDC to maintain association.

An example of the conversion of a serial data stream, which can represent bits of characters on a line into a plurality of parallel data streams that can be processed locally at low data rates, is shown in FIG. 5. A character (20 bits in this example) is processed by the deserializer DES at the recovery clock rate RDC and routed within the module at clock rates of 1/2 and 1/4 of the line data rate. The logical association of the plane of the switch logic with the individual data bit locations is shown, with each plane operating at a switch clock rate (SWC). Other logical alignments of data into the lower clock-rate portion of the process are equally possible. In this example, the character's data is not fully parallelized. That is, four internal data paths (plane A-D) are used, and the data is not transmitted completely in parallel data characters. Equally, if the logic is fast enough, the characters can be treated as fully serial data characters, without serializing or parallelizing.

Rather than a logical representation, if a time period of bits is shown, the interval or time period between successive bits of data in the half rate region is twice the interval or time period of the incoming data, and It can be understood that the duration is four times the duration of the incoming data. In this example, the data is shown as even and odd bits of incoming characters, but this notation is for convenience only in the description and is not intended to limit the way data bits are routed within the module.

The phase alignment buffer PAB is used to align the output of the internal switch clock SWC and the parallelizer DES. The switch rate clock is shown to be distributed from the switch SW as SWC ¹ and SWC ¹¹ to mean that the SWC used for the PAB at the input and output of the switch has a clock rate that is the same frequency as that of the SWC, but It may differ from the SWC used in the switch SW itself but may have a substantially fixed phase or time offset.

In one aspect, the bit length of the PAB may not be longer than characters, and when the routing scheme described in US 11 / 405,083 is used, routing information may be obtained before the complete character is received by the receiving module. The PAB length may not be longer than 18 bits and may be 5 bits in length.

In this example, the character is parallelized into four packets of five data bits, and the clock rate (SWC) is one quarter of DC. The PAB may act to align the zeroth bit of the input character with the rising edge of the SWC. The next bits (first, second, third) of the character's parallelism data may be aligned with the relative phases 90, 180 and 270 ° of the SWC, respectively. The next bit of the character following in time can be processed as well.

The packet can be processed by the switch SW to route the packet to memory or to one of the output ports P ₁ , P ₂ . This routing may be dynamic, based on the received steering data characters. In addition, the switch SW can perform lane exchange, for example, by sending an input character on the line 1 of P ₀ to an output line 3 on P ₁ . With physical line associations, logical lanes can be static in nature, and a process can be called "binding". In this way, input characters can be routed along memory locations, paths to another module, and the alignment of lanes to lines can be used to manage differential skew between the characters of a frame, for example, at the destination module. Can be. The term redundant line is used to indicate any line connecting two ports that are not currently bound to a lane.

The interface between the switches SW and the on-module memory may be similar to the module-to-module interface and receive characters from packets intended to or from local memory. Prepare. This reception may include deskewing the characters in the frame and sending the frame to memory. Multiple such local data are described in patent application US 11 / 405,083.

After processing at the switch SW, where data is routed to an output port (e.g., P ₁ or P ₂ ), a packet of characters is output as input to the output phase alignment buffer PAB. The output PAB is used to maintain the alignment of the zeroth bit of the character with the clock edge of the data clock DC in the output serializer SER set during the configuration process. Since the processing at the switch SW is generally a synchronous process, the skew between the switch clock SWC and the data clock DC, once established, tends to remain relatively static. In other words, the propagation delay time of the bits through the switch is substantially constant. However, the phase or relative time of the data input to the switch SW may be different for each line with respect to the data clock DC. The output PAB can be used to align the bits of the character on the line with the appropriate clock cycle of the data clock DC for the line. This alignment is performed on a line by line basis, and the characters of the frame do not need to be resynchronized on each hop as they are processed individually.

The serializer combines the bits of a packet of characters into characters that have the same timing as previously transmitted characters on the line. That is, the association of bit 0 of the n th next molecule with the clock edge of the data clock DC delayed by 20 n clock cycles from the 0 th bit of the previous character is maintained even if no actual character is transmitted. The output of the serializer (SER) can be converted into a differential analog data signal in the TX circuit for transmission on the next hop.

Other characters on other lines in the frame are similarly processed. However, each of the lines has a separate CDR circuit, and the RDC for each line may have a different phase with respect to the SWC and with respect to each other. Changes on each line may also have other dependencies on time or temperature, corresponding to circuit details, such as line length, or the characteristics of an amplifier or filter.

6 shows a simple functional block diagram of the switch SW. The switch may be formed of two switches, a lane-exchange grid (LEG) and a port-exchange grid (PEG). As shown, there is only one LEG, but depending on the particular design, one or more LEG elements may also be provided at the switch output, or the LEG may be omitted at the input or output. Here, input lines 0-9 are associated with lanes (eg AJ), for example, where lane A is associated with line 0 at the input. Binding, switching, or routing of characters from input lines to output lines and ports can be performed by LEG, PEG. The binding of the remaining line BJ to the output lines 1-9 can be determined if the system is configured so that each lane in the input frame is routed to a particular physical line at the output. Lane A is associated with line 0 at the input. Routing can be at output port P ₁ , for example, where lane A can be associated with line 3 at the output. The binding, switching, or routing of characters from input lines to output lines and ports can be performed by LEG and PEG, respectively. The binding of the remaining line BJ to the output lines 1-9 can be determined when the system is configured, so that each lane in the input frame is routed to a specific physical line at the output. Lane A can likewise be bound to another line. The binding can be different between ports on one module, and can be different between one module and another module.

Even if the time delay of the transfer of data on the hop changes over time, the result of this process is that the bit of the character (eg 0) transmitted by the memory controller MC to the same clock edge of the data clock DC. Second bit), and the data clock is the data clock DC in any of the modules.

This concept is illustrated in FIGS. 7A-7B. In FIG. 7A, the propagation delay time D ₁ represents the propagation delay time between the output of the module MM2 and the output of the module MM1. This results in a set correspondence between the clock edge of the DC that remains the same as it exists at the completion of the configuration process that keeps this time constant for each lane of data and the first bit of the character transmitted on the line. Delay D1 may be considered as the sum of delays D2 and D3, D2 may be a delay not associated with switch SW and D3 is a delay associated with switch SW. Each of the delays D2 and D3 may vary depending on environmental factors over time, but the effect of the change in delay is compensated by the phase alignment buffer PAB so that the overall transmission time delay is substantially constant. This is shown in more detail in Figure 7b. Substantially constant is understood to mean that after synchronization the circuit is adjusted so that the entire transmission time of the first bit of the character does not change beyond the bit period, without resynchronization.

In the previous discussion, it is implicitly assumed that the time delay of the system clock SYSC in the module is constant relative to the clock edge in the SYSC clock source, which may not be on one module. This time delay may vary depending on environmental factors or aging, however, because the phase alignment buffers (PAB) on each module compensate for the relative time delay between DC and SWC and RDC, the change in the SYSC clock delay compensates. Can be.

The result of the described arrangement is that the entire character or more need not be buffered at the input PAB of the MM to adjust for changes in the delay of one hop. The depth of the PAB may be, for example, a change in skew over the operating temperature range for the line in the hop, rather than the overall change in skew in both lines of the hop. At the time of construction, the data may be located at appropriate points in the buffer for transmission between clock rate domains such that the data bit position is maintained in the buffer for the transmission period.

The result of the phase alignment buffer is that the total time delay through the memory module can be optimized, taking into account changes in skew due to environmental factors. When data propagates over a multi-hop path, the overall time delay between the MC and the destination module is compared to the approach to perform deskew by restoring the data in the frame at the intermediate module and adding a delay along the data path. , Can be reduced. The delay may be added to one or more lines in the destination module or source to handle the remaining skew, or may be added during configuration to optimize the overall skew.

Since the association of data of any line with the data clock (DC), which may be called synchronization, is maintained after the association is established, and maintained locally in the module pair, the data transmission on each line is responsible for latency performance reasons and system bandwidth. Unless required, it does not have to be continuous. If no lines are required, power can be removed from the transmitter and receiver elements as well as from the CDRs and other parts of the circuitry. If data is typically routed from an input port to one of two output ports or local memory, at least one of the output ports may be suspended if not needed.

One of the lines in one group of lines of sub-ports is active for signaling purposes, such as alerting the remaining data as data frames, or data transmitted for refresh purposes, in the previous inactive link. Can be maintained. However, signal presence indicators may also be used such that the presence on data on the line activates the receiver and associated circuitry. Thus, in steady state operation, the line may be either active or inactive, and if the data line is active, data may be coordination data, instructions (including addressing), data stored or read, or The clock hold transmission is sent.

Although only a few exemplary embodiments of the invention have been described in detail above, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without substantially departing from the advantages and novel techniques of the invention. Accordingly, all such modifications are intended to be included within the scope of this invention.

Included in the context of the present invention.

Claims (22)

A first node;
A second node in communication with the first node;
A first clock provided to the first node and the second node; And
A second having a first integral relationship to the first clock and having a time delay with respect to the first clock such that the transmission time of the bit between the output of the first node and the output of the second node remains substantially constant Interconnection system including a clock. The method of claim 1,
An interconnect system in which a first node communicates with a second node via a bus. The method of claim 2,
And a character consisting of bits is transmitted between the first node and the second node in series at a clock rate having a second integer relationship to the first clock. The method of claim 3, wherein
The first and second integer relationships are the same integer relationship. The method of claim 3, wherein
And wherein the received characters are converted into a plurality of serial data packets in a serial data format at a clock rate lower than that of the second clock. The method of claim 5, wherein
The time delay of the second clock is adjusted so that the same bit of the next character is converted to the same bit of the serial data packet. The method of claim 5, wherein
Wherein each plurality of serial data packets is routed to memory or to another node. The method of claim 7, wherein
An interconnect system in which routing is determined by received characters. The method of claim 7, wherein
If a character is routed to another node, the data is re-serialized using a second clock. The method of claim 9,
The reserialized data is transmitted on one of the plurality of lines connecting the two nodes. The method of claim 10,
Wherein the reserialized data is bound to a selected one of the plurality of lines. The method of claim 1,
Wherein the time delay of the lane is measured between the output of the first node and the output of the second node, such that the total time delay of the lane is substantially constant. The method of claim 12,
And a character received on one line of the plurality of lines is converted from a serial data format to a plurality of serial data packets irrespective of the received character on another line of the plurality of lines. The method of claim 13,
An interconnect system in which the number of serial data packets is equal to the number of bits in the character. Data receiving circuit;
Data transmission circuits;
A switch connecting the data receiving circuit and the data transmitting circuit;
Circuitry operable to maintain a sampling time in a fixed relationship to a first bit of a received character having a plurality of bits; And
And an input data buffer configured to resample the first bit such that the overall time delay measured between the resampled bit and the corresponding resampled bit at another node remains substantially constant. The method of claim 15,
The circuit is a node of an interconnect system that is a data clock recovery circuit. The method of claim 15,
An input data buffer is a node in an interconnect system that operates at a clock frequency that is an integral submultiple of the data clock rate. The method of claim 15,
A switch in an interconnect system that is operable to route received characters to at least one of a memory or another node. The method of claim 15,
A switch output is a node in an interconnect system that is transmitted at a data clock rate. The method of claim 15,
An input data buffer may be operable to parallelize characters received into a plurality of data packets, and the output data buffer may be operable to reserialize switch output data into characters. The method of claim 15,
And a character is received on a first line of the plurality of lines of the first port and the character is sent to another node on a selectable one of the plurality of lines of the second port. Transmitting a character of data comprising a plurality of bits from a first node to a second node;
Receiving the text at a second node;
Recovering the clock from the received data and aligning the sampling time of the characters; And
Re-sample the sampled data at a divisor of the clock frequency, and phase of the re-sampling clock such that the total time delay between the character transmitted from the first node and the character on the same lane transmitted by the first node is substantially constant; Adjusting the time delay;
A method of managing an interconnect system comprising a plurality of nodes in communication with each other.

Images