US7877536B2 - Scalable distributed routing scheme for PCI express switches - Google Patents
Scalable distributed routing scheme for PCI express switches Download PDFInfo
- Publication number
- US7877536B2 US7877536B2 US11/964,609 US96460907A US7877536B2 US 7877536 B2 US7877536 B2 US 7877536B2 US 96460907 A US96460907 A US 96460907A US 7877536 B2 US7877536 B2 US 7877536B2
- Authority
- US
- United States
- Prior art keywords
- routing
- routing information
- ports
- dedicated
- port
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/38—Information transfer, e.g. on bus
- G06F13/42—Bus transfer protocol, e.g. handshake; Synchronisation
- G06F13/4204—Bus transfer protocol, e.g. handshake; Synchronisation on a parallel bus
- G06F13/4221—Bus transfer protocol, e.g. handshake; Synchronisation on a parallel bus being an input/output bus, e.g. ISA bus, EISA bus, PCI bus, SCSI bus
Definitions
- the present application relates generally to computing devices, and more particularly to Peripheral Component Interconnect (PCI) express switches with distributed routing schemes.
- PCI Peripheral Component Interconnect
- PCI Peripheral Component Interconnect
- a PCI Express switch is a mix of PCI-to-PCI (P2P) bridges and internal endpoints, wherein each P2P bridge or internal end-point represents a single logical PCI function, and each PCI Express port represents a logical P2P bridge.
- P2P PCI-to-PCI
- PCI Express switch sometimes referred to as PCIe switch
- PCIe switch will include multiple PCI bridges
- the switching logic of the PCI Express switch needs to know all the information regarding the ports in the PCI Express switch. This brings several possible problems. First, since data packets may come into a PCI Express switch from different ports simultaneously, the processing of one packet may be blocked by the processing of another packet coming in from a different port. This adversely affects the performance of the PCI Express switch.
- the routers in the PCI Express switch need to have knowledge of all of the ports, when a port is added or removed (in a new product), or if the configuration/topology of the switch is changed over the previous switch products, the routers need to be re-designed every time a new PCI Express switch product is desired.
- IP intellectual property
- PCI Peripheral Component Interconnect Express switches.
- a PCI Express switch includes a first routing information bus connected to the first port; a second routing information bus connected to the second port; a third routing information bus connected to the third port; two routing slaves in the first port, each dedicated to listening to one of the second and the third routing information buses; two routing slaves in the second port, each dedicated to listening to one of the first and the third routing information buses; and two routing slaves in the third port, each dedicated to listening to one of the first and the second routing information buses.
- a PCI Express switch includes an upstream port; a plurality of downstream ports; a plurality of routing blocks, each in one of the upstream port and the plurality of downstream ports; a plurality of routing information buses, each connected to one of the upstream port and the plurality of downstream ports; a plurality of routing slaves in each of the plurality of the routing blocks, wherein each of the plurality of routing information buses is connected to a routing slave in each of the routing blocks; a plurality of “Claimed” daisy chains, each dedicated for propagating and coordinating “Claimed” assertions in response to a routing information received from one of the plurality of routing information buses; a plurality of “Malformed” daisy chains, each dedicated for propagating and coordinating “Malformed” assertions in response to the routing information received from one of the plurality of routing information buses; and a plurality of “Unsupported” daisy chains, each dedicated for propagating and coordinating “Unsupported” assertions in response to the
- a method for operating a PCI Express switch includes providing a first, a second, and a third port; providing a first daisy chain connecting the first, the second, and the third ports; providing a first, a second, and a third routing information bus connecting to, and dedicated to, the first, the second, and the third ports, respectively; providing a first, a second, and a third routing slave in the first, the second, and the third ports, respectively, wherein the second and the third routing slaves are dedicated to listening to the first routing information bus; receiving a first data packet into the first port; extracting a first routing information from the first data packet; broadcasting the first routing information on the first routing information bus; receiving the first routing information into the second and the third routing slaves; and making first routing assertions regarding the first data packet by the second and the third ports, wherein the first routing assertions are propagated and coordinated through the first daisy chain.
- An advantageous feature of the embodiments of the present application is that the design reusability is increased.
- the design of new PCI Express switch products may copy the routing slaves from the existing design of PCI Express switch products, and design new routing blocks by adding and deleting the routing slaves.
- the routing of incoming data packets can be processed in parallel, and non-blocking routing may be achieved.
- FIG. 1 illustrates a block diagram showing exemplary ports in a Peripheral Component Interconnect (PCI) Express switch embodiment with multifunction endpoints;
- PCI Peripheral Component Interconnect
- FIGS. 2A , 2 B, and 2 C illustrate exemplary routing slaves in the routing blocks of an upstream port, a downstream port, and an internal endpoint, respectively;
- FIG. 3 illustrates a connection scheme for connecting routing information buses to the routing slaves
- FIGS. 4A , 4 B, and 4 C illustrate a “Claimed” daisy chain, a “Malformed” daisy chain, and a “Unsupported” daisy chain, respectively.
- FIG. 1 illustrates a block diagram of exemplary ports in Peripheral Component Interconnect (PCI) Express switch 10 .
- PCI Express switch 10 includes a plurality of ports, including an upstream port, and three downstream ports, namely downstream port 0 , downstream port 1 , and downstream port 2 .
- PCI Express switches often have only upstream and downstream ports, in the embodiments of the present application, a PCI Express switch may also include internal endpoints.
- PCI Express switch 10 further includes two internal endpoints, namely internal endpoint 0 and internal endpoint 1 . It is appreciated that the number of downstream ports and the number of internal endpoints may be different from what is shown in FIG. 1 .
- the endpoints 0 and 1 are where PCI buses end, and are points at which peripherals such as universal serial buses (USB), firewires, hard disk controllers, and keyboard controllers are connected to PCI Express switch 10 .
- USB universal serial buses
- PCI Express switch 10 is configured to route data packets between the above-discussed ports. For example, a data packet coming into the upstream port may be routed to, and exit any of the downstream ports 0 through 2 and internal endpoints 0 and 1 . A data packet coming into one of the downstream ports 0 through 2 may exit out of another downstream port or the upstream port. A data packet coming into one of the endpoints 0 and 1 may exit out of the upstream port. Throughout the description, the data packets are alternatively referred to as incoming transaction layer packets (TLP).
- TLP incoming transaction layer packets
- PCI Express switch 10 includes a plurality of routing information buses connecting the upstream port, the downstream ports 0 through 2 and internal endpoints 0 and 1 .
- the number and the connection scheme of the routing information buses are discussed in detail in the subsequent paragraphs.
- the preferred number of the routing information buses is related to the number of ports in PCI Express switch 10 , and each of the ports preferably corresponds to one routing information bus.
- PCI Express switch 10 adopts a distributed routing scheme, with each of the ports taking a part in the routing of TLPs. Accordingly, each of the ports includes a routing block for performing the routing-related tasks.
- the exemplary routing blocks are illustrated in FIGS. 2A through 2C .
- a postfix “_UP” is used to represent the upstream port
- postfixes “_DP 0 ,” “_DP 1 ,” “DP 2 ,” “_IE 0 ,” and “_IE 1 ” are used to represent downstream ports, 0 , 1 , 2 and internal endpoints 0 and 1 , respectively.
- FIG. 2A illustrates routing block RB_UP in the upstream port. Since each of the ports may route TLPs to/from one or more of the other ports, in each of the ports, a routing slave is created and dedicated to listening to each of the other ports that communicate TLPs with it. Further, since a port may also be used to route other TLPs (including TLPs generated within the port) that may need to exit the PCI Express switch 10 through the port, an additional optional routing slave may be added to listen to the port's own routing information bus. The creation of the additional routing slave also improves the modularity in the design of the routing blocks.
- the routing block RB_UP includes six routing slaves, Slave_UP, Slave_DP 0 , Slave_DP 1 , Slave_DP 2 , Slave_IE 0 , and Slave_IE 1 .
- Routing slave Slave_UP listens to the upstream port itself.
- Routing slaves Slave_DP 0 , Slave_DP 1 , and Slave_DP 2 listen to downstream ports 0 , 1 , and 2 , respectively.
- Routing slaves Slave_IE 0 and Slave_IE 1 listen to internal endpoints 0 and 1 , respectively.
- routing slaves Slave_UP, Slave_DP 0 , Slave_DP 1 , Slave_DP 2 , Slave_IE 0 , and Slave_IE 1 will be very similar to each other.
- routing slaves Slave_DP 0 , Slave_DP 1 , and Slave_DP 2 may be identical.
- routing slaves Slave_IE 0 and Slave_IE 1 may be identical.
- FIG. 2B illustrates routing block RB_DP 0 in downstream port 0 .
- routing blocks RB_DP 1 and RB_DP 2 may be identical to routing block RB_DP 0 . Since each of the downstream ports 0 through 2 only route TLPs to/from other downstream ports or the upstream port, each of the routing blocks RB_DP 0 through RB_DP 2 only includes four routing slaves, namely Slave_UP, Slave_DP 0 , Slave_DP 1 , and Slave_DP 2 . Throughout the description, when two routing slaves in different ports are referred to by a same name, it indicates that they listen to a same routing information bus. Accordingly, routing slaves with a same name may be found in a plurality of routing blocks.
- FIG. 2C illustrates routing block RB_IE 0 . Since the internal endpoints only route TLPs through the upstream port, each of the internal endpoints 0 and 1 only includes a routing slave Slave_UP for listening to the upstream port, and optionally routing slave Slave_IE 0 or Slave_IE 1 for listening to the endpoints itself.
- FIG. 3 illustrates a connection scheme for connecting routing information buses and routing slaves of PCI Express switch 10 . Since there are six ports in PCI Express switch 10 , there are six routing information buses, namely, BUS_UP, BUS_DP 0 , BUS_DP 1 , BUS_DP 2 , BUS_IE 0 , and BUS_IE 1 , each for broadcasting the routing information extracted from the incoming TLP. Routing information bus BUS_UP is used by the upstream port for broadcasting the routing information of the TLPs received (from outside the PCI Express switch) by the upstream port.
- BUS_UP is used by the upstream port for broadcasting the routing information of the TLPs received (from outside the PCI Express switch) by the upstream port.
- Each of the downstream ports 0 through 2 and internal endpoints 1 and 2 has a routing slave Slave_UP dedicated to listening to the routing information carried on routing information bus BUS_UP, so that the downstream ports and the internal endpoints may make decisions as to how they want to act on the routing information.
- routing information buses BUS_DP 0 , BUS_DP 1 , and BUS_DP 2 are used for broadcasting the routing information of the TLPs received by the downstream ports 0 , 1 , and 2 , respectively.
- the routing information buses BUS_IE 0 and BUS_IE 1 are used for transferring the routing information of the TLPs received by the internal endpoints 0 and 1 , respectively.
- a routing slave may optionally be created in each of the routing blocks for listening to the routing information bus of the port (resident port) that the optional routing slave resides in.
- each of the ports makes routing decisions and assertions, for example, whether to claim a TLP for routing, whether to assert a TLP as malformed, and/or whether to assert a TLP as unsupported, by themselves.
- the individual routing decisions and assertions made by the potential egress ports may conflict with each other.
- two potential egress ports may both determine to route a TLP out of PCI Express switch 10 through them, and may both claim the same TLP.
- the integrity of PCI Express switch 10 is maintained using daisy chains, as illustrated in FIGS. 4A , 4 B, and 4 C.
- FIG. 4A illustrates a daisy chain for integrating the assertions for “claiming” TLPs, which daisy chain is referred to hereinafter as “Claimed” daisy chain.
- a TLP enters an ingress port, for example, the upstream port as shown in FIG. 4A
- the ingress port extracts the routing information of the TLP and broadcast the routing information on the respective routing information bus, for example, BUS_UP in this example.
- the “Claimed” daisy chain is formed by connecting the routing blocks into a chain.
- the order of a port/routing block in the daisy chain determines its priority in the processing of the TLPs.
- the ports are arranged in the order of: internal end point 0 , internal endpoint 1 , downstream port 2 , downstream port 1 , and downstream port 0 , with the ports in the up-stream of the daisy chain having higher priorities.
- Each of the routing blocks RB_IE 0 , RB_IE 1 , RB_DP 0 , RB_DP 1 and RB_DP 2 will make a decision as whether to claim the TLP or not.
- the port that successfully claims the packet will eventually be the egress port of the TLP.
- Each of the ports will make its decision based on two pieces of information, that is, whether a port in the up-stream of the “Claimed” daisy chain has claimed the TLP or not (through a “Claimed_in” node in the routing block), and whether the port itself wants to claim the TLP or not.
- the ports in the up-stream of the “Claimed” daisy chain have higher priorities over the ports in the down-stream, they may override decisions and assertions made by ports in the down-stream of the daisy chain. Therefore, if a port (referred to as current port hereinafter) finds that a port in the up-stream of the “Claimed” daisy chain has claimed a TLP, the current port will not claim it. However, the current port will set a signal at a “Claimed_out” node to inform the ports in the down-stream of the “Claimed” daisy chain that the TLP has been claimed.
- current port finds that a port in the up-stream of the “Claimed” daisy chain has claimed a TLP
- the current port may make the decision as whether to claim it or not, for example, based on the address of the TLP, the type of the TLP, and/or the target bus number of the TLP. If the current port decides to claim the TLP, it will set a signal at the “Claimed_out” node to inform the ports down-stream from it on the “Claimed” daisy chain that the TLP has been claimed. If, however, the current port decides not to claim the TLP, the ports in the down-stream may still claim it.
- the “Claimed” daisy chain ensures that at most one port may claim a TLP, and thus the TLP will only exit out of only one egress port.
- a Global Control logic (GCL) unit or any other final logic block may be used at the end of the daisy chain to collect the “Claimed” information. This final logic block will have the responsibility of handling all unclaimed TLPs as unsupported requests. This is needed because TLPs need to be claimed by a port to exit the PCI Express switch 10 . If no port claims a TLP, the TLP is deemed unsupported (by any of the ports).
- a TLP is claimed by a port in the up-stream of the “Claimed” daisy chain
- the information is rippled through the down-stream of the daisy chain, preferably in one clock cycle. Further, the entire process for propagating and coordinating which of the ports will claim the TLP is finished in one clock cycle, which yields maximum performance advantage.
- PCI Express switch 10 may include two other types of daisy chains, “Malformed” daisy chains, and “Unsupported” daisy chains.
- FIG. 4B illustrates a “Malformed” daisy chain for the upstream port, which daisy chain has a similar structure as the “Claimed” daisy chain. If one of the routing blocks in the “Malformed” daisy chain asserts the TLP as being malformed, the “Malformed” information is rippled through the “Malformed” daisy chain using “Mal_In” and “Mal_Out” nodes, and the respective TLP is put in a “Malformed” bucket for further processing. As a result, no routing of the TLP will be performed.
- an internal logic unit GCL may be used to collect the “Malformed” information.
- the “Malformed” status of the TLP is propagated and coordinated in one clock cycle.
- FIG. 4C illustrates an exemplary “Unsupported” daisy chain for the upstream port.
- the routing blocks may be determined by the GCL unit as being “Unsupported,” and the respective TLP is put into an “Unsupported” bucket for further processing by internal logics.
- one routing block may proactively assert the TLP as unsupported. In this case, the “Unsupported” assertion is rippled through the “Unsupported” daisy chain, and received by the GCL unit. Again, the “Unsupported” status of the TLP may be propagated and coordinated in one clock cycle.
- the decisions and assertions as whether the respective TLP is claimed, malformed, or unsupported may all be propagated and coordinated in clock cycle. As discussed in the preceding paragraphs, if the TLP is claimed, it is also determined in the same clock cycle which of the ports has the right to claim it.
- the upstream port which is the ingress port in this example, will forward the TLP to the egress port, which has claimed the TLP.
- the “Claimed,” “Malformed,” and “Unsupported” daisy chains are used for the routing information bus BUS_UP, which is for broadcasting routing information of TLPs incoming from the upstream port.
- Downstream ports 0 , 1 , and 2 require routing information buses BUS_DP 0 , BUS_DP 1 , and BUS_DP 2 , respectively.
- each of the routing information buses BUS_DP 0 , BUS_DP 1 , and BUS_DP 2 requires a “Claimed” daisy chain, a “Malformed” daisy chain, and an “Unsupported” daisy chain.
- the daisy chains for the downstream ports 0 through 2 are essentially the same as the daisy chains of the upstream port, except that the respective ingress downstream ports are the beginning point of the respective daisy chains.
- the internal endpoints 0 and 1 do not need daisy chains, since only the upstream port will route packets to/from the internal endpoints.
- Each of the above-discussed routing slaves regardless in which port they are located, may be identical to each other, and thus one intellectual property (IP) of a routing slave may be simply copied into the design of other ports.
- IP intellectual property
- the existing routing slave design may be reused.
- the routing blocks may be re-designed by simply adding/deleting routing slaves if the new PCI Express switch product has a different number of ports over existing PCI Express switch products. The routing logic of the routing blocks, however, does not have to be changed significantly.
- the routing of incoming data packets can be processed in parallel, and non-blocking routing may be achieved even if the packets are received simultaneously. Additional advantageous features of the embodiments of the present application include quick turn-around of the routing information, and hence improved performance, and robust conflict resolution and error handling by using daisy chains.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Small-Scale Networks (AREA)
Abstract
Description
Claims (20)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US11/964,609 US7877536B2 (en) | 2007-12-26 | 2007-12-26 | Scalable distributed routing scheme for PCI express switches |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US11/964,609 US7877536B2 (en) | 2007-12-26 | 2007-12-26 | Scalable distributed routing scheme for PCI express switches |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| US20090172237A1 US20090172237A1 (en) | 2009-07-02 |
| US7877536B2 true US7877536B2 (en) | 2011-01-25 |
Family
ID=40799979
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| US11/964,609 Active 2028-11-14 US7877536B2 (en) | 2007-12-26 | 2007-12-26 | Scalable distributed routing scheme for PCI express switches |
Country Status (1)
| Country | Link |
|---|---|
| US (1) | US7877536B2 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9292462B2 (en) | 2013-05-22 | 2016-03-22 | International Business Machines Corporation | Broadcast for a distributed switch network |
| US11258630B2 (en) * | 2018-08-28 | 2022-02-22 | Texas Instruments Incorporated | Controller area network receiver |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9292464B2 (en) * | 2012-12-13 | 2016-03-22 | Coherent Logix, Incorporated | Multiprocessor system with improved secondary interconnection network |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5787095A (en) * | 1994-10-25 | 1998-07-28 | Pyramid Technology Corporation | Multiprocessor computer backlane bus |
| US7058738B2 (en) * | 2004-04-28 | 2006-06-06 | Microsoft Corporation | Configurable PCI express switch which allows multiple CPUs to be connected to multiple I/O devices |
-
2007
- 2007-12-26 US US11/964,609 patent/US7877536B2/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5787095A (en) * | 1994-10-25 | 1998-07-28 | Pyramid Technology Corporation | Multiprocessor computer backlane bus |
| US7058738B2 (en) * | 2004-04-28 | 2006-06-06 | Microsoft Corporation | Configurable PCI express switch which allows multiple CPUs to be connected to multiple I/O devices |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9292462B2 (en) | 2013-05-22 | 2016-03-22 | International Business Machines Corporation | Broadcast for a distributed switch network |
| US11258630B2 (en) * | 2018-08-28 | 2022-02-22 | Texas Instruments Incorporated | Controller area network receiver |
Also Published As
| Publication number | Publication date |
|---|---|
| US20090172237A1 (en) | 2009-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI364663B (en) | Configurable pci express switch and method controlling the same | |
| CN106797349B (en) | Dynamic cascading clustering for dynamic VNF | |
| JP2006515941A (en) | Pipeline accelerator having multiple pipeline units, associated computing machine, and method | |
| US20070214301A1 (en) | PCI-E Automatic allocation system | |
| US20120221764A1 (en) | Low latency precedence ordering in a pci express multiple root i/o virtualization environment | |
| US20060004837A1 (en) | Advanced switching peer-to-peer protocol | |
| US8589614B2 (en) | Network system with crossbar switch and bypass route directly coupling crossbar interfaces | |
| JPH11328098A (en) | Method and system for supporting pci bus | |
| US7877536B2 (en) | Scalable distributed routing scheme for PCI express switches | |
| US20180183672A1 (en) | System and method for grouping of network on chip (noc) elements | |
| US20070168452A1 (en) | Method of processing data, a network analyser card, a host and an intrusion detection system | |
| CN109379248B (en) | Protocol stack testing system and method | |
| CN120762879A (en) | Device resource management method, related system and storage medium | |
| CN113986969B (en) | Data processing method, device, electronic device and storage medium | |
| CN112492039B (en) | Data distribution service method and device running in multiple cores | |
| US9007943B2 (en) | Methods and structure for reduced layout congestion in a serial attached SCSI expander | |
| EP3913490B1 (en) | Routing traffics having primary and secondary destinations in communication networks | |
| WO2020107460A1 (en) | Computing method, chip, and system, readable storage medium, and computer program product | |
| US10938904B2 (en) | Multi-processor/endpoint data splitting system | |
| CN112532545A (en) | Communication device between external nodes and server | |
| CN118748654A (en) | Data packet sending method, device, electronic device and readable storage medium | |
| CN115758975A (en) | Layout optimization method, device and equipment and computer readable storage medium | |
| CN113691434B (en) | Data transmission system, method, electronic device, and storage medium | |
| CN103425620B (en) | The coupled structure of accelerator and processor based on multiple token-ring | |
| US9404968B1 (en) | System and methods for debug connectivity discovery |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| AS | Assignment |
Owner name: TEXAS INSTRUMENTS INCORPORATED, TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WOJCIECHOWSKI, ROY D.;MADHAVAPEDDI, SRINADH;MORRISON, SCOTT ADAM;REEL/FRAME:020323/0695;SIGNING DATES FROM 20071220 TO 20071221 Owner name: TEXAS INSTRUMENTS INCORPORATED, TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WOJCIECHOWSKI, ROY D.;MADHAVAPEDDI, SRINADH;MORRISON, SCOTT ADAM;SIGNING DATES FROM 20071220 TO 20071221;REEL/FRAME:020323/0695 |
|
| AS | Assignment |
Owner name: TEXAS INSTRUMENTS INCORPORATED,TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WOJCIECHOWSKI, ROY D.;MADHAVAPEDDI, SRINADH;MORRISON, SCOTT A.;AND OTHERS;SIGNING DATES FROM 20071220 TO 20090413;REEL/FRAME:024458/0339 Owner name: TEXAS INSTRUMENTS INCORPORATED, TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WOJCIECHOWSKI, ROY D.;MADHAVAPEDDI, SRINADH;MORRISON, SCOTT A.;AND OTHERS;SIGNING DATES FROM 20071220 TO 20090413;REEL/FRAME:024458/0339 |
|
| STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
| FPAY | Fee payment |
Year of fee payment: 4 |
|
| MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552) Year of fee payment: 8 |
|
| MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 12 |