EP1417577A1 - Systeme et procede de vcd sur demande - Google Patents

Systeme et procede de vcd sur demande

Info

Publication number
EP1417577A1
EP1417577A1 EP01965946A EP01965946A EP1417577A1 EP 1417577 A1 EP1417577 A1 EP 1417577A1 EP 01965946 A EP01965946 A EP 01965946A EP 01965946 A EP01965946 A EP 01965946A EP 1417577 A1 EP1417577 A1 EP 1417577A1
Authority
EP
European Patent Office
Prior art keywords
simulation
hardware
logic
user
software
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP01965946A
Other languages
German (de)
English (en)
Other versions
EP1417577A4 (fr
Inventor
Ping-Sheng Tseng
Yogesh Kumar Goel
Quincy Kun-Hsu Shen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cadence Design Systems Inc
Original Assignee
Axis Systems Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Axis Systems Inc filed Critical Axis Systems Inc
Publication of EP1417577A1 publication Critical patent/EP1417577A1/fr
Publication of EP1417577A4 publication Critical patent/EP1417577A4/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/30—Circuit design
    • G06F30/32—Circuit design at the digital level
    • G06F30/33—Design verification, e.g. functional simulation or model checking

Definitions

  • the present invention generally relates to electronic design automation (EDA). More particularly, the present invention relates to value change dump (VCD) improvements to accelerate design debug sessions.
  • EDA electronic design automation
  • VCD value change dump
  • EDA electronic design automation
  • EDA is a computer-based tool configured in various workstations to provide designers with automated or semi-automated tools for designing and verifying user's custom circuit designs.
  • EDA is generally used for creating, analyzing, and editing any electronic design for the purpose of simulation, emulation, prototyping, execution, or computing.
  • EDA technology can also be used to develop systems (i.e., target systems) which will use the user-designed subsystem or component.
  • the end result of EDA is a modified and enhanced design, typically in the form of discrete integrated circuits or printed circuit boards, that is an improvement over the original design while maintaining the spirit of the original design.
  • Co-simulation arose out of a need to address some problems with the cumbersome nature of using two separate and independent processes of pure software simulation and pure hardware emulation/acceleration, and to make the overall system more user-friendly.
  • co-simulators still have a number of drawbacks: (1) co-simulation systems require manual partitioning, (2) co-simulation uses two loosely coupled engines, (3) co-simulation speed is as slow as software simulation speed, and (4) co-simulation systems encounter race conditions.
  • partitioning between software and hardware is done manually, instead of automatically, further burdening the user.
  • co-simulation requires the user to partition the design (starting with behavior level, then RTL, and then gate level) and to test the models themselves among the software and hardware at very large functional blocks. Such a constraint requires some degree of sophistication by the user.
  • co-simulation systems utilize two loosely coupled and independent engines, which raise inter-engine synchronization, coordination, and flexibility issues.
  • Co-simulation requires synchronization of two different verification engines - software simulation and hardware emulation.
  • the software simulator side is coupled to the hardware accelerator side, only external pin-out data is available for inspection and loading. Values inside the modeled circuit at the register and combinational logic level are not available for easy inspection and downloading from one side to the other, limiting the utility of these co-simulator systems.
  • the user may have to re-simulate the whole design if the user switches from software simulation to hardware acceleration and back.
  • co-simulator systems do not provide this capability.
  • co-simulation speed is as slow as simulation speed.
  • Co-simulation requires synchronization of two different verification engines - software simulation and hardware emulation.
  • Each of the engines has its own control mechanism for driving the simulation or emulation. This implies that the synchronization between the software and hardware pushes the overall performance to a speed that is as low as software simulation.
  • the additional overhead to coordinate the operation of these two engines adds to the slow speed of co-simulation systems.
  • Co-simulators use hardware driven clocks, which may find themselves at the inputs to different logic elements at different times due to different wire line lengths. This raises the uncertainty level of evaluation results as some logic elements evaluate data at some time period and other logic elements evaluate data at different time periods, when these logic elements should be evaluating the data together.
  • Another problem encountered by a typical designer is the relatively slow process of isolating and identifying design problems while debugging. Although the designer's own limited problem-solving ability may contribute to some of this straggling pace, the main source of the problem is the simulator itself. Not only is the simulator slow because of its software-based engine, debugging with a simulator requires the entire simulation to be rerun. A further explanation of this problem will now be provided.
  • a typical ASIC chip designer debugs his design using a simulator; that is, the designer simulates or tests his design using, among other things, test bench processes to observe its reactions to various stimuli. Based on an examination of some key nodes and outputs of his design, the designer can generally determine whether or not his design has a problem. Of course, if the design is in its early stages, it invariably has some problems.
  • the simulator saves the entire simulation as a VCD file from simulation time tO to the end of the simulation.
  • This VCD file is then analyzed by the designer to isolate the bug.
  • the designer makes a reasonable guess as to its general location so that he can analyze this location with some fine-stepping; that is, if the designer somehow suspects that the bug occurred somewhere between simulation time t350 and t400, he will proceed to a simulation time located just before the suspected simulation time, such as simulation time t345. He will then proceed to examine this suspected area (i.e., t345 to t400) very carefully.
  • the entire simulation is not saved; rather, the simulator saves a designer- selected portion of the simulation.
  • selective VCD does not save the designer from having to rerun the entire simulation from the beginning.
  • the designer runs the simulation and invariably observes a problem with his design. He then makes a guess as to where the problem is located. If the designer suspects that the problem will occur somewhere between simulation time t350 and t400, the designer reruns the simulation and instructs the simulator to save this simulation time range as a VCD file. Thereafter, the designer can examine the VCD file corresponding to his guess. If his guess is incorrect in isolating the problem, he must make another guess, instruct the simulator to save the new simulation range as the VCD file, and then rerun the simulation. The designer then analyzes the VCD file again.
  • selective VCD does not require as much disk space since the entire simulation is not saved.
  • selective VCD still requires the entire simulation to be rerun. If the designer makes a wrong guess in locating his bug, he must rerun the simulation again to save the new simulation range in the VCD file. In any event, the selective VCD method is still time consuming that is also exacerbated by wrong guesses.
  • the VCD on-demand feature is incorporated in the RCC System, which includes an RCC Computing System and an RCC Hardware Accelerator.
  • the RCC Computing System contains the computational resources that are necessary to allow the user to simulate the user's entire software-modeled design in software and control the hardware acceleration of the hardware-modeled portion of the design.
  • the RCC Hardware Accelerator contains the reconfigurable array of logic elements (e.g., FPGA) that can model at least a portion of the user's design in hardware so that the user can accelerate the debugging process.
  • the RCC Computing System is tightly coupled to the RCC Hardware Accelerator via the software clock.
  • the VCD on-demand feature allows the user to select a portion of the simulation history for detailed debugging analysis without rerunning the simulation.
  • the RCC System allows the user to select two simulation time ranges - the broader "simulation session range" and a narrower subset of this range called the "simulation target range.”
  • the VCD file will be created for this narrower "simulation target range.”
  • the RCC System fast simulates the design for the entire duration of the simulation session range by providing the primary inputs from the test bench process to the hardware model in the RCC Hardware Accelerator for evaluation. These same primary inputs are also compressed and recorded in a simulation history file. With this simulation history file, the RCC System can reproduce any portion of the simulation within the simulation session range at any time.
  • the RCC System saves the hardware state information of the design at this point so that the user may simulate off-line as necessary.
  • the RCC System saves the hardware state information of the design at this point so that the user may quickly return to this point where he last left off to simulate beyond this simulation session range at any time without rewinding the simulation.
  • the RCC System When the user has selected a "simulation target range,” the RCC System fast simulates to the beginning of the simulation target range by decompressing the compressed primary inputs in the simulation history file and providing these decompressed primary inputs to the RCC Hardware Accelerator for evaluation. In the simulation target range, the RCC System dumps the evaluated results or primary outputs from the hardware model into a VCD file for storage in system disk. At the end of the simulation target range, the RCC System ceases the dump process.
  • the user can then view the VCD file with a waveform viewer to debug his design in greater detail. This is accomplished without rerunning the simulation. If the bug is not located within this simulation target range, the user can select another simulation target range within the same simulation session range. Once the new simulation target range has been selected, the RCC System creates a new VCD file in the manner described above. The user can then analyze this new VCD file to isolate the bug.
  • the user can then move on to simulate beyond the current simulation session range to the next simulation session range.
  • the saved hardware state information from the end of the current simulation session range is loaded into the RCC System. The user can then commence simulating.
  • the VCD on- demand feature is available both on-line and off-line.
  • FIG. 1 shows a high level overview of one embodiment of the present invention, including the workstation, reconfigurable hardware emulation model, emulation interface, and the target system coupled to a PCI bus.
  • FIG.2 shows one particular usage flow diagram of the present invention.
  • FIG. 3 shows a high level diagram of the software compilation and hardware configuration during compile time and run time in accordance with one embodiment of the present invention.
  • FIG. 4 shows a flow diagram of the compilation process, which includes generating the software/hardware models and the software kernel code.
  • FIG. 5 shows the software kernel that controls the overall SEmulation system.
  • FIG. 6 shows a method of mapping hardware models to reconfigurable boards through mapping, placement, and routing.
  • FIG. 7 shows the connectivity matrix for the FPGA array shown in FIG. 8.
  • FIG. 8 shows one embodiment of the 4x4 FPGA array and their interconnections.
  • FIGS. 9(A), 9(B), and 9(C) illustrate one embodiment of the time division multiplexed (TDM) circuit which allows a group of wires to be coupled together in a time multiplexed fashion so that one pin, instead of a plurality of pins, can be used for this group of wires in a chip.
  • FIG. 9(A) presents an overview of the pin-out problem
  • FIG. 9(B) provides a TDM circuit for the transmission side
  • FIG. 9(C) provides a TDM circuit for the receiver side.
  • FIG. 10 shows a SEmulation system architecture in accordance with one embodiment of the present invention.
  • FIG. 11 shows one embodiment of address pointer of the present invention.
  • FIG. 12 shows a state transition diagram of the address pointer initialization for the address pointer of FIG. 11.
  • FIG. 13 shows one embodiment of the MOVE signal generator for derivatively generating the various MOVE signals for the address pointer.
  • FIG. 14 shows the chain of multiplexed address pointers in each FPGA chip.
  • FIG. 15 shows one embodiment of the multiplexed cross chip address pointer chain in accordance with one embodiment of the present invention.
  • FIG. 16 shows a flow diagram of the clock/data network analysis that is critical for the software clock implementation and the evaluation of logic components in the hardware model.
  • FIG. 17 shows a basic building block of the hardware model in accordance with one embodiment of the present invention.
  • FIGS. 18(A) and 18(B) show the register model implementation for latches and flip-flops.
  • FIG. 19 shows one embodiment of the clock edge detection logic in accordance with one embodiment of the present invention.
  • FIG. 20 shows a four state finite state machine to control the clock edge detection logic of FIG. 19 in accordance with one embodiment of the present invention.
  • FIG.21 shows the interconnection, JTAG, FPGA bus, and global signal pin designations for each FPGA chip in accordance with one embodiment of the present invention.
  • FIG.22 shows one embodiment of the FPGA controller between the PCI bus and the FPGA array.
  • FIG.23 shows a more detailed illustration of the CTRLJFPGA unit and data buffer which were discussed with respect to FIG. 22.
  • FIG. 24 shows the 4x4 FPGA array, its relationship to the FPGA banks, and expansion capability.
  • FIG.25 shows one embodiment of the hardware start-up method.
  • FIG.26 shows the HDL code for one example of a user circuit design to be modeled and simulated.
  • FIG.27 shows a circuit diagram that symbolically represent the circuit design of the HDL code in FIG.26.
  • FIG. 28 shows the component type analysis for the HDL code of FIG.26.
  • FIG. 29 shows a signal network analysis of a structured RTL HDL code based on the user's custom circuit design shown in FIG.26.
  • FIG. 30 shows the software/hardware partition result for the same hypothetical example.
  • FIG. 31 shows a hardware model for the same hypothetical example.
  • FIG. 32 shows one particular hardware model-to-chip partition result for the same hypothetical example of a user's custom circuit design.
  • FIG. 33 shows another particular hardware model-to-chip partition result for the same hypothetical example of a user's custom circuit design.
  • FIG. 34 shows the logic patching operation for the same hypothetical example of a user's custom circuit design.
  • FIGS. 35(A) to 35(D) illustrate the principle of "hops" and interconnections with two examples.
  • FIG. 36 shows an overview of the FPGA chip used in the present invention.
  • FIG. 37 shows the FPGA interconnection buses on the FPGA chip.
  • FIGS. 38(A) and 38(B) show side views of the FPGA board connection scheme in accordance with one embodiment of the present invention.
  • FIG. 39 shows a direct-neighbor and one-hop six-board interconnection layout of the FPGA array in accordance with one embodiment of the present invention.
  • FIGS.40(A) and 40(B) show FPGA inter-board interconnection scheme.
  • FIGS. 41(A) to 41(F) show top views of the board interconnection connectors.
  • FIG.42 shows on-board connectors and some components in a representative FPGA board.
  • FIG.43 shows a legend of the connectors in FIGS.41(A) to 41(F) and 42.
  • FIG. 44 shows a direct-neighbor and one-hop dual-board interconnection layout of the FPGA array in accordance with another embodiment of the present invention.
  • FIG. 45 shows a workstation with multiprocessors in accordance with another embodiment of the present invention.
  • FIG.46 shows an environment in accordance with another embodiment of the present invention in which multiple users share a single simulation/emulation system on a time-shared basis.
  • FIG. 47 shows a high level structure of the Simulation server in accordance with one embodiment of the present invention.
  • FIG. 48 shows the architecture of the Simulation server in accordance with one embodiment of the present invention.
  • FIG.49 shows a flow diagram of the Simulation server.
  • FIG. 50 shows a flow diagram of the job swapping process.
  • FIG. 51 shows the signals between the device driver and the reconfigurable hardware unit.
  • FIG. 52 illustrates the time-sharing feature of the Simulation server for handling multiple jobs with different levels of priorities.
  • FIG. 53 shows the communication handshake signals between the device driver and the reconfigurable hardware unit.
  • FIG. 54 shows the state diagram of the communication handshake protocol.
  • FIG. 55 shows an overview of the client-server model of the Simulation server in accordance with one embodiment of the present invention.
  • FIG. 56 shows a high level block diagram of the Simulation system for implementing memory mapping in accordance with one embodiment of the present invention.
  • FIG. 57 shows a more detailed block diagram of the memory mapping aspect of the Simulation system with supporting components for the memory finite state machine (MEMFSM) and the evaluation finite state machine for each FPGA logic device (EVALFSMx).
  • MEMFSM memory finite state machine
  • EVALFSMx evaluation finite state machine for each FPGA logic device
  • FIG. 58 shows a state diagram of a finite state machine of the MEMFSM unit in the CTRL_FPGA unit in accordance with one embodiment of the present invention.
  • FIG. 59 shows a state diagram of a finite state machine in each FPGA chip in accordance with one embodiment of the present invention.
  • FIG. 60 shows the memory read data double buffer.
  • FIG. 61 shows the Simulation write/read cycle in accordance with one embodiment of the present invention.
  • FIG. 62 shows a timing diagram of the Simulation data transfer operation when the DMA read operation occurs after the CLK_EN signal.
  • FIG. 63 shows a timing diagram of the Simulation data transfer operation when the DMA read operation occurs near the end of the EVAL period.
  • FIG. 64 shows a typical user design implemented as a PCI add-on card.
  • FIG. 65 shows a typical hardware/software coverification system using an ASIC as the device-under-test.
  • FIG. 66 shows a typical coverification system using an emulator where the device-under-test is programmed in the emulator.
  • FIG. 67 shows a simulation system in accordance with one embodiment of the present invention.
  • FIG. 68 shows a coverification system without external I/O devices in accordance with one embodiment of the present invention, where the RCC computing system contains a software model of the various I/O devices and the target system.
  • FIG. 69 shows a coverification system with actual external I/O devices and the target system in accordance with another embodiment of the present invention.
  • FIG. 70 shows a more detailed logic diagram of the data-in portion of the control logic in accordance with one embodiment of the present invention.
  • FIG. 71 shows a more detailed logic diagram of the data-out portion of the control logic in accordance with one embodiment of the present invention.
  • FIG. 72 shows the timing diagram of the data-in portion of the control logic.
  • FIG. 73 shows the timing diagram of the data-out portion of the control logic.
  • FIG. 74 shows a board layout of the RCC hardware array in accordance with one embodiment of the present invention.
  • FIG. 75(A) shows an exemplary shift register circuit which will be used to explain the hold time and clock glitch problems.
  • FIG. 75(B) shows a timing diagram of the shift register circuit shown in FIG. 75(A) to illustrate hold time.
  • FIG. 76(A) shows the same shift register circuit of FIG. 75(A) placed across multiple FPGA chips.
  • FIG.76(B) shows a timing diagram of the shift register circuit shown in FIG. 76(A) to illustrate hold time violation.
  • FIG. 77(A) shows an exemplary logic circuit which will be used to illustrate a clock glitch problem.
  • FIG. 77(B) shows a timing diagram of the logic circuit of FIG. 77(A) to illustrate the clock glitch problem.
  • FIG. 78 shows a prior art timing adjustment technique for solving the hold time violation problem.
  • FIG. 79 shows a prior art timing resynthesis technique for solving the hold time violation problem.
  • FIG. 80(A) shows the original latch and FIG. 80(B) shows a timing insensitive and glitch-free latch in accordance with one embodiment of the present invention.
  • FIG. 81(A) shows the original design flip-flop and FIG. 81(B) shows a timing insensitive and glitch-free design type flip-flop in accordance with one embodiment of the present invention.
  • FIG. 82 shows a timing diagram of the trigger mechanism of the timing insensitive and glitch-free latch and flip-flop in accordance with one embodiment of the present invention.
  • FIG. 83 shows a high level view of the components of the RCC system which incorporates one embodiment of the present invention.
  • FIG. 84 shows several simulation time periods to illustrate the VCD on-demand operation in accordance with one embodiment of the present invention.
  • SEmulator or “SEmulation” system.
  • SEmulation system software simulation
  • SEmulator system simulation through hardware acceleration
  • ICE in-circuit emulation
  • post- simulation analysis including their respective set-up or pre-processing stages.
  • SEmulation may be used. This term refers to the novel processes described herein.
  • RRCC Reconfigurable Computing
  • RRC computing system refers to that portion of the simulation/coverif ⁇ cation system that contains the main processor, software kernel and the software model of the user design.
  • Terms such as “Reconfigurable hardware array” or “RCC hardware array” refers to that portion of the simulation/coverif ⁇ cation system that contains the hardware model of the user design and which contains the array of reconfigurable logic elements, in one embodiment.
  • the specification also makes references to a "user” and a user's “circuit design” or “electronic design.”
  • the "user” is a person who uses the SEmulation system through its interfaces and may be the designer of a circuit or a test/debugger who played little or no part in the design process.
  • the "circuit design” or “electronic design” is a custom designed system or component, whether software or hardware, which can be modeled by the SEmulation system for test/debug purposes. In many cases, the "user” also designed the “circuit design” or “electronic design.”
  • wire refers to various electrically conducting lines. Each line may be a single wire between two points or several wires between points. These terms are interchangeable in that a “wire” may comprise one or more conducting lines and a “bus” may also comprise one or more conducting lines.
  • the various embodiments of the present invention have four general modes of operation: (1) software simulation, (2) simulation through hardware acceleration, (3) in-circuit emulation, and (4) post-simulation analysis.
  • the various embodiments include the system and method of these modes with at least some of the following features:
  • the end result is a flexible and fast simulator/emulator system and method with full HDL functionality and emulator execution performance.
  • the SEmulator system through automatic component type analysis, can model the user's custom circuit design in software and hardware.
  • the entire user circuit design is modeled in software, whereas evaluation components (i.e., register component, combinational component) are modeled in hardware.
  • Hardware modeling is facilitated by the component type analysis.
  • a software kernel residing in the main memory of the general purpose processor system, serves as the SEmulator system's main program that controls the overall operation and execution of its various modes and features. So long as any test-bench processes are active, the kernel evaluates active test-bench components, evaluates clock components, detects clock edges to update registers and memories as well as propagating combinational logic data, and advances the simulation time.
  • This software kernel provides for the tightly coupled nature of the simulator engine with the hardware acceleration engine.
  • the SEmulator system provides a number of I/O address spaces - REG (register), CLK (software clock), S2H (software to hardware), and H2S (hardware to software).
  • the SEmulator has the capability to selectively switch among the four modes of operation.
  • the user of the system can start simulation, stop simulation, assert input values, inspect values, single step cycle by cycle, and switch back and forth among the four different modes.
  • the system can simulate the circuit in software for a time period, accelerate the simulation through the hardware model, and return back to software simulation mode.
  • the SEmulation system provides the user with the capability to "see” every modeled component, regardless of whether it's modeled in software or hardware. For a variety of reasons, combinational components are not as “visible” as registers, and thus, obtaining combinational component data is difficult.
  • FPGAs which are used in the reconfigurable board to model the hardware portion of the user's circuit design, typically model combinational components as look-up tables (LUT), instead of actual combinational components. Accordingly, the SEmulation system reads register values and then regenerates combinational components. Because some overhead is needed to regenerate the combinational components, this regeneration process is not performed all the time; rather, it is done only upon the user's request.
  • a clock edge detection mechanism is provided to trigger the generation of a so-called software clock that drives the enable input to the various registers in the hardware model.
  • the timing is strictly controlled through a double-buffered circuit implementation so that the software clock enable signal enters the register model before the data to these models.
  • the software clock gates the data synchronously to ensure that all data values are gated together without any risk of hold-time violations.
  • Software simulation is also fast because the system logs all input values and only selected register values/states, thus overhead is minimized by decreasing the number of I/O operations.
  • the user can selectively select the logging frequency.
  • the SEmulation system is capable of emulating the user's circuit within its target system environment.
  • the target system outputs data to the hardware model for evaluation and the hardware model also outputs data to the target system.
  • the software kernel controls the operation of this mode so that the user still has the option to start, stop, assert values, inspect values, single step, and switch from one mode to another.
  • Logs provide the user with a historical record of the simulation session. Unlike known simulation systems, the SEmulation system does not log every single value, internal state, or value change during the simulation process. The SEmulation system logs only selected values and states based on a logging frequency (i.e., log 1 record every N cycles).
  • a logging frequency i.e., log 1 record every N cycles.
  • VCD on-demand system allows the user to view any simulation target range (i.e., simulation times) on demand without simulation rerun.
  • the SEmulation system implements an array of FPGA chips on a reconfigurable board. Based on the hardware model, the SEmulation system partitions, maps, places, and routes each selected portion of the user's circuit design onto the FPGA chips. Thus, for example, a 4x4 array of 16 chips may be modeling a large circuit spread out across these 16 chips.
  • the interconnect scheme allows each chip to access another chip within 2 "jumps" or links.
  • Each FPGA chip implements an address pointer for each of the I/O address spaces (i.e., REG, CLK, S2H, H2S).
  • the combination of all address pointers associated with a particular address space are chained together. So, during data transfer, word data in each chip is sequentially selected from/to the main FPGA bus and PCI bus, one word at a time for the selected address space in each chip, and one chip at a time, until the desired word data have been accessed for that selected address space.
  • This sequential selection of word data is accomplished by a propagating word selection signal. This word selection signal travels through the address pointer in a chip and then propagates to the address pointer in the next chip and continues on till the last chip or the system initializes the address pointer.
  • the FPGA bus system in the reconfigurable board operates at twice the PCI bus bandwidth but at half the PCI bus speed.
  • the FPGA chips are thus separated into banks to utilize the larger bandwidth bus.
  • the throughput of this FPGA bus system can track the throughput of the PCI bus system so performance is not lost by reducing the bus speed. Expansion is possible through piggyback boards that extend the bank length.
  • denser FPGA chips are used.
  • One such denser chip is the Altera 10K130V and 10K250V chips. Use of these chips alters the board design such that only four FPGA chips, instead of eight less dense FPGA chips (e.g., Altera 10K100), are used per board.
  • the FPGA array in the Simulation system is provided on the motherboard through a particular board interconnect structure.
  • Each chip may have up to eight sets of interconnections, where the interconnections are arranged according to adjacent direct-neighbor interconnects (i.e., N[73:0], S[73:0], W[73:0], E[73:0]), and one-hop neighbor interconnects (i.e., NH[27:0], SH[27:0], XH[36:0], XH[72:37]), excluding the local bus connections, within a single board and across different boards.
  • Each chip is capable of being interconnected directly to adjacent neighbor chips, or in one hop to a non-adjacent chip located above, below, left, and right.
  • the array In the X direction (east-west), the array is a torus.
  • the array In the Y direction (north-south), the array is a mesh.
  • interconnects alone can couple logic devices and other components within a single board.
  • inter- board connectors are provided to couple these boards and interconnects together across different boards to carry signals between (1) the PCI bus via the motherboard and the array boards, and (2) any two array boards.
  • a motherboard connector connects the board to the motherboard, and hence, to the PCI bus, power, and ground.
  • the motherboard connector is not used for direct connection to the motherboard.
  • every other board is directly connected to the motherboard, and interconnects and local buses of these boards are coupled together via inter-board connectors arranged solder-side to component-side.
  • PCI signals are routed through one of the boards (typically the first board) only. Power and ground are applied to the other motherboard connectors for those boards. Placed solder-side to component-side, the various inter-board connectors allow communication among the PCI bus components, the FPGA logic devices, memory devices, and various Simulation system control circuits.
  • a Simulation server is provided to allow multiple users to access the same reconfigurable hardware unit.
  • multiple workstations across a network or multiple users/processes in a non-network environment can access the same server-based reconfigurable hardware unit to review/debug the same or different user circuit design.
  • the access is accomplished via a time-shared process in which a scheduler determines access priorities for the multiple users, swaps jobs, and selectively locks hardware model access among the scheduled users.
  • each user can access the server to map his/her separate user design to the reconfigurable hardware model for the first time, in which case the system compiles the design to generate the software and hardware models, performs the clustering operation, performs place-and-route operations, generates a bitstream configuration file, and reconfigures the FPGA chips in the reconfigurable hardware unit to model the hardware portion of the user's design.
  • the hardware unit can be released for access by another user.
  • the server provides the multiple users or processes to access the reconfigurable hardware unit for acceleration and hardware state swapping purposes.
  • the Simulation server includes the scheduler, one or more device drivers, and the reconfigurable hardware unit.
  • the scheduler in the Simulation server is based on a preemptive round robin algorithm.
  • the server scheduler includes a simulation job queue table, a priority sorter, and a job swapper.
  • the restore and playback function of the present invention facilitates the non-network multiprocessing environment as well as the network multi-user environment in which previous checkpoint state data can be downloaded and the entire simulation state associated with that checkpoint can be restored for playback debugging or cycle-by-cycle stepping.
  • the Memory Simulation or memory mapping aspect of the present invention provides an effective way for the Simulation system to manage the various memory blocks associated with the configured hardware model of the user's design, which was programmed into the array of FPGA chips in the reconfigurable hardware unit.
  • the memory Simulation aspect of the invention provides a structure and scheme where the numerous memory blocks associated with the user's design is mapped into the SRAM memory devices in the Simulation system instead of inside the logic devices, which are used to configure and model the user's design.
  • the memory Simulation system includes a memory state machine, an evaluation state machine, and their associated logic to control and interface with: (1) the main computing system and its associated memory system, (2) the SRAM memory devices coupled to the FPGA buses in the Simulation system, and (3) the FPGA logic devices which contain the configured and programmed user design that is being debugged.
  • the operation of the memory Simulation system in accordance with one embodiment of the present invention is generally as follows.
  • the Simulation write/read cycle is divided into three periods - DMA data transfer, evaluation, and memory access.
  • the FPGA logic device side of the memory Simulation system includes an evaluation state machine, an FPGA bus driver, and a logic interface for each memory block N to interface with the user's own memory interface in the user design to handle: (1) data evaluations among the FPGA logic devices, and (2) write/read memory access between the FPGA logic devices and the SRAM memory devices.
  • the FPGA I/O controller side includes a memory state machine and interface logic to handle DMA, write, and read operations between: (1) main computing system and SRAM memory devices, and (2) FPGA logic devices and the SRAM memory devices.
  • One embodiment of the present invention is a coverification system that includes a reconfigurable computing system (hereinafter “RCC computing system”) and a reconfigurable computing hardware array (hereinafter “RCC hardware array”).
  • RCC computing system reconfigurable computing system
  • RCC hardware array reconfigurable computing hardware array
  • the target system and the external I/O devices are not necessary since they can be modeled in software.
  • the target system and the external I/O devices are actually coupled to the coverification system to obtain speed and use actual data, rather than simulated test bench data.
  • a coverification system can incorporate the RCC computing system and RCC hardware array along with other functionality to debug the software portion and hardware portion of a user's design while using the actual target system and/or I/O devices.
  • the RCC computing system also contains clock logic (for clock edge detection and software clock generation), test bench processes for testing the user design, and device models for any I/O device that the user decides to model in software instead of using an actual physical I/O device.
  • clock logic for clock edge detection and software clock generation
  • test bench processes for testing the user design
  • device models for any I/O device that the user decides to model in software instead of using an actual physical I/O device.
  • the software clock is provided to the external interface to function as the external clock source for the target system and the external I/O devices.
  • the use of this software clock provides the synchronization necessary to process incoming and outgoing data. Because the RCC computing system-generated software clock is the time base for the debug session, simulated and hardware-accelerated data are synchronized with any data that is delivered between the coverification system and the external interface.
  • the coverification system contains a control logic that provides traffic control between: (1) the RCC computing system and the RCC hardware array, and (2) the external interface (which are coupled to the target system and the external I/O devices) and the RCC hardware array. Because the RCC computing system has the model of the entire design in software, including that portion of the user design modeled in the RCC hardware array, the RCC computing system must also have access to all data that passes between the external interface and the RCC hardware array. The control logic ensures that the RCC computing system has access to these data.
  • FIG. 1 shows a high level overview of one embodiment of the present invention.
  • a workstation 10 is coupled to a reconfigurable hardware model 20 and emulation interface 30 via PCI bus system 50.
  • the reconfigurable hardware model 20 is coupled to the emulation interface 30 via PCI bus 50, as well as cable 61.
  • a target system 40 is coupled to the emulation interface 30 via cables 60.
  • the in-circuit emulation set-up 70 which comprises the emulation interface 30 and target system 40 (as shown in the dotted line box) are not provided in this set-up when emulation of the user's circuit design within the target system's environment is not desired during a particular test/debug session. Without the in-circuit emulation set-up 70, the reconfigurable hardware model 20 communicates with the workstation 10 via the PCI bus 50.
  • the reconfigurable hardware model 20 imitates or mimics the user's circuit design of some electronic subsystem in the target system.
  • input and output signals between the target system 40 and the modeled electronic subsystem must be provided to the reconfigurable hardware model 20 for evaluation.
  • the input and output signals of the target system 40 to/from the reconfigurable hardware model 20 are delivered via cables 60 through the emulation interface 30 and the PCI bus 50.
  • input/output signals of the target system 40 can be delivered to the reconfigurable hardware model 20 via emulation interface 30 and cables 61.
  • control data and some substantive simulation data pass between the reconfigurable hardware model 20 and the workstation 10 via the PCI bus 50.
  • the workstation 10 runs the software kernel that controls the operation of the entire SEmulation system and must have access (read/write) to the reconfigurable hardware model 20.
  • a workstation 10 complete with a computer, keyboard, mouse, monitor and appropriate bus/network interface allows a user to enter and modify data describing the circuit design of an electronic system.
  • Exemplary workstations include a Sun Microsystems SPARC or ULTRA-SPARC workstation or an Intel/Microsoft-based computing station.
  • the workstation 10 comprises a CPU 11, a local bus 12, a host/PCI bridge 13, memory bus 14, and main memory 15.
  • the various software simulation, simulation by hardware acceleration, in-circuit emulation, and post-simulation analysis aspects of the present invention are provided in the workstation 10, reconfigurable hardware model 20, and emulation interface 30.
  • the algorithm embodied in software is stored in main memory 15 during a test/debug session and executed through the CPU 11 via the workstation's operating system.
  • control passes to its initialization code to set up necessary data structures, and load and initialize device drivers. Control is then passed to the command line interpreter (CLI), which prompts the user to indicate the program to be run.
  • CLI command line interpreter
  • the operating system determines the amount of memory needed to run the program, locates the block of memory, or allocates a block of memory and accesses the memory either directly or through BIOS. After completion of the memory loading process, the application program begins execution.
  • One embodiment of the present invention is a particular application program for SEmulation.
  • the application program may require numerous services from the operating system, including, but not limited to, reading from and writing to disk files, performing data communications, and interfacing with the display/keyboard/mouse.
  • the workstation 10 has the appropriate user interface to allow the user to enter the circuit design data, edit the circuit design data, monitor the progress of simulations and emulations while obtaining results, and essentially control the simulation and emulation process.
  • the user interface includes user-accessible menu- driven options and command sets which can be entered with the keyboard and mouse and viewed with a monitor.
  • the user uses a computing station 80 with a keyboard 90.
  • the user typically creates a particular circuit design of an electronic system and enters a HDL (usually structured RTL level) code description of his designed system into the workstation 10.
  • the SEmulation system of the present invention performs component type analysis, among other operations, for partitioning the modeling between software and hardware.
  • the SEmulation system models behavior, RTL, and gate level code in software.
  • the system can model RTL and gate level code; however, the RTL level must be synthesized to gate level prior to hardware modeling.
  • the gate level code can be processed directly into usable source design database format for hardware modeling. Using the RTL and gate level codes, the system automatically performs component type analysis to complete the partition step.
  • the system maps some portion of the circuit design into hardware for fast simulation via hardware acceleration.
  • the user can also couple the modeled circuit design to the target system for real environment in-circuit emulation. Because the software simulation and the hardware acceleration engines are tightly coupled, through the software kernel, the user can then simulate the overall circuit design using software simulation, accelerate the test/debug process by using the hardware model of the mapped circuit design, return to the simulation portion, and return to the hardware acceleration until the test/debug process is complete.
  • the ability to switch between software simulation and hardware acceleration cycle-by-cycle and at will by the user is one of the valuable features of this embodiment.
  • This feature is particularly useful in the debug process by allowing the user to go to a particular point or cycle very quickly using the hardware acceleration mode and then using software simulation to examine various points thereafter to debug the circuit design.
  • the SEmulation system makes all components visible to the user whether the internal realization of the component is in hardware or software. The SEmulation system accomplishes this by reading the register values from the hardware model and then rebuilding the combinational components using the software model when the user requests such a read.
  • the workstation 10 is coupled to a bus system 50.
  • the bus system can be any available bus system that allows various agents, such as the workstation 10, reconfigurable hardware model 20, and emulation interface 30, to be operably coupled together.
  • the bus system is fast enough to provide real-time or near real-time results to the user.
  • One such bus system is the bus system described in the Peripheral Component Interconnect (PCI) standard, which is incorporated herein by reference.
  • PCI Peripheral Component Interconnect
  • revision 2.0 of the PCI standard provides for a 33 MHz bus speed.
  • Revision 2.1 provides support for 66 MHz bus speed.
  • the workstation 10, reconfigurable hardware model 20, and emulation interface 30 may comply with the PCI standard.
  • PCI bus communication between the workstation 10 and the reconfigurable hardware model 20 is handled on the PCI bus.
  • Other PCI-compliant devices may be found in this bus system. These devices may be coupled to the PCI bus at the same level as the workstation 10, reconfigurable hardware model 20, and emulation interface 30, or other levels.
  • Each PCI bus at a different level, such as PCI bus 52 is coupled to another PCI bus level, such as PCI bus 50, if it exists at all, through a PCI-to-PCI bridge 51.
  • PCI bus 52 two PCI devices 53 and 54 may be coupled therewith.
  • the reconfigurable hardware model 20 comprises an array of field-programmable gate array (FPGA) chips that can be programmably configured and reconfigured to model the hardware portion of the user's electronic system design.
  • the hardware model is reconfigurable; that is, it can reconfigure its hardware to suit the particular computation or user circuit design at hand. If, for example, many adders or multiplexers are required, the system is configured to include many adders and multiplexers. As other computing elements or functions are needed, they may also be modeled or formed in the system. In this way, the system can be optimized to perform specialized computations or logic operations. Reconfigurable systems are also flexible, so that users can work around minor hardware defects that arise during manufacture, testing, or use.
  • the reconfigurable hardware model 20 comprises a two-dimensional array of computing elements consisting of FPGA chips to provide the computational resources for various user circuit designs and applications. More details on the hardware configuration process will be provided.
  • the reconfigurable hardware model is reconfigurable via the use of field programmable devices.
  • ASIC application specific integrated circuit
  • Still other embodiments may be in the form of a custom integrated circuit.
  • reconfigurable devices will be used to simulate/emulate the user's circuit design so that appropriate changes can be made prior to actual prototype manufacturing.
  • an actual ASIC or custom integrated circuit can be used, although this deprives the user of the ability to quickly and cost-effectively change a possibly non-functional circuit design for re-simulation and re-emulation.
  • an ASIC or custom IC has already been manufactured and readily available so that emulation with an actual non-reconfigurable chip may be preferable.
  • the software in the workstation along with its integration with an external hardware model, provides a greater degree of flexibility, control, and performance for the end user over existing systems.
  • a model of the circuit design and the relevant parameters e.g., input test-bench stimulus, overall system output, intermediate results
  • the user can use either schematic capture tools or synthesis tools to define the system circuit design.
  • the user starts with a circuit design of an electronic system, usually in draft schematic form, which is then converted to HDL form using synthesis tools.
  • the HDL can also be directly written by the user.
  • Exemplary HDL languages include Verilog and VHDL; however, other languages are also available.
  • a circuit design represented in HDL comprises many concurrent components. Each component is a sequence of code which either defines the behavior of a circuit element or controls the execution of the simulation.
  • the SEmulation system analyzes these components to determine their component types and the compiler uses this component type information to build different execution models in software and hardware. Thereafter, the user can use the SEmulation system of the present invention. The designer can verify the accuracy of the circuit through simulation by applying various stimuli such as input signals and test vector patterns to the simulated model. If, during the simulation, the circuit does not behave as planned, the user re-defines the circuit by modifying the circuit schematic or the HDL file.
  • the algorithm starts at step 100. After loading the HDL file into the system, the system compiles, partitions, and maps the circuit design to appropriate hardware models. The compilation, partition, and mapping steps are discussed in more detail below.
  • One embodiment of the present invention uses a 2-bit wide data path to provide a 4-state value for the bus signal - "00" is logic low, “01” is logic high, “10” is “z,” and “11” is “x.”
  • software models can deal with “0,” “1,” "x”(bus conflicts or unknown value), and "z” (no driver or high impedance).
  • hardware cannot deal with the unknown values "x,” so the reset sequence, which varies depending on the particular applicable code, resets the register values to all "0” or all “ 1.”
  • step 105 the user decides whether to simulate the circuit design. Typically, a user will start the system with software simulation first. Thus, if the decision at step 105 resolves to "YES,” software simulation occurs at step 110.
  • step 115 The user can stop the simulation to inspect values as shown in step 115. Indeed, the user can stop the simulation at any time during the test/debug session as shown by the dotted lines extending from step 115 to various nodes in the hardware acceleration mode, ICE mode, and post-simulation mode. Executing step 115 takes the user to step 160.
  • step 115 branches to the stop/value inspect routine.
  • the stop/value inspect routine starts at step 160.
  • step 165 the user must decide whether to stop the simulation at this point and inspect values. If step 165 resolves to "YES,” step 170 stops the simulation that may be currently underway and inspects various values to check for correctness of the circuit design.
  • step 175 the algorithm returns to the point at which it branched, which is at step 115.
  • the user can continue to simulate and stop/inspect values for the remainder of the test/debug session or proceed forward to the in-circuit emulation step.
  • step 120 the user decides whether to accelerate the test/debug process by accelerating the simulation through the hardware portion of the modeled circuit design. If the decision at step 120 resolves to "YES,” then hardware model acceleration occurs at step 125.
  • the SEmulation system mapped some portions into a hardware model.
  • the system moves register and combinational components into the hardware model and moves the input and evaluation values to the hardware model.
  • the evaluation occurs in the hardware model for a long time period at the accelerated speed.
  • the kernel writes test-bench output to the hardware model, updates the software clock, then reads the hardware model output values cycle-by-cycle.
  • values from the entire software model of the user's circuit design which is the entire circuit design, can be made available by outputting register values and combinational components by regenerating combinational components with the register values. Because of the need for software intervention to regenerate these combinational components, outputs of values for the entire software model are not provided at every cycle; rather, values are provided to the user only if the user wants such values. This specification will discuss the combinational component regeneration process later.
  • the user can stop the hardware acceleration mode at any time as indicated by step 115. If the user wants to stop, the algorithm proceeds to steps 115 and 160 to branch to the stop/value inspect routine.
  • the user can stop the hardware accelerated simulation process at any time and inspect values resulting from the simulation process, or the user can continue with the hardware-accelerated simulation process.
  • the stop/value inspect routine branches to steps 160, 165, 170, and 175, which were discussed above in the context of stopping the simulation.
  • the user can decide to continue with the hardware-accelerated simulation or perform pure simulation instead at step 135. If the user wants to simulate further, the algorithm proceeds to step 105. If not, the algorithm proceeds to the post-simulation analysis at step 140.
  • the SEmulation system provides a number of post-simulation analysis features.
  • the system logs all inputs to the hardware model.
  • the system logs all values of hardware register components at a user-defined logging frequency (e.g., 1/10,000 record cycle).
  • the logging frequency determines how often the output values are recorded.
  • output values are recorded once every 10,000 cycles.
  • the higher the logging frequency the more information is recorded for later post-simulation analysis.
  • the selected logging frequency has a causal relationship to the SEmulation speed, the user selects the logging frequency with care.
  • a higher logging frequency will decrease the SEmulation speed because the system must spend time and resources to record the output data by performing I/O operations to memory before further simulation can be performed.
  • the user selects a particular point at which simulation is desired.
  • the user can then perform analysis after SEmulation by running the software simulation with input logs to the hardware model to compute the value changes and internal states of all hardware components.
  • the hardware accelerator is used to simulate the data from the selected logging point to analyze simulation results.
  • This post-simulation analysis method can link to any simulation waveform viewer for post-simulation analysis. More detailed discussion will follow.
  • step 145 the user can opt to emulate the simulated circuit design within its target system environment. If step 145 resolves to "NO,” the algorithm ends and the SEmulation process ends at step 155. If emulation with the target system is desired, the algorithm proceeds to step 150. This step involves activating the emulation interface board, plugging the cable and chip pin adapter to the target system, and running the target system to obtain the system I/O from the target system.
  • the system I/O from the target system includes signals between the target system and the emulation of the circuit design.
  • the emulated circuit design receives input signals from the target system, processes these, sends them to the SEmulation system for further processing, and outputs the processed signals to the target system.
  • the emulated circuit design sends output signals to the target system, which processes these, and possibly outputs the processed signals back to the emulated circuit design.
  • the performance of the circuit design can be evaluated in its natural target system environment.
  • the user After the emulation with the target system, the user has results that validate the circuit design or reveal non-functional aspects. At this point, the user can simulate/emulate again as indicated at step 135, stop altogether to modify the circuit design, or proceed to integrated circuit fabrication based on the validated circuit design.
  • FIG. 3 shows two sets of information: one set of information distinguishes the operations performed during compile time and simulation/emulation run time; and the other set of information shows the partitioning between software models and hardware models.
  • the SEmulation system in accordance with one embodiment of the present invention needs the user circuit design as input data 200.
  • the user circuit design is in some form of HDL file (e.g., Verilog, VHDL).
  • the SEmulation system parses the HDL file so that behavior level code, register transfer level code, and gate level code can be reduced to a form usable by the SEmulation system.
  • the system generates a source design database for front end processing step 205.
  • the processed HDL file is now usable by the SEmulation system.
  • the parsing process converts ASCII data to an internal binary data structure and is known to those ordinarily skilled in the art. Please refer to ALFRED V. AHO, RAVI SETHI, AND JEFFREY D. ULLMAN, COMPILERS: PRINCIPLES, TECHNIQUES, AND TOOLS (1988), which is incorporated by reference herein.
  • Compile time is represented by processes 225 and run time is represented by processes/elements 230.
  • the SEmulation system compiles the processed HDL file by performing component type analysis.
  • the component type analysis classifies HDL components into combinational components, register components, clock components, memory components, and test-bench components. Essentially, the system partitions the user circuit design into control and evaluation components.
  • the SEmulation compiler 210 essentially maps the control components of the simulation into software and the evaluation components into software and hardware.
  • the compiler 210 generates a software model for all HDL components.
  • the software model is cast in code 215. Additionally, the SEmulation compiler 210 uses the component type information of the HDL file, selects or generates hardware logic blocks/elements from a library or module generator, and generates a hardware model for certain HDL components. The end result is a so-called "bitstream" configuration file 220.
  • the software model in code form is stored in main memory where the application program associated with the SEmulation program in accordance with one embodiment of the present invention is stored.
  • This code is processed in the general purpose processor or workstation 240.
  • the configuration file 220 for the hardware model is used to map the user circuit design into the reconfigurable hardware boards 250.
  • those portions of the circuit design that have been modeled in hardware are mapped and partitioned into the FPGA chips in the reconfigurable hardware boards 250.
  • test-bench stimulus and test vector data as well as other test-bench resources 235 are applied to the general purpose processor or workstation 240 for simulation purposes.
  • the user can perform emulation of the circuit design via software control.
  • the reconfigurable hardware boards 250 contain the user's emulated circuit design. This SEmulation system has the ability to let the user selectively switch between software simulation and hardware emulation, as well as stop either the simulation or emulation process at any time, cycle-by- cycle, to inspect values from every component in the model, whether register or combinational.
  • the SEmulation system passes data between the test-bench 235 and the processor/workstation 240 for simulation and the test-bench 235 and the reconfigurable hardware boards 250 via data bus 245 and processor/workstation 240 for emulation.
  • emulation data can pass between the reconfigurable hardware boards 250 and the target system 260 via the emulation interface 255 and data bus 245.
  • the kernel is found in the software simulation model in the memory of the processor/workstation 240 so data necessarily pass between the processor/workstation 240 and the reconfigurable hardware boards 250 via data bus 245.
  • FIG. 4 shows a flow chart of the compilation process in accordance with one embodiment of the present invention.
  • the compilation process is represented as processes 205 and 210 in FIG. 3.
  • the compilation process in FIG. 4 starts at step 300.
  • Step 301 processes the front end information.
  • gate level HDL code is generated.
  • the user has converted the initial circuit design into HDL form by directly handwriting the code or using some form of schematic or synthesis tool to generate the gate level HDL representations of the code.
  • the SEmulation system parses the HDL file (in ASCII format) into a binary format so that behavior level code, register transfer level (RTL) code, and gate level code can be reduced to an internal data structure form usable by the SEmulation system.
  • the system generates a source design database containing the parsed HDL code.
  • Step 302 performs component type analysis by classifying HDL components into combinational components, register components, clock components, memory components, and test-bench components as shown in component type resource 303.
  • the SEmulation system generates hardware models for register and combinational components, with some exceptions as discussed below.
  • Test-bench and memory components are mapped in software.
  • Some clock components e.g., derived clocks
  • Combinational components are stateless logic components whose output values are a function of current input values and do not depend on the history of input values.
  • Examples of combinational components include primitive gates (e.g., AND, OR, XOR, NOT), selector, adder, multiplier, shifter, and bus drivers.
  • Register components are simple storage components. The state transition of a register is controlled by a clock signal.
  • One form of register is edge-triggered which may change states when an edge is detected.
  • Another form of register is a latch, which is level triggered. Examples include flip-flops (D-type, JK-type) and level-sensitive latches.
  • Clock components are components that deliver periodic signals to logic devices to control their behavior. Typically, clock signals control the update of registers.
  • the clock signal is initially at logic "0.” After 5 time units, the clock signal changes to logic "1.” After 5 time units, the clock signal reverts back to logic "0.”
  • the primary clock signals are generated in software and only a few (i.e., 1-10) primary clocks are found in a typical user circuit design. Derived or gated clocks are generated from a network of combinational logic and registers that are in turn driven by the primary clocks. Many (i.e., 1,000 or more) derived clocks are found in a typical user circuit design.
  • Memory components are block storage components with address and control lines to access individual data in specific memory locations. Examples include ROM, asynchronous RAM, and synchronous RAM.
  • Test-bench components are software processes used to control and monitor the simulation processes. Accordingly, these components are not part of the hardware circuit design under test. Test-bench components control the simulation by generating clock signals, initializing simulation data, and reading simulation test vector patterns from disk/memory. Test-bench components also monitor the simulation by checking for changes in value, performing value change dump, checking asserted constraints on signal value relations, writing output test vectors to disk/memory, and interfacing with various waveform viewers and debuggers.
  • the SEmulation system performs component type analysis as follows. The system examines the binary source design database. Based on the source design database, the system can characterize or classify the elements as one of the above component types. Continuous assignment statements are classified as combinational components. Gate primitives are either combinational type or latch form of register type by language definition. Initialization code are treated as test-benches of initialization type.
  • An always process that drives nets without using the nets is a test-bench of driver type.
  • An always process that reads nets without driving the nets is a test-bench of monitor type.
  • An always process with delay controls or multiple event controls are test-benches of general type.
  • An always process with a single event control and driving a single net can be one of the following: (1) If the event control is edge-triggered event, then the process is an edge-triggered type register component. (2) If a net driven in a process is not defined in all possible execution paths, then the net is a latch type of register. (3) If a net driven in a process is defined in all possible execution paths, then the net is a combinational component.
  • An always process with a single event control but driving multiple nets can be decomposed into several processes driving each net separately to derive their respective component types separately.
  • the decomposed processes can then be used to determine component type.
  • Step 304 generates a software model for all HDL components, regardless of component type.
  • the user is capable of simulating the entire circuit design using the complete software model.
  • Test-bench processes are used to drive the stimulus input, test vector patterns, control the overall simulation, and monitor the simulation process.
  • Step 305 performs clock analysis.
  • the clock analysis includes two general steps: (1) clock extraction and sequential mapping, and (2) clock network analysis.
  • the clock extraction and sequential mapping step includes mapping the user's register components into the SEmulation system's hardware register model and then extracting clock signals out of the system's hardware register components.
  • the clock network analysis step includes determining primary clocks and derived clocks based on the extracted clock signals, and separating the gated clock network and gated data network. A more detailed description will be provided with respect to FIG. 16.
  • Step 306 performs residence selection.
  • the system in conjunction with the user, selects the components for hardware models; that is, of the universe of possible hardware components that can be implemented in the hardware model of the user's circuit design, some hardware components will not be modeled in hardware for a variety of reasons. These reasons include component types, hardware resource constraints (i.e., floating point operations and large multiply operations stay in software), simulation and communication overhead (i.e., small bridge logic between test-bench processes stay in software, and signals that are monitored by test-bench processes stay in software), and user preferences. For a variety of reasons including performance and simulation monitoring, the user can force certain components that would otherwise be modeled in hardware to stay in software.
  • Step 307 maps the selected hardware models into a reconfigurable hardware emulation board.
  • step 307 maps takes the netlist and maps the circuit design into specific FPGA chips. This step involves grouping or clustering logic elements together. The system then assigns each group to a unique FPGA chip or several groups to a single FPGA chip. The system may also split groups to assign them to different FPGA chips. In general, the system assigns groups to FPGA chips. More detailed discussion will be provided below with respect to FIG. 6. The system places the hardware model components into a mesh of FPGA chips to minimize inter-chip communication overhead.
  • the array comprises a 4x4 array of FPGAs, a PCI interface unit, and a software clock control unit.
  • the array of FPGAs implements a portion of the user's hardware circuit design, as determined above in steps 302-306 of this software compilation process.
  • the PCI interface unit allows the reconfigurable hardware emulation model to communicate with the workstation via the PCI bus.
  • the software clock avoids race conditions for the various clock signals to the array of FPGAs.
  • step 307 routes the FPGA chips according to the communication schedule among the hardware models.
  • Step 308 inserts the control circuits.
  • These control circuits include the I/O address pointers and data bus logic for communicating with the DMA engine to the simulator (discussed below with respect to FIGS. 11, 12, and 14), and the evaluation control logic to control hardware state transitions and wire multiplexing (discussed below with respect to FIGS. 19 and 20).
  • a direct memory access (DMA) unit provides an additional data channel between peripherals and main memory in which the peripherals can directly access (i.e., read, write) the main memory without the intervention of the CPU.
  • the address pointer in each FPGA chip allows data to move between the software model and the hardware model in light of the bus size limitations.
  • the evaluation control logic is essentially a finite state machine that ensures that the clock enable inputs to registers to be asserted before the clock and data inputs enter these registers.
  • Step 309 generates the configuration files for mapping the hardware model to FPGA chips. In essence, step 309 assigns circuit design components to specific cells or gate level components in each chip. Whereas step 307 determines the mapping of hardware model groups to specific FPGA chips, step 309 takes this mapping result and generates a configuration file for each FPGA chip.
  • Step 310 generates the software kernel code.
  • the kernel is a sequence of software code that controls the overall SEmulation system. The kernel cannot be generated until this point because portions of the code require updating and evaluating hardware components. Only after step 309 has the appropriate mapping to hardware models and FPGA chips occurred. More detailed discussion will be provided below with respect to FIG. 5. The compilation ends at step 311.
  • the software kernel code is generated in step 310 after the software and hardware models have been determined.
  • the kernel is a piece of software in the SEmulation system that controls the operation of the overall system.
  • the kernel controls the execution of the software simulation as well as the hardware emulation. Because the kernel also resides in the center of the hardware model, the simulator is integrated with the emulator.
  • the SEmulation system in accordance with one embodiment of the present invention does not require the simulator to interact with the emulator from the outside.
  • One embodiment of the kernel is a control loop shown in FIG. 5.
  • the kernel begins at step 330.
  • Step 331 evaluates the initialization code.
  • the control loop begins and cycles repeatedly until the system observes no active test-bench processes, in which case the simulation or emulation session has completed.
  • Step 332 evaluates the active test-bench components for the simulation or emulation.
  • Step 333 evaluates clock components. These clock components are from the test-bench process. Usually, the user dictates what type of clock signal will be generated to the simulation system. In one example (discussed above with respect to component type analysis and reproduced here), a clock component as designed by a user in the test- bench process is as follows: always begin
  • the user has decided, in this clock component example, that a logic "0" signal will be generated first, and then after 5 simulation times later, a logic "1" signal will be generated. This clock generation process will cycle continuously until stopped by the user. These simulation times are advanced by the kernel.
  • Step 334 inquires whether any active clock edge is detected, which would result in some kind of logic evaluation in the software and possible hardware model (if emulation is running).
  • the clock signal which the kernel uses to detect an active clock edge, is the clock signal from the test-bench process. If the decision step 334 evaluates to "NO,” then the kernel proceeds to step 337. If the decision step 334 evaluates to "YES,” resulting in step 335 updating registers and memories, and step 336 propagating combinational components. Step 336 essentially takes care of combinational logic which needs some time to propagate values through the combinational logic network after a clock signal has been asserted. Once the values have propagated through the combinational components and stabilized, the kernel proceeds to step 337.
  • the kernel controls the emulator portion of the SEmulation system.
  • the kernel can accelerate the evaluation of the hardware model in steps 334 and 335 whenever any active clock edge is detected.
  • the SEmulation system in accordance with one embodiment of the present invention can accelerate the hardware emulator through the software kernel and based on component type (e.g., register, combinational).
  • the kernel controls the execution of the software and hardware model cycle by cycle.
  • the emulator hardware model can be characterized as a simulation coprocessor to the general-purpose processor running the simulation kernel. The coprocessor speeds up the simulation task.
  • Step 337 evaluates active test-bench components.
  • Step 338 advances the simulation time.
  • Step 339 provides the boundary for the control loop that begins at step 332.
  • Step 339 determines whether any test-bench processes are active. If so, the simulation and/or emulation is still running and more data should be evaluated.
  • the kernel loops to step 332 to evaluate any active test-bench components. If no test-bench processes are active, then the simulation and emulation processes have completed.
  • Step 340 ends the simulation/emulation process.
  • the kernel is the main control loop that controls the operation of the overall SEmulation system. So long as any test-bench processes are active, the kernel evaluates active test-bench components, evaluates clocks components, detects clock edges to update registers and memories as well as propagate combinational logic data, and advances the simulation time.
  • FIG. 6 shows one embodiment of a method of automatically mapping hardware models to reconfigurable boards.
  • a netlist file provides the input to the hardware implementation process.
  • the netlist describes logic functions and their interconnections.
  • the hardware model-to-FPGA implementation process includes three independent tasks: mapping, placement, and routing.
  • the tools are generally referred to as "place-and-route" tools.
  • the design tool used may be Viewlogic Viewdraw, a schematic capture system, and Xilinx Xact place and route software, or Altera's MAX+PLUS II system.
  • the mapping task partitions the circuit design into the logic blocks, I/O blocks, and other FPGA resources. Although some logic functions such as flip-flops and buffers may map directly into the corresponding FPGA resource, other logic functions such as combinational logic must be implemented in logic blocks using mapping algorithms. The user can usually select mapping for optimal density or optimal performance.
  • the placement task involves taking the logic and I/O blocks from the mapping task and assigning them to physical locations within the FPGA array.
  • Current FPGA tools generally use some combination of three techniques: mincut, simulating annealing, and general force-directed relaxation (GFDR). These techniques essentially determine optimal placement based on various cost functions which depend on total net length of interconnections or the delay along a set of critical signal paths, among other variables.
  • the Xilinx XC4000 series FPGA tools use a variation of the mincut technique for initial placement followed by a GFDR technique for fine improvement in the placement.
  • the routing task involves determining the routing paths used to interconnect the various mapped and placed blocks.
  • One such router called a maze router, seeks the shortest path between two points. Since the routing task provides for direct interconnection among the chips, the placement of the circuits with respect to the chips is critical.
  • the hardware model can be described in either gate netlist 350 or RTL 357.
  • the RTL level code can be further synthesized to gate level netlist.
  • a synthesizer server 360 such as the Altera MAX+PLUS II programmable logic development tool system and software, can be used to produce output files for mapping purposes.
  • the synthesizer server 360 has the ability to match the user's circuit design components to any standard existing logic elements found in a library 361 (e.g., standard adders or standard multipliers), generate any parameterized and frequently used logic module 362 (e.g., non-standard multiplexers or non-standard adders), and synthesize random logic elements 363 (e.g., look-up table-based logic that implements a customized logic function).
  • the synthesizer server also removes redundant logic and unused logic.
  • the output files essentially synthesize or optimize the logic required by the user's circuit design.
  • the synthesizer server is capable of generating any logic element based on variations of standard logic elements or random logic elements that may not have any parallels in these variations or library standard logic elements.
  • the SEmulation system will initially perform the grouping or clustering operation 351.
  • the hardware model construction is based on the clustering process because the combinational logic and registers are separated from the clock. Thus, logic elements that share a common primary clock or gated clock signal may be better served by grouping them together and placed on a chip together.
  • the clustering algorithm is based on connectivity driven, hierarchical extraction, and regular structure extraction. If the description is in structured RTL 358, the SEmulation system can decompose the function into smaller units as represented by the logic function decomposition operation 359. At any stage, if logic synthesis or logic optimization is required, a synthesizer server 360 is available to transform the circuit design to a more efficient representation based on user directives.
  • the link to the synthesizer server is represented by dotted arrow 364.
  • the link to the synthesizer server 360 is represented by arrow 365.
  • the link to the synthesizer server 360 is represented by arrow 366.
  • the clustering operation 351 groups the logic components together in a selective manner based on function and size.
  • the clustering may involve only one cluster for a small circuit design or several clusters for a large circuit design. Regardless, these clusters of logic elements will be used in later steps to map them into the designated FPGA chips; that is, one cluster will be targeted for a particular chip and another cluster will be targeted for a different chip or possibly the same chip as the first cluster.
  • the logic elements in a cluster will stay together with the cluster in a chip, but for optimization purposes, a cluster may have to be split up into more than one chip.
  • the system After the clusters are formed in the clustering operation 351, the system performs a place-and-route operation. Initially, a coarse-grain placement operation 352 of the clusters into the FPGA chips is performed. The coarse-grain placement operation 352 initially places clusters of logic elements to selected FPGA chips. If necessary, the system makes the synthesizer server 360 available to the coarse-grain placement operation 352 as represented by arrow 367. A fine-grain placement operation is performed after the coarse-grain placement operation to fine-tune the initial placement. The SEmulation system uses a cost function based on pin usage requirements, gate usage requirements, and gate-to-gate hops to determine the optimal placement for both the coarse-grain and fine-grain placement operations.
  • the user's circuit design that is modeled in the hardware model comprises the total combination of circuits CKTQ.
  • Each cost function is defined such that the computed values of the calculated placement cost tend to generally promote: (1) a minimum number of "hops" between any two circuits CKTN-1 and CKTN in the FPGA array, and (2) placement of circuits CKTN-1 and CKTN in the FPGA array such that pin usage is minimized.
  • the cost function F(P, G, D) is defined as:
  • f(P,G,D) [CO* MAX each _ FPGA _ Mp (- ⁇ -)] + [Cl* MAX each FPGA chip (- ⁇ -)] +
  • the first term (i.e., C0*P) generates a first placement cost value based on the number of pins used and the number pins available.
  • the second term (i.e., C1*G) generates a second placement cost value based on the number of gates used and the number of gates available.
  • the third term (i.e., C2*D) generates a placement cost value based on the number of hops present between various interconnecting gates in the circuits CKTQ (i.e., CKT1, CKT2, . . . , CKTN).
  • the overall placement cost value is generated by iteratively summing these three placement cost values.
  • Constants CO, Cl, and C2 represent weighting constants that selectively skew the overall placement cost value generated from this cost function toward the factor or factors (i.e., pin usage, gate usage, or gate-to-gate hops) that is/are most important during any iterative placement cost calculation.
  • the placement cost is calculated repeatedly as the system selects different relative values for the weighting constants CO, Cl, and C2.
  • the system selects large values for CO and Cl relative to C2.
  • the system determines that optimizing pin usage/availability and gate usage/availability are more important than optimizing gate-to-gate hops in the initial placement of the circuits CKTQ in the array of FPGA chips.
  • the system selects small values for CO and Cl relative to C2.
  • the system determines that optimizing gate-to-gate hops is more important than optimizing pin usage/availability and gate usage/availability.
  • the system uses the same cost function.
  • the iterative steps with respect to the selection of CO, Cl, and C2 are the same as for the coarse-grain operation.
  • the fine-grain placement operation involves having the system select small values for CO and Cl relative to C2.
  • the cost function examines pin usage/availability (P), gate usage/availability (G), and gate-to-gate hops (D). Based on the cost function variables, P, G, and D, the cost function f(P, G, D) generates a placement cost value for placing circuits CKTQ in particular locations in the FPGA array.
  • Pin usage/availability P also represents the I/O capacity.
  • P used is the number of used pins by the circuits CKTQ for each FPGA chip.
  • P ava ii ab i e is the number of available pins in the FPGA chip. In one embodiment, is P aV aiia b i e is 264 (44 pins x 6 interconnections/chip), while hi another embodiment, P aVa ii ab i e is 265 (44 pins x 6 interconnections/chip + 1 extra pin).
  • the specific number of available pins depends on the type of FPGA chip used, the total number of interconnections used per chip, and the number of pins used for each interconnection.
  • P ava ii ab i e can vary considerably. So, to evaluate the first term of the cost function F(P, G, D) equation (i.e., C0*P), the ratio P USed /P a a ii ab i e is calculated for each FPGA chip. Thus, for a 4x4 array of FPGA chips, sixteen ratios P USed /P ava ii a bi e are calculated. The more pins are used for a given number of available pins, the higher the ratio. Of the sixteen calculated ratios, the ratio yielding the highest number is selected.
  • the first placement cost value is calculated from the first term C0*P by multiplying the selected maximum ratio P USed /P ava ii ab i e with the weighting constant CO. Because this first term depends on the calculated ratio P USed /P ava ii ab i e and the particular maximum ratio among the ratios calculated for each FPGA chip, the placement cost value will be higher for higher pin usage, all other factors being equal.
  • the system selects the placement yielding the lowest placement cost.
  • the particular placement yielding a maximum ratio P USed /P ava ii ab i e that is the lowest among all the maximums calculated for various placements is generally considered as the optimum placement in the FPGA array, all other factors being equal.
  • the gate usage/availability G is based on the number of gates allowable by each FPGA chip. In one embodiment, based on the location of the circuits CKTQ in the array, if the number of gates used G used in each chip is above a certain threshold, then this second placement cost (C1*G) will be assigned a value indicating that the placement is not feasible. Analogously, if the number of gates used in each chip containing circuits CKTQ is at or below a certain threshold, then this second term (CPG) will be assigned a value indicating that the placement is feasible.
  • the system may conclude through the cost function that this particular placement is infeasible.
  • the high number e.g., infinity
  • G ensures that the cost function will generate a high placement cost value indicating that the desired placement of the circuits CKTQ is not feasible and that an alternative placement should be determined.
  • the ratio G use ⁇ /Gavaii ab i e is calculated for each chip, where G used is the number of gates used by the circuits CKTQ in each FPGA chip, and G avai ⁇ abIe is the number of gates available in each chip.
  • the system uses the FLEX 1 OKI 00 chip for the FPGA array.
  • the FLEX 1 OKI 00 chip contains approximately 100,000 gates.
  • G available is equal to 100,000 gates.
  • sixteen ratios G used /G ava ii ab i e are calculated. The more gates are used for a given number of available gates, the higher the ratio.
  • the second placement cost value is calculated from the second term C1*G by multiplying the selected maximum ratio G used /G ava u ab i e with the weighting constant Cl. Because this second term depends on the calculated ratio G used /G ava ii ab i e and the particular maximum ratio among the ratios calculated for each FPGA chip, the placement cost value will be higher for higher gate usage, all other factors being equal.
  • the system selects the circuit placement yielding the lowest placement cost.
  • the particular placement yielding a maximum ratio G used /G ava ji ab i. that is the lowest among all the maximums calculated for various placements is generally considered as the optimum placement in the FPGA array, all other factors being equal.
  • the system selects some value for Cl initially. If the ratio G use ⁇ /G ava ii ab i e is greater than "1," then this particular placement is infeasible (i.e., at least one chip does not have enough gates for this particular placement of circuits). As a result, the system modifies Cl with a very high number (e.g., infinity) and accordingly, the second term C 1 *G will also be a very high number and the overall placement cost value f(P, G, D) will also be very high.
  • the third term C2*D represents the number of hops between all gates that require interconnection.
  • the number of hops also depends on the interconnection matrix.
  • the connectivity matrix provides the foundation for determining circuit paths between any two gates that need chip-to-chip interconnection. Not every gate needs the gate- to-gate interconnection. Based on the user's original circuit design and the partitioning of clusters to certain chips, some gates will not need any interconnection whatsoever because the logic element(s) connected to their respective input(s) and output(s) is/are located in the same chip. Other gates, however, need the interconnections because the logic elements) connected to their respective input(s) and output(s) is/are located in different chips.
  • each interconnection between chips such as interconnection 602 between chip Fl 1 and chip F14, represents 44 pins or 44 wire lines. In other embodiments, each interconnection represents more than 44 pins. In still other embodiments, each interconnection represents less than 44 pins.
  • data can pass from one chip to another chip within two "hops" or "jumps."
  • data can pass from chip Fl 1 to chip F12 in one hop via interconnection 601
  • data can pass from chip Fl 1 to chip F33 in two hops via either interconnections 600 and 606, or interconnections 603 and 610.
  • These exemplary hops are the shortest path hops between these sets of chips.
  • signals may be routed through various chips such that the number of hops between a gate in one chip and a gate in another chip exceeds the shortest path hop. The only circuit paths that must be examined in determining the number of gate-to-gate hops are the ones that need the interconnections.
  • the connectivity is represented by the sum of all hops between the gates that need the inter-chip mterconnections.
  • the shortest path between any two chips can be represented by one or two "hops" using the connectivity matrix of FIGS.7 and 8.
  • I/O capacity may limit the number of direct shortest path connections between any two gates in the array and hence, these signals must be routed through longer paths (and therefore more than two hops) to reach their destinations. Accordingly, the number of hops may exceed two for some gate-to-gate connections. Generally, all things being equal, a smaller number of hops results in a smaller placement cost.
  • This third term is the product of a weighting constant C2 and a summation component (S ).
  • the summation component is essentially the sum of all hops between each gate i and gate j in the user's circuit design that require chip- to-chip interconnections. As discussed above, not all gates need inter-chip interconnections. For those gates i and gates j that need inter-chip interconnections, the number of hops is determined. For all gates i and gates j, the total number of hops is added together.
  • the distance calculation can also be defined as:
  • M is the connectivity matrix.
  • One embodiment of the connectivity matrix is shown in FIG. 7. The distance is calculated for each gate-to-gate connection requiring an interconnection. Thus, for each gate i and gate j comparison, the connectivity matrix M is examined. More specifically,
  • a matrix is set up with all chips in the array such that each chip is identifiably numbered. These identifying numbers are set up at the top of the matrix as a column header. Similarly, these identifying numbers are set up along the side of the matrix as a row header. A particular entry at the intersection of a row and column in this matrix provides the direct connectivity data between the chip identified by the row and the chip identified by the column at which the intersections occur. For any distance calculation between chip i and chip j, an entry in the matrix My contains either a "1" for a direct connection or "0" for no direct connection.
  • the index k refers to the number of hops necessary to interconnect any gate in chip i to any gate in chip j requiring the interconnections.
  • the operation includes ANDing matrices M together and then ORing the ANDed results. If the AND operation between matrix my and my results in a logic "1" value, then a connection exists between a selected gate in chip i and a selected gate in chip j through any chip 1 within hop k; if not, no connection exists within this particular hop k and further calculation is necessary to.
  • the matrices my and my are the connectivity matrix M as defined for this hardware modeling.
  • the row containing the FPGA chip for gate i in matrix m is logically ANDed to the column containing the FPGA chip for gate j and my.
  • the individual ANDed components are ORed to determine if the resulting My value for index or hop k is a "1" or "0.” If the result is a "1,” then a connection exists and the index k is designated as the number of hops. If the result is "0,” then no connection exists.
  • FIG. 35(A) shows a user's circuit design represented as a cloud 1090.
  • This circuit design 1090 may be simple or complex.
  • a portion of the circuit design 1090 includes an OR gate 1091 and two AND gates 1092 and 1093.
  • the outputs of AND gates 1092 and 1093 are coupled to the inputs of OR gate 1091.
  • These gates 1091, 1092, and 1093 may also be coupled to other portions of the circuit design 1090.
  • the components of this circuit 1090 may be configured and placed in FPGA chips 1094, 1095, and 1096.
  • This particular exemplary array of FPGA chips has the interconnection scheme as shown; that is, a set of interconnections 1097 couple chip 1094 to chip 1095, and another set of interconnections 1098 couple chip 1095 to chip 1096. No direct interconnections are provided between chip 1094 and chip 1096.
  • the system uses the pre-designed interconnection scheme to connect circuit paths across different chips.
  • OR gate 1091 placed in chip 1094 AND gate 1092 placed in chip 1095, and AND gate 1093 placed in chip 1096.
  • Other portions of the circuit 1090 are not shown for pedagogic purposes.
  • the connection between OR gate 1091 and AND gate 1092 requires an interconnection because they are located in different chips so the set of interconnections 1097 is used. The number of hops for this interconnection is "1.”
  • the connection between OR gate 1091 and AND gate 1093 also requires interconnections so sets of interconnections 1097 and 1098 are used. The number of hops is "2.” For this placement example, the total number of hops is "3," discounting the contribution from other gates and their interconnections in the remainder of circuit 1090 that are not shown.
  • FIG. 35(D) shows another placement example.
  • OR gate 1091 is placed in chip 1094, and AND gates 1092 and 1093 are placed in chip 1095. Again, other portions of the circuit 1090 are not shown for pedagogic purposes.
  • the connection between OR gate 1091 and AND gate 1092 requires an interconnection because they are located in different chips so the set of interconnections 1097 is used. The number of hops for this interconnection is " 1.”
  • the connection between OR gate 1091 and AND gate 1093 also requires interconnections so the set of interconnections 1097 is used. The number of hops is also "1.” For this placement example, the total number of hops is "2,” discounting the contribution from other gates and their interconnections in the remainder of circuit 1090 that are not shown.
  • the cost function calculates a lower cost function for the placement example of FIG. 35(D) than the placement example of FIG. 35(C). However, all other factors are not equal. More than likely, the cost function for FIG. 35(D) is also based on the gate usage/availability G. In FIG. 35(D), one more gate is used in chip 1095 than that used in the same chip in FIG. 35(C). Furthermore, the pin usage/availability P for chip 1095 in the placement example illustrated in FIG. 35(C) is greater than the pin usage/availability for the same chip in the other placement example illustrated in FIG. 35(D).
  • This fine-grain placement operation 353 refines the placement initially selected by the coarse- grain placement operation 352.
  • initial clusters may be split up if such an arrangement will increase the optimization.
  • logic elements X and Y are originally part of cluster A and designated for FPGA chip 1. Due to the fine-grain placement operation 353, logic elements X and Y may now be designated as a separate cluster B or made part of another cluster C and designated for placement in FPGA chip 2.
  • An FPGA netlist 354 which ties the user's circuit design to specific FPGAs, is then generated.
  • the determination of how clusters are split up and placed in certain chips is also based on placement cost, which is calculated through a cost function f(P, G, D) for circuits CKTQ.
  • the cost function used for the fine-grain placement process is the same as the cost function used for the coarse-grain placement process. The only difference between the two placement processes is the size of the clusters placed, not in the processes themselves. The coarse-grain placement process uses larger clusters than the fine-grain placement process.
  • the cost functions for the coarse-grain and fine-grain placement processes are different from each other, as described above with respect to selecting weighting constants CO, Cl, and C2.
  • a routing task 355 among the chips is performed. If the number of routing wires to connect circuits located in different chips exceeds the available pins in these FPGA chips allocated for the circuit-to-circuit routing, time division multiplex (TDM) circuits can be used. For example, if each FPGA chip allows only 44 pins for connecting circuits located in two different FPGA chips, and a particular model implementation requires 45 wires between chips, a special time division multiplex circuit will also be implemented in each chip. This special TDM circuit couples at least two of the wires together. One embodiment of the TDM circuit is shown in FIGS. 9(A), 9(B), and 9(C), which will be discussed later. Thus, the routing task can always be completed because the pins can be arranged into time division multiplex form among the chips.
  • TDM time division multiplex
  • each FPGA can be configured into optimized and working circuits and accordingly, the system generates a "bitstream" configuration file 356.
  • the system generates one or more Programmer Object Files (.pof).
  • Other generated files include SRAM Object Files (.sof), JEDEC Files (.jed), Hexadecimal (Intel-format) Files (.hex), and Tabular Text Files (.ttf).
  • the Altera MAX+PLUS II Programmer uses POFs, SOFs, and JEDEC Files along with Altera hardware programmable devices to program the FPGA array.
  • the system generates one or more raw binary files (.rbf).
  • the CPU revises .rbf files and programs the FPGA array through the PCI bus.
  • the configured hardware is ready for hardware start-up 370. This completes the automatic construction of hardware models on the reconfigurable boards.
  • the TDM circuit is essentially a multiplexer with at least two inputs (for the two wires), one output, and a couple of registers configured in a loop as the selector signal. If the SEmulation system requires more wires to be grouped together, than more inputs and loop registers can be provided. As the selector signal to this TDM circuit, several registers configured in a loop provide the appropriate signals to the multiplexer so that at one time period, one of the inputs is selected as the output, and at another time period, another input is selected as the output.
  • the TDM circuit manages to use only one output wire between chips so that, for this example, the hardware model of the circuit implemented in a particular chip can be accomplished using 44 pins, instead of 45 pins.
  • the routing task can always be completed because the pins can be arranged into time division multiplex form among the chips.
  • FIG. 9(A) shows an overview of the pin-out problem. Since this requires the TDM circuit, FIG. 9(B) provides a TDM circuit for the transmission side, and FIG. 9(C) provides a TDM circuit for the receiver side.
  • FIG. 9(B) provides a TDM circuit for the transmission side
  • FIG. 9(C) provides a TDM circuit for the receiver side.
  • FIG. 9(A) shows one embodiment of the TDM circuit in which the SEmulation system couples two wires in a TDM configuration.
  • Two chips, 990 and 991 are provided.
  • a circuit 960 which is portion of a complete user circuit design is modeled and placed in chip 991.
  • a circuit 973 which is portion of a complete user circuit design is modeled and placed in chip 990.
  • Several interconnections, including a group of interconnections 994, interconnection 992, and interconnection 993, are provided between circuit 960 and circuit 973.
  • the number of interconnections in this example, total 45. If, in one embodiment, each chip provides only 44 pins at most for these interconnections, one embodiment of the present invention provides for at least two of the interconnections to be time multiplexed to require only one interconnection between these chips 990 and 991.
  • the group of mterconnections 994 will continue to use the 43 pins.
  • a TDM circuit in accordance with one embodiment of the present invention can be used to couple interconnections 992 and 993 together in time division multiplexed form.
  • FIG. 9(B) shows one embodiment of the TDM circuit.
  • a modeled circuit (or a portion thereof) 960 within a FPGA chip 991 provides two signals on wires 966 and 967. To the circuit 960, these wires 966 and 967 are outputs. These outputs would normally be coupled to modeled circuit 973 in chip 990 (see FIGS. 9(A) and 9(C)). However, the availability of only one pin for these two output wires 966 and 967 precludes a direct pin-for-pin connection. Because the outputs 966 and 967 are uni-directionally transmitted to the other chip, appropriate transmission and receiver TDM circuits must be provided to couple these lines together.
  • FIG. 9(B) One embodiment of the transmission side TDM circuit is shown in FIG. 9(B).
  • the transmission side TDM circuit includes AND gates 961 and 962, whose respective outputs 970 and 971 are coupled to the inputs of OR gate 963.
  • the output 972 of OR gate 963 is the output of the chip assigned to a pin and connected to another chip 990.
  • One set of inputs 966 and 967 to AND gates 961 and 962, respectively, is provided by the circuit model 960.
  • the other set of inputs 968 and 969 is provided by a looped register scheme which functions as the time division multiplexed selector signal.
  • the looped register scheme includes registers 964 and 965.
  • the output 995 of register 964 is provided to the input of register 965 and the input 968 of AND gate 961.
  • the output 996 of register 965 is coupled to the input of register 964 and the input 969 to AND gate 962.
  • Each register 964 and 965 is controlled by a common clock source. At any given instant in time, only one of the outputs 995 or 996 provides a logic "1.” The other is at logic "0.” Thus, after each clock edge, the logic "1" shifts between output 995 and output 996. This in turn provides either a "1” to AND gate 961 or AND gate 962, "selecting" either the signal on wire 966 or wire 967.
  • the data on wire 972 is from circuit 960 on either wire 966 or wire 967.
  • FIG. 9(C) One embodiment of the receiver side portion of the TDM circuit is shown in FIG. 9(C).
  • the signals from circuit 960 on wires 966 and wire 967 in chip 991 (FIGS. 9(A) and 9(B)) must be coupled to the appropriate wires 985 or 986 to the circuit 973 in FIG. 9(C).
  • the time division multiplexed signals from chip 991 enter from wire/pin 978.
  • the receiver side TDM circuit can couple these signals on wire/pin 978 to the appropriate wires 985 and 986 to circuit 973.
  • the TDM circuit includes input registers 974 and 975.
  • the signals on wire/pin 978 are provided to these input registers 974 and 975 via wires 979 and 980, respectively.
  • the output 985 of input register 974 is provided to the appropriate port in circuit 973.
  • the output 986 of input register 975 is provided to the appropriate port in circuit 973.
  • the output 984 of register 976 is coupled to the input of register 977 and the clock input 981 of register 974.
  • the output 983 of register 977 is coupled to the input of register 976 and the clock input 982 of register 975.
  • Each register 976 and 977 is controlled by a common clock source. At any given instant in time, only one of the enable inputs 981 or 982 is a logic "1.” The other is at logic "0.” Thus, after each clock edge, the logic "1" shifts between enable input 981 and output 982. This in turn “selects” either the signal on wire 979 or wire 980. Thus, the data on wire 978 from circuit 960 is appropriately coupled to circuit 973 via either wire 985 or wire 986.
  • address pointer in accordance with one embodiment of the present invention, as discussed briefly with respect to FIG. 4, will now be discussed in greater detail. To reiterate, several address pointers are located in each FPGA chip in the hardware model. Generally, the primary purpose for implementing the address pointers is to enable the system to deliver data between the software model 315 and the specific FPGA chip in the hardware model 325 via the 32-bit PCI bus 328 (refer to FIG. 10).
  • the primary purpose of the address pointer is to selectively control the data delivery between each of the address spaces (i.e., REG, S2H, H2S, and CLK) in the software/hardware boundary and each FPGA chip among the banks 326a-326d of FPGA chips in light of the bandwidth limitations of the 32-bit PCI bus. Even if a 64-bit PCI bus is implemented, these address pointers are still needed to control the data delivery. Thus, if the software model has 5 address spaces (i.e., REG read, REG write, S2H read, H2S write, and CLK write), each FPGA chip has 5 address pointers corresponding to these 5 address spaces. Each FPGA needs these 5 address pointers because the particular selected word in the selected address space being processed may reside in any one or more of the FPGA chips.
  • the software model has 5 address spaces (i.e., REG read, REG write, S2H read, H2S write, and CLK write)
  • each FPGA chip has 5 address pointers corresponding to these
  • the FPGA I/O controller 381 selects the particular address space (i.e., REG, S2H, H2S, and CLK) corresponding to the software/hardware boundary by using a SPACE index. Once the address space is selected, the particular address pointer corresponding to the selected address space in each FPGA chip selects the particular word corresponding to the same word in the selected address space.
  • the maximum sizes of the address spaces in the software/hardware boundary and the address pointers in each FPGA chip depend on the memory/word capacity of the selected FPGA chip. For example, one embodiment of the present invention uses the Altera FLEX 10K family of FPGA chips. Accordingly, estimated maximum sizes for each address space are: REG, 3,000 words; CLK, 1 word; S2H, 10 words; and H2S, 10 words. Each FPGA chip is capable of holding approximately 100 words.
  • the SEmulator system also has the feature of allowing the user to start, stop, assert input values, and inspect values at any time in the SEmulation process.
  • the SEmulator must also make all the components visible to the user regardless of whether the internal realization of a component is in software or hardware. In software, combinational components are modeled and values are computed during the simulation process. Thus, these values are clearly "visible" for the user to access at any time during the simulation process.
  • combinational component values in the hardware model are not so directly “visible.” Although registers are readily and directly accessible (i.e., read/write) by the software kernel, combinational components are more difficult to determine. In FPGAs, most combinational components are modeled as look-up tables in order to achieve high gate utilization. As a result, the look-up table mapping provides efficient hardware modeling but loses visibility of most of the combinational logic signals.
  • the SEmulation system can rebuild or regenerate combinational components for inspection by the user after the hardware acceleration mode. If a user's circuit design has only combinational and register components, the values of all the combinational components can be derived from the register components. That is, combinational components are constructed from or contain registers in various arrangements in accordance with the specific logic function required by the circuit design.
  • the SEmulator has hardware models of register and combinational components only, and as a result, the SEmulator will read all the register values from the hardware model and then rebuild or regenerate all the combinational components. Because of the overhead required to perform this regeneration process, combinational component regeneration is not performed all the time; rather, it is performed only upon request by the user. Indeed, one of the benefits of using the hardware model is to accelerate the simulation process. Determining combinational component values at every cycle (or even most cycles) further decreases the speed of simulation. In any event, inspection of register values alone should be sufficient for most simulation analyses.
  • the process of regenerating combinational component values from register values assumes that the SEmulation system was in the hardware acceleration mode or ICE mode. Otherwise, software simulation already provides combinational component values to the user.
  • the SEmulation system maintains combinational component values as well as register values that were resident in the software model prior to the onset of hardware acceleration. These values remain in the software model until further over-writing action by the system. Because the software model already has register values and combinational component values from the time period immediately before the onset of the hardware acceleration run, the combinational component regeneration process involves updating some or all of these values in the software model in response to updated input register values.
  • the combinational component regeneration process is as follows: First, if requested by the user, the software kernel reads all the output values of the hardware register components from the FPGA chips into the REG buffer. This process involves a DMA transfer of register values in the FPGA chips via the chain of address pointers to the REG address space. Placing register values that were in the hardware model into the REG buffer, which is in the software/hardware boundary, allows the software model to access data for further processing.
  • the software kernel compares the register values before the hardware acceleration run and after the hardware acceleration run. If the register values before the hardware acceleration run are the same as the values after the hardware acceleration run, the values in the combinational components have not changed. Instead of expending time and resources to regenerating combinational components, these values can be read from the software model, which already has combinational component values stored therein from the time immediately before the hardware acceleration run. On the other hand, if one or more of these register values have changed, one or more combinational components that depend on the changed register values may also change values. These combinational components must be regenerated through the following third step.
  • the software kernel schedules their fan-out combinational components into the event queue.
  • those registers that changed values during this acceleration run have detected an event. More than likely, these combinational components that depend on these changed register values will produce different values. Regardless of any change in value in these combinational components, the system ensures that these combinational components evaluate these changed register values in the next step.
  • the software kernel executes the standard event simulation algorithms to propagate the value changes from the registers to all the combinational components in the software model.
  • the register values that changed during the before-acceleration to after-acceleration time interval are propagated to all combinational components downstream that depend on these register values.
  • These combinational components then evaluate these new register values.
  • other second-level combinational components that are located downstream from the first-level combinational components that in turn directly rely on the changed register values must also evaluate the changed data, if any. This process of propagating register values to other components downstream that may be affected continues to the end of the fan-out network. Thus, only those combinational components located downstream and affected by the changed register values are updated in the software model.
  • the system is ready for any mode of operation.
  • the : user desires to inspect values after a long run. After the combinational component regeneration process, the user will continue with pure software simulation for debug/test purposes. However, at other times, the user may wish to continue with the hardware acceleration to the next desired point. Still in other cases, the user may wish to proceed further with ICE mode.
  • combinational component regeneration involves using register values to update combinational component values in the software model.
  • the changed register value will be propagated through that register's fan-out network as values are updated.
  • the values in the software model also will not change, so the system does not need to regenerate combinational components.
  • the hardware acceleration run will occur for some time.
  • many register values may change, affecting many combinational component values located downstream in the fan-out network of these registers that have the changed values.
  • the combinational component regeneration process may be relatively slow.
  • only a few register values may change.
  • the fan-out network for registers that had the changed register values may be small and thus, the combinational component regeneration process may be relatively fast.
  • FIG. 10 shows a SEmulation system architecture in accordance with one embodiment of the present invention.
  • FIG. 10 also shows a relationship between the software model, hardware model, the emulation interface, and the target system when the system is operating in in-circuit emulation mode.
  • the SEmulation system comprises a general purpose microprocessor and a reconfigurable hardware board interconnected by a high-speed bus, such as a PCI bus.
  • the SEmulation system compiles the user's circuit design and generates the emulation hardware configuration data for the hardware model-to-reconfigurable board mapping process.
  • the user can then simulate the circuit through the general purpose processor, hardware accelerate the simulation process, emulate the circuit design with the target system through the emulation interface, and later perform post-simulation analysis.
  • the software model 315 and hardware model 325 are determined during the compilation process.
  • the emulation interface 382 and the target system 387 are also provided in the system for in-circuit emulation mode. Under the user's discretion, the emulation interface and the target system need not be coupled to the system at the outset.
  • the software model 315 includes the kernel 316, which controls the overall system, and four address spaces for the software/hardware boundary - REG, S2H, H2S, and CLK.
  • the SEmulation system maps the hardware model into four address spaces in main memory according to different component types and control functions: REG space 317 is designated for the register components; CLK space 320 is designated for the software clocks; S2H space 318 is designated for the output of the software test-bench components to the hardware model; and H2S space 319 is designated for the output of the hardware model to the software test-bench components.
  • REG space 317 is designated for the register components
  • CLK space 320 is designated for the software clocks
  • S2H space 318 is designated for the output of the software test-bench components to the hardware model
  • H2S space 319 is designated for the output of the hardware model to the software test-bench components.
  • the hardware model includes several banks 326a-326d of FPGA chips and FPGA I/O controller 327.
  • Each bank e.g., 326b
  • each bank contains 4 FPGA chips.
  • banks 326b and 326d may be the low bank and banks 326a and 326c may be the high bank.
  • the mapping, placement, and routing of specific hardware-modeled user circuit design elements to specific chips and their interconnections are discussed with respect to FIG. 6.
  • the interconnection 328 between the software model 315 and the hardware model 325 is a PCI bus system.
  • the hardware model also includes the FPGA I/O controller 327 which includes a PCI interface 380 and a control unit 381 for controlling the data traffic between the PCI bus and the banks 326a-326d of FPGA chips while maintaining the throughput of the PCI bus.
  • Each FPGA chip further includes several address pointers, where each address pointer corresponds to each address space (i.e., REG, S2H, H2S, and CLK) in the software/hardware boundary, to couple data between each of these address spaces and each FPGA chip in the banks 326a-326d of FPGA chips.
  • Communication between the software model 315 and the hardware model 325 occurs through a DMA engine or address pointer in the hardware model. Alternatively, communication also occurs through both the DMA engine and the address pointer in the hardware model.
  • the kernel initiates DMA transfers together with evaluation requests through direct mapped I O control registers.
  • REG space 317, CLK space 320, S2H space 318, and H2S space 319 use I/O datapath lines 321, 322, 323, and 324, respectively, for data delivery between the software model 315 and the hardware model 325.
  • Double buffering is required for all primary inputs to the S2H and CLK spaces because these spaces take several clock cycles to complete the updating process. Double buffering avoids disturbing the internal hardware model states which may cause race conditions.
  • the S2H and CLK space are the primary input from the kernel to the hardware model.
  • the hardware model holds substantially all the register components and the combinational components of the user's circuit design.
  • the software clock is modeled in software and provided in the CLK I/O address space to interface with the hardware model.
  • the kernel advances simulation time, looks for active test-bench components, and evaluates clock components. When any clock edge is detected by the kernel, registers and memories are updated and values through combinational components are propagated. Thus, any changes in values in these spaces will trigger the hardware model to change logic states if the hardware acceleration mode is selected.
  • emulation interface 382 is coupled to the PCI bus 328 so that it can communicate with the hardware model 325 and the software model 315.
  • the kernel 316 controls not only the software model, but also the hardware model during the hardware accelerated simulation mode and the in-circuit emulation mode.
  • the emulation interface 382 is also coupled to the target system 387 via cable 390.
  • the emulation interface 382 also includes the interface port 385, emulation I/O control 386, the target-to-hardware I/O buffer (T2H) 384, and the hardware-to-target I/O buffer (H2T) 383.
  • the target system 387 includes a connector 389, a signal-in/signal-out interface socket 388, and other modules or chips that are part of the target system 387.
  • the target system 387 could be an EGA video controller
  • the user's circuit design may be one particular I/O controller circuit.
  • the user's circuit design of the I/O controller for the EGA video controller is completely modeled in software model 315 and partially modeled in hardware model 325.
  • the kernel 316 in the software model 315 also controls the in-circuit emulation mode.
  • the control of the emulation clock is still in the software via the software clock, the gated clock logic, and the gated data logic so no set-up and hold-time problems will arise during in-circuit emulation mode.
  • the user can start, stop, single-step, assert values, and inspect values at any time during the in-circuit emulation process.
  • the SEmulation system uses the software clock to control the hardware model instead of the target system's clock.
  • the primary input (signal-in) and output (signal-out) signals between the target system 40 and the modeled circuit design are provided to the hardware model 325 for evaluation. This is accomplished through two buffers, the target-to-hardware buffer (T2H) 384 and the hardware-to-target buffer (H2T) 383.
  • the target system 387 uses the T2H buffer 384 to apply input signals to the hardware model 325.
  • the hardware model 325 uses the H2T buffer 383 to deliver output signals to the target system 387.
  • the hardware model send and receive I/O signals through the T2H and H2T buffers instead of the S2H and H2S buffers because the system is now using the target system 387, instead of test- bench processes in the software model 315 to evaluate the data. Because the target system runs at a speed substantially higher than the speed of the software simulation, the in-circuit emulation mode will also run at a higher speed. The transmission of these input and output signals occurs on the PCI bus 328.
  • a bus 61 is provided between the emulation interface 382 and the hardware model 325.
  • This bus is analogous to the bus 61 in FIG. 1.
  • This bus 61 allows the emulation interface 382 and the hardware model 325 to communicate via the T2H buffer 384 and the H2T buffer 383.
  • the target system 387 is not coupled to the PCI bus. However, such a coupling may be feasible if the emulation interface 382 is incorporated in the design of the target system 387. In this set-up, the cable 390 will not be present. Signals between the target system 387 and the hardware model 325 will still pass tlirough the emulation interface.
  • the SEmulation system of the present invention can support value change dump (VCD), a widely used simulator function for post-simulation analysis.
  • VCD value change dump
  • the VCD provides a historical record of all inputs and selected register outputs of the hardware model so that later, during post-simulation analysis, the user can review the various inputs and resulting outputs of the simulation process.
  • the system logs all inputs to the hardware model. For outputs, the system logs all values of hardware register components at a user-defined logging frequency (e.g., 1/10,000 record/cycle). The logging frequency determines how often the output values are recorded. For a logging frequency of 1/10,000 record/cycle, output values are recorded once every 10,000 cycles.
  • the user selects a particular point at which simulation is desired. If the logging frequency is 1/500 records/cycle, register values are record for points 0, 500, 1000, 1500, and so on every 500 cycles. If the user wants results at point 610, for example, the user selects point 500, which is recorded, and simulates forward in time until the simulation reaches point 610. During the analysis stage, the analysis speed is the same as the simulation speed because the user initially accesses data for point 500 and then simulates forward to point 610. Note that at higher logging frequencies, more data is stored for post-simulation analysis. Thus, for a logging frequency of 1/300 records/cycle, data is stored for points 0, 300, 600, 900, and so on every 300 cycles.
  • point 610 the user initially selects point 600, which is recorded, and simulates forward to point 610. Notice that the system can reach the desired point 610 faster during post-simulation analysis when the logging frequency is 1/300 than 1/500. However, this is not always the case.
  • the particular analysis point in conjunction with the logging frequency determines how fast the post-simulation analysis point is reached. For example, the system can reach point 523 faster if the VCD logging frequency was 1/500 rather than 1/300.
  • the user can then perform analysis after SEmulation by running the software simulation with input logs to the hardware model to compute the value change dump of all hardware components.
  • the user can also select any register log point in time and start the value change dump from that log point forward in time.
  • This value change dump method can link to any simulation waveform viewer for post-simulation analysis.
  • One embodiment of the present invention is a system that generates VCD on demand without simulation rerun.
  • the VCD on-demand technology as described herein incorporates the following high level attributes: (1) RCC-based parallel simulation history compression and recording, (2) RCC-based parallel simulation history decompression and VCD file generation, and (3) On-demand software regeneration for a selected simulation target range and design review without simulation rerun.
  • RCC-based parallel simulation history compression and recording incorporates the following high level attributes: (1) RCC-based parallel simulation history compression and recording, (2) RCC-based parallel simulation history decompression and VCD file generation, and (3) On-demand software regeneration for a selected simulation target range and design review without simulation rerun.
  • the EDA tool (hereinafter referred to as the RCC System, which incorporates the various aspects of the present invention) records the primary inputs from a test bench process so that any portion of the simulation can be reproduced.
  • the user can then selectively command the EDA tool, or RCC System, to dump the hardware state information from any simulation time range into a VCD file for later analysis. Thereafter, the user can immediately begin debugging his design in the selected simulation time range. If the selected simulation time range does not include the bug that the user is seeking to fix, he can select another simulation time range for dump into the VCD file. The user can then analyze this new VCD file. With this VCD on-demand feature, the user can cease simulation at any point and request the generation of another selective VCD file on-demand from any desired simulation time starting point to any simulation time end point.
  • the user debugs his design using the RCC System illustrated in FIG. 83.
  • the user fast simulates his design from a desired beginning simulation time to any desired end simulation time, referred to herein as a simulation session range.
  • a highly compressed form of the primary inputs is recorded in an "input history" file so that any portion of the simulation session can be reproduced.
  • the RCC System saves the hardware state information from this end point in a "simulation history" file so that the user can return to debugging the design past this end point if desired.
  • the user will analyze the results and invariably detect some problem with his design.
  • the user then makes a guess that the source of the problem (i.e., bug) is located in a particular narrow simulation time range, referred to herein as the simulation target range, which is within the broader simulation session range.
  • the simulation target range For example, if the simulation session range encompassed 1,000 simulation time steps, the narrower simulation target range might include only 100 simulation time steps at a particular location within the broader simulation session range.
  • the RCC System fast simulates from the beginning by decompressing the compressed primary inputs in the input history file and delivering the decompressed primary inputs into the hardware model for evaluation.
  • the RCC System reaches the simulation target range, it dumps the evaluated results (e.g., hardware node values and register states) into a VCD file.
  • the user can analyze this region more carefully by replaying his design using the VCD file starting from the beginning of the simulation target range, rather than having to rerun the simulation from the beginning of the simulation session range, or even from the very beginning of the simulation.
  • This feature of saving the hardware states from the simulation target range as a VCD file saves the user an enormous amount of debug time - time that is not otherwise wasted on simulation rerun.
  • the RCC System includes an RCC Computing System 2600 and an RCC Hardware Accelerator 2620.
  • the RCC Computing System 2600 contains the computational resources that are necessary to allow the user to simulate the user's entire software-modeled design in software and control the hardware acceleration of the hardware-modeled portion of the design.
  • the RCC Computing System 2600 contains the CPU 2601, various clocks 2602 (including the software clock that is described elsewhere in this patent specification) that are needed by the various components of the RCC System, test bench processes 2603, and system disk 2604.
  • the system disk is used to record the compressed data rather than a small hardware RAM buffer.
  • the RCC Computing System 2600 includes other logic components and bus subsystems that provide the circuit designer with the computational power to run diagnostics, various software, and manage files, among other tasks that a computing system performs.
  • the RCC Hardware Accelerator 2620 which is also referred to as the RCC Array in other sections of this patent specification, contains the reconfigurable array of logic elements (e.g., FPGA) that can model at least a portion of the user's design in hardware so that the user can accelerate the debugging process.
  • the RCC Hardware Accelerator 2620 includes the array of reconfigurable logic elements 2621 which provides the hardware model of a portion of the user design.
  • the RCC Computing System 2600 is tightly coupled to the RCC Hardware Accelerator 2620 via the software clock as described elsewhere in this patent specification and a bus system, a portion of which is shown as lines 2610 and 2611 in FIG. 83.
  • FIG. 84 shows a timeline of several simulation times - tO, tl, t2, and t3.
  • the simulation session range is between simulation time tO and simulation time t3, which of course includes simulation times tl and t2.
  • Simulation time tO represents the first simulation time in the simulation session range where fast simulation begins. This simulation time tO represents the first simulation time for any separable simulation session, or simulation session range.
  • simulation time t3 represents the last simulation time for the selected simulation session range.
  • simulation time t3 represents the very last simulation time for the user design's last debug session.
  • the user may continue to simulate beyond this simulation time t3 if desired but for the moment, he is focused on debugging his design for the simulation times tO to t3, the current simulation session range. Typically, when the bugs have been ironed out for the current simulation session range, the user will then proceed to simulate his design beyond simulation time t3 into the next simulation session range.
  • simulation time periods tO-t3 are not necessarily contiguous to each other; that is, simulation time tO and tl are not immediately adjacent to each other. Indeed, simulation times tO and tl may be thousands of simulation time periods apart.
  • references to various components of the RCC System shown in FIG. 83 will be made.
  • This generation operation includes some form of data compression for the primary inputs and recordation of the compressed primary inputs.
  • the RCC System's VCD generation operation will be discussed. This VCD generation operation includes decompressing the' primary inputs to reproduce the simulation history and dumping the hardware states into a VCD file for the simulation target range.
  • the VCD file review process is then discussed. Although the term "simulation history" is used at times, this does not mean that the entire debug session involves software simulation. Indeed, the RCC System generates VCD files from hardware states and the software model is used only for later analysis of the VCD file.
  • the user models the design in software in the RCC Computing System 2600 of FIG. 83.
  • the RCC Computing System 2600 automatically generates a hardware model of the design based on the hardware description language (e.g., VHDL).
  • the hardware model is configured in the array of reconfigurable logic elements 2621, which is a portion of the RCC Hardware Accelerator 2620.
  • the user can simulate the design in software in the RCC Computing System 2600, accelerate a portion (i.e., simulation time step or distinct physical section of the circuit) of the design using the RCC Hardware Accelerator 2620, or a combination of simulation and hardware acceleration.
  • the simulation session range should be selected to be short enough to isolate a few bugs in the design and long enough to quickly move the debugging process and minimize the number of debug sessions necessary to fully debug a design.
  • a simulation session range of two or three simulation time steps will not reveal the existence of any bug.
  • this small simulation session range will force the user to conduct many repetitive tasks that will slow the debug process. If the selected simulation session range is a million simulation time steps, too many bugs may manifest themselves and thus, the user will be find difficulty in implementing a more focused attack of the problem.
  • simulation time tO represents the beginning of the simulation
  • simulation time t3 represents the last simulation time for this simulation session range.
  • fast simulation begins in the RCC Computing System 2600.
  • Fast simulation is performed from simulation time tO to simulation time t3 instead of normal simulation mode because no regeneration of the software model is needed during this time period.
  • the regeneration operation requires the RCC Computing System 2620 to receive hardware state information (e.g., node values, register states) so that more sophisticated logic elements (e.g., combinational logic) can be regenerated in software for further analysis by the user.
  • hardware state information e.g., node values, register states
  • more sophisticated logic elements e.g., combinational logic
  • the simulation process is much slower due to the extra time needed by the RCC Computing System 2600 to regenerate the software model from the primary outputs of the hardware model.
  • the full states of the design such as the software model states and hardware model register and node values, are saved at simulation time tO into a file, called "simulation history" file, in the system disk.
  • This allows the user to load the states of the design into the RCC System at any time in the future for debugging purposes.
  • the RCC Computing System 2600 applies two distinct processes to the primary inputs I P in parallel.
  • the raw primary inputs from the test bench processes 2603 are provided on line 2610 to the RCC Hardware Accelerator 2620 for evaluation.
  • the same primary inputs from the test bench processes are compressed and recorded in system disk as a separate file, called an "input history" file, so that the entire history of the primary inputs can be collected to allow the user to reproduce any part of the simulation later.
  • the primary inputs corresponding to simulation time tO to simulation time t3 are compressed and saved in system disk.
  • the RCC Hardware Accelerator 2620 When the RCC Hardware Accelerator 2620 receives the primary inputs I P from the test bench processes 2603, it processes the primary inputs. As a result, hardware states in the hardware model will most likely change as the various logic and other circuit devices evaluate the data. During this period from simulation time tO to simulation time t3, the RCC System need not wait for the RCC Computing System 2600 to perform its logic regeneration since the user is not interested in finely debugging the design during this fast simulation period. The RCC System also does not save the primary outputs (e.g., hardware node values and register states) yet.
  • the primary outputs e.g., hardware node values and register states
  • the RCC Computing System 2600 compresses the primary inputs for recording into the "input history" file
  • the RCC Hardware Accelerator 2620 evaluates the raw and uncompressed primary inputs. In other embodiments, the RCC System does not compress the primary inputs for recording into the input history file.
  • the RCC Computing System 2600 delivers the primary inputs to the RCC Hardware Accelerator for evaluation when these outputs will not be saved at all during the fast simulation period?
  • the RCC System needs to save the hardware states of the design based on its evaluation of the primary inputs from the beginning of the simulation to simulation time t3.
  • An accurate snapshot of the hardware model states cannot be obtained at simulation time t3 unless the hardware model has evaluated the entire history of primary inputs from the beginning to this point t3, not the inputs from just simulation time t3.
  • Logic circuits have memory attributes that will affect the results of the evaluation based on the order of the inputs. Thus, if the primary inputs from just simulation time t3 (or the simulation time immediately prior to simulation time t3) are fed to the hardware model for evaluation, the hardware model will probably exhibit the wrong states at this simulation time t3. .
  • the hardware model in the RCC Hardware Accelerator 2620 provides internal hardware states on line 2611 to the RCC Computing System 2600, so that the RCC Computing System 2600 can build or regenerate the various logic elements (e.g., combinational logic) in the software model, if necessary and desired by the user. But, as noted above, the user is not concerned with observing the software simulation during the fast simulation of the simulation session range. Accordingly, these internal hardware states from the RCC Hardware Accelerator are not saved in the system disk, since the internal hardware states will not be examined by the user for bugs for now.
  • logic elements e.g., combinational logic
  • this particular fast simulation operation ceases.
  • the evaluation results or primary outputs (e.g., register values) from the design's hardware model in the RCC Hardware Accelerator 2620 corresponding to simulation time t3 are saved in the simulation history file. This is done so that when the user has debugged the design from simulation time tO to simulation time t3, he can then proceed straight to simulation time t3 for further debugging as necessary. The user need not rerun the simulation from simulation time tO to debug his design at some point beyond simulation time t3.
  • simulation time tO to simulation time t3 i.e., simulation session range
  • the user is essentially accelerating the design by feeding the RCC Hardware Accelerator 2620 with the primary inputs from the test bench process 2603 on line 2610 while at the same time compressing the same primary inputs and saving them into system disk for future reference.
  • the RCC Computing System 2600 needs to save the primary inputs (compressed or otherwise) in the input history file to reproduce the debug session.
  • the compression operation also occurs in parallel with the data evaluation in the RCC Hardware Accelerator 2620.
  • simulation time t3 at the end of the simulation session range the RCC System saves the state information of the hardware model into a simulation history file.
  • all recorded compressed primary inputs from the simulation session range are part of the same file that will be modified later for the hardware state information from simulation time t3.
  • the saved information from the simulation session range and the hardware state information from simulation time t3 are each saved as distinct files in system disk.
  • any of the above described files may be modified with the VCD on-demand information that is created later for the simulation target range.
  • the VCD on-demand information may be saved in a distinct VCD file in system disk that is separate from the compressed primary input file and the simulation time t3 hardware state information file.
  • the input history file,, the simulation history file, and the VCD file may be incorporated together in one file.
  • the input history file, the simulation history file, and the VCD file may be separate files.
  • the input history file and the simulation history file may be incorporated in one file that is separate from the VCD file. , .
  • the RCC System's compression logic allows for a compression ratio of 20X for the primary input events with 10% input events per simulation time step.
  • a large ASIC design having over a million gates may require 200 primary input events.
  • 10% input events per simulation time step approximately 20 inputs need to be compressed and recorded. If each input signal is 2 bytes long, 20 input signals results in 40 bytes of data need to be processed at the primary inputs per simulation time step.
  • the 40 bytes of data can be compressed to 2 bytes of data per simulation time step.
  • the RCC System compresses the primary inputs to 2 Mega bytes of data. A file of this size can be easily managed by any computing file system and the waveform viewer. In one embodiment, ZIP compression is used.
  • the primary input compression is performed in parallel with the primary input evaluation by the RCC Hardware Accelerator 2620; input history file generation occurs concurrently with the primary input evaluation. Accordingly, the compression scheme provides no direct negative impact on the RCC System's performance. The only possible bottleneck is the process of recording the compressed primary inputs into the system disk. However, since the data is highly compressed, the RCC System experiences less than 5% slowdown for • most designs running at 50,000 simulation time steps per second.
  • the "name” argument is the record name for the current simulation session range. Different names are required to distinguish different simulation runs of the same design. A distinct record name is needed especially for off-line VCD on-demand debugging.
  • the ⁇ disk space> argument is an optional parameter to specify the maximum disk space (in units of MB) allocated for the RCC System recording process. The default value is 100 MB. The RCC System only records the latest part of the current simulation session range within the specified disk space.
  • the RCC System records only the last 100 MB while discarding the first 40 MB of compressed primary inputs.
  • This aspect of the invention provides one benefit for failure analysis.
  • the test bench process has some self- testing functions to detect simulation failures and stop the simulation. The latest history of the RCC simulation can provide most of the information for such failure analysis.
  • the Checkpoint control> argument is an optional parameter that specifies the number of simulation time steps ; needed to perform a full-state checkpoint. The default is 1,000,000 time steps.
  • the compressed primary inputs are also based on the state difference between successive simulation time steps.
  • checkpoints for the full RCC states at a given low frequency can greatly facilitate simulation history extraction.
  • the RCC System can extract (i.e., reproduction of the simulation from the primary inputs and selected VCD file generation) any simulation history within 5 to 50 seconds.
  • the RCC System When this $rcc(record) command is invoked, the RCC System will record the simulation history; that is, the primary inputs will be compressed and recorded in a file for storage in the system disk. The primary outputs from the RCC Hardware Accelerator are ignored since software logic regeneration is not needed at this time. The recording process can be terminated with either the commands $rcc(stop) or $rcc(off), at which point the RCC System switches control of the simulation back to the software model. At this point, the primary outputs are processed for software logic regeneration.
  • the RCC System has saved the software model and hardware model at the beginning of the simulation session range at simulation time tO, recorded the compressed primary inputs for the entire simulation session range in the input history file, and saved the hardware model states for the design at the end of the simulation session range at simulation time t3 in the simulation history file.
  • the user now has enough information to load the design at the start of the simulation session range from the design information from simulation time tO.
  • the user can software simulate any portion of his design.
  • the VCD on- demand feature the user will probably not want to software simulate his design at this point. Rather, the user will want to generate a VCD file for the selected simulation target range for fine analysis to isolate and fix the bug.
  • the RCC System can reproduce any point within the simulation session range.
  • the RCC System can simulate beyond the current simulation session range if desired by loading the previously saved hardware state information from simulation time t3.
  • the user reviews the results to determine if a bug exists. If no bug is apparent to the user, the design may be free of bugs for the current simulation session range. The user can then proceed to simulate beyond the current simulation session range to the next simulation session range, whatever selected range this may be. If, however, the user has determmed that the design has some sort of problem, he must analyze the simulation more carefully to isolate and fix the bug. Because the entire simulation session range is too large for careful and detailed analysis, the user must target a particular narrower range for further study. Based on the user's familiarity of the design and perhaps past debugging efforts, the user makes a reasonable guess as to the location of the bug within the simulation session range.
  • the user will focus on a selected simulation target range that should correspond with the user's guess as to the location of the bug (or where the bug will manifest itself).
  • the user determines that the simulation target range is between simulation time tl and simulation time t2 as shown in FIG. 84.
  • the RCC System loads the software model of the design in the RCC Computing System 2600 and the hardware model in the RCC Hardware Accelerator 2620 with the previously saved configuration information from simulation state tO.
  • the RCC System then fast simulates from simulation time tO to simulation time tl.
  • the RCC Computing System loads the previously saved file containing the compressed primary inputs.
  • the RCC Computing System decompresses the compressed primary inputs and delivers the decompressed primary inputs to the RCC Hardware Accelerator 2620 for evaluation.
  • the primary outputs which are the evaluated results are not saved during the fast simulation operation from simulation time tO to simulation time tl.
  • the RCC System then dumps the evaluated results (i.e., primary outputs 0 P ) from the hardware model in the RCC Hardware Accelerator 2620 into a VCD file in the system disk.
  • the RCC Computing System 2600 does not perform any compression.
  • the RCC Computing System 2600 does not perform any regeneration operation for the software model since the user need not view the evaluation results at this time. By not performing any regeneration operation for the software model, the RCC System can quickly generate the VCD file.
  • the user may concurrently view the software model of his design for this simulation time period from tl to t2 while saving the primary outputs. If so, the RCC Computing System 2600 performs the software model regeneration operation to allow the user to view any and all states from any aspect of his design.
  • the RCC Computing System 2600 ceases saving the evaluation outputs from the RCC Hardware Accelerator 2620 in the VCD file. At this point, the user can stop fast simulating.
  • the RCC System now has the complete VCD file for the simulation target range and the user can proceed to analyze the VCD file in greater detail.
  • the user When the user wants to analyze the VCD file, he need not rerun the simulation from the very beginning (e.g., simulation time tO). Instead, the user can command the RCC System to load the saved hardware state information from the beginning of the simulation target range and view the simulated results with the software model. This will be described in more detail below in the Simulation History Review section.
  • the user may or may not discover the bug. If the bug is found, the user will of course commence fixing the design. If the bug is not found, the user may have made a wrong guess of the simulation target range that he suspects has the bug. The user must employ the same process that he used above with respect to the decompress and VCD file dump. The user makes another guess with, hopefully, a better simulation target range within the simulation session range. Having done so, the RCC System fast simulates from the beginning of the simulation session range to the beginning of the new simulation target range, decompressing the primary inputs and delivering them to the RCC Hardware Accelerator 2620 for evaluation.
  • the RCC System When the RCC System reaches the beginning of the new simulation target range, the primary outputs from the RCC Hardware Accelerator 2620 are dumped into a VCD file. At the end of the new simulation target range, the RCC System ceases dumping the hardware state information into the VCD file. At this point, the user can then view the VCD file for isolating the bug.
  • the RCC System fast simulates the design by decompressing the previously compressed primary inputs and delivering them to the hardware model for evaluation.
  • the RCC System dumps the primary outputs from the hardware model into a VCD file.
  • the user can cease fast simulating the design.
  • the user can then view the VCD file by going directly to simulation time tl without rerunning the simulation from the very beginning at simulation time tO.
  • This new simulation session range begins at simulation time t3.
  • the particular length of the new simulation target range which can be the same length as the previous simulation session range, is selected by the user.
  • the RCC System loads the previously saved hardware state information corresponding to simulation time t3.
  • the RCC System is now ready for fast simulation of this new simulation session range. Note that this new simulation session range corresponds to the range from simulation time tO to t3, where the loaded hardware state now corresponds to simulation time tO.
  • the fast simulation, VCD on-demand dump, and VCD review process is similar to that described above.
  • the decompression step does not negatively impact performance.
  • the RCC System can decompress the simulation history (i.e., compressed and recorded primary inputs) at a rate of 20,000 to 200,000 simulation time steps per second. With proper checkpoint control, the RCC System can extract (i.e., reproduction of the simulation from the primary inputs and selected VCD file generation) the simulation history within 50 seconds.
  • the $axis_rpd is an interactive command to extract the RCC evaluation record and create a VCD file on demand.
  • the execution of the $axis_rpd command neither rewinds the internal simulation state nor corrupts the external PLI and file I/O states.
  • the user can continue simulation after invoking the $axis_rpd command in the same manner as the user is capable of simulating after the $stop command.
  • $axis_rpd command displays all available simulation time periods within the simulation session range; that is, the user can select the simulation target range.
  • the time unit is the same time unit in the command line interface.
  • $axis_rpd shows the recorded simulation windows.
  • the start-time and end-time specify the simulation time window, or the simulation target range, for the VCD file.
  • the unit of the time control arguments is the time unit used in the command line interface.
  • the "dump-file-name" is the name of the VCD file.
  • the dump ⁇ level and scope control> parameters are identical to the standard Sdumpvars command in the IEEE Verilog.
  • This $axis_rpd command creates a VCD file called "fl.dump" for the simulation target range from simulation time 50505 to 50600. Just like Sdumpvars, if no level and scope control parameters are provided, the Saxisjrpd command will dump the entire hardware states or primary outputs.
  • This $axis_rpd command creates a 2-level VCD file "f2.dump" on the scope dpO from time 40000 to 50600. Since the simulation swaps back to software control during time 50000 to 50500, Saxisjrpd skips that window because no simulation record is available.
  • VCD on-demand is also available after the user terminates the simulation process.
  • the user starts the simulation program named "vlg" with the +rccplay option. With this option, the RCC System is instructed to extract the simulation record instead of executing the normal initialization sequence for simulation.
  • the user can use the same $axis_rpd command to obtain VCD on demand.
  • An example of this procedure is as follows:
  • the simulation record "rl" is used to extract the simulation history and produce the VCD on the entire design from time 40000 to 45000.
  • the user need not fast simulate from simulation time t2 to t3. Instead, the RCC System allows the user to cease simulation and proceed directly to the beginning of the simulation target range, or simulation time tl . Thus, in contrast to the prior art, the user does not have to rerun the simulation from the very beginning (e.g., simulation time tO).
  • the hardware states that have been dumped into the VCD file reflects the evaluation of the entire history of primary inputs from simulation time tO, including the primary inputs from simulation times tl to t2.
  • the RCC System loads the VCD file. Thereafter, the saved primary outputs are delivered to the RCC Computing System 2600 so that the software model, and all of its many combinational logic circuits, can be regenerated with the correct state information.
  • the user views the software model with a waveform viewer for debugging.
  • the VCD the user can step through his software model very carefully step-by-step until the bug is isolated.
  • the user can select any simulation target range within the simulation session range and perform software simulation to isolate the bug. If the bug cannot be found in the selected simulation target range, the user can select another different simulation target range on demand.
  • this VCD on-demand feature is supported on-line in the middle of the simulation process as well as off-line after the simulation process has terminated.
  • This on-line support is possible the hardware states at simulation time tO can be saved in system disk and the primary inputs can be compressed and recorded for any length of the simulation session range. Thereafter, the user can then specify a simulation target range for a more focused analysis of the primary outputs.
  • the off-line support is possible because the hardware states at simulation time tO, the entire primary inputs for the simulation session range, and the hardware states at simulation time tl are all saved in the system disk.
  • the user can return to debugging his design by loading the design corresponding to simulation time tO and then specifying the simulation target range. Also, the user can proceed directly to the next simulation target range by loading the hardware states corresponding to simulation time t3.
  • the SEmulation system implements an array of FPGA chips on a reconfigurable board. Based on the hardware model, the SEmulation system partitions, maps, places, and routes each selected portion of the user's circuit design onto the FPGA chips. Thus, for example, a 4x4 array of 16 chips may be modeling a large circuit spread out across these 16 chips.
  • the interconnect scheme allows each chip to access another chip within 2 "jumps" or links.
  • Each FPGA chip implements an address pointer for each of the I/O address spaces (Le., REG, CLK, S2H, H2S).
  • the combination of all address pointers associated with a particular address space are chained together. So, during data transfer, word data in each chip is sequentially selected from/to the main FPGA bus and PCI bus, one word at a time for the selected address space in each chip, and one chip at a time, until the desired word data have been accessed for that selected address space.
  • This sequential selection of word data is accomplished by a propagating word selection signal. This word selection signal travels through the address pointer in a chip and then propagates to the address pointer in the next chip and continues on till the last chip or the system initializes the address pointer.
  • the FPGA bus system in the reconfigurable board operates at twice the PCI bus bandwidth but at half the PCI bus speed.
  • the FPGA chips are thus separated into banks to utilize the larger bandwidth bus.
  • the throughput of this FPGA bus system can track the throughput of the PCI bus system so performance is not lost by reducing the bus speed. Expansion is possible through bigger boards which contains more FPGA chips or piggyback boards that extend the bank length.
  • FIG. 11 shows one embodiment of the address pointer of the present invention. All I/O operations go through DMA streaming. Because the system has only one bus, the system accesses data sequentially one word at a time. Thus, one embodiment of the address pointer uses a shift register chain to sequentially access the selected words in these address spaces.
  • the address pointer 400 includes flip-flops 401-405, an AND gate 406, and a couple of control signals, INITIALIZE 407 and MOVE 408.
  • Each address pointer has n outputs (W0, WI, W2, . . . , Wn-1) for selecting a word out of n possible words in each FPGA chip corresponding to the same word in the selected address space.
  • the number of words n may vary from circuit design to circuit design and, for a given circuit design, n varies from FPGA chip to FPGA chip.
  • this particular FPGA chip which contains this 5-word address pointer for a particular address space has only 5 words to select.
  • the address pointer 400 can implement any number of words n.
  • This output signal Wn can also be called the word selection signal. When this word selection signal reaches the output of the last flip-flop in this address pointer, it is called an OUT signal to be propagated to the inputs of the address pointers of the next FPGA chip.
  • the address pointer When the INITIALIZE signal is asserted, the address pointer is initialized.
  • the first flip-flop 401 is set to "1" and all other flip-flops 402-405 are set to "0.” At this point, the initialization of the address pointer will not enable any word selection; that is, all the Wn outputs are still at "0" after initialization.
  • the address pointer initialization procedure will also be discussed with respect to FIG. 12.
  • the MOVE signal controls the advance of the pointer for word selection.
  • This MOVE signal is derived from the READ, WRITE, and SPACE index control signals from the FPGA I/O controller. Because every operation is essentially a read or a write, the SPACE index signal essentially determines which address pointer will be applied with the MOVE signal. Thus, the system activates only one address pointer associated with a selected I/O address space at a time, and during that time, the system applies the MOVE signal only to that address pointer.
  • the MOVE signal generation is discussed further with respect to FIG. 13. Referring to FIG. 11, when the MOVE signal is asserted, the MOVE signal is provided to an input to an AND gate 406 and the enable input of the flip-flops 401-405.
  • FIG. 12 shows a state transition diagram of the address pointer initialization for the address pointer of FIG. 11.
  • state 460 is idle.
  • the DATA_XSFR is set to "1," the system goes to state 461, where the address pointer is initialized.
  • the INITIALIZE signal is asserted.
  • the first flip-flop in each address pointer is set to "1" and all other flip-flops in the address pointer are set to "0.” At this point, the initialization of the address pointer will not enable any word selection; that is, all the Wn outputs are still at “0.”
  • the next state is wait state 462 while the DATA_XSFR is still “1.” When the DATA_XSFR is "0,” the address pointer initialization procedure has completed and the system returns to the idle state 460.
  • the MOVE signal generator for generating the various MOVE signals for the address pointer will now be discussed.
  • the SPACE index which is generated by the FPGA I/O controller (item 327 in FIG.
  • the system of the present invention sequentially selects the particular word to be accessed. The sequential word selection is accomplished in each address pointer by the MOVE signal.
  • Each FPGA chip 450 has address pointers that correspond to the various software/hardware boundary address spaces (i.e., REG, S2H, H2S, and CLK).
  • the MOVE signal generator 470 is provided in the FPGA chip 450.
  • the MOVE signal generator 470 includes an address space decoder 451 and several AND gates 452-456.
  • the input signals are the FPGA read signal (F_RD) on wire line 457, FPGA write signal (F_WR) on wire line 458, and the address space signal 459.
  • the output MOVE signal for each address pointer corresponds to REGR-move on wire line 464, REGW-move on wire line 465, S2H-move on wire line 466, H2S-move on wire line 467, and CLK-move on wire line 468, depending on which address space's address pointer is applicable. These output signals correspond to the MOVE signal on wire line 408 (FIG. 11).
  • the address space decoder 451 receives a 3-bit input signal 459. It can also receive just a 2-bit input signal. The 2-bit signal provides for 4 possible address spaces, whereas the 3-bit input provides for 8 possible address spaces.
  • CLK is assigned to "00”
  • S2H is assigned to "01”
  • H2S is assigned to "10”
  • REG is assigned to "11.”
  • the output of the address space decoder outputs a "1" on one of the wire lines 460-463, corresponding to REG, H2S, S2H, and CLK, respectively, while the remaining wire lines are set to “0.”
  • the corresponding output of the AND gates 452-456 is “0.”
  • Wire line 461 is “1” while the remaining wire lines 460, 462, and 463 are “0.” Accordingly, wire line 466 is “1,” while the remaining output wire lines 464, 465, 467, and 468 are “0.” Similarly, if wire line 460 is “1,” The REG space is selected and depending on whether a read (F_RD) or write (F_WR) operation is selected, either the REGR-move signal on wire line 464 or the REGW-move signal on wire line 465 will be “1.”
  • F_RD read
  • F_WR write
  • the SPACE index is generated by the FPGA I/O controller.
  • the MOVE controls are:
  • each FPGA chip has the same number of address pointers as address spaces in the software/hardware boundary. If the software/hardware boundary has 4 address spaces (i.e., REG, S2H, H2S, and CLK), each FPGA chip has 4 address pointers corresponding to these 4 address spaces. Each FPGA needs these 4 address pointers because the particular selected word in the selected address space being processed may reside in any one or more of the FPGA chips, or the data in the selected address space affects the various circuit elements modeled and implemented in each FPGA chip.
  • each set of address pointers associated with a given software/hardware boundary address space i.e., REG, S2H, H2S, and CLK
  • REG, S2H, H2S, and CLK software/hardware boundary address space
  • One embodiment of the system in accordance with the present invention uses a multiplexed cross chip address pointer chain which allows the hardware model to use only one wire between chips and only 1 input pin and 1 output pin in each chip (2 I/O pins in a chip).
  • One embodiment of the multiplexed cross chip address pointer chain is shown in FIG. 14.
  • each address pointer for example address pointer 427, has a structure and function similar to the address pointer shown in FIG. 11, except that the number of words Wn and hence the number of flip-flops may vary depending on how many words are implemented in each chip for the user's custom circuit design.
  • the FPGA chip 415 has address pointer 421, FPGA chip 416 has address pointer 425, and FPGA chip 417 has address pointer 429.
  • the FPGA chip 415 has address pointer 422, FPGA chip 416 has address pointer 426, and FPGA chip 417 has address pointer 430.
  • the FPGA chip 415 has address pointer 423, FPGA chip 416 has address pointer 427, and FPGA chip 417 has address pointer 431.
  • the FPGA chip 415 has address pointer 424, FPGA chip 416 has address pointer 428, and FPGA chip 417 has address pointer 432.
  • Each chip 415-417 has a multiplexer 418-420, respectively.
  • these multiplexers 418-420 may be models and the actual implementation may be a combination of registers and logic elements, as known to those ordinarily skilled in the art.
  • the multiplexer may be several AND gates feeding into an OR gate as shown in FIG. 15.
  • the multiplexer 487 includes four AND gates 481-484 and an OR gate 485.
  • the inputs to the multiplexer 487 are the OUT and MOVE signals from each address pointer in the chip.
  • the output 486 of the multiplexer 487 is a chain-out signal which is passed to the inputs to the next FPGA chip.
  • this particular FPGA chip has four address pointers 475-478, corresponding to I/O address spaces.
  • the outputs of the address pointers, the OUT and MOVE signals, are inputs to the multiplexer 487.
  • address pointer 475 has an OUT signal on wire line 479 and a MOVE signal on wire line 480. These signals are inputs to AND gate 481.
  • the output of this AND gate 481 is an input to OR gate 485.
  • the output of the OR gate 485 is the output of this multiplexer 487.
  • the OUT signal at the output of each address pointer 475-478 in combination with their corresponding MOVE signals and the SPACE index serve as a selector signal for the multiplexer 487; that is, both the OUT and MOVE signals (which are derived from the SPACE index signals) have to be asserted active (e.g., logic "1") to propagate the word selection signal out of the multiplexer to the chain-out wire line.
  • the MOVE signal will be asserted periodically to move the word selection signal through the flip-flops in the address pointer so that it can be characterized as the input MUX data signal.
  • these multiplexers 418-420 have four sets of inputs and one output.
  • Each set of inputs includes: (1) the OUT signal found on the last output Wn-1 wire line for the address pointer (e.g., wire line 413 in the address pointer shown in FIG. 11) associated with a particular address space, and (2) the MOVE signal.
  • the output of each multiplexer 418-420 is the chain-out signal.
  • the word selection signal Wn through the flip-flops in each address pointer becomes the OUT signal when it reaches the output of the last flip-flop in the address pointer.
  • the chain-out signal on wire lines 433-435 will become "1" only when an OUT signal and a MOVE signal associated with the same address pointer are both asserted active (e.g., asserted "1").
  • the inputs are MOVE signals 436-439 and OUT signals 440-443 corresponding to OUT ⁇ and MOVE signals from address pointers 421-424, respectively.
  • the inputs are MOVE signals 444-447 and OUT signals 452-455 corresponding to OUT and MOVE signals from address pointers 425-428, respectively.
  • the inputs are MOVE signals 448-451 and OUT signals 456-459 corresponding to OUT and MOVE signals from address pointers 429-432, respectively.
  • any given shift of words Wn only those address pointers or chain of address pointers associated with a selected I/O address space in the software/hardware boundary are active.
  • the address pointers in chips 415, 416, and 417 associated with one of the address spaces REGR, REGW, S2H, or H2S are active for a given shift.
  • the selected word is accessed sequentially because of limitations on the bus bandwidth.
  • the bus is 32 bits wide and a word is 32 bits, so only one word can be accessed at a time and delivered to the appropriate resource.
  • the output chain-out signal is not activated (e.g., not "1") and thus, this multiplexer in this chip is not yet ready to propagate the word selection signal to the next FPGA chip.
  • the OUT signal is asserted active (e.g., "1")
  • the chain-out signal is asserted active (e.g.,”l") indicating that the system is ready to propagate or shift the word selection signal to the next FPGA chip.
  • accesses occur one chip at a time; that is, the word selection signal is shifted through the flip-flops in one chip before the word selection shift operation is performed for another chip.
  • the chain-out signal is asserted only when the word selection signal reaches the end of the address pointer in each chip.
  • the chain-out signal is:
  • Chain-out (REGR-move & REGR-out)
  • each FPGA has X address pointers, one address pointer for each address space.
  • the size of each address pointer depends on the number of words required for modeling the user's custom circuit design in each FPGA chip. Assuming n words for a particular FPGA chip and hence, n words for the address pointer, this particular address pointer has n outputs (i.e., W0, WI, W2, . . . , Wn-1). These outputs Wi are also called word selection signals. When a particular word Wi is selected, the Wi signal is asserted active (i.e., "1").
  • This word selection signal shifts or propagates down the address pointer of this chip until it reaches the end of the address pointer in this chip, at which point, it triggers the generation of a chain-out signal that starts the propagation of the word selection signal Wi through the address pointer in the next chip.
  • a chain of address pointers associated with a given I/O address space can be implemented across all of the FPGA chips in this reconfigurable hardware board.
  • the various embodiments of the present invention perform clock analysis in association with gated data logic and gated clock logic analysis.
  • the gated clock logic (or clock network) and the gated data network determinations are critical to the successful implementation of the software clock and the logic evaluation in the hardware model during emulation.
  • the clock analysis is performed in step 305.
  • FIG. 16 shows a flow diagram in accordance with one embodiment of the present invention.
  • FIG. 16 also shows the gated data analysis.
  • the SEmulation system has the complete model of the user's circuit design in software and some portions of the user's circuit design in hardware. These hardware portions include the clock components, especially the derived clocks. Clock delivery timing issues arise due to this boundary between software and hardware. Because the complete model is in software, the software can detect clock edges that affect register values. In addition to the software model of the registers, these registers are physically located in the hardware model. To ensure that the hardware registers also evaluate their respective inputs (i.e., moving the data at the D input to the Q output), the softwarehardware boundary includes a software clock. The software clock ensures that the registers in the hardware model evaluate correctly. The software clock essentially controls the enable input of the hardware register rather than controlling the clock input to the hardware register components.
  • the clock network and gated data logic analysis process shown in FIG. 16 provides a way of modeling and implementing the clock and data delivery system to the hardware registers such that race conditions are avoided and a flexible software/hardware boundary implementation is provided.
  • primary clocks are clock signals from test-bench processes. All other clocks, such as those clock signals derived from combinational components, are derived or gated clocks.
  • a primary clock can derive both gated clocks and gated data signals. For the most part, only a few (e.g., 1-10) derived or gated clocks are in the user's circuit design. These derived clocks can be implemented as software clocks and will stay in software. If a relatively large number (e.g., more than 10) of derived clocks are present in the circuit design, the SEmulation system will model them into hardware to reduce I/O overhead and maintain the SEmulation system's performance.
  • Gated data is data or control input of a register other than the clock driven from the primary clock through some combinational logic.
  • Step 501 takes the usable source design database code generated from the HDL code and maps the user's register elements to the SEmulation system's register components. This one-to-one mapping of user registers to SEmulation registers facilitates later modeling steps. In some cases, this mapping is necessary to handle user circuit designs which describe register elements with specific primitives.
  • SEmulation registers can be used quite readily because the RTL level code is at a high enough level, allowing for varying lower level implementations.
  • the SEmulation system will access the cell library of components and modify them to suit the particular circuit design-specific logic elements.
  • Step 502 extracts clock signals out of the hardware model's register components. This step allows the system to determine primary clocks and derived clocks. This step also determines all the clock signals needed by various components in the circuit design. The information from this step facilitates the software/hardware clock modeling step.
  • Step 503 determines primary clocks and derived clocks.
  • Primary clocks originate from test-bench components and are modeled in software only.
  • Derived clocks are derived from combinational logic, which are in turn driven by primary clocks.
  • the SEmulation system of the present invention will keep the derived clocks in software. If the number of derived clocks is small (e.g., less than 10), then these derived clocks can be modeled as software clocks. The number of combinational components to generate these derived clocks is small, so significant I/O overhead is not added by keeping these combinational components residing in software. If, however, the number of derived clocks is large (e.g., more than 10), these derived clocks may be modeled in hardware to minimize I/O overhead. Sometimes, the user's circuit design uses a large number of derived clock components derived from primary clocks. The system thus builds the clocks in hardware to keep the number of software clocks small.
  • step 504 requires the system to determine if any derived clocks are found in the user's circuit design. If not, step 504 resolves to "NO" and the clock analysis ends at step 508 because all the clocks in the user's circuit design are primary clocks and these clocks are simply modeled in software. If derived clocks are found in the user's circuit design, step 504 resolves to "YES" and the algorithm proceeds to step 505.
  • Step 505 determines the fan-out combinational components from the primary clocks to the derived clocks. In other words, this step traces the clock signal datapaths from the primary clocks through the combinational components.
  • Step 506 determines the fan-in combinational components from the derived clocks. In other words, this step traces the clock signal datapaths from the combinational components to the derived clocks. Determining fan-out and fan-in sets in the system is done recursively in software. The fan-in set of a net N is as follows:
  • Fanln Set of a net N find all the components driving net N; for each component X driving net N do: if the component X is not a combinational component then return; else for each input net Y of the component X add the Fanln set W of net Y to the Fanln Set of net N end for add the component X into N; end if endfor
  • a gated clock or data logic network is determined by recursively determining the fan-in set and fan-out set of net N, and determining their intersection. The ultimate goal here is to determine the so-called Fan-In Set of net N.
  • the net N is typically a clock input node for determining the gated clock logic from a fan-in perspective.
  • net N is a clock input node associated with the data input at hand. If the node is on a register, the net N is the clock input to that register for the data input associated with that register.
  • the system finds all the components driving net N. For each component X driving net N, the system determines if the component X is a combinational component or not. If each component X is not a combinational component, then the fan-in set of net N has no combinational components and net N is a primary clock.
  • the system determines the input net Y of the component X.
  • the system is looking further back in the circuit design by finding the input nodes to the component X.
  • a fan-in set W may exist which is coupled to net Y. This fan- in set W of net Y is added to the Fan-In Set of net N, then the component X is added into set N.
  • the fan-out set of a net N is determined in a similar manner.
  • the fan-out set of net N is determined as follows:
  • FanOut Set of a net N find all the components using the net N; for each component X using the net N do: if the component X is not a combinational component then return; else for each output net Y of component X add the FanOut Set of net Y to the FanOut Set of Net N end for add the component X into N; end if end for
  • the gated clock or data logic network is determined by recursively determining the fan-in set and fan- out set of net N, and determining their intersection.
  • the ultimate goal here is to determine the so-called Fan-Out Set of net N.
  • the net N is typically a clock output node for determining the gated clock logic from a fan-out perspective.
  • net N is a clock output node associated with the data output at hand. If the node is on a register, the net N is the output of that register for the primary clock-driven input associated with that register. The system finds all the components using net N.
  • the system determines if the component X is a combinational component or not. If each component X is not a combinational component, then the fan-out set of net N has no combinational components and net N is a primary clock.
  • the system determines the output net Y of the component X.
  • the system is looking further forward from the primary clock in the circuit design by finding the output nodes from the component X.
  • a fan-out set W may exist which is coupled to net Y. This fan-out set W of net Y is added to the Fan-Out Set of net N, then the component X is added into set N.
  • Step 507 determines the clock network or gated clock logic.
  • the clock network is the intersection of the fan-in and fan-out combinational components.
  • gated data is the data or control input of a register (except for the clock) driven by a primary clock through some combinational logic.
  • Gated data logic is the intersection of the fan-in of the gated data and fan-out from the primary clock.
  • the clock analysis and gated data analysis result in a gated clock network/logic through some combinational logic and a gated data logic.
  • the gated clock network and the gated data network determinations are critical to the successful implementation of the software clock and the logic evaluation in the hardware model during emulation.
  • the clock/data network analysis ends at step 508.
  • FIG. 17 shows a basic building block of the hardware model in accordance with one embodiment of the present invention.
  • the SEmulation system uses a D-type flip-flop with asynchronous load control as the basic block for building both edge trigger (i.e., flip-flops) and level sensitive (i.e., latches) register hardware models.
  • This register model building block has the following ports: Q (the output state); A_E (asynchronous enable); A_D (asynchronous data); S_E (synchronous enable); S_D (synchronous data); and of course, System.clk (system clock).
  • This SEmulation register model is triggered by a positive edge of the system clock or a positive level of the asynchronous enable (A_E) input.
  • the register model looks for the asynchronous enable (A_E) input. If the asynchronous enable (A_E) input is enabled, the output Q takes on the value of the asynchronous data (A D); otherwise, if the synchronous enable (S_E) input is enabled, the output Q takes on the value of the synchronous data (S_D). If, on the other hand, neither the asynchronous enable (A_E) nor the synchronous enable (S_E) input is enabled, the output Q is not evaluated despite the detection of a positive edge of the system clock. In this way, the inputs to these enable ports control the operation of this basic building block register model.
  • the system uses software clocks, which are special enable registers, to control the enable inputs of these register models.
  • software clocks which are special enable registers, to control the enable inputs of these register models.
  • millions of elements are found in the circuit design and accordingly, the SEmulator system will implement millions of elements in the hardware model. Controlling all of these elements individually is costly because the overhead of sending millions of control signals to the hardware model will take a longer time than evaluating these elements in software.
  • this complex circuit design usually calls for only a few (from 1-10) clocks and clocks alone are sufficient to control the state changes of a system with register and combinational components only.
  • the hardware model of the SEmulator system uses only register and combinational components.
  • the SEmulator system also controls the evaluation of the hardware model through software clocks.
  • the hardware models for registers do not have the clock directly connected to other hardware components; rather, the software kernel controls the value of all clocks. By controlling a few clock signals, the kernel has the full control over the evaluation of the hardware models with negligible amount of coprocessor intervention overhead.
  • the software clock will be input to either the asynchronous enable (A_E) or synchronous enable (S_E) wire lines.
  • A_E asynchronous enable
  • S_E synchronous enable
  • the application of the software clock from the software model to the hardware model is triggered by edge detection of clock components.
  • the software kernel detects the edge of clock components, it sets the clock-edge register through the CLK address space.
  • This clock- edge register controls the enable input, not the clock input, to the hardware register model.
  • the global system clock still provides the clock input to the hardware register model.
  • the clock-edge register provides the software clock signal to the hardware register model through a double-buffered interface.
  • a double-buffer interface from the software clock to the hardware model ensures that all the register models will be updated synchronously with respect to the global system clock.
  • the use of the software clock eliminates the risk of hold time violations.
  • FIGS. 18(A) and 18(B) show the implementation of the building block register model for latches and flip- flops. These register models are software clock-controlled via the appropriate enable inputs. Depending on whether the register model is used as a flip-flop or latch, the asynchronous ports (A__E, A_D) and synchronous ports (S_E, S_D) are either used for the software clock or I/O operations.
  • FIG. 18(A) shows the register model implementation if it is used as a latch. Latches are level-sensitive; that is, so long as the clock signal has been asserted (e.g., "1"), the output Q follows the input (D).
  • the software clock signal is provided to the asynchronous enable (A_E) input and the data input is the provided to the asynchronous data (A_D) input.
  • the software kernel uses the synchronous ' enable (S_E) and synchronous data (S_D) inputs to download values into the Q port.
  • the S_E port is used as a REG space address pointer and the S_D is used to access data to/from the local data bus.
  • FIG. 18(B) shows the register model implementation if it is used as a design flip-flop.
  • Design flip-flops use the following ports for determining the next state logic: data (D), set (S), reset (R), and enable (E). All the next state logic of a design flip-flop is factored into a hardware combinational component which feeds into the synchronous data (S D) input.
  • the software clock is input to the synchronous enable (SJE) input.
  • the software kernel uses the asynchronous enable (A_E) and asynchronous data (A_D) inputs to download values into the Q port.
  • the A E port is used as a REG space write address pointer and the A_D port is used to access data to/from the local data bus.
  • One embodiment of the software clock of the present invention is a clock enable signal to the hardware register model such that the data at the inputs to these hardware register models are evaluated together and synchronously with the system clock. This eliminates race conditions and hold-time violations.
  • One implementation of the software clock logic includes clock edge detection logic in software which triggers additional logic in the hardware upon clock edge detection. Such enable signal logic generates an enable signal to the enable inputs to hardware register models before the arrival of the data to these hardware register models.
  • the gated clock network and the gated data network determinations are critical to the successful implementation of the software clock and the logic evaluation in the hardware model during hardware acceleration mode.
  • the clock network or gated clock logic is the intersection of the fan-in of the gated clock and fan-out of the primary clock.
  • the gated data logic is also the intersection of the fan-in of the gated data and fan-out of the primary clock for the data signals.
  • primary clocks are generated by test-bench processes in software.
  • Derived or gated clocks are generated from a network of combmational logic and registers which are in turn driven by the primary clocks.
  • the SEmulation system of the present invention will also keep the derived clocks in software. If the number of derived clocks is small (e.g., less than 10), then these derived clocks can be modeled as software clocks. The number of combinational components to generate these derived clocks is small, so significant I/O overhead is not added by modeling these combinational components in software. If, however, the number of derived clocks is large (e.g., more than 10), these derived clocks and their combinational components may be modeled in hardware to minimize I/O overhead.
  • clock edge detection occurring in software can be translated to clock detection in hardware (via the input to a clock edge register).
  • the clock edge detection in software triggers an event in hardware so that the registers in the hardware model receive the clock enable signal before the data signal to ensure that the evaluation of the data signal occurs in synchronization with the system clock to avoid hold-time violations.
  • the SEmulation system has the complete model of the user's circuit design in software and some portions of the user's circuit design in hardware.
  • the software can detect clock edges that affect hardware register values.
  • the software/hardware boundary includes a software clock.
  • the software clock ensures that the registers in the hardware model evaluate in synchronization with the system clock and without any hold-time violations.
  • the software clock essentially controls the enable input of the hardware register components, rather than controlling the clock, input to the hardware register components.
  • the double-buffered approach to implementing the software clocks ensures that the registers evaluate in synchronization with the system clock to avoid race conditions and eliminates the need for precise timing controls to avoid hold-time violations.
  • FIG. 19 shows one embodiment of the clock implementation system in accordance with the present invention.
  • the gated clock logic and the gated data logic are determined by the SEmulator system, as discussed above with respect to FIG. 16.
  • the gated clock logic and the gated data logic are then separated.
  • the driving source and the double-buffered primary logic must also be separated. Accordingly, the gated data logic 513 and gated clock logic 514, from the fan-in and fan-out analysis, have been separated.
  • the modeled primary clock register 510 includes a first buffer 511 and a second buffer 512, which are both D registers. This primary clock is modeled in software but the double-buffer implementation is modeled in both software and hardware. Clock edge detection occurs in the primary clock register 510 in software to trigger the hardware model to generate the software clock signal to the hardware model. Data and address enter the first buffer 511 at wire lines 519 and 520, respectively. The Q output of this first buffer 511 on wire line 521 is coupled to the D input of second buffer 512. The Q output of this first buffer 511 is also provided on wire line 522 to the gated clock logic 514 to eventually drive the clock input of the first buffer 516 of the clock edge register 515.
  • the Q output of the second buffer 512 on wire line 523 is provided to the gated data logic 513 to eventually drive the input of register 518 via wire line 530 in the user' s custom-designed circuit model.
  • the enable input to the second buffer 512 in the primary clock register 510 is the INPUT-EN signal on wire line 533 from a state machine, which determines evaluation cycles and controls various signals accordingly.
  • the clock edge register 515 also includes a first buffer 516 and a second buffer 517.
  • the clock edge register 515 is implemented in hardware. When a clock edge detection occurs in software (via the input to the primary clock register 510), this can trigger the same clock edge detection in hardware (via clock edge register 515) in hardware.
  • the D input to the first buffer 516 on wire line 524 is set to logic " 1."
  • the clock signal on wire line 525 is derived from the gated clock logic 514 and ultimately from the primary clock register 510 at the output on wire line 522 of the first buffer 511. This clock signal on wire line 525 is the gated clock signal.
  • the enable wire line 526 for the first buffer 516 is the ⁇ EVAL signal from the state machine that controls the I/O and evaluation cycles (to be discussed later).
  • the first buffer 516 also has a RESET signal on wire line 527. This same RESET signal is also provided to the second buffer 517 in the clock edge register 515.
  • the Q output of the first buffer 516 on wire line 529 is provided to the D input to the second buffer 517.
  • the second buffer 517 also has an enable input on wire line 528 for the CLK-EN signal and a RESET input on wire line 527.
  • the Q output of the second buffer 517 on wire line 532 is provided to the enable input of the register 518 in the user's custom-designed circuit model. Buffers 511, 512, and 517 along with register 518 are clockedby the system clock. Only buffer 516 in the clock edge register 515 is clocked by a gated clock from a gated clock logic 514.
  • Register 518 is a typical D-type register model that is modeled in hardware and is part of the user's custom circuit design. Its evaluation is strictly controlled by this embodiment of the clock implementation scheme of the present invention. The ultimate goal of this clock set-up is to ensure that the clock enable signal at wire line 532 arrives at the register 518 before the data signal at wire line 530 so that the evaluation of the data signal by this register will be synchronized with the system clock and without race conditions.
  • the modeled primary clock register 510 is modeled in software but its double buffer implementation is modeled in both software and hardware.
  • the clock edge register 515 is implemented in hardware.
  • the gated data logic 513 and gated clock logic 514, from the fan-in and fan-out analysis, have also been separated for modeling purposes, and can be modeled in software (if the number of gated data and gated clocks is small) or hardware (if the number of gated data and gated clocks is large).
  • the gated clock network and the gated data network determinations are critical to the successful implementation of the software clock and the logic evaluation in the hardware model during hardware acceleration mode.
  • the software clock implementation relies primarily on the clock set-up shown on FIG. 19 along with the timing of the assertions of signals ⁇ EVAL, INPUT-EN, CLK-EN, and RESET.
  • the primary clock register 510 detects clock edges to trigger the software clock generation for the hardware model. This clock edge detection event triggers the "activation" of the clock edge register 515 via the clock input on wire line 525, gated clock logic 514, and wire line 522 so that the clock edge register 515 also detects the same clock edge. In this way, clock detection occurring in software (via the inputs 519 and 520 to the primary clock register 510) can be translated to clock edge detection in hardware (via the input 525 in clock edge register 515).
  • the INPUT-EN wire line 533 to second buffer 512 in the primary clock register 510 and the CLK-EN wire line 528 to second buffer 517 in the clock edge register 515 have not been asserted and thus, no data evaluation will take place.
  • the clock edges will be detected before the data are evaluated in the hardware register model. Note that at this stage, the data from the data bus on wire line 519 has not even propagated out to the gated data logic 513 and into the hardware-modeled user register 518. Indeed, the data have not even reached the second buffer 512 in the primary clock register 510 because the INPUT-EN signal on wire line 533 has not been asserted yet.
  • the ⁇ EVAL signal on wire line 526 is asserted to enable the first buffer 516 in the clock edge register 515.
  • the ⁇ EVAL signal also goes through the gated clock logic 514 to monitor the gated clock signal as it makes its way through the gated clock logic to the clock input on wire line 525 of first buffer 516.
  • the ⁇ EVAL signal can be maintained as long as necessary to stabilize the data and the clock signals through that portion of the system illustrated in FIG. 19.
  • the ⁇ EVAL is deasserted to disable the first buffer 516.
  • the CLK-EN signal is asserted and applied to second buffer 517 via wire line 528 to enable the second buffer 517 and send the logic "1" value on wire line 529 to the Q output on wire line 532 to the enable input for register 518.
  • Register 518 is now enabled and any data present on wire line 530 will be synchronously clocked into the register 518 by the system clock. As the reader can observe, the enable signal to the register 518 runs faster than the evaluation of the data signal to this register 518.
  • the INPUT-EN signal on wire line 533 is not asserted to the second buffer 512. Also, the RESET edge register signal on wire line 527 is asserted to buffers 516 and 517 in the clock edge register 515 to reset these buffers and ensuring that their outputs are logic "0.” Now that the INPUT-EN signal has been asserted for buffer 512, the data on wire line 521 now propagates to the gated data logic 513 to the user's circuit register 518 on wire line 530. Because the enable input to this register 518 is now logic "0,” the data on wire line 530 is cannot be clocked into the register 518.
  • FIG. 20 shows a four state finite state machine to control the software clock logic of FIG. 19 in accordance with one embodiment of the present invention.
  • the EVAL signal is logic "0.”
  • the EVAL signal determines the evaluation cycle, is generated by the system controller, and lasts as many clock cycles as needed to stabilize the logic in the system.
  • the duration of the EVAL signal is determmed by the placement scheme during compilation and is based on the length of the longest direct wire and the length of the longest segmented multiplexed wires (i.e., TDM circuits).
  • EVAL signal is at logic "1.”
  • the clock is enabled.
  • the CLK-EN signal is asserted at logic "1" and thus, the enable signal to the hardware register model is asserted.
  • previously gated data at the hardware register model is evaluated synchronously without risk of hold-time violation.
  • the new data is enabled when INPUT-EN signal is asserted at logic "1."
  • the RESET signal is also asserted to remove the enable signal from the hardware register model.
  • the new data that had been enabled into the hardware register model through the gated data logic network continues to propagate to its intended hardware register model destination or has reached its destination and is waiting to be clocked into the hardware register model if and when the enable signal is asserted again.
  • the propagating new data is stabilizing in the logic while the EVAL signal remain at logic "1.”
  • the muxed-wire as discussed above for the time division multiplexed (TDM) circuit in association with FIGS. 9(A), 9(B), and 9(C), is also at logic "1.”
  • TDM time division multiplexed
  • the SEmulator system initially compiles the user circuit design data into software and hardware models based on a variety of controls including component type. During the hardware compilation process, the system performs the mapping, placement, and routing process as described above with respect to FIG. 6 to optimally partition, place, and interconnect the various components that make up the user's circuit design.
  • the bitstream configuration files or Programmer Object Files (.pof) or alternatively, raw binary files (.rbf)
  • .pof Programmer Object Files
  • Each chip contains a portion of the hardware model corresponding to the user's circuit design.
  • the SEmulator system uses a 4x4 array of FPGA chips, totaling 16 chips.
  • Exemplary FPGA chips include Xilinx XC4000 series family of FPGA logic devices and the Altera FLEX 10K devices.
  • the Xilinx XC4000 series of FPGAs can be used, including the XC4000, XC4000A, XC4000D, XC4000H, XC4000E, XC4000EX, XC4000L, and XC4000XL.
  • Particular FPGAs include the Xilinx XC4005H, XC4025, and Xilinx 4028EX.
  • the Xilinx XC4028EX FPGA engines approach half a million gates in capacity on a single PCI board. Details of these Xilinx FPGAs can be obtained in their data book, Xilinx, The Programmable Logic Data Book (9/96), which is incorporated herein by reference. For Altera FPGAs, details can be found in their data book, Altera, The 1996 Data Book (June 1996), which is incorporated herein by reference.
  • Each array chip consists of a 240-pin Xilinx chip.
  • the array board populated with Xilinx XC4025 chips contains approximately 440,000 configurable gates, and is capable of performing computationally-intensive tasks.
  • the Xilinx XC4025 FPGA consists of 1024 configurable logic blocks (CLBs). Each CLB can implement 32 bits of asynchronous SRAM, or a small amount of general Boolean logic, and two strobed registers. On the periphery of the chip, unsfrobed I/O registers are provided.
  • An alternative to the XC4025 is the XC4005H.
  • the XC4005H devices have high-power 24 mA drive circuits, but are missing the input/output flip/flops of the standard XC4000 series. Details of these and other Xilinx FPGAs can be obtained through their publicly available data sheets, which are incorporated herein by reference.
  • Xilinx XC4000 series FPGAs can be customized by loading configuration data into internal memory cells.
  • the values stored in these memory cells determine the logic functions and interconnections in the FPGA.
  • the configuration data of these FPGAs can be stored in on-chip memory and can be loaded from external memory.
  • the FPGAs can either read configuration data from an external serial or parallel PROM, or the configuration data can be written into the FPGAs from an external device.
  • the XC4000 series FPGAs has up to 1024 CLBs.
  • Each CLB has two levels of look-up tables, with two four-input look-up tables (or function generators F and G) providing some of the inputs to a third three-input lookup table (or function generator H), and two flip-flops or latches.
  • the outputs of these look-up tables can be driven independent of these flip-flops or latches.
  • the CLB can implement the following combination of arbitrary Boolean functions: (1) any function of four or five variables, (2) any function of four variables, any second function of up to four unrelated variables, and any third function of up to three unrelated variables, (3) one function of four variables and another function of six variables, (4) any two functions of four variables, and (5) some functions of nine variables.
  • Two D type flip-flops or latches are available for registering CLB inputs or for storing look-up table outputs. These flip- flops can be used independently from the look-up tables. DIN can be used as a direct input to either one of these two flip-flops or latches and HI can drive the other through the H function generator.
  • Each four-input function generators in the CLB contains dedicated arithmetic logic for the fast generation of carry and borrow signals, which can be configured to implement a two-bit adder with carry-in and carry- out.
  • These function generators can also be implemented as read/write random access memory (RAM).
  • RAM read/write random access memory
  • Altera FLEX 10K chips are somewhat similar in concept. These chips are SRAM-based programmable logic devices (PLDs) having multiple 32-bit buses. In particular, each FLEX 10K100 chip contains approximately 100,000 gates, 12 embedded array blocks (EABs), 624 logic array blocks (LABs), 8 logic elements (LEs) per LAB (or 4,992 LEs), 5,392 flip-flops or registers, 406 I/O pins, and 503 total pins.
  • EABs embedded array blocks
  • LABs logic array blocks
  • LEs logic elements per LAB (or 4,992 LEs)
  • 406 I/O pins and 503 total pins.
  • the Altera FLEX 10K chips contain an embedded array of embedded array blocks (EABs) and a logic array of logic array blocks (LABs).
  • An EAB can be used to implement various memory (e.g., RAM, ROM, FIFO) and complex logic functions (e.g., digital signal processors (DSPs), microcontrollers, multipliers, data transformation functions, state machines).
  • DSPs digital signal processors
  • the EAB provides 2,048 bits.
  • the EAB provides 100 to 600 gates.
  • a LAB via the LEs, can be used to implement medium sized blocks of logic.
  • Each LAB represents approximately 96 logic gates and contains 8 LEs and a local interconnect.
  • An LE contains a 4-input look-up table, a programmable flip-flop, and dedicated signal paths for carry and cascade functions.
  • Typical logic functions that can be created include counters, address decoders, or small state machines.
  • FIG. 8 shows one embodiment of the 4x4 FPGA array and their interconnections. Note that this embodiment of the SEmulator does not use cross bar or partial cross bar connections for the FPGA chips.
  • the FPGA chips include chips Fl 1 to F14 in the first row, chips F21 to F24 in the second row, chips F31 to F34 in the third row, and chips F41 to F44 in the fourth row.
  • each FPGA chip e.g., chip F23
  • each FPGA chip uses only 41 pins for interfacing with the SEmulator system. These pins will be discussed further with respect to FIG. 22.
  • Each interconnection between chips such as interconnection 602 between chip Fll and chip F14, represents 44 pins or 44 wire lines. In other embodiments, each interconnection represents more than 44 pins. Still in other embodiments, each interconnection represents less than 44 pins.
  • Each chip has six interconnections.
  • chip Fll has interconnections 600 to 605.
  • chip F33 has interconnections 606 to 611. These interconnections run horizontally along a row and vertically along a column.
  • Each interconnection provides a direct connection between two chips along a row or between two chips along a column.
  • interconnection 600 directly connects chip Fl 1 and F 13;
  • interconnection 601 directly connects chip Fll and F12;
  • interconnection 602 directly connects chip Fl 1 and F14;
  • interconnection 603 directly connects chip Fl 1 and F31,
  • interconnection 604 directly connects chip Fll andF21;
  • interconnection 605 directly connects chip Fl 1 and F41.
  • interconnection 606 directly connects chip F33 and F13; interconnection 607 directly connects chip F33 and F23; interconnection 608 directly connects chip F33 and F34; interconnection 609 directly connects chip F33 and F43, interconnection 610 directly connects chip F33 andF31; and interconnection 611 directly connects chip F33 andF32.
  • interconnection 600 is labeled as "1.” Because chip Fll is located within one hop from chip F12, interconnection 601 is labeled as “1.” Similarly, because chip Fll is located within one hop from chip F14, interconnection 602 is labeled as “1.” Similarly, for chip F33, all interconnections are labeled as "1.”
  • chip Fl 1 is connected to chip F33 through either of the following two paths: (1) interconnection 600 to interconnection 606; or (2) interconnection 603 to interconnection 610.
  • the path can be either: (1) along a row first and then along a column, or (2) along a column first and then along a row.
  • FIG. 8 shows the FPGA chips configured in a 4x4 array with horizontal and vertical interconnections
  • the actual physical implementation on a board is through low and high banks with an expansion piggyback board.
  • chips F41-F44 and chips F21-F24 are in the low bank.
  • Chips F31-F34 and chips Fl 1-F14 are in the high bank.
  • the piggyback board contains chips Fl 1-F14 and chips F21-F24.
  • piggyback boards containing a number (e.g., 8) of chips are added to the banks and hence, above the row currently containing chips Fl 1-F14.
  • the piggyback board will expand the array below the row currently containing chips F41-F44.
  • FIG. 7 shows a connectivity matrix for the 4x4 FPGA array of FIG. 8. This connectivity matrix is used to generate a placement cost result from a cost function used in the hardware mapping, placement, and routing process for this SEmulation system. The cost function was discussed above with respect to FIG. 6. As an example, chip Fl 1 is located within one hop from chip F13, so the connectivity matrix entry for Fl 1-F13 is
  • FIG. 21 shows the interconnect pin-outs for a single FPGA chip in accordance with one embodiment of the present invention.
  • Each chip has six sets of interconnections, where each set comprises a particular number of pins. In one embodiment, each set has 44 pins.
  • the interconnections for each FPGA chip are oriented horizontally (East- West) and vertically (North-South).
  • the set of interconnections for the West direction is labeled as W[43:0].
  • the set of interconnections for the East direction is labeled as E[43:0].
  • the set of interconnections for the North direction is labeled as N[43:0].
  • the set of interconnections for the South direction is labeled as S[43:0].
  • chip F33 has interconnection 607 for N[43:0], interconnection 608 for E[43:0], interconnection 609 for S[43:0], and interconnection 611 for W[43:0].
  • One set of interconnections is for the non-adjacent interconnections running vertically - YH[21:0] and YH[43:22].
  • the other set of interconnections is for the non-adjacent interconnections running horizontally - XH[21:0] and XH[43:22].
  • Each set, YH[...] and XH[...] are divided into two, where each half of a set contains 22 pins. This configuration allows each chip to be manufactured identically. Thus, each chip is capable of being interconnected in one hop to a non-adjacent chip located above, below, left, and right.
  • This FPGA chip also shows the pin(s) for global signals, the FPGA bus, and JTAG signals.
  • FPGA I/O controller manages the data and control traffic between the PCI bus and the FPGA array.
  • FIG. 22 shows one embodiment of the FPGA controller between the PCI bus and the FPGA array, along with the banks of FPGA chips.
  • the FPGA I/O controller 700 includes CTRL_FPGA unit 701, clock buffer 702, PCI controller 703, EEPROM 704, FPGA serial configuration mterface 705, boundary scan test interface 706, and buffer 707.
  • Appropriate power/voltage regulating circuitry as known to those skilled in the art is provided.
  • Exemplary sources include Vcc coupled to a voltage detector/regulator and a sense amplifier to substantially maintain the voltage in various environmental conditions.
  • the Vcc to each FPGA chip is provided with fast acting thin-film fuses therebetween.
  • the Vcc-HI is provided to the CONFIG# to all FPGA chips and LINTI# to a LOCAL_BUS 708.
  • the CTRL_FPGA unit 701 is the primary controller for FPGA I/O controller 700 to handle the various control, test, and read/write substantive data among the various units and buses.
  • CTRL_FPGA unit 701 is coupled to the low and high banks of FPGA chips.
  • FPGA chips F41-F44 and F21-F24 i.e., low bank
  • FPGA chips F31-F34 and Fl 1-F14 i.e., high bank
  • These FPGA chips Fl 1-F14, F21-F24, F31-F34, and F41-F44 correspond to the FPGA chips in FIG. 8, retaining their reference numbers.
  • the group of resistors 713 coupled to the low bank bus 718 includes, for example, resistor 716 and resistor 717.
  • the group of resistors 712 coupled to the high bank bus 719 includes, for example, resistor 714 and resistor 715. If expansion is desired, more FPGA chips may be installed on the low bank bus 718 and the high bank bus 719 in the direction to the right of FPGA chips Fl 1 and F21. In one embodiment, expansion is done through piggyback boards resembling piggyback board 720.
  • piggyback board 720 which contains FPGA chips F24-F21 in the low bank and chips F14-F11 in the high bank.
  • the piggyback board 720 also includes the additional low and high bank bus, and the thick film chip resistors.
  • the PCI controller 703 is the primary mterface between the FPGA I/O controller 700 and the 32-bit PCI bus 709. If the PCI bus expands to 64 bits and/or 66 MHz, appropriate adjustments can be made in this system without departing from the spirit and scope of the present invention. These adjustments will be discussed below.
  • One example of a PCI controller 703 that may be used in the system is PLX Technology's PCI9080 or 9060.
  • the PCI 9080 has the appropriate local bus interface, control registers, FIFOs, and PCI interface to the PCI bus.
  • the data book PLX Technology, PCI 9080 Data Sheet (ver. 0.93, Feb. 28, 1997) is incorporated herein by reference.
  • the PCI controller 703 passes data between the CTRL_FPGA unit 701 and the PCI bus 709 via a LOCALJBUS 708.
  • LOCALJBUS includes control bus portion, address bus portion, and data bus portion for control signals, address signals, and data signals, respectively. If the PCI bus expands to 64 bits, the data bus portion of LOCALJBUS 708 can also expand to 64 bits.
  • the PCI controller 703 is coupled to EEPROM 704, which contains the configuration data for the PCI controller 703.
  • An exemplary EEPROM 704 is National Semiconductor's 93CS46.
  • the PCI bus 709 supplies a clock signal at 33 MHz to the FPGA I O controller 700.
  • the clock signal is provided to clock buffer 702 via wire line 710 for synchronization purposes and for low timing skew.
  • the output of this clock buffer 702 is the global clock (GL_CLK) signal at 33 MHz supplied to all the FPGA chips via wire line 711 and to the CTRLJFPGA unit 701 via wire line 721. If the PCI bus expands to 66 MHz, the clock buffer will also supply 66 MHz to the system.
  • FPGA serial configuration interface 705 provides configuration data to configure the FPGA chips Fl 1-F14, F21-F24, F31-F34, and F41-F44.
  • the Altera data book, Altera, 1996 DATA BOOK (June 1996) provides detailed information on the configuration devices and processes.
  • FPGA serial configuration interface 705 is also coupled to LOCAL_BUS 708 and the parallel port 721.
  • the FPGA serial configuration interface 705 is coupled to CTRL_FPGA unit 701 and the FPGA chips Fl 1-F14, F21-F24, F31-F34, and F41-F44 via CONFJNTF wire line 723.
  • the boundary scan test interface 706 provides JTAG implementations of certain specified test command set to externally check a processor's or system's logic units and circuits by software. This interface 706 complies with the IEEE Std. 1149.1-1990 specification. Refer to the Altera data book, Altera, 1996 DATA BOOK (June 1996) and Application Note 39 (JTAG Boundary-Scan Testing in Altera Devices), both of which are incorporated herein by reference, for more information. Boundary scan test interface 706 is also coupled to LOCALJBUS 708 and the parallel port 722. Furthermore, the boundary scan test interface 706 is coupled to CTRLJFPGA unit 701 and the FPGA chips Fl 1-F14, F21-F24, F31-F34, and F41-F44 via BST_INTF wire line 724.
  • CTRL_FPGA unit 701 passes data to/from the low (chips F41-F44 and F21-F24) and high (chips F31-F34 and Fl 1-F14) banks of FPGA chips via low bank 32-bit bus 718 and high bank 32-bit bus 719, respectively, along with buffer 707, and F_BUS 725 for the low bank 32 bits FD[31 :0] and F_BUS 726 for the high bank 32 bits FD[63:32].
  • One embodiment duplicates the throughput of the PCI bus 709 in the low bank bus 718 and the high bank bus 719.
  • the performance of the low and high bank buses tracks the performance of the PCI bus. In other words, the performance limitations are in the PCI bus, not in the low and high bank buses.
  • Address pointers are also implemented in each FPGA chip for each software/hardware boundary address space. These address pointers are chained across several FPGA chips through the multiplexed cross chip address pointer chain. Please refer to the address pointer discussion above with respect to FIGS. 9, 11, 12, 14, and 15. To move the word selection signal across the chain of address pointers associated with a given address space and across several chips, chain-out wire lines must be provided. These chain-out wire lines are shown as the arrows between the chips. One such chain-out wire line for the low bank is wire line 730 between chips F23 and F22. Another such chain-out wire line for the high bank is wire line 731 between chips F31 and F32.
  • the chain-out wire line 732 at the end of low bank chip F21 is coupled to the CTRLJFPGA unit 701 as LAST_SHIFT_L.
  • the chain-out wire line 733 at the end of high bank chip Fl 1 is coupled to the CTRL_FPGA unit 701 as LAST_SHIFT_H.
  • LAST_SHIFT_L and LAST_SHIFT_H are the word selection signals for their respective banks as the word selection signals are propagated through the FPGA chips.
  • LAST_SHIFT_L and LAST_SHIFT_H presents a logic "1" to the CTRL_FPGA unit 701, this indicates that the word selection signal has made its way to the end of its respective bank of chips.
  • the CTRLJFPGA unit 701 provides a write signal (F_WR) on wire line 734, a read signal (F_RD) on wire line 735, a DATA_XSFR signal on wire line 736, an EVAL signal on wire line 737, and a SPACE[2:0] signal on wire line 738 to and from the FPGA chips.
  • the CTRLJFPGA unit 701 receives the EVAL_REQ# signal on wire line 739.
  • the write signal (F_WR), read signals (F_RD), DATA_XSFR signal, and SPACE[2:0] signal work together for the address pointers in the FPGA chips.
  • the write signal (F_WR), read signals (F_RD), and SPACE[2:0] signal are used to generate the MOVE signal for the address pointers associated with the selected address space as determined by the SPACE index (SPACE[2:0]).
  • the DATA_XSFR signal is used to initialize the address pointers and begin the word-by-word data transfer process.
  • the EVAL_REQ# signal is used to start the evaluation cycle all over again if any of the FPGA chips asserts this signal.
  • data is transferred or written from main memory in the host processor's computing station to the FPGAs via the PCI bus.
  • the evaluation cycle begins including address pointer initialization and the operation of the software clocks to facilitate the evaluation process.
  • a particular FPGA chip may need to evaluate the data all over again.
  • This FPGA chip asserts the EVAL_REQ# signal and the CNTL_FPGA chip 701 starts the evaluation cycle all over again.
  • FIG.23 shows a more detailed illustration of the CTRLJFPGA unit 701 and buffer 707 of FIG. 22.
  • the same input/output signals and their corresponding reference numbers for CTRLJFPGA unit 701 shown in FIG.22 are also retained and used in FIG.23.
  • additional signals and wire/bus lines not shown in FIG.22 will be described with new reference numbers, such as SEM_FPGA output enable 1016, local interrupt output (Local INTO) 708a, local read/write control signals 708b, local address bus 708c, local interrupt input (Local INTI#) 708d, and local data bus 708e.
  • CTRLJFPGA unit 701 contains a Transfer Done Checking Logic (XSFR_DONE Logic) 1000, Evaluation Control Logic (EVAL Logic) 1001, DMA Descriptor Block 1002, Control Register 1003, Evaluation Timer Logic (EVAL timer) 1004, Address Decoder 1005, Write Flag Sequencer Logic 1006, FPGA Chip Read/Write Control Logic SEMJFPGA R/W Logic) 1007, Demultiplexer and Latch (DEMUX logic) 1008, and latches 1009-1012, which correspond to buffer 707 in FIG.22.
  • a global clock signal (CTRL_FPGA_CLK) on wire/bus 721 is provided to all logic elements/blocks in CTRLJFPGA unit 701.
  • the Transfer Done Checking Logic (XSFRJDONE) 1000 receives LAST_SHIFT_H 733, LAST_SHIFT_L 732 and local INTO 708a.
  • XSFRJDONE logic 1000 outputs a transfer done signal (XSFRJDONE) on wire/bus 1013 to EVAL Logic 1001. Based on the reception of LAST_SHIFT_H 733 and LAST_SHIFT_L 732, the XSFRJDONE logic 1000 checks for the completion of the data transfer so that the evaluation cycle can begin, if desired.
  • the EVAL Logic 1001 receives the EVAL_REQ# signal on wire/bus 739 and WRJXSFR/RDJtSFR signal on wire/bus 1015, in addition to transfer done signal (XSFRJDONE) on wire bus 1013. EVAL Logic 1001 generates two output signals, Start EVAL on wire/bus 1014 and DATAJ SFR on wire/bus 736. The EVAL logic indicates when data transfer between the FPGA bus and the PCI bus will begin to initialize the address pointers. It receives the XSFRJDONE signal when the data transfer is complete. The WR_XSFR RD_XSFR signal indicates whether the transfer is a read or a write.
  • the EVAL logic can start the evaluation cycle with the start EVAL signal t the EVAL timer.
  • the EVAL timer dictates the duration of the evaluation cycle and ensures the successful operation of the software clock mechanism by keeping the evaluation cycle active for as long as necessary to stabilize the data propagation to all the registers and combinational components.
  • DMA descriptor block 1002 receives the local bus address on wire bus 1019, a write enable signal on wire/bus 1020 from address decoder 1005, and local bus data on wire/bus 1029 via local data bus 708e. The output is DMA descriptor output on wire/bus 1046 to DEMUX logic 1008 on wire/bus 1045.
  • the DMA descriptor block 1002 contains the descriptor block information corresponding to that in the host memory, including PCI address, local address, transfer count, transfer direction, and address of the next descriptor block.
  • the host will also set up the address of the initial descriptor block in the descriptor pointer register of the PCI controller. Transfers can be initiated by setting a control bit.
  • the PCI loads the first descriptor block and initiates the data transfer.
  • the PCI controller continues to load descriptor blocks and transfer data until it detects the end of the chain bit is set in the next descriptor pointer register.
  • Address decoder 1005 receives and transmits local R W control signals on bus 708b, and receives and transmits local address signals on bus 708c.
  • the address decoder 1005 generates a write enable signal on wire/bus 1020 to the DMA descriptor 1002, a write enable signal on wire/bus 1021 to control register 1003, the FPGA address SPACE index on wire/bus 738, a control signal on wire bus 1027, and another control signal on wire/bus 1024 to DEMUX logic 1008.
  • Control register 1003 receives the write enable signal on wire/bus 1021 from address decoder 1005, and data from wire/bus 1030 via local data bus 708e.
  • the control register 1003 generates a WR_XSFR RD_XSFR signal on wire/bus 1015 to EVAL logic 1001, a Set EVAL time signal on wire/bus 1041 to EVAL timer 1004, and a SEMJFPGA output enable signal on wire/bus 1016 to the FPGA chips.
  • the system uses the SEMJFPGA output enable signal to turn on or enable each FPGA chip selectively. Typically, the system enables each FPGA chip one at a time.
  • EVAL timer 1004 receives the Start EVAL signal on wire/bus 1014, and the Set EVAL time on wire/bus 1041. EVAL timer 1004 generates the EVAL signal on wire/bus 737, an evaluation done (EVALJDONE) signal on wire/bus 1017, and a Start write flag signal on wire/bus 1018 to the Write Flag Sequencer logic 1006. In one embodiment, the EVAL timer is 6 bits long
  • the Write Flag Sequencer logic 1006 receives the Start write flag signal on wire/bus 1018 from EVAL timer
  • the Write Flag Sequencer logic 1006 generates a local R/W control signal on wire/bus 1022 to local R/W wire/bus 708b, local address signal on wire/bus 1023 to local address bus 708c, a local data signal on wire/bus 1028 to local data bus 708e, and local INTI# on wire/bus 708d.
  • the write flag sequencer logic Upon receiving the start write flag signal, the write flag sequencer logic begins the sequence of control signals to begin the memory write cycles to the PCI bus.
  • the SEMJFPGA R W Control logic 1007 receives control signals on wire/bus 1027 from the address decoder
  • the SEMJFPGA R W Control logic 1007 generates enable signal on wire/bus 1035 to latch 1009, a control signal on wire/bus 1025 to the DEMUX logic 1008, an enable signal on wire/bus 1037 to latch 1011, an enable signal on wire/bus 1040 to latch 1012, a FJVVR signal on wire/bus 734, and a F-RD signal on wire/bus 735.
  • the SEMJFPGA R/W Control logic 1007 controls the various write and read data transfers to/from the FPGA low bank and high bank buses.
  • the DEMUX logic 1008 is a multiplexer and a latch which receives four sets of input signals and outputs one set of signals on wire/bus 1026 to the local data bus 708e.
  • the selector signals are the control signal on wire/bus 1025 from SEMJFPGA R/W control logic 1007 and the control signal on wire/bus 1024 from address decoder 1005.
  • the DEMUX logic 1008 receives one set of inputs from EVALJDONE signal on wire/bus 1042, XSFRJDONE signal on wire/bus 1043, and EVAL signal on wire/bus 1044. This single set of signals is labeled as reference number 1048.
  • DEMUX logic 1008 At any one time period, only one of these three signals, EVALJDONE, XSFRJDONE, and EVAL will be provided to DEMUX logic 1008 for possible selection.
  • the DEMUX logic 1008 also receives, as the other three sets of input signals, the DMA descriptor output signal on wire/bus 1045 from the DMA descriptor block 1002, a data output on wire/bus 1039 from latch 1012, and another data output on wire/bus 1034 from latch 1010.
  • the data buffer between the CTRLJFPGA unit 701 and the low and high FPGA bank bus comprise latches 1009 to 1012.
  • Latch 1009 receives local bus data on wire/bus 1032 via wire/bus 1031 and local data bus 708e, and an enable signal on wire/bus 1035 from SEMJFPGA R/W Control logic 1007.
  • Latch 1009 outputs data on wire/bus 1033 to latch 1010.
  • Latch 1010 receives data on wire/bus 1033 from latch 1009, and an enable signal on wire/bus 1036 via wire/bus 1037 from SEMJFPGA R/W Control logic 1007. Latch 1010 outputs data on wire/bus 725 to the FPGA low bank bus and the DEMUX logic 1008 via wire/bus 1034.
  • Latch 1011 receives data on wire bus 1031 from local data bus 708e, and an enable signal on wire/bus 1037 from SEMJFPGA R/W Control logic 1007. Latch 1011 outputs data on wire/bus 726 to the FPGA high bank bus and on wire/bus 1038 to latch 1012.
  • Latch 1012 receives data on wire/bus 1038 from latch 1011, and an enable signal on wire/bus 1040 from SEMJ ⁇ GA R/W Control logic 1007. Latch 1012 outputs data on wire/bus 1039 to DEMUX 1008.
  • FIG.24 shows the 4x4 FPGA array, its relationship to the FPGA banks, and the expansion capability. Like FIG. 8, FIG.24 shows the same 4x4 array.
  • the CTRLJFPGA unit 740 is also shown.
  • Low bank chips chips F41-F44 and F21-F24
  • high bank chips chips F31-F34 and Fl 1-F14
  • the data transfer chain follows the banks in a predetermined order.
  • the data transfer chain for the low bank is shown by arrow 741.
  • the data transfer chain for the high bank is shown by arrow 742.
  • the JTAG configuration chain is shown by arrow 743, which runs through the entire array of 16 chips from F41 to F44, F34 to F31, F21 to F24, and F14 to Fl 1, and back to the CTRLJ F PGA unit 740.
  • Expansion can be accomplished with piggyback boards. Assuming in FIG. 24 that the original array of FPGA chips included F41-F44 and F31-F34, the addition of two more rows of chips F21-F24 and Fl 1-F14 can be accomplished with piggyback board 745.
  • the piggyback board 745 also includes the appropriate buses to extend the banks. Further expansion can be accomplished with more piggyback boards placed one on top of the other in the array.
  • FIG. 25 shows one embodiment of the hardware start-up method.
  • Step 800 initiates the power on or warm boot sequence.
  • the PCI controller reads the EEPROM for initialization.
  • Step 802 reads and writes PCI controller registers in light of the initialization sequence.
  • Step 803 boundary scan tests for all the FPGA chips in the array.
  • Step 804 configures the CTRLJFPGA unit in the FPGA I/O controller.
  • Step 802 reads and writes the registers in the CTRLJFPGA unit.
  • Step 806 sets up the PCI controller for DMA master read/write modes. Thereafter, the data is transferred and verified.
  • Step 807 configures all the FPGA chips with a test design and verifies its correctness.
  • the hardware is ready for use. At this point, the system assumes all the steps resulted in a positive confirmation of the operability of the hardware, otherwise, the system would never reach step 808.
  • the FPGA logic devices are provided on individual boards. If more FPGA logic devices are required to model the user's circuit design than is provided in the board, multiple boards with more FPGA logic devices can be provided. The ability to add more boards into the Simulation system is a desirable feature of the present invention.
  • denser FPGA chips such as Altera 10K130V and 10K250V, are used. Use of these chips alters the board design such that only four FPGA chips, instead of eight less dense FPGA chips (e.g., Altera 10K100), are used per board.
  • the coupling of these boards to the motherboard of the Simulation system presents a challenge.
  • the interconnection and connection schemes must compensate for the lack of a backplane.
  • the FPGA array in the Simulation system is provided on the motherboard through a particular board interconnect structure.
  • Each chip may have up to eight sets of interconnections, where the interconnections are arranged according to adjacent direct-neighbor interconnects (i.e., N[73:0], S[73:0], W[73:0], E[73:0]), and one-hop neighbor interconnects (i.e., NH[27:0], SH[27:0], XH[36:0], XH[72:37]), excluding the local bus connections, within a single board and across different boards.
  • adjacent direct-neighbor interconnects i.e., N[73:0], S[73:0], W[73:0], E[73:0]
  • one-hop neighbor interconnects i.e., NH[27:0], SH[27
  • Each chip is capable of being interconnected directly to adjacent neighbor chips, or in one hop to a non-adjacent chip located above, below, left, and right.
  • the array In the X direction (east-west), the array is a torus. In the Y direction (north-south), the array is a mesh.
  • the interconnects alone can couple logic devices and other components within a single board. However, inter- board connectors are provided to couple these boards and interconnects together across different boards to carry signals between (1) the PCI bus via the motherboard and the array boards, and (2) any two array boards.
  • Each board contains its own FPGA bus FD[63:0] that allows the FPGA logic devices to communicate with each other, the SRAM memory devices, and the CTRLJFPGA unit (FPGA I/O controller).
  • the FPGA bus FD[63:0] is not provided across the multiple boards.
  • the FPGA interconnects however, provide connectivity among the FPGA logic devices across multiple boards although these mterconnects are not related to the FPGA bus. On the other hand, the local bus is provided across all the boards.
  • a motherboard connector connects the board to the motherboard, and hence, to the PCI bus, power, and ground.
  • the motherboard connector is not used for direct connection to the motherboard.
  • every other board is directly connected to the motherboard, and interconnects and local buses of these boards are coupled together via inter-board connectors arranged solder-side to component-side.
  • PCI signals are routed through one of the boards (typically the first board) only. Power and ground are applied to the other motherboard connectors for those boards. Placed solder-side to component-side, the various inter-board connectors allow communication among the PCI bus components, the FPGA logic devices, memory devices, and various Simulation system control circuits.
  • FIG. 56 shows a high level block diagram of the array of FPGA chip configuration in accordance with one embodiment of the present invention.
  • a CTRLJFPGA unit 1200 described above, is coupled to bus 1210 via line 1209.
  • the CTRLJFPGA unit 1200 is a programmable logic device (PLD) in the form of an FPGA chip, such as an Altera 10K50 chip.
  • Bus 1210 allows the CTRLJFPGA unit 1200 to be coupled to other Simulation array boards (if any) and other chips (e.g., PCI controller, EEPROM, clock buffer).
  • FIG. 56 shows other major functional blocks in the form of logic devices and memory devices.
  • the logic device is a programmable logic device (PLD) in the form of an FPGA chip, such as an Altera 10K130V or 10K250V chip.
  • PLD programmable logic device
  • the 10K130V and 10K250V are pin compatible and each is a 599-pin PGA package.
  • this embodiment uses only four chips of Altera' s FLEX 10K130.
  • One embodiment of the present invention describes the board containing these four logic devices and their interconnections.
  • inter- FPGA logic device communication is necessary to connect one part of the user's circuit design to another part.
  • initial configuration information and boundary scan tests are also supported by the inter-FPGA interconnects.
  • the necessary Simulation system control signals must be accessible between the Simulation system and the FPGA logic devices.
  • FIG. 36 shows the hardware architecture an FPGA logic device used in the present invention.
  • the FPGA logic device 1500 includes 102 top I/O pins, 102 bottom I/O pins, 111 left I/O pins, and 111 right I/O pins. Thus, the total number of interconnect pins is 425.
  • an additional 45 I/O pins are dedicated for GCLK, FPGA bus FD[31 :0] (for the high bank, FD[63 :32] is dedicated), FJ D, F_WR, DATAXSFR, SHIFTIN, SHIFTOUT, SPACE[2:0], EVAL, EVAL_REQ_N, DEVICE_OE (signal from CTRLJFPGA unit to turn on the output pins of FPGA logic devices), and DEV_CLRN (signal from CTRLJFPGA unit to clear all the internal flip-flops before starting the simulation).
  • any data and control signals that cross between any two FPGA logic devices are carried by these interconnections.
  • the remaining pins are dedicated for power and ground.
  • FIG. 37 shows the FPGA interconnect pin-outs for a single FPGA chip in accordance with one embodiment of the present invention.
  • Each chip 1510 may have up to eight sets of interconnections, where each set comprises a particular number of pins. Some chips may have less than eight sets of interconnections depending on their respective positions on the board. In the preferred embodiment, all chips have seven sets of interconnections, although the specific sets of interconnections used may vary from chip to chip depending on their respective location on the board.
  • the interconnections for each FPGA chip are oriented horizontally (East- West) and vertically (North-South). The set of interconnections for the West direction is labeled as W[73:0]. The set of interconnections for the East direction is labeled as E[73:0].
  • the set of interconnections for the North direction is labeled as N[73:0].
  • the set of interconnections for the South direction is labeled as S[73:0].
  • These complete sets of interconnections are for the connections to adjacent chips; that is, these interconnections do not "hop" over any chip.
  • chip 1570 has interconnection 1540 for N[73:0], interconnection 1542 for W[73:0], interconnection 1543 forE[73:0], and interconnection 1545 for S[73:0].
  • this FPGA chip 1570 which is also the FPGA2 chip, has all four sets of adjacent interconnections - N[73:0], S[73:0], W[73:0], and E[73:0].
  • the West interconnection sof FPGAO connects to the east interconnections of FPGA3 through wire 1539 via a torus-style interconnections.
  • wire 1539 allows the chips 1569 QFPGA0) and 1572 (FPGA3) to be directly coupled to each other in a manner akin to wrapping the west-east ends of the board to be wrapped around to meet each other.
  • FPGA2 chip 1570 in FIG. 39 shows NH interconnect 1541 and SH interconnect 1546.
  • the other two sets of interconnections are for the non-adjacent interconnections running horizontally - XH[36:0] and XH[72:37].
  • FPGA2 chip 1570 in FIG. 39 shows XH interconnect 1544.
  • the vertical hopping interconnections NH[27:0] and SH[27:0] have 28 pins each.
  • the horizontal interconnections have 73 pins, XH[36:0] and XH[72:37].
  • the horizontal interconnection pins, XH[36:0] and XH[72:37] can be used on the west side (e.g., for FPGA3 chip 1576, interconnect 1605 in FIG. 39) and/or the east side (e.g., for FPGAO chip 1573, interconnect 1602 in FIG. 39).
  • This configuration allows each chip to be manufactured identically.
  • each chip is capable of being interconnected in one hop to a non-adjacent chip located above, below, left, and right.
  • FIG. 39 shows a direct-neighbor and one-hop neighbor FPGA array layout of the six boards on a single motherboard in accordance with one embodiment of the present invention. This figure will be used to illustrate two possible configurations - a six-board system and a dual-board system.
  • Position indicator 1550 shows that the "Y" direction is north-south and the "X" direction is east-west. In the X direction, the array is a torus. In the Y direction, the array is a mesh.
  • FIG. 39 only the boards, FPGA logic devices, interconnects, and connectors at a high level are shown.
  • the motherboard and other supporting components e.g., SRAM memory devices
  • wire lines e.g., FPGA bus
  • FIG. 39 provides an array view of the boards and their components, interconnects, and connectors.
  • the actual physical configuration and installation involves placing these boards on their respective edges component- side to solder-side. Approximately half of the boards are directly connected to the motherboard while the other half of the boards are connected to their respective neighbor boards.
  • the six boards 1551 (boardl), 1552 (board2), 1553 (board3), 1554 (board4), 1555 (board5), and 1556 (board6) are provided on the motherboard (not shown) as part of the reconfigurable hardware unit 20 in FIG. 1.
  • Each board contains an almost identical set of components and connectors.
  • the sixth board 1556 contains FPGA logic devices 1565 to 1568, and connectors 1557 to 1560 and 1581;
  • the fifth board 1555 contains FPGA logic devices 1569 to 1572 and connectors 1582 and 1583;
  • the fourth board 1554 contains FPGA logic devices 1573 to 1576, and connectors 1584 and 1585.
  • boardl 1551 andboard ⁇ 1556 are provided as "bookend” boards that contain the Y-mesh terminations such as R-pack terminations 1557 to 1560 on board6 1556 and terminations 1591 to 1594 on boardl 1551.
  • Intermediately placed boards i.e., boards 1552 (board2), 1553 (board3), 1554 (board4), and 1555 (board5) are also provided to complete the array.
  • the interconnects are arranged according to adjacent direct-neighbor interconnects (i.e., N[73:0], S[73:0], W[73:0], E[73:0]), and one-hop neighbor interconnects (i.e., NH[27:0], SH[27:0], XH[36:0], XH[72:37]), excluding the local bus connections, within a single board and across different boards.
  • the interconnects alone can couple logic devices and other components within a single board.
  • inter-board connectors 1581 to 1590 allow communication among the FPGA logic devices across different boards (i.e., boardl to board ⁇ ).
  • the FPGA bus is part of the inter-board connectors 1581 to 1590.
  • These connectors 1581 to 1590 are 600-pin connectors carrying 520 signals and 80 power/ground connections between two adjacent array boards.
  • inter-board connectors 1581 to 1590 are provided between board 1551 and 1552.
  • inter-board connectors 1589 and 1590 are provided between board 1551 and 1552.
  • Interconnect 1515 connects FPGA logic devices 1511 and 1577 together and according to connectors 1589 and 1590, this connection is symmetrical.
  • interconnect 1603 is not symmetrical; it connects an FPGA logic device in the third board 1553 to the FPGA logic device 1577 in board 1551. With respect to connectors 1589 and 1590, such an interconnect is not symmetrical.
  • interconnect 1600 is not symmetrical with respect to connectors 1589 and 1590 because it connects FPGA logic device 1577 to the termination 1591, which connects to FPGA logic device 1577 via interconnect 1601.
  • Other similar interconnects exist which further shows the non- symmetry.
  • the interconnects are routed tlirough the inter-board connectors in two different ways - one for symmetric interconnects like interconnect 1515 and another for non-symmetric interconnects like interconnects 1603 and 1600.
  • the interconnection routing scheme is shown in FIGS. 40(A) and 40(B).
  • an example of a direct-neighbor connection within a single board is interconnect 1543 which couples logic device 1570 to logic device 1571 along the east-west direction in board 1555.
  • Another example of a direct-neighbor connection within a single board is interconnect 1607 which couples logic device 1573 to logic device . 1576 in board 1554.
  • An example of a direct-neighbor connection between two different boards is interconnect 1545 which couples logic device 1570 in board 1555 to logic device 1574 in board 1554 via connectors 1583 and 1584 along the north-south direction.
  • two inter-board connectors 1583 and 1584 are used to transport signals across.
  • interconnect 1544 which couples logic device 1570 to logic device 1572 in board 1555 along the east-west direction.
  • An example of a one-hop interconnect between two different boards is interconnect 1599 which couples logic device 1565 in board 1556 to logic device 1573 in board 1554 via connectors 1581 to 1584.
  • four inter-board connectors 1581 to 1584 are used to transport signals across.
  • the sixth board 1556 includes the 10-ohm R-pack connectors 1557 to 1560
  • the first board 1551 includes the 10-ohm R-pack connectors 1591 to 1594.
  • the sixth board 1556 contains R-pack connector 1557 for interconnects 1970 and 1971, R-pack connector 1558 for interconnects 1972 and 1541, R-pack connector 1559 for interconnects 1973 and 1974, and R-pack connector 1560 for interconnects 1975 and 1976. Moreover, interconnects 1561 to 1564 are not connected to anything. These north-south interconnections, unlike the east-west torus-type interconnections, are arranged in mesh-type fashion.
  • FPGA logic devices 1511 and 1577 already have one set of direct interconnection 1515. Additional interconnections are also provided for these two FPGA logic devices via R-pack 1591 and interconnects 1600 and 1601; that is, R-pack 1591 connects interconnects 1600 and 1601 together. This increases the number of direct connections between FPGA logic devices 1511 and 1577.
  • Logic devices 1577, 1578, 1579, and 1580 on board 1551 are coupled to logic devices 1511, 1512, 1513, and 1514 on board 1552 via interconnects 1515, 1516, 1517, and 1518 and inter-board connectors 1589 and 1590.
  • interconnect 1515 couples the logic device 1511 on board 1552 to logic device 1577 on board 1551 via connectors 1589 and 1590;
  • interconnect 1516 couples the logic device 1512 on board 1552 to logic device 1578 on board 1551 via connectors 1589 and 1590;
  • interconnect 1517 couples the logic device 1513 on board 1552 to logic device 1579 on board 1551 via connectors 1589 and 1590;
  • interconnect 1518 couples the logic device 1514 on board 1552 to logic device 1580 on board 1551 via connectors 1589 and 1590.
  • R-pack 1591 connects interconnects 1600 and 1601 to increase the number of north-south interconnects.
  • FIG. 44 A dual-board embodiment of the present invention is illustrated in FIG. 44.
  • the dual-board configuration of FIG. 44 uses the same two boards for "bookends" - boardl 1551 and board ⁇ 1556, which are provided on a motherboard as part of the reconfigurable hardware unit 20 in FIG. 1.
  • one bookend board is boardl and the second bookend board is board6.
  • Board6 is used in FIG. 44 to show its similarity to board6 in FIG. 39; that is, the bookend boards like boardl and board ⁇ should have the requisite terminations for the north-south mesh connections.
  • This dual-board configuration contains four FPGA logic devices 1577 (FPGAO), 1578 (FPGA1), 1579 (FPGA2), and 1580 (FPGA3) on boardl 1551, and four FPGA logic devices 1565 (FPGAO), 1566 (FPGA1), 1567 (FPGA2), and 1568 (FPGA3) on board ⁇ 1556. These two boards are connected by inter-board connectors 1581 and 1590.
  • Board 1551 contains 10-ohm R-pack connectors 1591, 1592, 1593, and 1594 as resistive terminations.
  • the second board 1556 also contains the 10-ohm R-pack connectors 1557 to 1560.
  • Board 1551 has connector 1590 and board 1556 has connector 1581 for inter-board communication.
  • the inter-board connectors 1590 and 1581 carry control data and control signals on the FPGA buses.
  • boardl and board ⁇ provide the bookend boards, while board2 1552 and board3 1553 (see FIG. 39) are the intermediate boards.
  • boardl and board2 are paired and board3 and board6 are paired.
  • boardl and board6 provide the bookend boards as discussed above, while board2 1552, board3 1553, board4 1554, and board5 1555 (see FIG. 39) are the intermediate boards.
  • boardl and board2 are paired, board3 and board4 are paired, and board5 and board ⁇ are paired.
  • the bookend boards (such boardl and board ⁇ of FIG. 39) should have the requisite terminations that complete the mesh array connections.
  • the minimum configuration is the dual-board configuration of FIG. 44. More boards can be added by two-board increments. If the initial configuration had boardl and board ⁇ , a future modification to a four-board configuration involves moving the board ⁇ further out and pairing boardl and board2 together, and then pairing board3 and board ⁇ together, as mentioned above.
  • each logic device is coupled to its adjacent neighbor logic device and its non-adjacent neighbor logic device within one hop.
  • logic device 1577 is coupled to adjacent neighbor logic device 1578 via interconnect 1547.
  • Logic device 1577 is also coupled to non-adjacent logic device 1579 via one- hop interconnect 1548.
  • logic device 1580 is considered to be adjacent to logic device 1577 due to the wraparound torus configuration with interconnect 1549 providing the coupling.
  • FIG. 42 shows a top view (component side) of the on-board components and connectors for a single board.
  • only one board is necessary to model the user's design in the Simulation system.
  • multiple boards i.e., at least 2 boards
  • FIG. 39 shows six boards 1551 to 1556 coupled together through various 600-pin connectors 1581 to 1590. At the top and bottom ends, board 1551 is terminated by one set of 10-ohm R-packs and board 1556 is terminated by another set of 10-ohm R- packs.
  • board 1820 contains four FPGA logic devices 1822 (FPGAO), 1823 (FPGA1), 1824 (FPGA2), and 1825 (FPGA3).
  • FPGAO FPGA logic devices
  • FPGA1 1823
  • FPGA2 1824
  • FPGA3 1825
  • SRAM memory devices 1828 and 1829 are also provided. These SRAM memory devices 1828 and 1829 will be used to map the memory blocks from the logic devices on this board; in other words, the memory Simulation aspect of the present invention maps memory blocks from the logic devices on this board to the SRAM memory devices on this board.
  • Other boards will contain other logic devices and memory devices to accomplish a similar mapping operation.
  • the memory mapping is dependent on the boards; that is, memory mapping for boardl is limited to logic devices and memory devices on boardl while disregarding other boards. In other embodiments, the memory mapping is independent of the boards.
  • LEDs 1821 are also provided to visually indicate some select activities.
  • the LED display is as follows in Table A in accordance with one embodiment of the present invention:
  • PLX PCI controller 1826 and CTRLJFPGA unit 1827 control inter- FPGA and PCI communications.
  • PLX PCI controller 1826 that may be used in the system is PLX Technology's PCI9080 or 9060.
  • the PCI 9080 has the appropriate local bus interface, control registers, FIFOs, and PCI interface to the PCI bus.
  • the data book PLX Technology, PCI 9080 Data Sheet (ver. 0.93, Feb. 28, 1997) is incorporated herein by reference.
  • CTRLJFPGA unit 1827 is a programmable logic device (PLD) in the form of an FPGA chip, such as an Altera 10K50 chip. In multiple board configurations, only the first board coupled to the PCI bus contains the PCI controller.
  • PLD programmable logic device
  • Connector 1830 connects the board 1820 to the motherboard (not shown), and hence, the PCI bus, power, and ground. For some boards, the connector 1830 is not used to for direct connection to the motherboard. Thus, in a dual- board configuration, only the first board is directly coupled to the motherboard. In a six-board configuration, only boards 1, 3, and 5 are directly connected to the motherboard while the remaining boards 2, 4, and 6 rely on their neighbor boards for motherboard connectivity. Inter-board connectors JI to J28 are also provided. As the name implies, these connectors JI to J28 allow connections across different boards.
  • Connector JI is for external power and ground connections.
  • Table B shows the pins and corresponding description for the external power connector JI in accordance with one embodiment of the present invention:
  • Connector J2 is for the parallel port connection.
  • Connectors JI and J2 are used for stand-alone single-board boundary scan test during production.
  • Table C shows the pins and corresponding description for the parallel JTAG port connector J2 in accordance with one embodiment of the present invention:
  • Connectors J3 and J4 are for the local bus connections across boards.
  • Connectors J5 to J16 are one set of FPGA interconnect connections.
  • Connectors J17 to J28 are a second set of FPGA interconnect connections. When placed component-side to solder-side, these connectors provide effective connections between one component in one board with another component in another board.
  • Tables D and E provide a complete list and description of the connectors JI to J28 in accordance with one embodiment of the present invention:
  • VO direction is for Boardl.
  • FIG.43 shows a legend of the connectors JI to J28 in FIGS. 41(A) to 41(F) and 42.
  • the clear filled blocks indicate surface mount
  • the gray filled blocks represent the through hole types.
  • the solid outline block represents the connectors located on the component side.
  • the dotted outline block represents the connectors located on the solder side.
  • the block 1840 with the clear fill and the solid outline represents a 2x30 header, surface mount and located on the component side.
  • Block 1841 with the clear fill and the dotted outline represents a 2x30 receptacle, surface mount and located on the solder side of the board.
  • Block 1842 with the gray fill and solid outline represents a 2x30 or 2x45 header, through hole and located on the component side.
  • Block 1843 with the gray fill and the dotted outline represents a 2x45 or 2x30 receptacle, through hole and located on the solder side.
  • the Simulation system uses Samtec's SFM and TFM series of 2x30 or 2x45 micro strip connectors for both surface mount and through hole types.
  • Block 1844 with the cross-hatched fill and the solid outline is an R-pack, surface mount and located on the component side of the board.
  • Block 1845 with the cross-hatched fill and the dotted outline is an R-pack, surface mount and located on the solder side.
  • the Samtec specification from Samtec's catalog on their website is incorporated by reference herein.
  • connectors J3 to J28 are the type as indicated in the legend of FIG. 43.
  • FIGS. 41(A) to 41(F) show top views of each board and their respective connectors.
  • FIG. 41(A) shows the connectors for board ⁇ .
  • board 1660 contains connectors 1661 to 1681 along with motherboard connector 1682.
  • FIG.41(B) shows the connectors for board5.
  • board 1690 contains connectors 1691 to 1708 along with motherboard connector 1709.
  • FIG.41(C) shows the connectors for board4.
  • board 1715 contains connectors 1716 to 1733 along with motherboard connector 1734.
  • FIG.41(D) shows the connectors for board3.
  • board 1740 contains connectors 1741 to 1758 along with motherboard connector 1759.
  • FIG. 41(E) shows the connectors for board2.
  • board 1765 contains connectors 1766 to 1783 along with motherboard connector 1784.
  • FIG.41(F) shows the connectors for boardl.
  • board 1790 contains connectors 1791 to 1812 along with motherboard connector 1813.
  • these connectors for the six boards are various combinations of (1) surface mount or through hole, (2) component side or solder side, and (3) header or receptacle or R-pack.
  • these connectors are used for inter-board communications. Related buses and signals are grouped together and supported by these inter-board connectors for routing signals between any two boards. Also, only half of the boards are directly coupled to the motherboard.
  • board ⁇ 1660 contains connectors 1661 to 1668 designated for one set of the FPGA interconnects, connectors 1 69 to 1674, 1676, and 1679 designated for another set of FPGA interconnects, and connector 1681 designated for the local bus. Because board ⁇ 1660 is positioned as one of the boards at the end of the motherboard (along with boardl 1790 in FIG.
  • connectors 1675, 1677, 1678, and 1680 are designated for the 10-ohm R-pack connections for certain north-south interconnects.
  • the motherboard connector 1682 is not used for board ⁇ 1660, as shown in FIG. 38(B) where the sixth board 1535 is coupled to the fifth board 1534 but not directly coupled to the motherboard 1520.
  • board5 1690 contains connectors 1691 to 1698 designated for one set of the FPGA interconnects, connectors 1699 to 1706 designated for another set of FPGA interconnects, and connectors 1707 and 1708 designated for the local bus.
  • Connector 1709 is used to couple board5 1690 to the motherboard.
  • board4 1715 contains connectors 1716 to 1723 designated for one set of the FPGA interconnects, connectors 1724 to 1731 designated for another set of FPGA interconnects, and connectors 1732 and 1733 designated for the local bus.
  • Connector 1709 is not used to couple board4 1715 directly to the motherboard. This configuration is also shown in FIG. 38(B) where the fourth board 1533 is coupled to the third board 1532 and the fifth board 1534 but not directly coupled to the motherboard 1520.
  • board3 1740 contains connectors 1741 to 1748 designated for one set of the FPGA interconnects, connectors 1749 to 1756 designated for another set of FPGA interconnects, and connectors 1757 and 1758 designated for the local bus.
  • Connector 1759 is used to couple board3 1740 to the motherboard.
  • board2 1765 contains connectors 1766 to 1773 designated for one set of the FPGA interconnects, connectors 1774 to 1781 designated for another set of FPGA interconnects, and connectors 1782 and 1783 designated for the local bus.
  • Connector 1784 is not used to couple board2 1765 directly to the motherboard. This configuration is also shown in FIG. 38(B) where the second board 1525 is coupled to the third board 1532 and the first board 1526 but not directly coupled to the motherboard 1520.
  • boardl 1790 contains connectors 1791 to 1798 designated for one set of the FPGA interconnects, connectors 1799 to 1804, 1806, and 1809 designated for another set of FPGA interconnects, and connectors 1811 and 1812 designated for the local bus.
  • Connector 1813 is used to couple boardl 1790 to the motherboard. Because boardl 1790 is positioned as one of the boards at the end of the motherboard (along with board ⁇ 1660 in FIG. 41(A) at the other end), connectors 1805, 1807, 1808, and 1810 are designated for the 10-ohm R-pack connections for certain north-south interconnects.
  • multiple boards are coupled to the motherboard and to each other in a unique manner. Multiple boards are coupled together component-side to solder-side.
  • One of the boards say the first board, is coupled to the motherboard and hence, the PCI bus, via a motherboard connector.
  • the FPGA interconnect bus on the first board is coupled to the FPGA mterconnect bus of the other board, say the second board, via a pair of FPGA interconnect connectors.
  • the FPGA interconnect connector on the first board is on the component side and the FPGA interconnect connector on the second board is on the solder side.
  • the component-side and solder-side connectors on the first board and second board, respectively, allow the FPGA interconnect buses to be coupled together.
  • the local buses on the two boards are coupled together via local bus connectors.
  • the local bus connector on the first board is on the component side and the local bus connector on the second board is on the solder side.
  • the component-side and solder-side connectors on the first board and second board, respectively allow the local buses to be coupled together.
  • a third board can be added with its solder-side to the component-side of the second board. Similar FPGA interconnects and local bus inter-board connections are also made. The third board is also coupled to the motherboard via another connector but this connector merely provides power and ground to the third board, to be discussed further below.
  • FIG. 38(A) shows side views of the FPGA board connection on the motherboard in accordance with one embodiment of the present invention.
  • FIG. 38(A) shows the dual-board configuration where, as the name implies ⁇ only two boards are utilized. These two boards 1525 0 oard2) and 1526 (boardl) in FIG. 38(A) coincide with the two boards 1552 and 1551 in FIG. 39.
  • the component sides of the boards 1525 and 1526 are represented by reference numeral 1989.
  • the solder side of the two boards 1525 and 1526 are represented by reference numeral 1988. As shown in FIG. 38(A), these two boards 1525 and 1526 are coupled to the motherboard 1520 via motherboard connector 1523.
  • motherboard connectors 1521, 1522, and 1524 can also be provided for expansion purposes. Signals between the PCI bus and the boards 1525 and 1526 are routed via the motherboard connector 1523. PCI signals are routed between the dual-board structure and the PCI bus via the first board 1526 first. Thus, signals from the PCI bus encounter the first board 1526 first before they travel to the second board 1525. Analogously, signals to the PCI bus from the dual-board structure are sent from the first board 1526. Power is also applied to the boards 1525 and 1526 via motherboard connector 1523 from a power supply (not shown).
  • board 1526 contains several components and connectors.
  • One such component is an FPGA logic device 1530.
  • Connectors 1528A and 1531 A are also provided.
  • board 1525 contains several components and connectors.
  • One such component is an FPGA logic device 1529.
  • Connectors 1528B and 1531B are also provided.
  • connectors 1528 A and 1528B are the inter-board connectors for the FPGA bus such as 1590 and 1581 (FIG. 44). These inter-board connectors provide the inter-board connectivity for the various FPGA interconnects, such as N[73:0], S[73:0], W[73:0], E[73:0],NH[27:0], SH[27:0], XH[36:0] andXH[72:37], excluding the local bus connections.
  • connectors 1531 A and 153 IB are the inter-board connectors for the local bus.
  • the local bus handles the signals between the PCI bus (via the PCI controller) and the FPGA bus (via the FPGA I/O controller (CTRLJFPGA) unit)).
  • the local bus also handles configuration and boundary scan test information between the PCI controller and the FPGA logic devices and the FPGA I/O controller (CTRLJ F PGA) unit.
  • the motherboard connector couples one board in a pair of boards to the PCI bus and power.
  • One set of connectors couples the FPGA interconnects via the component side of one board to the solder side of the other board.
  • Another set of connectors couples the local buses via the component side of one board to the solder side of the other board.
  • FIG.38(B) shows a six-board configuration.
  • the configuration is analogous to that of FIG. 38(A), in which every other board is directly connected to the motherboard, and interconnects and local buses of these boards are coupled together via inter-board connectors arranged solder-side to component-side.
  • FIG. 38(B) shows six boards 1526 (first board), 1525 (second board), 1532 (third board), 1533 (fourth board), 1534 (fifth board), and 1535 (sixth board). These six boards are coupled to the motherboard 1520 via the connectors on boards 1526 (first board), 1532 (third board), and 1534 (fifth board).
  • the other boards 1525 (second board), 1533 (fourth board), and 1535 (sixth board) are not directly coupled to the motherboard 1520; rather, they are indirectly coupled to the motherboard through their respective connections to their respective neighbor boards.
  • the various inter-board connectors allow communication among the PCI bus components, the FPGA logic devices, memory devices, and various Simulation system control circuits.
  • the first set of inter-board connectors 1990 correspond to connectors J5 to J16 in FIG. 42.
  • the second set of inter-board connectors 1991 correspond to connectors J17 to J28 in FIG.42.
  • the third set of inter-board connectors 1992 correspond to connectors J3 and J4 in FIG.42.
  • Motherboard connectors 1521 to 1524 are provided on the motherboard 1520 to couple the motherboard (and hence the PCI bus) to the six boards. As mentioned above, boards 1526 (first board), 1532 (third board), and 1534 (fifth board) are directly coupled to the connectors 1523, 1522, and 1521, respectively. The other boards 1525 (second board), 1533 (fourth board), and 1535 (sixth board) are not directly coupled to the motherboard 1520. Because only one PCI controller is needed for all six boards, only the first board 1526 contains a PCI controller. Also, the motherboard connector 1523 which is coupled to the first board 1526 provides access to/from the PCI bus. Connectors 1522 and 1521 are only coupled to power and ground. The center-to-center spacing between adjacent motherboard connectors is approximately 20.32 mm in one embodiment.
  • the J5 to J16 connectors are located on the component side
  • the J17 to J28 connectors are located on the solder side
  • the J3 to J4 local bus connectors are located on the component side
  • the J5 to J16 connectors are located on the solder side
  • the J17 to J28 connectors are located on the component side
  • the J3 to J4 local bus connectors are located on the solder side.
  • parts of the J17 to J28 connectors are 10- ohm R-pack terminations.
  • FIGS.40(A) and 40(B) show array connection across different boards. To facilitate the manufacturing process, a single layout design is used for all the boards. As explained above, boards connect to other boards through connectors without a backplane.
  • FIG.40(A) shows two exemplary boards 1611 (board2) and 1610 (boardl). The component side of board 1610 is facing the solder side of board 1611.
  • Board 1611 contains numerous FPGA logic devices, other components, and wire lines. Particular nodes of these logic devices and other components on board 1611 are represented by nodes A' (reference numeral 1612) and B' (reference numeral 1614). Node A' is coupled to connector pad 1616 via PCB trace 1620. Similarly, node B' is connected to connector pad 1617 via PCB trace 1623.
  • board 1610 also contains numerous FPGA logic devices, other components, and wire lines. Particular nodes of these logic devices and other components on board 1610 are represented by nodes A (reference numeral 1613) and B (reference numeral 1615). Node A is coupled to connector pad 1618 via PCB trace 1625. Similarly, node B is connected to connector pad 1619 via PCB trace 1622.
  • FIG. 40(A) the desired connections are between: (1) node A and node B' as indicated by imaginary path 1620, 1621, and 1622, and (2) node B and node A' as indicated by imaginary path 1623, 1624, and 1625. These connections are for paths such as the asymmetric interconnect 1600 between board 1551 and board 1552 in FIG. 39.
  • Other asymmetric interconnects include the NH to SH interconnects 1977, 1979, and 1981 on both sides of connectors 1589 and 1590.
  • A-A' and B-B' correspond to symmetrical interconnections like interconnect 1515 (N, S).
  • N and S interconnections use through hole connectors, whereas NH and SH asymmetric interconnections use SMD connectors.
  • Table D Refer to Table D.
  • board 1611 shows node A' on the component side coupled to component-side connector pad 1636 via PCB trace 1620.
  • the component-side connector pad 1636 is coupled to the solder-side connector pad 1639 via conductive path 1651.
  • Solder-side connector pad 1639 is coupled to the component- side connector pad 1642 on board 1610 via conductive path 1648.
  • component-side connector pad 1642 is coupled to node B via PCB trace 1622.
  • node A' on board 1611 can be coupled to node B on board 1610.
  • board 1611 shows node B' on the component side coupled to component-side connector pad 1638 via PCB trace 1623.
  • the component-side connector pad 1638 is coupled to the solder-side connector pad 1637 via conductive path 1650.
  • Solder-side connector pad 1637 is coupled to the component-side connector pad 1640 via conductive path 1645.
  • component-side connector pad 1640 is coupled to node A via PCB trace 1625.
  • node B' on board 1611 can be coupled to node A on board 1610. Because these boards share the same layout, conductive paths 1652 and 653 could be used in the same manner as conductive paths 1650 and 1651 for other boards placed adjacent to board 1610.
  • a unique inter-board connectivity scheme is provided using surface mount and through hole connectors without using switching components.
  • One embodiment of the present invention solves both the hold time and clock glitch problems.
  • standard logic devices e.g., latches, flip-flops
  • emulation logic devices e.g., emulation logic devices, or timing- insensitive glitch-free (TIGF) logic devices, in accordance with one embodiment of the present invention.
  • TIGF timing- insensitive glitch-free
  • a trigger signal that has been incorporated into the EVAL signal is used to update the values stored in these TIGF logic devices.
  • the trigger signal is provided to update the values stored or latched by the TIGF logic devices. Thereafter, a new evaluation period begins. This evaluation period- trigger period is cyclical, in one embodiment.
  • Hold time is defined as the minimum amount of time that the data input(s) of a logic element must be held stable after the control input (e.g., clock input) changes to latch, capture or store the value indicated by the data input(s); otherwise, the logic element will fail to work properly.
  • a shift register example will now be discussed to illustrate the hold time requirement.
  • FIG. 75(A) shows an exemplary shift register in which three D-type flip-flops are connected serially; that is, the output of flip-flop 2400 is coupled to the input of flip-flop 2401, whose output is in turn coupled to the input of flip-flop 2402.
  • the overall input signal S m - is coupled to the input of flip-flop 2400 and the overall output signal S out is generated from the output of flip- flop 2402. All three flip-flops receive a common clock signal at their respective clock inputs. This shift register design is based on the assumption that (1) the clock signal will reach all the flip-flops at the same time, and (2) after detecting the edge of the clock signal, the input of the flip-flop will not change for the duration of the hold time.
  • the hold time assumption is illustrated where the system does not violate hold time requirements.
  • the hold time varies from one logic element to the next but is always specified in the specification sheets.
  • the clock input changes from logic 0 to logic 1 at time to.
  • the clock input is provided to each flip-flop 2400-2402. From this clock edge at to, the input S; n must be stable for the duration of the hold time T H , which lasts from time t 0 to time ti.
  • the inputs to flip-flops 2401 (i.e., D 2 ) and 2402 i.e., D 3 ) must also be stable for the duration of the hold time from the trigger edge of the clock signal.
  • the clock signal will not reach all the logic elements at the same time; rather, the circuit is designed such that the clock signal will reach all the logic elements in almost the same time or substantially the same time.
  • the circuit must be designed such that the clock skew, or the timing difference between the clock signals reaching each flip-flop, is much smaller than the hold time requirement. Accordingly, all the logic elements will capture the appropriate input values.
  • hold time violation due to clock signals arriving at different times at the flip-flops 2400-2402 may result in some flip-flops capturing the old input values while another flip-flop capturing a new input value. As a result, the shift register will not operate properly.
  • a reconfigurable logic e.g., FPGA
  • the circuit can be designed so that the low skew network can distribute the clock signal to all the logic elements such that the logic elements will detect the clock edge at substantially the same time.
  • Primary clocks are generated from self-timed test-bench processes. Usually, the primary clock signals are generated in software and only a few (i.e., 1-10) primary clocks are found in a typical user circuit design.
  • clock signal is generated from internal logic instead of the primary input, hold time becomes more of an issue.
  • Derived or gated clocks are generated from a network of combinational logic and registers that are in turn driven by the primary clocks. Many (i.e., 1,000 or more) derived clocks are found in a typical user circuit design. Without extra precautions or additional controls, these clock signals may reach each logic element at different times and the clock skew may be longer than the hold time. This may result in the failure of a circuit design, such as the shift register circuit illustrated in FIGS. 75(A) and 75(B).
  • the first FPGA chip 2411 contains the internally derived clock logic 2410 which will feed its clock signal CLK to some components of FPGA chips 2412-2416.
  • the internally generated clock signal CLK will be provided to flip-flops 2400-2402 of the shift register circuit.
  • 2412 contains flip-flop 2400
  • chip 2415 contains flip-flop 2401
  • chip 2416 contains flip-flop 2402. Two other chips
  • 2413 and 2414 are provided to illustrate the hold time violation concept.
  • the clock logic 2410 in chip 2411 receives a primary clock input (or possibly another derived clock input) to generate an internal clock signal CLK.
  • This internal clock signal CLK will travel to chip 2412 and is labeled CLKl .
  • the internal clock signal CLK from clock logic 2410 will also travel to chip 2415 as CLK2 via chips 2413 and 2414.
  • CLKl is input to flip-flop 2400 and CLK2 is input to flip-flop 2401.
  • Both CLKl and CLK2 experience wire trace delays such that the edges of CLKl and CLK2 will be delayed from the edge of the internal clock signal CLK. Furthermore, CLK2 will experience additional delays because it traveled through two other chips 2413 and 2414.
  • the internal clock signal CLK is generated and triggered at time t 2 . Because of wire trace delays, CLKl does not arrive at flip-flop 2400 in chip 2412 until time t 3 , which is a delay of time Tl. As shown in the table above, the output at Qi (or input D 2 ) is at logic 0 before the arrival of the clock edge of CLKl. After the edge of CLKl is sensed at flip-flop 2400, the input at Dj must remain stable for the requisite hold time H2 (i.e., until time t 4 ). At this point, flip-flop 2400 shifts in or stores the input logic 1 so that the output at Q ! (or D 2 ) is at logic 1.
  • FIG. 77(A) shows an exemplary logic circuit where some logic elements generate a clock signal for another set of logic elements; that is, D-type flip-flop 2420, D-type flip-flop 2421, and exclusive-or (XOR) gate 2422 generate a clock signal (CLK3) for D-type flip-flop 2423.
  • Flip-flop 2420 receives its data input at Di on line 2425 and outputs data at Qi on line 2427. It receives its clock input (CLKl) from a clock logic 2424.
  • CLK refers to the originally generated clock signal from the clock logic 2424 and CLKl refers to the same signal that is delayed in time when it reaches flip-flop 2420.
  • Flip-flop 2421 receives its data input at D on line 2426 and outputs data at Q 2 on line 2428. It receives its clock input (CLK2) from a clock logic 2424.
  • CLK refers to the originally generated clock signal from the clock logic 2424
  • CLK2 refers to the same signal that is delayed in time when it reaches flip-flop 2421.
  • flip-flops 2420 and 2421 on lines 2427 and 2428, respectively are inputs to XOR gate 2422.
  • XOR gate 2422 outputs data labeled as CLK3 to the clock input of flip-flop 2423.
  • Flip-flop 2423 also inputs data at D 3 on line 2429 and outputs data at Q 3 .
  • the clock glitch problem that may arise for this circuit will now be discussed with reference to the timing diagram illustrated in FIG. 77(B).
  • the CLK signal is triggered at time t 0 .
  • this clock signal i.e., CLKl
  • CLK2 does not reach flip-flop 2421 until time t 2 .
  • Timing adjustment requires the insertion of sufficient delay elements (such as buffers) in certain signal paths to prolong the hold time of the logic elements. For example, adding sufficient delay on the inputs D 2 and D 3 in the shift register circuit above may avoid hold time violation.
  • FIG. 78 the same shift register circuit is shown with delay elements 2430 and 2431 added to the inputs D 2 and D 3 , respectively.
  • the delay element 2430 can be designed such that time t 4 occurs after time t 5 so that T2 ⁇ T1+H2 (FIG. 76(B)), and hence, no hold time violation will occur.
  • a potential problem with the timing adjustment solution is that it relies on the specification sheet of the FPGA chips too heavily.
  • reconfigurable logic chips like FPGA chips, implement logic elements with look-up tables.
  • the delay of look-up tables in the chips is provided in the specification sheets and designers using the timing adjustment method of avoiding hold time violations rely on this specified time delay.
  • this delay is just an estimate and varies from chip to chip.
  • Another potential problem with the timing adjustment method is that designers must also compensate for the wiring delays present throughout the circuit design. Although this is not an impossible task, the estimation of wiring delay is time-consuming and prone to errors. Moreover, the timing adjustment method does not solve clock glitch problems.
  • Timing resynthesis a technique introduced by IKOS's VirtualWires technology.
  • the timing resynthesis concept involves transforming a user's circuit design into a functionally equivalent design while strictly controlling the timing of clock and pin-out signals via finite state machines and registers.
  • Timing resynthesis retimes a user's circuit design by introducing a single high speed clock. It also converts latches, gated clocks, and multiple synchronous and asynchronous clocks into a flip-flop based single-clock synchronous design.
  • timing resynthesis uses registers at the input and output pin-outs of each chip to control the precise inter-chip signal movement so that no inter-chip hold-time violation will occur.
  • Timing resynthesis also uses a finite state machine in each chip to schedule inputs from other chips, schedule outputs to other chips, and schedule updates of internal flip-flops based on the reference clock.
  • FIG. 79 shows one example of the timing resynthesis circuit.
  • the basic three flip-flop shift register design has been transformed into a functionally equivalent circuit.
  • Chip 2430 includes the original internal clock generating logic 2435 coupled to a register 2443 via line 2448.
  • the clock logic 2435 generates the CLK signal.
  • a first finite state machine 2438 is also coupled to the register 2443 via line 2449. Both the register 2443 and the first finite state machine 2438 are controlled by a design-independent global reference clock.
  • the CLK signal is also delivered across chips 2432 and 2433 before it arrives at chip 2434.
  • a second finite state machine 2440 controls a register 2445 via line 2462.
  • the CLK signal travels to register 2445 via line 2461 from register 2443.
  • Register 2445 outputs the CLK signal to the next chip 2433 via line 2463.
  • Chip 2433 includes a third finite state machine 2441 which controls a register 2446 via line 2464.
  • the register 2446 outputs the CLK signal to chip 2434.
  • Chip 2431 includes the original flip-flop 2436.
  • a register 2444 receives the input Sj n and outputs the input S in to the Di input of flip-flop 2436 via line 2452.
  • the Qi output of the flip-flop 2436 is coupled to register 2466 via line 2454.
  • a fourth finite state machine 2439 controls the register 2444 via line 2451, register 2466 via line 2455, and the flip-flop 2436 via the latch enable line 2453.
  • the fourth finite state machine 2439 also receives the original clock signal CLK from chip 2430 via line 2450.
  • Chip 2434 includes the original flip-flop 2437, which receives the signal from register 2466 in the chip 2431 at its D 2 input via line 2456.
  • the Q 2 output of the flip-flop 2437 is coupled to register 2447 via line 2457.
  • a fifth finite state machine 2439 controls the register 2447 via line 2459, and the flip-flop 2437 via the latch enable line 2458.
  • the fifth finite state machine 2442 also receives the original clock signal CLK from chip 2430 via chips 2432 and 2433.
  • the finite state machines 2438-2442, registers 2443-2447 and 2466, and the single global reference clock are used to control signal flow across multiple chips and update internal flip-flops.
  • the distribution of the CLK signal to other chips is scheduled by the first finite state machine 2438 via the register 2443.
  • the fourth finite state machine 2439 schedules the delivery of the input Sj owned to the flip-flop 2436 via register 2444 as well as the Qi output via register 2466.
  • the latching function of the flip-flop 2436 is also controlled by a latch enable signal from the fourth finite state machine 2439.
  • timing resynthesis technique requires the transformation of the user's circuit design into a much larger functionally equivalent circuit including the addition of finite state machines and registers. Typically, the additional logic necessary to implement this technique takes up to 20% of the useful logic in each chip. Furthermore, this technique is not immune to clock glitch problems. To avoid clock glitches, designers using the timing resynthesis technique must take additional precautionary steps. One conservative design approach is to design the circuit so that the inputs to a logic device utilizing gated clocks are not changed at the same time. An aggressive approach uses the gate delays to filter the glitches so that they do not impact the rest of the circuit. However, as stated above, timing resynthesis requires some additional non-trivial measures to avoid clock glitches.
  • latches shown in FIG. 18(A) are emulated with a timing insensitive glitch-free (TIGF) latch in accordance with one embodiment of the present invention.
  • design flip-flops shown in FIG. 18(B) are emulated with a TIGF flip-flop in accordance with one embodiment of the present invention.
  • TIGF logic devices whether in the form of a latch or flip-flop, can also be called emulation logic devices.
  • the updates of the TIGF latches and flip-flops are controlled with a global trigger signal.
  • not all of the logic devices found in the user"design circuit are replaced with the TIGF logic devices.
  • a user design circuit includes those portions that are enabled or clocked by the primary clocks and other portions that are controlled by gated or derived clocks. Because hold time violations and clock glitches are issues for the latter case where logic devices are controlled by gated or derived clocks, only these particular logic devices that are controlled by gated or derived clocks are replaced with the TIGF logic devices in accordance with the present invention. In other embodiments, all logic devices found in the user design circuit are replaced with the TIGF logic devices.
  • the global trigger signal is used to allow the TIGF latches and flip-flops to keep its state (i.e., keep the old input value) during the evaluation period and update its state (i.e., store the new input value) during a short trigger period.
  • the global trigger signal shown in FIG. 82, is separate from and derived from the EVAL signal discussed above.
  • the global trigger signal has a long evaluation period followed by a short trigger period. The global trigger signal tracks the EVAL signal during the evaluation period and at the conclusion of the EVAL cycle, a short trigger signal is generated to update the TIGF latches and flip-flops.
  • the EVAL signal is the global trigger signal, where the EVAL signal is at one logic state (e.g., logic 0) during the evaluation period and at another logic state (e.g., logic 1) during non-evaluation or TIGF latch/flip-flop update periods.
  • the EVAL signal is at one logic state (e.g., logic 0) during the evaluation period and at another logic state (e.g., logic 1) during non-evaluation or TIGF latch/flip-flop update periods.
  • the evaluation period is used to propagate all the primary inputs and flip-flop/latch device changes into the entire user design, one simulation cycle at a time. During the propagation, the RCC system waits until all the signals in the system achieve steady-state.
  • the evaluation period is calculated after the user design has been mapped and placed into the appropriate reconfigurable logic devices (e.g., FPGA chips) of the RCC array. Accordingly, the evaluation period is design- specific; that is, the evaluation period for one user design may be different from the evaluation period for another user design. This evaluation period must be long enough to assure that all the signals in the system are propagated through the entire system and reach steady-state before the next short trigger period.
  • the short trigger period occurs adjacent in time to the evaluation period, as shown in FIG. 82.
  • the short trigger period occurs after the evaluation period.
  • the input, signals are propagated throughout the hardware model-configured portion of the user design circuit during the evaluation period.
  • the short trigger period marked by a change in the logic state of the EVAL signal in accordance with one embodiment of the present invention, controls all the TIGF latches and flip-flops in the user design so that they can be updated with the new values that have been propagated from the evaluation period after steady-state has been achieved.
  • This short trigger period is globally distributed with a low skew network and can be as short (i.e., duration from to to ti, as well as duration t 2 to t 3 , as shown in FIG. 82) as the reconfigurable logic devices will allow for proper operation.
  • the new primary inputs are sampled at every input stage of the TIGF latches and flip-flops and the old stored values at the same TIGF latches and flip-flops are exported out to the next stage in the RCC hardware model of the user design.
  • the portion of the global trigger signal that occurs during the short trigger period will be referred to as the TIGF trigger, TIGF trigger signal, trigger signal, or simply the trigger.
  • FIG. 80(A) shows the latch 2470 originally shown in FIG. 18(A). This latch operates as follows:
  • this latch is level-sensitive and asynchronous, so long as the clock input is enabled and the latch enable input is enabled, the output Q tracks the input D.
  • FIG. 80(B) shows the TIGF latch in accordance with one embodiment of the present invention.
  • the TIGF latch has a D input, an enable input, a set (S), a reset (R), and an output Q. Additionally, it has a trigger input.
  • the TIGF latch includes a D flip-flop 2471, a multiplexer 2472, an OR gate 2473, an AND gate 2474, and various interconnections.
  • D flip-flop 2471 receives its input from the output of AND gate 2474 via line 2476.
  • the D flip-flop is also triggered at its clock input by a trigger signal on line 2477, which is globally distributed by the RCC system in accordance with a strict schedule dependent on the evaluation cycle.
  • the output of D flip-flop 2471 is coupled to one input of multiplexer 2472 via line 2478.
  • the other input of multiplexer 2472 is coupled to the TIGF latch D input on line 2475.
  • the multiplexer is controlled by an enable signal on line 2484.
  • the output of the multiplexer 2472 is coupled to one input of OR gate 2473 via line 2479.
  • the other input of OR gate 2473 is coupled to the set (S) input on line 2480.
  • the output of the OR gate 2473 is coupled to one input of AND gate 2474 via line 2481.
  • the other input of AND gate 2474 is coupled to the reset (R) signal on line 2482.
  • the output of AND gate 2474 is fed back to the input of the D flip-flop 2471 via line 2476, as mentioned above.
  • the D flip-flop 2471 holds the current state (i.e., old value) of the TIGF latch.
  • Line 2476 at the input of D flip-flop 2471 presents the new input value that has yet to be latched into the TIGF latch.
  • Line 2476 presents the new value because the main input (D input) of the TIGF latch on line 2475 ultimately makes its way from the input of the multiplexer 2472 (with the proper enable signal on line 2484, which will ultimately be presented) through the OR gate 2473, and finally through the AND gate 2474 onto line 2483, which feeds back the new input signal of the TIGF latch to the D flip-flop 2471 on line 2476.
  • a trigger signal on line 2477 updates the TIGF latch, by clocking the new input value on line 2476 into the D flip-flop 2471.
  • the output on line 2478 of the D flip-flop 2471 indicates the current state (i.e., old value) of the TIGF latch, while the input on line 2476 indicates the new input value that has yet to be latched by the TIGF latch.
  • the multiplexer 2472 receives the current state from D flip-flop 2471 as well as the new input value on line 2475.
  • the enable line 2484 functions as the selector signal for the multiplexer 2472. Because the TIGF latch will not update (i.e., store new input value) until the trigger signal is provided on line 2477, the D input of the TIGF latch on line 2475 and the enable input on line 2484 can arrive at the TIGF latch in any order. If this TIGF latch (and other TIGF latches in the hardware model of the user design) encounters a situation that would normally cause hold time violation in a circuit that used a conventional latch, such as in the discussion above with respect to FIGS. 76(A) and 76(B) where one clock signal arrived much later than another clock signal, this TIGF latch will function properly by keeping the proper old value until the trigger signal is provided on line 2477.
  • the trigger signal is distributed through the low-skew global clock network. ***
  • This TIGF latch also solves the clock glitch problem. Note that the clock signal is replaced by the enable signal in the TIGF latch.
  • the enable signal on line 2484 can glitch often during the evaluation period but the TIGF latch will continue to hold the current state without fail.
  • the only mechanism by which the TIGF latch can be updated is through the trigger signal, which is provided after the evaluation period, in one embodiment, when the signals have attained steady-state.
  • FIG. 81(A) shows a flip-flop 2490 originally shown in FIG. 18(B). This flip-flop operates as follows:
  • FIG. 81(B) shows the TIGF D-type flip-flop in accordance with one embodiment of the present invention.
  • the TIGF flip-flop has a D input, a clock input, a set (S), a reset (R), and an output Q ; Additionally, it has a trigger input.
  • the TIGF flip-flop includes three D flip-flops 2491, 2492, and 2496, a multiplexer 2493, an OR gate 2494, two AND gates 2495 and 2497, and various interconnections.
  • Flip-flop 2491 receives the TIGF D input on line 2498, the trigger input on line 2499, and provides a Q output on line 2500.
  • This output line 2500 also serves as one of the inputs to multiplexer 2493.
  • the other input to the multiplexer 2493 comes from the Q output of flip-flop 2492 via line 2503.
  • the output of multiplexer 2493 is coupled to one of the inputs of OR gate 2494 via line 2505.
  • the other input of OR gate 2492 is the set (S) signal on line 2506.
  • the output of OR gate 2494 is coupled to one of the inputs of AND gate 2495 via line 2507.
  • the other input of AND gate 2495 is the reset (R) signal on line 2508.
  • the output of AND gate 2495 (which is also the overall TIGF output Q) is coupled to the input of flip-flop 2492 via line 2501.
  • Flip-flop 2492 also has a trigger input on line 2502.
  • AND gate 2497 receives one of its inputs from the CLK signal on line 2510 and the other input from the output of flip- flop 2496 via line 2512.
  • Flip-flop 2496 also receives its input from the CLK signal on line 2511 and its trigger input on line 2513.
  • the TIGF flip-flop receives the trigger signal at three different points - the D flip-flop 2491 via line 2499, the D flip-flop 2492 via line 2502, and the D flip-flop 2496 via line 2513.
  • the TIGF flip-flop stores the input value only when an edge of the clock signal has been detected.
  • the required edge is the positive edge of the clock signal.
  • an edge detector 2515 To detect this positive edge of the clock signal, an edge detector 2515 has been provided.
  • the edge detector 2515 includes a D flip-flop 2496 and an AND gate 2497.
  • the edge detector 2515 is also updated via the trigger signal on line 2513 of the D flip-flop 2496.
  • the D flip-flop 2491 holds the new input value of the TIGF flip-flop and resists any changes to the D input on line 2498 until the trigger signal is provided on line 2499. Thus, before each evaluation period of the TIGF flip-flop, the new value is stored in the D flip-flop 2491. Accordingly, the TIGF flip-flop avoids hold time violations by pre- storing the new value until the TIGF flip-flop is updated by the trigger signal.
  • D flip-flop 2492 holds the current value (or old value) of the TIGF flip-flop until the trigger signal is provided on line 2502. This value is the state of the emulated TIGF flip-flop after it has been updated and before the next evaluation period.
  • the input to the D flip-flop 2492 on line 2501 holds the new value (which is the same value on line 2500, for a significant duration of the evaluation period).
  • the multiplexer 2493 receives the new input value on line 2500 and the old value that is currently stored in the TIGF flip-flop on line 2503. Based on the selector signal on line 2504, the multiplexer outputs either the new value (line 2500) or the old value (line 2503) as the output of the emulated TIGF flip-flop. This output changes with any clock glitches before all of the propagated signals in the user design's hardware model approach steady-state. Thus, the input on line 2501 will present the new value that is stored in flip-flop 2491 by the end of the evaluation period. When the trigger signal is received by the TIGF flip-flop, flip-flop 2492 now stores the new value that was present in line 2501 and the flip-flop 2491 stores the next new value on line 2498. Thus, the TIGF flip-flop in accordance with one embodiment of the present invention is not negatively affected by clock glitches.
  • this TIGF flip-flop also provides some immunity against clock glitches.
  • clock glitches will not impact any circuit utilizing this TIGF flip-flop. Referring to FIGS. 77(A) and 77(B) for a moment, a clock glitch negatively impacted the circuit of FIG. 77(A) because for the time between time t ! and t 2 , the flip-flop 2423 clocked in a new value when it should not have clocked in a new value.
  • TIGF flip-flop updates only with the trigger signal and this trigger signal will not be presented to the TIGF flip-flop until after the evaluation period when the signals propagating through the circuit have achieved steady-state.
  • TIGF flip-flop is a D-type flip-flop
  • other flip-flops e.g., T, JK, SR
  • edge-triggered flip-flops can be derived from the D flip-flop by adding some AND/OR logic before the D input.
  • a Simulation server in accordance with another embodiment of the present invention is provided to allow multiple users to access the same reconfigurable hardware unit to effectively simulate and accelerate the same or different user designs in a time-shared manner.
  • a high speed simulation scheduler and state swapping mechanisms are employed to feed the Simulation server with active simulation processes which results in a high throughput.
  • the server provides the multiple users or processes to access the reconfigurable hardware unit for acceleration and hardware state swapping purposes. Once the acceleration has been accomplished or the hardware state has been accessed, each user or process can then simulate in software only, thus releasing control of the reconfigurable hardware unit to other users or processes.
  • jobs In the Simulation server portion of this specification, terms such as "job” and “process” are used. In this specification, the terms “job” and “process” are generally used interchangeably. In the past, batch systems executed “jobs” and time-shared systems stored and executed “processes” or programs. In today's systems, these jobs and processes are similar.
  • job is not limited to batch-type systems and "process” is not limited to time-shared systems; rather, at one extreme, a “job” is equivalent to a “process” if the "process” can be executed within a time slice or without interruption by any other time-shared intervenor, and at the other extreme, a “job” is a subset of a “process” if the "job” requires multiple time slices to complete.
  • a "process” requires multiple time slices to execute to completion due to the presence of other equal priority users/processes, the "process” is divided up into “jobs.” Moreover, if the "process” does not require multiple time slices to execute to completion because it is the sole high priority user or the process is short enough to complete within a time slice, the "process” is equivalent to a "job.” Thus, a user can interact with one or more “processes” or programs that have been loaded and executed in the Simulation system, and each "process” may require one or more "jobs” to complete in a time-shared system.
  • multiple users via remote terminals can utilize the same multiprocessor workstation in a non-network environment to access the same reconfigurable hardware unit to review/debug the same or different user circuit design.
  • remote terminals are connected to a main computing system for access to its processing functions.
  • This non-network configuration allows multiple users to share access to the same user design for parallel debugging purposes. The access is accomplished via a time-shared process in which a scheduler determines access priorities for the multiple users, swaps jobs, and selectively locks hardware unit access among the scheduled users.
  • multiple users may access the same reconfigurable hardware unit via the server for his/her own separate and different user design for debugging purposes.
  • the multiple users or processes are sharing the multiple microprocessors in the workstation with the operating system.
  • multiple users or processes in separate microprocessor-based workstations can access the same reconfigurable hardware unit to review/debug the same or different user circuit design across a network.
  • the access is accomplished via a time-shared process in which a scheduler determines access priorities for the multiple users, swaps jobs, and selectively locks hardware unit access among the scheduled users.
  • the scheduler listens for network requests through UNIX socket system calls. The operating system uses sockets to send commands to the scheduler.
  • the Simulation scheduler uses a preemptive multiple priority round robin algorithm. In other words, higher priority users or processes are served first until the user or process completes the job and ends the session.
  • a preemptive round robin algorithm is used in which each user or process is assigned an equal time slice to execute its operations until completed.
  • the time slice is short enough such that multiple users or process will not have to wait a long time before being served.
  • the time slice is also long enough such that sufficient operations are executed before the Simulation server's scheduler interrupts one user or process to swap in and execute the new user's job.
  • the default time slice is 5 seconds and is user settable.
  • the scheduler makes specific calls to the operating system's built-in scheduler.
  • FIG. 45 shows a non-network environment with a multiprocessor workstation in accordance with one embodiment of the present invention.
  • FIG. 45 is a variation of FIG. 1, and accordingly, like reference numerals will be used for like components/units.
  • Workstation 1100 includes local bus 1105, a host/PCI bridge 1106, memory bus 1107, and main memory 1108.
  • a cache memory subsystem (not shown) may also be provided.
  • Other user interface units e.g., monitor, keyboard
  • Workstation 1100 also includes multiple microprocessors 1101, 1102, 1103, and 1104 coupled to the local bus 1105 via a scheduler 1117 and connections/path 1118.
  • an operating system 1121 provides the user-hardware interface foundation for the entire computing environment for managing files and allocating resources for the various users, processes, and devices in the computing environment.
  • the operating system 1121 along with a bus 1122 are shown.
  • References to operating systems can be made in Abraham Silberschatz and James L. Peterson, OPERATING SYSTEM CONCEPTS (1988) and William Stallings, MODERN OPERATING SYSTEMS (1996), which are incorporated herein by reference.
  • the workstation 1100 is a Sun Microsystems Enterprise 450 system which employs UltraSPARC II processors. Instead of the memory access via the local bus, the Sun 450 system allows the multiprocessors to access the memory via dedicated buses to the memory through a crossbar switch. Thus, multiple processes can be running with multiple microprocessors executing their respective instructions and accessing the memory without going through the local bus.
  • the Sun 450 system along with the Sun UltraSPARC multiprocessor specifications are incorporated herein by reference.
  • the Sun Ultra 60 system is another example of a microprocessor system although it allows only two processors.
  • the scheduler 1117 provides the time-shared access to the reconfigurable hardware unit 20 via the device driver 1119 and connections/path 1120.
  • Scheduler 1117 is implemented mostly in software to interact with the operating system of the host computing system and partially in hardware to interact with the Simulation server by supporting the simulation job interruption and swapping in/out the simulation sessions.
  • the scheduler 1117 and device driver 1119 will be discussed in more detail below.
  • Each microprocessor 1101-1104 is capable of processing independently of the other microprocessors in the workstation 1101.
  • the workstation 1100 is operating under a UNIX-based operating system, although in other embodiments, the workstation 1100 can operate under a Windows-based or Macintosh-based operating system.
  • the user is equipped with X- Windows for the user interface to manage programs, tasks, and files as necessary.
  • Maurice J. Bach THE DESIGN OF THE UNIX OPERATING SYSTEM (1986).
  • multiple users can access workstation 1100 via remote terminals.
  • each user may be using a particular CPU to run its processes.
  • each user uses different CPUs depending on the resource limitations.
  • the operating system 1121 determines such accesses and indeed, the operating system itself may jump from one CPU to another to accomplish its tasks.
  • the scheduler listens for network requests through socket system calls makes system calls to the operating system 1121, which in turn handles preemption by initiating the generation of interrupt signals by the device driver 1119 to the reconfigurable hardware unit 20.
  • interrupt signal generation is one of many steps in the scheduling algorithm which includes stopping the current job, saving state information for the currently interrupted job, swappingjobs, and executing the new job.
  • the server scheduling algorithm will be discussed below.
  • the UNIX operating system in one embodiment, can operate on a time-sharing mode.
  • the UNIX kernel allocates the CPU to a process for a period of time (e.g., time slice) and at the end of the time slice, preempts the process and schedules another one for the next time slice.
  • the preempted process from the previous time slice is rescheduled for execution at a later time slice.
  • the kernel has three layers that function in the context of a client-server model. These three layers include the socket layer, the protocol layer, and the device layer.
  • the top layer, the socket layer provides the interface between the system calls and the lower layers (protocol layer and device layer).
  • the socket has end points that couple client processes with server processes.
  • the socket end points can be on different machines.
  • the middle layer, the protocol layer provides the protocol modules for communication, such as TCP and IP.
  • the bottom layer, the device layer contains the device drivers that control the network devices.
  • One example of a device driver is an Ethernet driver over an Ethernet-based network.
  • Processes communicate using the client-server model where the server process listens to a socket at one end point and a client process to the server process over another socket at the other end point of the two-way communication path.
  • the kernel maintains internal connections among the three layers of each client and server and routes data from client to the server as needed.
  • the socket contains several system calls including a socket system call which establishes the end points of a communication path. Many processes use the socket descriptor sd in many system calls.
  • the bind system call associates a name with a socket descriptor.
  • Some other exemplary system calls include the connect system call requests that the kernel make a connection to a socket, the close system call closes sockets, the shutdown system call closes a socket connection, and the send and recv system calls transmit data over a connected socket.
  • FIG. 46 shows another embodiment in accordance with the present invention in which multiple workstations share a single Simulation system on a time-shared basis across a network.
  • the multiple workstations are coupled to the Simulation system via a scheduler 1117.
  • a single CPU 11 is coupled to the local bus 12 in station 1110. Multiple CPUs may also be provided in this system.
  • an operating system 1118 is also provided and nearly all processes and applications reside on top of the operating system. For conceptual purposes the operating system 1121 along with a bus 1122 are shown.
  • workstation 1110 includes those components/units found in FIG. 1 along with scheduler 1117 and scheduler bus 1118 coupled to the local bus 12 via the operating system 1121.
  • Scheduler 1117 controls the time-shared access for the user stations 1111, 1112, and 1113 by making socket calls to the operating system 1121.
  • Scheduler 1117 is implemented mostly in software and partially in hardware.
  • Each user accesses the system via remote stations 1111, 1112, or 1113.
  • Remote user stations 1111, 1112, and 1113 are coupled to the scheduler 1117 via network connections 1114, 1115, and 1116, respectively.
  • device driver 1119 is coupled between the PCI bus 50 and the reconfigurable hardware unit 20.
  • Connection or electrically conductive path 1120 are provided between the device driver 1119 and the reconfigurable hardware unit 20.
  • the scheduler 1117 interfaces with the device driver 1119 via the operating system 1121 to communicate and control the reconfigurable hardware unit 20 for hardware acceleration and simulation after hardware state restoration purposes.
  • the Simulation workstation 1100 is a Sun Microsystems Enterprise 450 system which employs UltraSPARC II multiprocessors. Instead of the memory access via the local bus, the Sun 450 system allows the multiprocessors to access the memory via dedicated buses to the memory through a crossbar switch instead of tying up the local bus.
  • FIG.47 shows a high level structure of the Simulation server in accordance with the network embodiment of the present invention.
  • the operating system is not explicitly shown but, as known to those skilled in the art, it is always present for file management and resource allocation purposes to serve the various users, processes, and devices in the Simulation computing environment.
  • Simulation server 1130 includes the scheduler 1137, one or more device drivers 1138, and the reconfigurable hardware unit 1139.
  • the Simulation server comprises the scheduler 1117, device driver 1119, and the reconfigurable hardware unit 20.
  • the Simulation server 1130 is coupled to three workstations (or users) 1131, 1132, and 1133 via network connections/paths 1134, 1135, and 1136, respectively. As stated above, more than three or less than three workstations may be coupled to the Simulation server 1130.
  • the scheduler in the Simulation server is based on a preemptive round robin algorithm.
  • the round robin scheme allows several users or processes to execute sequentially to completion with a cyclic executive.
  • each simulation job (which is associated with a workstation in a network environment or a user/process in a multiprocessing non-network environment) is assigned a priority level and a fixed time slice in which to execute.
  • the higher priority jobs execute first to completion.
  • the user with the highest priority is served first until this user's job(s) is/are completed and the user with the lowest priority is served last.
  • no time slice is used because each user has a different priority and the scheduler merely serves users according to priority. This scenario is analogous to having only one user accessing the Simulation system until completion.
  • the different users have equal priority.
  • the time slice concept with a first-in first- out (FIFO) queue are employed.
  • each job executes until it completes or the fixed time slice expires, whichever comes first. If the job does not execute to completion during its time slice, the simulation image associated with whatever tasks it has completed must be saved for later restoration and execution. This job is then placed at the end of the queue. The saved simulation image, if any, for the next job is then restored and executed in the next time slice.
  • a higher priority job can preempt a lower priority job.
  • jobs of equal priority run in round robin fashion until they execute through the time slices to completion.
  • jobs of lower priority run in round robin fashion. If a job of higher priority is inserted in the queue while a lower priority job is running, the higher priority job will preempt the lower priority job until the higher priority job executes to completion. Thus, jobs of higher priority run to completion before jobs of lower priority begin execution. If the lower priority job has already begun execution, the lower priority job will not be further executed to completion until the higher priority job executes to completion.
  • the UNIX operating system provides the basic and foundational preemptive round robin scheduling algorithm.
  • the Simulation server's scheduling algorithm in accordance with one embodiment of the present invention works in conjunction with the operating system's scheduling algorithm.
  • the preemptive nature of the scheduling algorithm provides the operating system to preempt user-defined schedules.
  • the Simulation scheduler uses a preemptive multiple priority round robin algorithm on top of the operating system's own scheduling algorithm.
  • the relationship between the multiple users and the Simulation server in accordance with one embodiment of the present invention follows a client-server model, where the multiple users are clients and the Simulation server is the server. Communication between the user clients and the server occurs via socket calls.
  • the client includes client program 1109, a socket system call component 1123, UNIX kernel 1124, and a TCP/IP protocol component 1125.
  • the server includes a TCP/IP protocol component 1126, a UNIX kernel 1127, socket system call component 1128, and the Simulation server 1129. Multiple clients may request simulation jobs to be simulated in the server through UNIX socket calls from the client application program.
  • a typical sequence of events includes multiple clients sending requests to the server via the UNIX socket protocol. For each request, the server acknowledges the requests as to whether the command was successfully executed. For the request of server queue status, however, the server replies with the current queue state so that it can be properly displayed to the user. Table F below lists the relevant socket commands from the client:
  • each command encoded in integers may be followed with additional parameters such as ⁇ design> which represents the design name.
  • Response from the Simulation server will be "0” if the command is executed successfully or a "1” if the command failed.
  • command "5" which requests queue status one embodiment of the command's return response is ASCII text terminated by a " ⁇ 0" character for display onto the user's screen.
  • FIG.48 shows one embodiment of the architecture of the Simulation server in accordance with the present invention.
  • multiple users or multiple processes may be served by the single Simulation server for simulation and hardware acceleration of the users' designs in a time-shared manner.
  • user/process 1147, 1148, and 1149 are coupled to the Simulation server 1140 via inter-process communication paths 1150, 1151, and 1152, respectively.
  • the inter-process communication paths 1150, 1151, and 1152 may reside in the same workstation for multiprocessor configuration and operation, or in the network for multiple workstations.
  • Each simulation session contains software simulation states along with hardware states for communication with the reconfigurable hardware unit.
  • Inter-process communication among the software sessions is performed using UNIX socket or system calls which provide the capability to have the simulation session reside on the same workstation where the Simulator plug-in card is installed or on a separate workstation connected via a TCP/IP network. Communication with the Simulation server will be initiated automatically.
  • Simulation server 1140 includes the server monitor 1141, a simulation job queue table 1142, a priority sorter 1143, a job swapper 1144, device driver(s) 1145, and the reconfigurable hardware unit 1146.
  • the simulation job queue table 1142, priority sorter 1143, and job swapper 1144 make up the scheduler 1137 shown in FIG. 47.
  • the server monitor 1141 provides user interface functions for the administrator of the system. The user can monitor the status of the Simulation server state by commanding the system to display simulation jobs in the queue, scheduling priority, usage history, and simulation job swapping efficiency. Other utility functions include editing job priority, deleting simulation jobs, and resetting the simulation server state.
  • the simulation job queue table 1142 keeps a list of all outstanding simulation requests in the queue which was inserted by the scheduler.
  • the table entries include job number, software simulation process number, software simulation image, hardware simulation image file, design configuration file, priority number, hardware size, software size, cumulative time of the simulation run, and owner identification.
  • the job queue is implemented using a first-in first-out (FIFO) queue. Thus, when a new job is requested, it is placed at the end of the queue.
  • FIFO first-in first-out
  • the priority sorter 1143 decides which simulation job in the queue to execute.
  • the simulation job priority scheme is user definable (i.e., controllable and definable by the system administrator) to control which simulation process has priority for current execution.
  • the priority levels are fixed based on the urgency of specific processes or importance of specific users.
  • the priority levels are dynamic and can change during the course of the simulation.
  • priority is based on the user ID. Typically, one user will have a high priority and all other users will have lower but equal priority.
  • Priority levels are settable by the system administrator.
  • Simulator server obtains all user information from the UNIX facility, typically found in the UNIX user file called "/etc/passwd". Adding new users is consistent with the process of adding new users within the UNIX system. After all users are defined, the Simulator server monitor can be used to adjust priority levels for the users.
  • the job swapper 1144 temporarily replaces one simulation job associated with one process or one workstation for another simulation job associated with another process or workstation based on the priority determination programmed for the scheduler. If multiple users are simulating the same design, the job swapper swaps in only the stored simulation state for the simulation session. However, if multiple users are simulating multiple designs, the job swapper loads in the design for hardware configuration before swapping in the simulation state. In one embodiment, the job swapping mechanism enhances the performance of the time-sharing embodiment of the present invention because the job swapping need only be done for reconfigurable hardware unit access. So, if one user needs software simulation for some time period, the server swaps in another job for another user so that this other user can access the reconfigurable hardware unit for hardware acceleration. The frequency of the job swapping can be user adjustable and programmable. The device driver also communicates with the reconfigurable hardware unit to swap jobs.
  • FIG. 49 shows a flow diagram of the Simulation server during its operation.
  • the system is idle.
  • the Simulation server is not necessarily inactive or that a simulation task is not running. Indeed, idleness may mean one of several things: (1) no simulation is running; (2) only one user/workstation is active in a single processor environment so that time-sharing is not required; or (3) only one user/workstation in a multiprocessing environment is active but only one process is running.
  • conditions 2 and 3 above indicate that the Simulation server has only one job to process so that queuing jobs, determining priorities, and swappingjobs are not necessary and essentially, the Simulation server is idle because it receives no requests (event 1161) from other workstations or processes.
  • the Simulation server queues the incoming simulation job or jobs at step 1162.
  • the scheduler keeps a simulation job queue table to insert all outstanding simulation requests onto its queue and list all outstanding simulation requests.
  • the scheduler in the server queues all the incoming simulation requests and automatically processes the tasks without human intervention.
  • the Simulation server sorts the queued jobs to determine priority at step 1163. This step is particularly important for multiple jobs where the server has to prioritize among them to provide access to the reconflgurable hardware unit.
  • the priority sorter decides which simulation job in the queue to execute.
  • the simulation job priority scheme is user definable (i.e., controllable and definable by the system administrator) to control which simulation process has priority for current execution if a resource contention exists.
  • the server then swaps simulation jobs, if necessary, at step 1164.
  • This step temporarily replaces one simulation job associated with one process or one workstation for another simulation job associated with another process or workstation based on the priority determination programmed for the scheduler in the server. If multiple users are simulating the same design, the job swapper swaps in only the stored simulation state for the simulation session. However, if multiple users are simulating multiple designs, the job swapper loads in the design first before swapping in the simulation state.
  • the device driver also communicates with the reconfigurable hardware unit to swap jobs.
  • the job swapping mechanism enhances the performance of the time-sharing embodiment of the present invention because the job swapping need only be done for reconfigurable hardware unit access. So, if one user needs software simulation for some time period, the server swaps in another job for another user so that this other user can access the reconfigurable hardware unit for hardware acceleration. For example, assume that two users, user 1 and user 2, are coupled to the Simulation server for access to the reconfigurable hardware unit. At one time, user 1 has access to the system so that debugging can be performed for his/her user design. If user 1 is debugging in software mode only, the server can release the reconflgurable hardware unit so that user 2 can access it. The server swaps in the job for user 2 and user 2 can then either software simulate or hardware accelerate the model.
  • user 2 can continue accessing the reconfigurable hardware unit for some predetermined time or, if user 1 needs the reconfigurable hardware unit for acceleration, the server can preempt the job for user 2 so that the job for user 1 can be swapped in for hardware acceleration using the reconfigurable hardware unit.
  • the predetermined time refers to the pre-emption of simulator jobs based on multiple requests of the same priority. In one embodiment, the default time is 5 minutes although this time is user settable. This 5 minute setting represents one form of a time-out timer.
  • the Simulation system of the present invention uses the time-out timer to stop the execution of the current simulation job because it is excessively time consuming and the system decides that other pending jobs of equal priority should gain access to the reconfigurable hardware model.
  • the device driver in the server locks the reconfigurable hardware unit so that only the currently scheduled user or process can simulate and use the hardware . model.
  • the locking and simulation step occurs at step 1165.
  • the server returns to the priority sorter step 1163 to determine priority of pending simulation jobs and later swap simulation jobs if necessary. Similarly, the server may preempt the running of the currently active simulation job at event 1167 to return the server to the priority sorter state 1163.
  • the preemption occurs only under certain conditions. One such condition is when a higher priority task or job is pending. Another such condition is when the system is currently running a computationally intensive simulation task, in which case the scheduler can be programmed to preempt the currently running job to schedule a task or job with equal priority by utilizing a time-out timer. In one embodiment, the time-out timer is set at 5 minutes and if the current job executes for 5 minutes, the system preempts the current job and swaps in the pending job even though it is at the same priority level.
  • FIG. 50 shows a flow diagram of the job swapping process.
  • the job swapping function is performed in step 1164 of FIG.49 and is shown in the Simulation server hardware as job swapper 1144 in FIG. 48.
  • the job swapper sends an interrupt to the reconfigurable hardware unit at step 1180. If the reconfigurable hardware unit is not currently running any jobs (i.e. s the system is idle or the user is operating in software simulation mode only without any hardware acceleration intervention), the interrupt immediately prepares the reconfigurable hardware unit for job swapping.
  • the interrupt signal is recognized but the reconfigurable unit continues to execute the currently pending instruction and process the data for the current job. If the reconfigurable hardware unit receives the interrupt signal while the current simulation job is not in the middle of executing an instruction or processing data, then the interrupt signal essentially terminates the operation of the reconfigurable hardware unit immediately.
  • the Simulation system saves the current simulation image (i.e., hardware and software states). By saving this image, users can later restore the simulation run without re-running the whole simulation up to that saved point.
  • the Simulation system configures the reconflgurable hardware unit with the new user design. This configuration step is only necessary if the new job is associated with a different user design than the one already configured and loaded in the reconfigurable hardware unit and whose execution has just been interrupted.
  • the saved hardware simulation image is reloaded at step 1183 and the saved software simulation image is reloaded at step 1184. If the new simulation job is associated with the same design, then no additional configuration is needed.
  • the Simulation system loads the desired hardware simulation image associated with the new simulation job for that same design at step 1183 because the simulation image for the new job is probably different from the simulation image for the just interrupted job.
  • the details of the configuration step are provided herein in this patent specification.
  • the associated software simulation image is reloaded at step 1184.
  • the simulation can begin at step 1185 for this new job, while the previous interrupted job can only proceed in software simulation mode only because it has no access to the reconfigurable hardware unit for the moment.
  • FIG. 51 shows the signals between the device driver and the reconfigurable hardware unit.
  • the device driver 1171 provides the interface between the scheduler 1170 and the reconfigurable hardware unit 1172.
  • the device driver 1171 also provides the interface between the entire computing environment (i.e., workstation(s), PCI bus, PCI devices) and the reconfigurable hardware unit 1172 as shown in FIGS. 45 and 46, but FIG.51 shows the Simulation server portion only.
  • the signals between the device driver and the reconfigurable hardware unit includes the bi-directional communication handshake signals, the unidirectional design configuration information from the computing environment via the scheduler to the reconflgurable hardware unit, the swapped in simulation state information, the swapped out simulation state information, and the mterrupt signal from the device driver to the reconfigurable hardware unit so that the simulation jobs can be swapped.
  • Line 1173 carries the bi-directional communication handshake signals.
  • Line 1174 carries the unidirectional design configuration information from the computing environment via the scheduler 1170 to the reconfigurable hardware unit 1172.
  • Initial configuration information can be transmitted to the reconfigurable hardware unit 1172 for modelmg purposes on this line 1170. Additionally, when users are modeling and simulating different user designs, the configuration information must be sent to the reconfigurable hardware unit 1172 during a time slice. When different users are modeling the same user design, no new design configuration is necessary; rather, different simulation hardware states associated with the same design may need to be transmitted to the reconfigurable hardware unit 1172 for different simulation runs.
  • Line 1175 carries the swapped in simulation state information to the reconflgurable hardware unit 1172.
  • Line 1176 carries the swapped out simulation state information from the reconfigurable hardware unit to the computing environment (i.e., usually memory).
  • the swapped in simulation state information includes previously saved hardware model state information and the hardware memory state that the reconfigurable hardware unit 1172 needs to accelerate.
  • the swapped in state information is sent at the beginning of a time slice so that the scheduled current user can access the reconfigurable hardware unit 1172 for acceleration.
  • the swapped out state information includes hardware model and memory state information that must be saved in memory at the end of a time slice upon the reconfigurable hardware unit 1172 receiving an interrupt signal to move on to the next time slice associated with a different user/process.
  • the saving of the state information allows the current user/process to restore this state at a later time, such as at the next time slice that is assigned to this current user/process.
  • Line 1177 sends the interrupt signal from the device driver 1171 to the reconflgurable hardware unit so that the simulation jobs can be swapped. This interrupt signal is sent between time slices to swap out the current simulation job in the current time slice and swap in the new simulation job for the next time slice.
  • FIG. 53 shows the communication handshake signals between the device driver and the reconfigurable hardware unit via a handshake logic interface.
  • FIG. 54 shows a state diagram of the communication protocol.
  • FIG. 51 shows the communication handshake signals on line 1173.
  • FIG. 53 shows a detailed view of the communication handshake signals between the device driver 1171 and the reconfigurable hardware unit 1172.
  • a handshake logic interface 1234 is provided in the reconfigurable hardware unit 1172.
  • the handshake logic interface 1234 can be installed external to the reconflgurable hardware unit 1172.
  • Four sets of signals are provided between the device driver 1171 and the handshake logic interface 1234. These signals are the 3-bit SPACE signal on line 1230, a single-bit read/write signal on line 1231, a 4-bit COMMAND signal on line 1232, and a single bit DONE signal on line 1233.
  • the handshake logic interface includes logic circuitry that processes these signals to place the reconfigurable hardware unit in the proper mode for the various operations that need to be performed.
  • the interface is coupled to the CTRLJFPGA unit (or FPGA I/O controller).
  • the data transfers between the Simulation system's computing environment over the PCI bus and the reconfigurable hardware unit are designated for certain I/O address spaces in the software/hardware boundary — REG (register), CLK (software clock), S2H (software to hardware), and H2S (hardware to software).
  • REG register
  • CLK software clock
  • S2H software to hardware
  • H2S hardware to software
  • the read/write signal on line 1231 indicates whether the data transfer is a read or a write.
  • the DONE signal on line 1233 indicates the completion of a DMA data transfer period.
  • the 4-bit COMMAND indicates whether the data transfer operation should be a write, read, configure new user design into the reconfigurable hardware unit, or interrupt the simulation.
  • the COMMAND protocol is as follows:
  • the Simulation system at the device driver is idle. As long as no new command is presented, the system remains idle as indicated by path 1401.
  • the command processor processes the new command at state 1402.
  • the command processor is the FPGA I/O controller.
  • FIG. 52 illustrates one example.
  • Four jobs (job A, job B, job C, job D) are the incoming jobs in the simulation job queue.
  • the priority levels for these four jobs are different; that is, jobs A and B are assigned high priority I, whereas jobs C and D are assigned lower priority II.
  • the time-shared reconfigurable hardware unit usage depends on the priority levels of the queued incoming jobs.
  • the simulation starts with job A given access to the reconflgurable hardware unit.
  • job A is preempted by job B because job B has the same priority as job A and the scheduler provides equal time-shared access to the two jobs.
  • Job B now has access to the reconfigurable hardware unit.
  • job A preempts job B and job A executes to completion at time 1193.
  • job B takes over and it executes to completion to time 1194.
  • job C which is next in the queue but with a lower priority level than jobs A and B, now has access to the reconflgurable hardware unit for execution.
  • job D preempts job C for time-shared access because it has the same priority level as job C.
  • Job D now has access until time 1196 where it is preempted by job C.
  • Job C executes to completion at time 1197.
  • Job D then takes over at time 1197 and executes to completion until time 1198.
  • the Memory Simulation or memory mapping aspect of the present invention provides an effective way for the Simulation system to manage the various memory blocks associated with the configured hardware model of the user's design, which was programmed into the array of FPGA chips in the reconfigurable hardware unit.
  • the memory Simulation scheme does not require any dedicated pins in the FPGA chips to handle the memory access.
  • the phrase "memory access” refers to either a write access or a read access between the FPGA logic devices where the user's design is configured and the SRAM memory devices which stores all the memory blocks associated with the user's design.
  • a write operation involves data transfer from the FPGA logic devices to the SRAM memory devices
  • a read operation involves data transfer from the SRAM memory devices to the FPGA logic devices.
  • the FPGA logic devices include 1201 (FPGA1), 1202 (FPGA3), 1203 (FPGAO), and 1204 (FPGA2).
  • the SRAM memory devices include memory devices 1205 and 1206.
  • DMA data transfer refers to data transfer between the computing system and the Simulation system, in addition to its common usage among those ordinarily skilled in the art.
  • the computing system is shown in FIGS. 1, 45, and 46 as the entire PCI-based system with memory that supports the Simulation system, which resides in software as well as the reconfigurable hardware unit. Selected device drivers, socket/system calls to/from the operating system are also part of the Simulation system that allow the proper interface with the operating system and the reconfigurable hardware unit.
  • a DMA read transfer involves the transfer of data from the FPGA logic devices (and FPGA SRAM memory devices for initialization and memory content dump) to the host computing system.
  • a DMA write transfer involves the transfer of data from the host computing system to the FPGA logic devices (and FPGA SRAM memory devices for initialization and memory content dump).
  • FPGA data bus refers to the high bank bus FD[63 :32] and low bank bus FD[31 :0] coupling the FPGA logic devices which contain the configured and programmed user design to be debugged and the SRAM memory devices.
  • the memory Simulation system includes a memory state machine, an evaluation state machine, and their associated logic to control and interface with: (1) the main computing system and its associated memory system, (2) the SRAM memory devices coupled to the FPGA buses in the Simulation system, and (3) the FPGA logic devices which contain the configured and programmed user design that is being debugged.
  • the FPGA logic device side of the memory Simulation system includes an evaluation state machine, ⁇ an FPGA bus driver, and a logic interface for each memory block N to interface with the user's own memory interface in the user design to handle: (1) data evaluations among the FPGA logic devices, and (2) write/read memory access between the FPGA logic devices and the SRAM memory devices.
  • the FPGA I/O controller side includes a memory state machine and interface logic to handle DMA, write, and read operations between: (1) main computing system and SRAM memory devices, and (2) FPGA logic devices and the SRAM memory devices.
  • the operation of the memory Simulation system in accordance with one embodiment of the present invention is generally as follows.
  • the Simulation write/read cycle is divided into three periods - DMA data transfer, evaluation, and memory access.
  • the DATAXSFR signal indicates the occurrence of the DMA data transfer period where the computing system and the SRAM memory units are transferring data to each other via the FPGA data bus — high bank bus (FD[63:32]) 1212 and low bank bus (FD[31:0]) 1213.
  • each FPGA logic device During the evaluation period, logic circuitry in each FPGA logic device generates the proper software clock, input enable, and mux enable signals to the user's design logic for data evaluation. Inter-FPGA logic device communication occurs in this period.
  • the memory Simulation system waits for the high and low bank FPGA logic devices to put their respective address and control signals onto their respective FPGA data buses. These address and control signals are latched in by the CTRLJ F PGA unit. If the operation is a write, then address, control, and data signals are transported from the FPGA logic devices to their respective SRAM memory devices. If the operation is a read, then address and control signals are provided to the designated SRAM memory devices, and data signals are transported from the SRAM memory devices to their respective FPGA logic devices. After all desired memory blocks in all FPGA logic devices have been accessed, the memory Simulation write/read cycle is complete and the memory Simulation system is idle until the onset of the next memory Simulation write/read cycle.
  • FIG. 56 shows a high level block diagram of the memory Simulation configuration in accordance with one embodiment of the present invention. Signals, connections, and buses that are not relevant to the memory Simulation aspect of the present invention are not shown.
  • a CTRLJFPGA unit 1200 is coupled to bus 1210 via line 1209.
  • the CTRLJFPGA unit 1200 is a programmable logic device (PLD) in the form of an FPGA chip, such as an Altera 10K50 chip.
  • Local bus- 1210 allows the CTRLJFPGA unit 1200 to be coupled to other Simulation array boards (if any) and other chips (e.g., PCI controller, EEPROM, clock buffer).
  • Line 1209 carries the DONE signal which indicates the completion of a Simulation DMA data transfer period.
  • FIG. 56 shows other major functional blocks in the form of logic devices and memory devices.
  • the logic device is a programmable logic device (PLD) in the form of an FPGA chip, such as an Altera 10K130 or 10K250 chip.
  • PLD programmable logic device
  • the memory device is a synchronous- pipelined cache SRAM, such as a Cypress 128Kx32 CY7C1335 or CY7C1336 chip.
  • the logic devices include 1201 (FPGAl), 1202 (FPGA3), 1203 (FPGAO), and 1204 (FPGA2).
  • the SRAM chips include low bank memory device 1205 (LJ5RAM) and high bank memory device 1206 (HJ5RAM).
  • These logic devices and memory devices are coupled to the CTRLJ F PGA unit 1200 via a high bank bus 1212 (FD[63:32]) and a low bank bus 1213 (FD[31 :0]).
  • Logic devices 1201 (FPGAl) and 1202 (FPGA3) are coupled to the high bank bus 1212 via bus 1223 and bus 1225, respectively, while logic devices 1203 (FPGAO) and 1204 (FPGA2) are coupled to the low bank data bus 1213 via bus 1224 and bus 1226, respectively.
  • High bank memory device 1206 is coupled to the high bank bus 1212 via bus 1220, while low bank memory device 1205 is coupled to the low bank bus 1213 via bus 1219.
  • the dual bank bus structure allows the Simulation system to access the devices on the high bank and the devices on the low bank in parallel at improved throughput rates.
  • the dual bank data bus structure supports other signals, such as control and address signals, so that the Simulation write/read cycles can be controlled.
  • each Simulation write/read cycle includes a DMA data transfer period, an evaluation period, and a memory access period.
  • the combination of the various control signals control and indicate whether the Simulation system is in one period as opposed to another.
  • DMA data transfer between the host computer system and the logic devices 1201 to 1204 in the reconfigurable hardware unit occurs across the PCI bus (e.g., bus 50 in FIG.46), the local bus 1210 and 1236, and the FPGA bus 1212 (FD[63:32j) and 1213 (FD[31 :0]).
  • the memory devices 1205 and 1206 are involved in DMA data transfer for initialization and memory content dumps.
  • Evaluation data transfer among the logic devices 1201-1204 in the reconfigurable hardware unit occurs across the interconnects (as described above) and the FPGA bus 1212 (FD[63:32]) and 1213 (FD[31:0]).
  • Memory access between the logic devices 1201 to 1204 and the memory devices 1205 and 1206 occurs across the FPGA bus 1212 (FD[63:32]) and 1213 (FD[31:0]).
  • the CTRLJFPGA unit 1200 provides and receives many control and address signals to control the Simulation write/read cycles.
  • the CTRLJFPGA unit 1200 provides DATAXSFR and EVAL signals on line 1211 to logic devices 1201 and 1203 via line 1221, respectively, and logic devices 1202 and 1204 via line 1222, respectively.
  • the CTRLJFPGA unit 1200 also provides memory address signals MA[18:2] to the low bank memory device 1205 and the high bank memory device 1206 via buses 1229 and 1214, respectively.
  • the CTRLJFPGA unit 1200 provides chip select write (and read) signals to the low bank memory device 1205 and the high bank memory device 1206 via lines 1216 and 1215, respectively.
  • the memory Simulation system can send and receive the DONE signal on line 1209 to the CTRLJ F PGA unit 1200 and the computing system.
  • the logic devices 1201-1204 are connected together by, among other things, the multiplexed cross chip address pointer chain represented here in FIG. 56 by the two sets of SHIFTIN/SHIFTOUT lines - lines 1207, 1227, and 1218, and lines 1208, 1228, and 1217. These sets are initialized at the beginning of the chain by Vcc at lines 1207 and 1208.
  • the SHIFTIN signal is sent from the preceding FPGA logic device in the bank to start the memory access for the current FPGA logic device.
  • the last logic device At the completion of the shifts through a given set of chain, the last logic device generates a LAST signal (i.e., LASTL or LASTH) to the CTRLJFPGA unit 1200.
  • LAST signal i.e., LASTL or LASTH
  • logic device 1202 For the high bank, logic device 1202 generates a LASTH shiftout signal on line 1218 to the CTRLJFPGA unit 1200, and for the low bank, logic device 1204 generates a LASTL signal on line 1217 to the CTRLJ F PGA unit 1200.
  • one embodiment of the present invention incorporates the components (e.g., logic devices 1201-1204, memory devices 1205-1206, and CTRLJFPGA unit 1200) and buses (e.g., FPGA buses 1212-1213 and local bus 1210) in one board.
  • This one board is coupled to the motherboard via motherboard connectors.
  • four logic devices two on each bank
  • two memory devices one on each bank
  • buses are provided.
  • a second board would contain its own complement of logic devices (typically four), memory devices (typically two), FPGA I/O controller (CTRLJFPGA unit) and buses.
  • the PCI controller would be installed on the first board only.
  • Inter-board connectors are provided between the boards so that the logic devices in all the boards can be connected together and communicate with each other during the evaluation period, and the local bus is provided across all the boards.
  • the FPGA buses FD[63:0] are provided only in each board but not across multiple boards.
  • the Simulation system performs memory mapping between logic devices and memory devices in each board. Memory mapping across different boards is not provided. Thus, logic devices in board5 map memory blocks to memory devices in board5 only, not to memory devices on other boards. In other embodiments, however, the Simulation system maps memory blocks from logic devices on one board to memory devices on another board.
  • each FPGA logic device 1201-1204 During the evaluation period, logic circuitry in each FPGA logic device 1201-1204 generates the proper software clock, input enable, and mux enable signals to the user's design logic for data evaluation. Inter-FPGA logic device communication occurs in this period.
  • the CTRLJFPGA unit 1200 also begins an evaluation counter to control the duration of the evaluation period. The number of counts, and hence the duration of the evaluation period, is set by the system by determining the longest path of the signals. The path length is associated with a specific number of steps. The system uses the step information and calculates the number of counts necessary to enable the evaluation cycle to run to its completion.
  • the memory Simulation system waits for the high and low bank FPGA logic devices 1201-1204 to put their respective address and control signals onto their respective FPGA data buses. These address and control signals are latched in by the CTRLJFPGA unit 1200. If the operation is a write, address, control, and data signals are transported from the FPGA logic devices 1201-1204 to their respective SRAM memory devices 1205 and 1206. If the operation is a read, address and control signals are transported from the FPGA logic devices 1201-1204 to their respective SRAM memory devices 1205 and 1206, and data signals are transported from the SRAM memory devices 1205, 1205 to their respective FPGA logic devices 1201-1204.
  • the FD bus driver places the address and control signals of a memory block onto the FPGA data bus (FD bus). If the operation is a write, the write data is placed on the FD bus for that memory block. If the operation is a read, the double buffer latches in the data for the memory block on the FD bus from the SRAM memory device. This operation continues for each memory block in each FPGA logic device in sequential order one memory block at a time. When all the desired memory blocks in an FPGA logic device has been accessed, the memory Simulation system proceeds to the next FPGA logic device in each bank and begins accessing the memory blocks in that FPGA logic device. After all desired memory blocks in all FPGA logic devices 1201-1204 have been accessed, the memory Simulation write/read cycle is complete and the memory Simulation system is idle until the onset of the next memory Simulation write/read cycle.
  • FD bus FPGA data bus
  • FIG. 57 shows a more detailed block diagram of the memory Simulation aspect of the present invention, including a more detailed structural diagram of the CTRLJFPGA 1200 and each logic device that are relevant to memory Simulation.
  • FIG.57 shows the CTRLJ F PGA 1200 and a portion of the logic device 1203 (which is structurally similar to that of the other logic devices 1201, 1202, and 1204).
  • the CTRLJFPGA.1200 includes the memory finite state machine (MEMFSM) 1240, AND gate 1241, evaluation (EVAL) counter 1242, a low bank memory address/control latch 1243, a low bank address/control multiplexer 1244, address counter 1245, a high bank memory address/control latch 1247, and a high bank address/control multiplexer 1246.
  • MEMFSM memory finite state machine
  • EVAL evaluation
  • Each logic device such as logic device 1203 shown here in FIG. 57, includes an evaluation finite state machine (EVALFSMx) 1248, data bus multiplexer (FDOJVlUXx for the FPGAO logic device 1203) 1249.
  • EVALFSMx evaluation finite state machine
  • FDOJVlUXx data bus multiplexer
  • the "x" designation appended to the end of EVALFSM identifies the particular logic device (FPGAO, FPGAl, FPGA2, FPGA3) with which it is associated, where "x" is a number from 0 to 3 in this example.
  • EVALFSM0 is associated with the FPGAO logic device 1203.
  • each logic device is associated with some number x and as N logic devices are used, the "x" represents a number from 0 to N-1.
  • memory block interface 1253 in the user's logic provides a means for the computing system to access the desired memory block in the array of FPGA logic devices.
  • the memory block interface 1253 also provides memory write data on bus 1295 to the FPGA data bus multiplexer (FDOJVlUXx) 1249 and receives memory read data on bus 1297 from the memory read data double buffer 1251.
  • FDOJVlUXx FPGA data bus multiplexer
  • a memory block data/logic interface 1298 is provided in each FPGA logic device. Each of these memory block data/logic interface 1298 is coupled to the FPGA data bus multiplexer (FDOJVlUXx) 1249, the evaluation finite state machine (EVALFSMx) 1248, and the FPGA bus FD[63:0].
  • the memory block data/logic interface 1298 includes a memory read data double buffer 1251, the address offset unit 1250, the memory model 1252, and the memory block interface for each memory block N (memJjlockJSf) 1253 which are all repeated in any given FPGA logic device 1201- 1204 for each memory block N.
  • five sets of the memory block data/logic interface 1298 are provided; that is, five sets of the memory read data double buffer 1251, the address offset unit 1250, the memory model 1252, and the memory block interface for each memory block N (memJ ockJSt) 1253 are provided.
  • FDOJVlUXx identifies the particular logic device (FPGAO, FPGAl, FPGA2, FPGA3) with which it is associated, where "x" is a number from 0 to 3.
  • the output of FDOJVlUXx 1249 is provided on bus 1282 which is coupled to the high bank bus FD[63:32] or the low bank bus FD[31 :0] depending on which chip (FPGAO, FPGAl, FPGA2, FPGA3) is associated with the FDOJVlUXx 1249.
  • FDOJVlUXx is FDOJV1UX0, which is associated with low bank logic device FPGAO 1203.
  • the output on bus 1282 is provided to low bank bus FD[31:0].
  • Portions of the bus 1283 are used for transporting read data from the high bank FD[63:32] or low bank FD[31 :0] bus to the read bus 1283 for input to the memory read data double buffer 1251.
  • write data is transported out via FDOJVlUXO 1249 from the memory block in each logic device 1201-1204 to the high bank FD[63 :32] or low bank FD[31 :0] bus
  • read data is transported in to the memory read data double buffer 1251 from the high bank FD[63:32] or low bank FD[31 :0] bus via read bus 1283.
  • the memory read data double buffer provides a double buffered mechanism to latch data in the first buffer and then buffered again to get the latched data out at the same time to minimize skew. This memory read data double buffer 1251 will be discussed in more detail below.
  • this memory block interface 1253 can also be unique to the user's design.
  • the user's memory type may be DRAM, flash memory, or EEPROM.
  • memory addresses and control signals e.g., read, write, chip select, mem_clk
  • One embodiment of the memory Simulation aspect of the present invention converts the user's memory type to the SRAM type used in the memory Simulation system. If the user's memory type is SRAM, the conversion to an SRAM type memory model is quite simple. Thus, memory addresses and control signals are provided on bus 1296 to the memory model 1252, which performs the conversion.
  • the memory model 1252 provides memory block address information on bus 1293 and control information on bus 1292.
  • Address offset unit 1250 receives address information for the various memory blocks and provides a modified offset address on bus 12 1 from the original address on bus 1293.
  • the offset is necessary because certain memory blocks' addresses may overlap each other. For example, one memory block may use and reside in space 0-2K, whereas another memory block may use and reside in space 0-3K. Because both memory blocks overlap in space 0-2K, individual addressing may be difficult without some sort of address offsetting mechanism. Thus, the first memory block may use and reside in space 0-2K, while the second memory block may use and reside in the space above 2K and up to 5K.
  • the offset addresses from address offset unit 1250 and the control signals on bus 1292 are combined and provided on bus 1299 to the FPGA bus multiplexer (FDOJVlUXx) 1249.
  • the FPGA data bus multiplexer FDOJVlUXx receives SPACE2 data on bus 1289, SPACE3 data on bus 1290, address/control signals on bus 1299, and memory write data on bus 1295.
  • SPACE2 and SPACE3 are specific space indices.
  • the SPACE index which is generated by the FPGA I/O controller (item 327 in FIG. 10; FIG. 22), selects the particular address space (i.e., REG read, REG write, S2H read, H2S write, and CLK write). Within this address space, the system of the present invention sequentially selects the particular word to be accessed.
  • SPACE2 refers to the memory space dedicated for the DMA read transfer for the hardware-to-software H2S data.
  • SPACE3 refers to the memory space dedicated for the DMA read transfer for REGISTER_READ data. Refer to Table G above.
  • FDOJVlUXx 1249 provides data on bus 1282 to either the low bank or high bank bus.
  • the selector signals are the output enable (output_en) signal on line 1284 and the select signal on line 1285 from the EVALFSMx unit 1248.
  • the output enable signal on line 1284 enables (or disables) the operation of the FDOJVlUXx 1249. For data accesses across the FPGA bus, the output enable signal is enabled to allow the FDOJVlUXx to function.
  • the select signal on line 1285 is generated by the EVALFSMx unit 1248 to select among the plurality of inputs from the SPACE2 data on bus 1289, SPACE3 data on bus 1290, address/control signals on bus 1299, and memory write data on bus 1295.
  • the generation of the select signal by the EVALFSMx unit 1248 will be discussed further below.
  • the EVALFSMx unit 1248 is at the operational core of each logic device 1201-1204 with respect to the memory Simulation system.
  • the EVALFSMx unit 1248 receives as its inputs the SHIFTIN signal on line 1279, the EVAL signal from the CTRLJFPGA unit 1200 on line 1274, and a write signal wrx on line 1287.
  • the EVALFSMx unit 1248 outputs the SHIFTOUT signal on line 1280, the read latch signal rd latx on line 1286 to the memory read data double buffer 1251, the output enable signal on line 1284 to the FDOJVlUXx 1249, the select signal on line 1285 to the FDOJ lUXx 1249, and three signals to the user's logic (input-en, mux_en, and clk_en) on lines 1281.
  • the operation of the FPGA logic devices 1201-1204 for the memory Simulation system in accordance with one embodiment of the present invention is generally as follows.
  • the EVAL signal is at logic 1
  • data evaluation within the FPGA logic devices 1201-1204 takes place; otherwise, the Simulation system is performing either DMA data transfer or memory access.
  • the EVALFSMx unit 1248 generates the clk_en signal, the input_en signal, and the mux_en signal to allow the user's logic to evaluate the data, latch relevant data, and multiplex signals across logic devices, respectively.
  • the EVALFSMx unit 1248 generates the clk_en signal to enable the second flip-flop of all the clock edge register flip-flops in the user's design logic (see FIG. 19).
  • the clk_en signal is otherwise known as the software clock. If the user's memory type is synchronous, clk en also enables the second clock of the memory read data double buffer 1251 in each memory block.
  • the EVALFSMx unit 1248 generates the input_en signal to the user's design logic to latch the input signals sent from the CPU by DMA transfer to the user's logic.
  • the input_en signal provides the enable input to the second flip-flop in the primary clock register (see FIG. 19).
  • the EVALFSMx unit 1248 generates the mux_en signal to turn on the multiplexing circuit in each FPGA logic device to start the communication with other FPGA logic devices in the array.
  • the memory Simulation system waits for the selected data to be shifted in to the selected FPGA logic device and then generates the output_en and select signals for the FPGA data bus driver to put the address and control signals of the memory block interface 1253 (mem_block_N) on the FD bus.
  • the write signal wrx on line 1287 is enabled (i.e., logic 1), then the select and output_en signals are enabled to place the write data onto either the low or high bank bus, depending on which bank the FPGA chip is coupled.
  • logic device 1203 is FPGAO and is coupled to the low bank bus FD[31 :0].
  • the write signal wrx on line 1287 is disabled (i.e., logic 0)
  • the select and output_en signals are disabled and the read latch signal rdjatx on line 1286 to let the memory read data double buffer 1251 latch and double buffer the selected data from the SRAM via either the low or high bank bus, depending on which bank the FPGA chip is coupled.
  • the wrx signal is the memory write signal which is derived from the memory interface of the of the user's design logic. Indeed, the wrx signal on line 1287 comes from memory model 1252 via control bus 1292.
  • This process of reading or writing data occurs for each FPGA logic device. After all memory blocks have been processed via SRAM access, the EVALFSMx unit 1248 generates the SHIFTOUT signal to allow SRAM access by the next FPGA logic device in the chain. Note that the memory access for the devices on the high and low banks occur in parallel. At times, the memory access for one bank may complete before the memory access for the other bank. For all of these accesses, appropriate wait cycles are inserted so that logic processes data only when it is ready and data is available.
  • the MEMFSM 1240 is at the core of the memory Simulation aspect of the present invention. It sends and receives many control signals to control the activation of the memory Simulation write/read cycles and the control of the various operations supported by the cycles.
  • the MEMFSM 1240 receives the DATAXSFR signal on line 1260 via line 1258. This signal is also provided to each logic device on line 1273. When DATAXSFR goes low (i.e., logic low), the DMA data transfer period ends and the evaluation and memory access periods begin.
  • the MEMFSM 1240 also receives a LASTH signal on line 1254 and a LASTL signal on line 1255 to indicate that the selected word associated with the selected address space has been accessed between the computing system and the Simulation system via the PCI bus and the FPGA bus.
  • the MOVE signal associated with this shift out process is propagated through each logic device (e.g., logic device 1201-1204) until the desired word has been accessed and the MOVE signal ultimately becomes the LAST signal (i.e., LASTH for the high bank and LASTL for the low bank) at the end of the chain.
  • the EVALFSM 1248 i.e., FIG.
  • the corresponding LAST signal is the SHIFTOUT signal on line 1280. Because the particular logic device 1203 is not the last logic device in the low bank chain as shown in FIG. 56 where logic device 1204 is the last logic device in the low bank chain, the SHIFTOUT signal for EVALFSM0 is not the LAST signal. If the EVALFSM 1248 corresponds to EVALFSM2 in FIG. 56, then the SHIFTOUT signal on line 1280 is the LASTL signal provided to line 1255 to the MEMFSM. Otherwise, the SHIFTOUT signal on line 1280 is provided to logic device 1204 (see FIG. 56). Similarly, the SHIFTIN signal on line 1279 represents Vcc for the FPGAO logic device (see FIG. 56) 1203.
  • the LASTL and LASTH signals are input to AND gate 1241 via lines 1256 and 1257, respectively.
  • AND gate 1241 provides an open drain.
  • the output of the AND gate 1241 generates the DONE signal on line 1259, which is provided to the computing system and the MEMFSM 1240.
  • the MEMFSM 1240 generates a start signal on line 1261 to the EVAL counter 1242.
  • the start signal triggers the start of the EVAL counter 1242 and is sent after the completion of the DMA data transfer period.
  • the start signal is generated upon the detection of a high to low (1 to 0) transition of the DATAXSFR signal.
  • the EVAL counter 1242 is a programmable counter that counts a predetermined number of clock cycles. The duration of the programmed counts in the EVAL counter 1242 determines the duration of the evaluation period.
  • the output of the EVAL counter 1242 on line 1274 is either a logic level 1 or 0 depending on whether the counter is counting or not.
  • the MEMJFN signal on line 1272 is asserted at logic 0 and provided to the MEMFSM unit 1240, in which case the memory Simulation system waits for another DMA data transfer period. If memory access is desired, the MEMJ3N signal on line 1272 is asserted at logic 1.
  • the MEM JEN signal is a control signal from the CPU to enable the on-board SRAM memory device for accessing the FPGA logic devices.
  • the MEMFSM unit 1240 waits for the FPGA logic devices 1201-1204 to place the address and control signals on the FPGA bus, FD[63:32] and FD[31:0].
  • the remainder of the functional units and their associated control signals and lines are for providing address/control information to the SRAM memory devices for writing and reading data.
  • These units include the memory address/control latch 1243 for the low bank, the address control mux 1244 for the low bank, the memory address/control latch 1247 for the high bank, the address control mux 1246 for the high bank, and the address counter 1245.
  • the memory address/control latch 1243 for the low bank receives address and control signals from the FPGA bus FD[31:0] 1275, which coincides with bus 1213, and a latch signal on line 1263.
  • the latch 1243 generates mem_wr_L signal on line 1264 and provides the incoming address/control signals from FPGA bus FD[31 :0] to the address/control mux 1244 via bus 1266.
  • This mem_wr signal is the same as the chip select write signal.
  • the address/control mux 1244 receives as inputs the address and control information on bus 1266 and the address information from address counter 1245 via bus 1268. As output, it sends address/control information on bus 1276 to the low bank SRAM memory device 1205.
  • the select signal on line 1265 provides the proper selection signal from the MEMFSM unit 1240.
  • the address/control information on bus 1276 corresponds to the MA[18:2] and chip select read/write signals on buses 1229 and 1216 in FIG. 56.
  • the address counter 1245 receives mformation from SPACE4 and SPACE5 via bus 1267.
  • SPACE4 includes the DMA write transfer information.
  • SPACE5 includes the DMA read transfer information.
  • these DMA transfers occur between the computing system (cache/main memory via the workstation CPU) and the Simulation system (SRAM memory devices 1205, 1206) across the PCI bus.
  • the address counter 1245 provides its output to bus 1288 and 1268 to address/control muxes 1244 and 1246.
  • the address/control mux 1244 places on bus 1276 either the address/control information on bus 1266 for write/read memory access between the SRAM devices 1205 and the FPGA logic devices 1203, 1204, or alternatively, the DMA write/read fransfer data from SPACE4 or SPACE5 on bus 1267.
  • the MEMFSM unit 1240 provides the latch signal on line 1263 to the memory address/control latch 1243 to fetch the inputs from the FPGA bus FD[31:0].
  • the MEMFSM unit 1240 extracts the mem_wr_L control information from the address/control signals on FD[31 :0] for further control. If the mem_wr_L signal on line 1264 is a logic 1, a write operation is desired and the appropriate select signal on line 1265 is generated by the MEMFSM unit 1240 to the address/control mux 1244 so that the address and control signals on bus 1266 are sent to the low bank SRAM on bus 1276. Thereafter, a write data transfer occurs from the FPGA logic devices to the SRAM memory devices.
  • the Simulation system waits for data on the FPGA bus FD[31 :0] placed there by the SRAM memory device. As soon as data is ready, the read data transfer occurs from the SRAM memory devices to the FPGA logic devices.
  • the memory address/control latch 1247 for the high bank receives address and control signals from the FPGA bus FD[63:32] 1278, which coincides with bus 1212, and a latch signal on line 1270.
  • the latch 1270 generates mem_wr_H signal on line 1271 and provides the incoming address/control signals from FPGA bus FD[63 :32] to the address/control mux 1246 via bus 1239.
  • the address/control mux 1246 receives as inputs the address and control information on bus 1239 and the address information from address counter 1245 via bus 1268. As output, it sends address/control information on bus 1277 to the high bank SRAM memory device 1206.
  • the select signal on line 1269 provides the proper selection signal from the MEMFSM unit 1240.
  • the address/control information on bus 1277 corresponds to the MA[18:2] and chip select read/write signals on buses 1214 and 1215 in FIG. 56.
  • the address counter 1245 receives information from SPACE4 and SPACE5 via bus 1267 as mentioned above for DMA write and read transfers.
  • the address counter 1245 provides its output to bus 1288 and 1268 to address/control muxes 1244 and 1246. With the appropriate select signal on line 1269 for the high bank, the address/control mux 1246 places on bus 1277 either the address/control information on bus 1239 for write/read memory access between the SRAM devices 1206 and the FPGA logic devices 1201, 1202, or alternatively, the DMA write/read transfer data from SPACE4 or SPACE5 on bus 1267.
  • the MEMFSM unit 1240 provides the latch signal on line 1270 to the memory address/control latch 1247 to fetch the inputs from the FPGA bus FD[63:32].
  • the MEMFSM unit 1240 extracts the mem_wr_H control information from the address/control signals on FD[63:32] for further control. If the mem_wr_H signal on line 1271 is a logic 1, a write operation is desired and the appropriate select signal on line 1269 is generated by the MEMFSM unit 1240 to the address/control mux 1246 so that the address and control signals on bus 1239 are sent to the high bank SRAM on bus 1277. Thereafter, a write data transfer occurs from the FPGA logic devices to the SRAM memory devices.
  • the Simulation system waits for data on the FPGA bus FD[63:32] placed there by the SRAM memory device. As soon as data is ready, the read data fransfer occurs from the SRAM memory devices to the FPGA logic devices.
  • address and control signals are provided to low bank SRAM memory device and the high bank memory device via bus 1276 and 1277, respectively.
  • the bus 1276 for the low bank corresponds to the combination of the buses 1229 and 1216 in FIG. 56.
  • the bus 1277 for the high bank corresponds to the combination of the buses 1214 and 1215 in FIG. 56.
  • the operation of the CTRLJFPGA unit 1200 for the memory Simulation system in accordance with one embodiment of the present invention is generally as follows.
  • the DONE signal on line 1259 which is provided to the computing system and the MEMFSM unit 1240 in the CTRLJFPGA unit 1200 indicates the completion of a Simulation write/read cycle.
  • the DATAXSFR signal on line 1260 indicates the occurrence of the DMA data transfer period of the Simulation write/read cycle.
  • Memory address/control signals on both of the FPGA bus FD[31:0] and FD[63:32] are provided to the memory address/control latch 1243 and 1247 for the high and low banks, respectively.
  • MEMFSM unit 1240 generates the latch signal (1263 or 1269) to latch the address and control information.
  • the mem_wr signal is used to determine if a write or a read operation is desired. If a write is desired, data is transferred from the FPGA logic devices 1201-1204 to the SRAM memory devices via the FPGA bus. If a read is desired, the Simulation system waits for the SRAM memory device to put the requested data onto the PFGA bus for transfer between the SRAM memory device to the FPGA logic devices. For DMA data transfers of SPACE4 and SPACE5, the select signal on lines 1265, 1269 can select the output of the address counter 1245 as the data to be transferred between the main computing system and the SRAM memory devices in the Simulation system. For all of these accesses, appropriate wait cycles are inserted so that logic processes data only when it is ready and data is available.
  • FIG. 60 shows a more detailed view of the memory read data double buffer 1251 (FIG. 57).
  • Each memory block N in each FPGA logic device has a double buffer to latch in the relevant data which may be coming in at different times, and then finally buffering out this relevant latched data at the same time.
  • double buffer 1391 for memory block 0 includes two D-type flip-flops 1340 and 1341.
  • the output 1343 of the first D flip-flop 1340 is coupled to the input of the second D flip-flop 1341.
  • the output 1344 of the second D flip-flop 1341 is the output of the double buffer, which is provided to the memory block N interface in the user's design logic.
  • the global clock input is provided to the first flip-flop 1340 on line 1393 and the second flip-flop 1341 on line 1394.
  • the first D flip-flop 1340 receives on line 1342 its data input from the SRAM memory devices via bus 1283 and the FPGA bus FD[63:32] for the high bank and FD[31 :0] for the low bank.
  • the enable input is coupled to line 1345 which receives the rdjatx (e.g., rd JatO) signal from the EVALFSMx unit for each FPGA logic device.
  • the EVALFSMx unit generates the rdjatx signal to latch in the data on line 1342 to line 1343.
  • the input data for all the double buffers of all memory blocks may come in at different times, the double buffer ensures that all of the data is latched in first.
  • the clk_en signal (i.e., the software clock) is provided on line 1346 as the clock input to the second D flip-flop 1341.
  • the clk_en signal is asserted, the latched data on line 1343 is buffered into D flip-flop 1341 to line 1344.
  • the clk_en (software clock) signal is input to the second flip-flop (not shown) in the double buffer 1392 on line
  • the rdjatx signal (e.g., rdjatl) for the second double buffer 1392 is provided on line 1395 separately from other rdjatx signals for other double buffers. More double buffers are provided for other memory blocks N.
  • FIG. 58 shows such a state diagram of the finite state machine of the MEMFSM unit in the CTRLJFPGA unit.
  • the state diagram in FIG. 58 has been structured so that the three periods within the Simulation write/read cycle are also shown with their corresponding states.
  • states 1300-1301 correspond to the DMA data transfer period
  • states 1302-1304 correspond to the evaluation period
  • states 1305-1314 correspond to the memory access period.
  • the sequence of signals for the DMA transfer, evaluation, and memory access is set.
  • the sequence is as follows: DATA SFR triggers the DMA data transfer, if any.
  • the LAST signals for both high and low banks are generated at the completion of the DMA data transfer and trigger the DONE signal to indicate the completion of the completion of the DMA data transfer period.
  • the XSFRJDONE signal is then generated and the EVAL cycle then begins. At the conclusion of EVAL, memory read write can begin.
  • state 1300 is idle whenever the DATAXSFR signal is at logic 0. This indicates that no DMA data transfers are occurring at the moment.
  • the MEMFSM unit 1240 proceeds to state 1301.
  • the computing system requires DMA data transfer between the computing system (main memory in FIGS. 1, 45, and 46) and the Simulation system (FPGA logic devices 1201-1204 or SRAM memory device 1205, 1206 in FIG. 56). Appropriate wait cycles are inserted until the DMA data transfer is complete.
  • the DATAXSFR signal returns to logic 0.
  • the generation of the start signal is triggered in the MEMFSM unit 1240 at state 1302.
  • the start signal starts the EVAL counter 1242, which is a programmable counter.
  • the duration of the programmed counts in the EVAL counter is equivalent to the duration of the evaluation period. So long as the EVAL counter is counting at state 1303, the EVAL signal is asserted at logic 1 and provided to the EVALFSMx in each FPGA logic device as well as the MEMFSM unit 1240.
  • the EVAL counter presents the EVAL signal at logic 0 to the EVALFSMx in each FPGA logic device and the MEMFSM unit 1240.
  • the MEMFSM unit 1240 When the MEMFSM unit 1240 receives the logic 0 EVAL signal, it turns on the EVALJDONE flag at state 1304.
  • the EVALJDONE flag is used by MEMFSM to indicate that the evaluation period has ended and the memory access period, if desired, can now proceed.
  • the CPU will check the EVALJDONE and XSFRJDONE by reading the XSFRJEVAL register (see Table K below) to confirm that DMA transfer and EVAL has completed successfully before starting the next DMA transfer.
  • the Simulation system may not want to perform memory access at the moment.
  • the Simulation system keeps the memory enable signal MEMJ3N at logic 0.
  • This disabled (logic 0) MEM JEN signal keeps the MEMFSM unit at idle state 1300, where it is waiting for DMA data transfer or evaluation of data by the FPGA logic devices.
  • the memory enable signal MEM JEN is at logic 1, the Simulation system is indicating the desire to conduct memory access.
  • state diagram is divided into two sections which proceed in parallel.
  • One section contains states 1305, 1306, 1307, 1308, and 1309 for the low bank memory access.
  • the other section contains states 1311, 1312, 1313, 1314, and 1309 for the high bank memory access.
  • the Simulation system waits one cycle for the currently selected FPGA logic device to place the address and control signals on the FPGA bus FD[31 :0].
  • the MEMFSM generates the latch signal on line 1263 to the memory address/control latch 1243 to fetch inputs from the FD[31 :0].
  • the data corresponding to this particular fetched address and control signal will either be read from the SRAM memory device or written to the SRAM memory device.
  • the proper select signal for the address/control mux 1244 is generated to send address and control signals to the low bank SRAM.
  • the MEMFSM proceeds to state 1308.
  • the Simulation system waits one cycle for the currently selected FPGA logic device to place the address and control signals on the FPGA bus FD[63:32].
  • the MEMFSM generates the latch signal on line 1270 to the memory address/control latch 1247 to fetch inputs from the FD[63:32].
  • the data corresponding to this particular fetched address and control signal will either be read from the SRAM memory device or written to the SRAM memory device.
  • the proper select signal for the address/control mux 1246 is generated to send address and control signals to the high bank SRAM.
  • the MEMFSM proceeds to state 1314.
  • states 1309 and 1310 are not implemented in accordance with another embodiment of the present invention.
  • FIG. 59 shows such a state diagram of the EVALFSMx finite state machine in each FPGA chip.
  • the state diagram in FIG. 59 has been structured so that two periods within the Simulation write/read cycle are also shown with their corresponding states.
  • states 1320-1326A correspond to the evaluation period
  • states 1326B-1336 correspond to the memory access period.
  • States 1321, 1322, and 1323 relate to inter-FPGA communication where data is evaluated by the user's design via the FPGA logic devices.
  • EVALFSMx generates the signals input_en, mux_en, and clk_en (item 1281 in FIG. 57) to the user's logic.
  • EVALFSMx generates the clk_en signal, which enables the second flip-flop of all the clock edge register flip-flops in the user's design logic in this cycle (see FIG. 19).
  • the clk_en signal is otherwise known as the software clock. If the user's memory type is synchronous, clk_en also enables the second clock of the memory read data double buffer 1251 in each memory block. The SRAM data output for each memory block are sent to the user's design logic in this cycle.
  • the EVALFSMx generates the input_en signal to the user's design logic to latch the input signals sent from the CPU by DMA transfer to the user's logic.
  • the input_en signal provides the enable input to the second flip-flop in the primary clock register (see FIG. 19).
  • EVALFSMx generates the mux_en signal to turn on the multiplexing circuit in each FPGA logic device to start the communication with other FPGA logic devices in the array.
  • inter-FPGA wire lines are often multiplexed to efficiently utilize the limited pin resources in each FPGA logic device chip.
  • the EVALFSMx receives as one of its inputs the wr signal on line 1287. Based on this wr signal, the proper select signal on line 1285 will be asserted.
  • EVALFSMx generates the rdjatch signal on line 1286 to let the memory read data double buffer 1251 of memory block N fetch the SRAM data out onto the FD bus.
  • the EVALFSMx proceeds to state 1329, where it generates the particular select and output_en signals for the FD bus driver based on whether the operation is a write or a read. Then, the write or read operation for this next memory block N will take place.
  • FIG. 61 shows the Simulation write/read cycle in accordance with one embodiment of the present invention.
  • FIG. 61 shows at reference numeral 1366 the three periods in the Simulation write/read cycle - DMA data transfer period, evaluation period, and memory access period.
  • the timing for data transfers to/from the low bank SRAM may differ from that of the high bank SRAM.
  • FIG. 61 shows one example where the access times for the low and high banks are identical.
  • a global clock GCLK 1350 provides the clocking signal for all components in the system.
  • the DATAXSFR signal 1351 indicates the occurrence of the DMA data transfer period.
  • DMA data transfer is taking place between the main computing system and the FPGA logic devices or SRAM memory devices.
  • data is provided on the FPGA high bank bus FD[63:32] 1359 and trace 1369, as well as the FPGA low bank bus FD[31:0] 1358 and trace 1368.
  • the DONE signal 1364 indicates the completion of the memory access period by a logic 0 to 1 signal (e.g., trace 1390) or otherwise indicates the duration of the Simulation write/read cycle with a logic 0 (e.g., combination of edge of trace 1370 and edge of trace 1390).
  • the DONE signal is at logic 0.
  • the DATAXSFR signal goes from logic 1 to 0, which triggers the onset of the evaluation period.
  • EVAL 1352 is at logic 1 as indicated by frace 1371.
  • the duration of the EVAL signal at logic 1- is predetermined and can be programmable.
  • the data in the user's design logic is evaluated with the clk_en signal 1353 which is at logic 1 as indicated by trace 1372, the input_en signal 1354 which is also at logic 1 as indicated by trace 1373, and the mux_en signal 1355 which is also at logic 1 for a longer duration than clk_en and input_en as indicated by trace 1374. Data is being evaluated within this particular FPGA logic device.
  • the mux_en signal 1355 goes from logic 1 to 0 at trace 1374 and at least one memory block is present in the FPGA logic devices, then the evaluation period ends and the memory access period begins.
  • the SHIFTIN signal 1356 is asserted with a logic 1 at trace 1375. This indicates that the preceding FPGA has completed its evaluations and all desired data have been accessed to/from this preceding FPGA logic device. Now, the next FPGA logic device in the bank is ready to begin memory accesses.
  • ACj_k indicates that the address and control signal is associated with FPGAj and memory block k, where j and k are integers including 0.
  • WDjJk indicates write data for FPGAj and memory block k.
  • RDj _k indicates read data for FPGAj and memory block k.
  • AC3 1 indicates the address and control signals associated with FPGA3 and memory block 1.
  • the low bank SRAM accesses and the high bank SRAM accesses 1361 are shown as trace 1387.
  • wrx Based on the logic level of wrx signal to the EVALFSMx and consequently, the mem_wr signal to the MEMFSM, either a write or read operation is performed. If a write operation is desired, the memory model interfaces with the user's memory block N interface (MemJ31ockJ>I interface 1253 in FIG. 57) to provide wrx as one of its control signals. This control signal wrx is provided to the FD bus driver as well as the EVALFSMx unit. If wrx is at logic 1, the proper select signal and output_en signal are provided to the FD bus driver to place the memory write data on the FD bus.
  • This same control signal which is now on the FD bus can be latched by the memory address/control latch in the CTRLJFPGA unit.
  • the memory address/control latch sends the address and control signals to the SRAM via a MA[18:2]/control bus.
  • the wrx control signal which is at logic 1, is extracted from the FD bus and because a write operation is requested, the data associated with the address and control signals on the FD bus is sent to the SRAM memory device.
  • this next FPGA logic device which is logic device FPGAO in the low bank, places AC0J) on FD[31 :0] as indicated by trace 1377.
  • the Simulation system performs a write operation for WD0 0.
  • AC0J. is placed on the FD[31 :0] bus. If, however, a read operation was requested, the placement of the AC0 1 on the FD bus FD[31 :0] would be followed by some time delay before RD0_0 instead of WD0 0 corresponding to AC0_0 is placed on the FD bus by the SRAM memory device.
  • the reading operation by the user's design logic via the Simulation memory block interface is accomplished when the EVALFSMx generates the rdJatO signal 1362 to the memory read data double buffer in the Simulation memory block interface as indicated by trace 1388.
  • This rdJatO signal is provided to both the low bank FPGAO and the high bank FPGAl.
  • write operations for the low and high banks occur together, followed by read operations on both banks. This may not always be the case.
  • the existence of low and high banks allows parallel operation of the devices coupled to these banks; that is, activity on the low bank is independent of activity on the high bank.
  • Other scenarios can be envisioned where the low bank is performing a series of write operations while the high bank is performing a series of read operations in parallel.
  • the SHIFTOUT signal 1357 is asserted as indicated by trace 1376.
  • a rdjatl signal 1363 corresponding to FPGA2 on the low bank and FPGA3 on the high bank is asserted as indicated by trace 1389 to read RD2 1 on frace 1379 and RD3 1 on trace 1382. Because the last data for the last FPGA units have been accessed, the completion of the Simulation write/read cycle is indicated by the DONE signal 1364 as indicated by trace 1390.
  • Table H lists and describes the various components on the Simulation system boards and their orresponding register/memory, PCI memory address, and local address.
  • the data format for the configuration file is shown below in Table J in accordance with one embodiment of the present invention.
  • the CPU sends one word through the PCI bus each time to configure one bit for all on-board FPGAs in parallel.
  • the following Table K lists the XSFR EVAL register. It resides in all the boards.
  • the XSFRJFVAL register is used by the host computing system to program the EVAL period, control DMA read/write, and read the status of the EVALJDONE and XSFRJDONE fields.
  • the host computing system also uses this register to enable memory access. The operation of the Simulation system with respect to this register is described below with in conjunction with FIGS. 62 and 63.
  • Table L lists the contents of the CONFIG JTAG [6:1] register.
  • the CPU configures the FPGA logic devices and runs the boundary scan test for FPGA logic devices through this register.
  • Each board has one dedicated register.
  • FIGS. 62 and 63 show timing diagrams of another embodiment of the present invention. These two figures show the operation of the Simulation system with respect to the XSFRJEVAL register.
  • the XSFR _E VAL register is used by the host computing system to program the EVAL period, control DMA read/write, and read the status of the EVALJDONE and XSFRJDONE fileds. The host computing system also uses this register to enable memory access.
  • One of the main differences between these two figures is the state of the WAITJ5VAL field. When WAITJ5VAL field is set to "0," as is the case for FIG. 62, the DMA read transfer starts after CLK_EN. When WAITJEVAL field is set to "1," as is the case for FIG. 63, the DMA read transfer starts after EVALJDONE.
  • both WR_XSFRJEN and RD_XSFR_EN are set to "1." These two fields enable DMA write/read transfers and can be cleared by XSFRJDONE. Because both fields are set to "1,” the CTRLJFPGA unit automatically executes DMA write transfer first and then DMA read transfer.
  • the WAITJ3VAL field is set to "0" indicating that the DMA read transfer starts after the assertion of CLKJ3N (and after the completion of the DMA write operation).
  • the DMA read operation occurs almost immediately after the completion of the DMA write operation as soon as the CLKJBN signal (software clock) is detected. The DMA read transfer operation does not wait for the completion of the EVAL period.
  • EVALJ EQJSf signals experience contention as multiple FPGA logic devices vie for attention.
  • EVALJ EQJ>I (or EVAL_REQ#) signal is used to start the evaluation cycle if any of the FPGA logic devices asserts this signal.
  • the evaluation cycle begins including address pointer initialization and the operation of the software clocks to facilitate the evaluation process.
  • the DONE signal which is generated at the conclusion of a DMA data transfer period, also experiences contention as multiple LAST signals (from the shiftin and shiftout signals at the output of each FPGA logic device) are generated and provided to the CTRLJFPGA unit.
  • LAST signals from the shiftin and shiftout signals at the output of each FPGA logic device
  • the DONE signal is generated and a new DMA data transfer operation can begin.
  • the EVAL_REQJST signal and the DONE signal use the same wire on a time-shared basis in a manner to be discussed below.
  • the system automatically initiates DMA write transfer first as is shown by the WRJ SFR signal at time 1409.
  • the initial portion of the WR XSFR signal includes some overhead associated with the PCI controller, the PCI9080 or
  • the host computing system performs a DMA write operation via the local bus
  • the WRJ ⁇ SFR signal is deactivated indicating the completion of the DMA write operation.
  • the EVAL signal is activated for a predetermmed time from time 1412 to time 1410.
  • the duration of the EVALTIME is programmable and initially set at 8+X, where X is derived from the longest signal trace path.
  • the XSFRJDONE signal is also activated for a brief time to indicate the completion of this DMA transfer operation in which the present operation is a DMA write.
  • the contention among EVALJREQJSI signals ceases but the wire that carries the DONE signal now delivers the EVAL_REQ_N signal to the CTRLJFPGA unit.
  • the EVAL_REQJNf signals are processed via the wire that carries the DONE signal.
  • the EVAL_REQ_N signals are no longer generated by the FPGA logic devices but the EVALJREQJSf signals that have previously been delivered to the CTRLJFPGA unit will be processed.
  • the maximum time that the EVALJREQJSf signals are no longer generated by the FPGA logic devices for gated clocks is roughly 23 clock cycles. EVAL_REQJSf signals longer than this period will be ignored.
  • the CTRLJFPGA unit sends a write address strobe WPLX ADS JST signal to the PCI controller (e.g., PLX PCI9080) to initiate the DMA read transfer.
  • the PCI controller e.g., PLX PCI9080
  • the PCI controller will start the DMA read transfer process and the DONE signal is also generated.
  • the RD XSFR signal is activated to enable the DMA read fransfer.
  • the DMA read data is placed on the FPGA bus FD[63:0] and the local bus LD[31 :0].
  • the PCI controller processes the DMA read data by transporting the data from the FPGA bus FD[63:0] and the local bus LD[31 :0] to the host computer system.
  • the DMA read data will continue to be processed while the EVAL signal will be deactivated and the EVALJDONE signal will be activated to indicate the completion of the EVAL cycle.
  • Contention among the FPGA logic devices also begins as they generate the EVALJREQJST signals.
  • the host computer system polls the PLX interrupt register to determine if the end of the DMA cycle is near.
  • the PCI controller knows how many cycles are necessary to complete the DMA data transfer process. After a predetermined number of cycles, the PCI . controller will set a particular bit in its interrupt register. The CPU in the host computer system polls this interrupt register in the PCI controller. If the bit is set, the CPU knows that the DMA period is almost done. The CPU in the host system does not poll the interrupt register all the time because then it will tie up the PCI bus with a read cycle. Thus, in one embodiment of the present invention, the CPU in the host computer system is programmed to wait a certain number of cycles before it polls the interrupt register.
  • the end of the DMA read period occurs at time 1416 as the RD XSFR is deactivated and the DMA read data is no longer on the FPGA bus FD[63 :0] or the local bus LD[31 :0].
  • the XSFRJDONE signal is also activated at time 1416 and contention among the LAST signals for generation of the DONE signal begins.
  • the CPU in the host computer system does not access the Simulation hardware system.
  • the duration of this period is the sum of (1) overhead time for the PCI controller times 2, (2) the number of words of WR XSFR and RD XSFR, and (3) the host computer system's (e.g., Sun ULTRASparc) PCI overhead.
  • the first access after the DMA period occurs at time 1419 when the CPU polls the interrupt register in the PCI controller.
  • the MEMJEN signal is activated to enable the onboard SRAM memory devices so that memory access between the FPGA logic devices and the SRAM memory devices can begin. Memory access continues until time 1419 and in one embodiment, 5 clock cycles are necessary per access. If no DMA read transfer is necessary, then the memory access can begin earlier at time 1410 instead of time 1411.
  • the CPU in the host computer system can communicate with the PCI controller and the CTRLJFPGA unit via the local bus LD[31:0] from time 1418 to time 1429. This occurs after the CPU has completed polling the interrupt register of the PCI controller. The CPU writes data onto various registers in preparation for the next data transfer. The duration of this period is greater than 4 ⁇ sec. If the memory access is shorter than this period, the FPGA bus FD[63:0] will not experience any conflicts. At time 1429, the XSFRJDONE signal is deactivated.
  • the timing diagram is somewhat different from that of FIG. 62 because in FIG. 63 the WAITJEVAL field is set to " 1."
  • the DMA read transfer period starts after the EVALJDONE signal has been activated and is almost completed. It waits for the near completion of the EVAL period instead of starting immediately after the completion of the DMA write operation.
  • the EVAL signal is activated for a predetermined time from time 1412 to time 1410. At time 1410, the EVALJDONE signal is activated to indicate the completion of the EVAL period.
  • the CTRLJFPGA unit does not generate the write address strobe signal WPLX ADSJM to the PCI controller until time 1420, which is about 16 clock cycles before the end of the EVAL period.
  • the XSFRJDONE signal is also extended to time 1423. At time 1423, the XSFRJDONE field is set and the WPLX ADS _N signal can then be generated to start the DMA read process.
  • the CTRLJFPGA unit sends a write address strobe WPLX ADS JST signal to the PCI controller (e.g., PLX PCI9080) to initiate the DMA read transfer.
  • the PCI controller e.g., PLX PCI9080
  • the PCI controller will start the DMA read transfer process and the DONE signal is also generated.
  • the RD XSFR signal is activated to enable the DMA read transfer.
  • the DMA read data is placed on the FPGA bus FD[63:0] and the local bus LD[31 :0].
  • the PCI controller processes the DMA read data by transporting the data from the FPGA bus FD[63:0] and the local bus LD[31 :0] to the host computer system.
  • the remainder of the timing diagram is equivalent to that of FIG. 62.
  • the RD_XSFR signal in FIG. 63 is activated later than in FIG. 62.
  • the RD_XSFR signal in FIG. 63 follows the near completion of the EVAL period so that the DMA read operation is delayed.
  • the RD XSFR signal in FIG. 62 follows the detection of the CLKJFN signal after the completion of the DMA write transfer.
  • the coverification system of the present invention can accelerate the design/development cycle by providing designers with the flexibility of software simulation and the faster speed derived from using a hardware model. Both the hardware and software portions of a design can be verified prior to ASIC fabrication and without the limitations of an emulator-based coverification tool.
  • the debugging feature is enhanced and overall debug time ,can be significantly reduced.
  • FIG. 64 shows a typical final design embodied as a PCI add-on card, such as a video, multimedia, Ethernet, or SCSI card.
  • This card 2000 includes a direct interface connector 2002 that allows communication with other peripheral devices.
  • the connector 2002 is coupled to bus 2001 to transport video signals from a VCR, camera, or television tuner; video and audio outputs to a monitor or speaker; and signals to communication or disk drive interface.
  • the bulk of the functionality of the design is in chip 2004 which is coupled to the interface connector 2002 via bus 2003, local oscillator 2005 via bus 2007 for generating a local clock signal, and memory 2006 via bus 2008.
  • the add-on card 2000 also includes a PCI connector 2009 for coupling with a PCI bus 2010.
  • the design Prior to implementing the design as an add-on card as shown in FIG. 64, the design is reduced to ASIC form for testing purposes.
  • a conventional hardware/software coverification tool is shown in FIG. 65.
  • the user's design is embodied in the form of an ASIC labeled as the device-under-test (or "DUT") 2024 in FIG. 65.
  • the device-under-test 2024 is placed in the target system 2020, which is a combination of the central computing system 2021 on the motherboard and several peripherals.
  • the target system 2020 includes a central computing system 2021 which includes a CPU and memory, and operates under some operating system such as Microsoft Windows or Sun Microsystem's Solaris to run a number of applications.
  • Sun MicroSystem's Solaris is an operating environment and set of software products which support Internet, Intranet and ente ⁇ rise-wide computing.
  • the Solaris operating environment is based on industry standard UNIX System V Release 4, and designed for client-server applications in a distributed networking environment, provide the appropriate resources for smaller workgroups, and provide the WebTone that is required for electronic commerce.
  • the device driver 2022 for the device-under-test 2024 is included in the central computing system 2021 to enable communication between the operating system (and any applications) and the device-under-test 2024.
  • a device driver is a particular software to control a hardware component or peripheral device of a computer system.
  • a device driver is responsible for accessing the hardware registers of the device and often includes an interrupt handler to service interrupts generated by the device.
  • Device drivers often form part of the lowest level of the operating system kernel, with which they are linked when the kernel is built. Some more recent systems have loadable device drivers which can be installed from files after the operating system is running.
  • the device-under-test 2024 and the central computing system 2021 are coupled to a PCI bus 2023.
  • Other peripherals in the target system 2020 include an Ethernet PCI add-on card 2025 used to couple the target system to a network 2030 via bus 2034, a SCSI PCI add-on card 2026 coupled to SCSI drives 2027 and 2031 via buses 2036 and 2035, a VCR 2028 coupled to the device-under-test 2024 via bus 2032 (if necessary for the design in the device-under- test 2024), and a monitor and/or speaker 2029 coupled to the device-under-test 2024 via bus 2033 (if necessary for the design in the device-under-test 2024).
  • SCSI Small Computer Systems Interface
  • a processor-independent standard for system-level interfacing between a computer and intelligent devices such as hard disks, floppy disks, CD-ROM, printers, scanners and many more.
  • the device-under-test 2024 can be examined with a variety of stimuli from the central computing system (i.e., operating system, applications) and the peripheral devices. If time is not a concern and the designers are only seeking a simple pass/fail test, this coverification tool should be adequate to fulfill their needs. However, in most situations, a design project is strictly budgeted and scheduled prior to release as a product. As explained above, this particular ASIC-based coverification tool is unsatisfactory because its debug feature is nonexistent (the designer cannot isolate the cause of a "failed” test without sophisticated techniques, and the number of "fixes" for every bug detected cannot be predicted at the outset of a project and thus makes scheduling and budgeting unpredictable.
  • FIG. 66 illustrates a conventional coverification tool with an emulator. Unlike the set-up illustrated in FIG. 64 and described above, the device-under-test is programmed in an emulator 2048 coupled to the target system 2040 and some peripheral devices and a test workstation 2052.
  • the emulator 2048 includes an emulation clock 2066 and the device-under-test which was programmed in the emulator.
  • the emulator 2048 is coupled to the target system 2040 via a PCI bus bridge 2044 and PCI bus 2057 and control lines 2056.
  • the target system 2040 includes a combination of the central computing system 2041 on the motherboard and several peripherals.
  • the target system 2040 mcludes a central computing system 2041 which includes a CPU and memory, and operates under some operating system such as Microsoft Windows or Sun MicroSystem's Solaris to run a number of applications.
  • the device driver 2042 for the device-under-test is included in the central computing system 2041 to enable communication between the operating system (and any applications) and the device- under-test in the emulator 2048.
  • the central computing system 2041 is coupled to the PCI bus 2043.
  • peripherals in the target system 2040 include an Ethernet PCI add-on card 2045 used to couple the target system to a network 2049 via bus 2058, and a SCSI PCI add-on card 2046 coupled to SCSI drives 2047 and 2050 via buses 2060 and 2059.
  • the emulator 2048 is also coupled to the test workstation 2052 via bus 2062.
  • the test workstation 2052 includes a CPU and memory to perform its functions.
  • the test workstation 2052 may also include test cases 2061 and device models 2068 for other devices that are modeled but not physically coupled to the emulator 2048.
  • the emulator 2048 is coupled to some other peripheral devices such as a frame buffer or data stream record/play system 2051 via bus 2061.
  • This frame buffer or data stream record/play system 2051 may also be coupled to a communication device or channel 2053 via bus 2063, a VCR 2054 via bus 2064, and a monitor and/or speaker 2055 via bus 2065.
  • the emulation clock operates at a speed much slower than the actual target system speed.
  • that portion of FIG. 66 that is shaded is running at emulation speed while the other unshaded portions are running at actual target system speed.
  • this coverification tool with the emulator has several limitations.
  • the designer When using a logic analyzer or a sample-and-hold device to get internal state information of the device-under-test, the designer must compile his design so that the relevant signals he is interested in examining for debug pmposes are provided on the output pins for sampling. If the designer wants to debug a different part of the design, he must make sure that that part has output signals that can be sampled by the logic analyzer or the sample-and-hold device or else he must re-compile his design in the emulator 2048 so that these signals can be presented on the output pins for sampling pmposes.
  • FIG. 67 illustrates a high level configuration of the single-engine reconfigurable computing (RCC) array system of the present invention which was previously described above in this patent specification.
  • This single-engine RCC system will be incorporated into the coverification system in accordance with one embodiment of the present invention.
  • the RCC array system 2080 includes a RCC computing system 2081, a reconfigurable computing (RCC) hardware array 2084, and a PCI bus 2089 coupling them together.
  • the RCC computing system 2081 includes the entire model of the user's design in software and the RCC hardware array 2084 includes a hardware model of the user's design.
  • the RCC computing system 2081 includes the CPU, memory, an operating system, and the necessary software to run the single-engine RCC system 2080.
  • a software clock 2082 is provided to enable the tight control of the software model in the RCC computing system 2081 and the hardware model in the RCC hardware array 2084.
  • Test bench data 2083 are also stored in the RCC computing system 2081.
  • the RCC hardware array system 2084 includes a PCI interface 2085, a set of RCC hardware array boards
  • the set of RCC hardware array boards 2086 includes at least a portion of the user's design modeled in hardware (i.e., hardware model 2087) and memory 2088 for the test bench data.
  • various portions of this hardware model are distributed among a plurality of reconfigurable logic elements (e.g., FPGA chips) during configuration time. As more reconfigurable logic elements or chips are used, more boards may be needed.
  • four reconfigurable logic elements are provided on a single board. In other embodiments, eight reconfigurable logic elements are provided on a single board.
  • the capacity and capabilities of the ⁇ reconfigurable logic elements in the four-chip boards can differ significantly from that of the reconfigurable logic • elements in the eight-chip board.
  • Bus 2090 provides various clocks for the hardware model from the PCI interface 2085 to the hardware model
  • Bus 2091 provides other I/O data between the PCI interface 2085 and the hardware model 2087 via connector. 2093 and internal bus 2094.
  • Bus 2092 functions as the PCI bus between the PCI interface 2085 and the hardware model 2087.
  • Test bench data can also be stored in memory in the hardware model 2087.
  • the hardware model 2087 includes other structures and functions other than the hardware model of the user's design that are needed to enable the hardware model to interface with the RCC computing system 2081.
  • This RCC system 2080 may be provided in a single workstation or alternatively, coupled to a network of workstations where each workstation is provided access to the RCC system 2080 on a time-shared basis.
  • the RCC array system 2080 serves as a simulation server having a simulation scheduler and state swapping mechanism.
  • the server allows each user at a workstation to access the RCC hardware array 2084 for high speed acceleration and hardware state swapping purposes. After the acceleration and state swapping, each user can locally simulate the user design in software while releasing control of the RCC hardware array 2084 to other users at other workstations.
  • This network model will also be used for the coverification system described below.
  • the RCC array system 2080 provides designers with the power and flexibility of simulating an entire design, accelerating part of the test points during selected cycles via the hardware model in the reconfigurable computing array, and obtaining internal state information of virtually any part of his design at any time.
  • the single-engine reconfigurable computing array (RCC) system which can be loosely described as a hardware-accelerated simulator, can be used to perform the following tasks in a single debug session: (1) simulation alone, (2) simulation with hardware acceleration where the user can start, stop, assert values, and inspect internal states of the design at any time, (3) post- simulation analyses, and (4) in-circuit emulation.
  • the hardware model in the reconfigurable computing array is tightly coupled to the software simulation model. This allows the designer to debug cycle-by-cycle as well as accelerate and decelerate the hardware model through a number of cycles to obtain valuable internal state information. Moreover, because this simulation system handles data instead of signals, no complex signal-to-data conversion/timing circuitry is needed. Furthermore, the hardware model in the reconfigurable computing array does not need to be re-compiled if the designer wishes to examine a different set of nodes, unlike the typical emulation system. For further details, review the description above.
  • One embodiment of the present invention is a coverification system which uses no actual and physical external I/O devices and target applications.
  • a coverification system in accordance with one embodiment of the present invention can incorporate the RCC system along with other functionality to debug the software portion and hardware portion of a user's design without using any actual target system or I/O devices.
  • the target system and external I O devices are, instead, modeled in software in the RCC computing system.
  • the coverification system 2100 includes a RCC computing system 2101, the RCC hardware array 2108, and a PCI bus 2114 coupling them together.
  • the RCC computing system 2101 includes the entire model of the user's design in software and the reconfigurable computing array 2108 includes a hardware model of the user's design.
  • the RCC computing system 2101 includes the CPU, memory, an operating system, and the necessary software to run the single-engine coverification system 2100.
  • a software clock 2104 is provided to enable the tight control of the software model in the RCC computing system 2101 and the hardware model in the reconfigurable computing array 2108. Test cases 2103 are also stored in the RCC computing system 2101.
  • the RCC computing system.2101 also includes the target applications 2102, a driver 2105 of the hardware model of the user's design, a model of a device (e.g., a video card) and its driver in software labeled as 2106, and a model of another device (e.g., a monitor) and its driver also in software labeled as 2107.
  • the RCC computing system 2101 contains as many device models and drivers as necessary to convey to the software model and the hardware model of the user's design that an actual target system and other I/O devices are part of this computing environment.
  • the RCC hardware array 2108 includes a PCI interface 2109, a set of RCC hardware array boards 2110, and various buses for interface purposes.
  • the set of RCC hardware array boards 2110 includes at least a portion of the user's design modeled in hardware 2112 and memory 2113 for the test bench data. As described above, each board contains a plurality of reconfigurable logic elements or chips.
  • Bus 2115 provides various clocks for the hardware model from the PCI interface 2109 to the hardware model 2112.
  • Bus 2116 provides other I/O data between the PCI interface 2109 and the hardware model 2112 via connector 2111 and internal bus 2118.
  • Bus 2117 functions as the PCI bus between the PCI interface 2109 and the hardware model 2112. Test bench data can also be stored in memory in the hardware model 2113.
  • the hardware model as described above, includes other structures and functions other than the hardware model of the user's design that are needed to enable the hardware model to interface with the RCC computing system 2101.
  • FIG. 66 shows the emulator 2048 coupled to the target system 2040, some I/O devices (e.g., frame buffer or data stream record/play system 2051), and a workstation 2052.
  • This emulator configuration provides numerous problems and set-up issues for the designer.
  • the emulator needs a logic analyzer or a sample-and-hold device to measure internal states of the user design modeled in the emulator. Because the logic analyzer and the sample-and-hold device needs signals, complex signal-to-data conversion circuitry is required. Additionally, complex signal-to-signal timing control circuitry is also required. The numerous wires needed for every signal that will be used to measure the internal states of the emulator further burden the user during set-up.
  • the user must re-compile the emulator each time he wants to examine a different set of internal logic circuitry so that the appropriate signals from that logic circuitry are provided as outputs for measurement and recording by the logic analyzer or the sample-and-hold device.
  • the long re-compilation time is too costly.
  • the target system and other I/O devices are modeled in software so that an actual physical target system and I/O devices are not physically necessary. Because the RCC computing system 2101 processes data, no complex signal-to-data conversion circuitry or signal-to-signal timing control circuitry are needed. The number of wires are also not tied to the number of signals and hence, set-up is relatively simple. Furthermore, debugging a different portion of the logic circuitry in the hardware model of the user design does not require re-compilation because the coverification system processes data and not signals.
  • the RCC computing system controls the RCC hardware array with the software-controlled clock (i.e., software clock and clock edge detection circuitry), starting and stopping the hardware model is facilitated. Reading data from the hardware model is also easy because the model of the entire user design is in software and the software clock enables synchronization. Thus, the user can debug by software simulation alone, accelerate part or all of the design in hardware, step through various desired test points cycle-by-cycle, inspect internal states of the software and hardware model (i.e., register and combinational logic states).
  • the software-controlled clock i.e., software clock and clock edge detection circuitry
  • the user can simulate the design with some test bench data, then download internal state information to the hardware model, accelerate the design with various test bench data with hardware model, inspect the resulting internal state values of the hardware model by register/combinational logic regeneration and loading values from the hardware model to the software model, and the user can finally simulate other parts of the user design in software using the results of the hardware model-accelerated process.
  • a workstation In a network configuration, a workstation may be remotely coupled to the coverification system to access debug data remotely. In a non-network configuration, a workstation may be locally coupled to the coverification system or in some other embodiments, the workstation may inco ⁇ orate the coverification system internally so that debug data can be accessed locally.
  • the various I/O devices and target applications were modeled in the RCC computing system 2101. However, when too many I/O devices and target applications are running in the RCC computing system 2101, the overall speed slows down. With only a single CPU in the RCC computing system 2101, more time is necessary to process the various data from all the device models and target applications. To increase the data throughput, actual I/O devices and target applications (instead of software models of these I/O devices and target applications) can be physically coupled to the coverification system.
  • a coverification system that uses actual and physical external I/O devices and target applications.
  • a coverification system can inco ⁇ orate the RCC system along with other functionality to debug the software portion and hardware portion of a user's design while using the actual target system and/or I/O devices.
  • the coverification system can use both test bench data from software and stimuli from the external interface (e.g., target system and external I/O devices).
  • Test bench data can be used to not only provide test data to pin-outs of the user design, but also test data to internal nodes in the user design.
  • Actual I O signals from external I/O devices (or target system) can only be directed to pin-outs of the user design.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Geometry (AREA)
  • Debugging And Monitoring (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Design And Manufacture Of Integrated Circuits (AREA)

Abstract

L'invention se rapporte à une technique appelée VCD sur demande. Dans un système type, l'outil EDA intégrant la technique VCD sur demande possède les attributs de haut niveau suivants : (1) compression et enregistrement de l'historique de la simulation parallèle de type RCC ; (2) décompression de l'historique de la simulation parallèle de type RCC, et génération de fichiers VCD ; et (3) régénération de logiciels sur demande pour une plage cible de simulation sélectionnée, et révision de la conception sans réexécution de la simulation. Chacun de ces attributs seront décrits plus en détail. Lorsque l'utilisateur sélectionne une plage de simulation (105), le système RCC enregistre une version hautement compressée des entrées primaires issues du processus d'évaluation des performances. L'utilisateur sélectionne ensuite une zone plus réduite, appelée la plage cible de simulation (135), dans la plage de session de simulation, en vue d'une analyse plus concentrée. Le système RCC vide les informations d'état de matériel (autrement dit, les entrées primaires) du modèle de matériel dans un fichier VCD. Le système RCC permet ensuite à l'utilisateur de visualiser directement le fichier VCD à partir du début de la plage cible de simulation (105), sans devoir réexécuter la simulation entière à partir du tout début de la plage de session de simulation.
EP01965946A 2001-08-14 2001-08-14 Systeme et procede de vcd sur demande Withdrawn EP1417577A4 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2001/025558 WO2003017099A1 (fr) 2001-08-14 2001-08-14 Systeme et procede de vcd sur demande

Publications (2)

Publication Number Publication Date
EP1417577A1 true EP1417577A1 (fr) 2004-05-12
EP1417577A4 EP1417577A4 (fr) 2009-08-26

Family

ID=21742775

Family Applications (1)

Application Number Title Priority Date Filing Date
EP01965946A Withdrawn EP1417577A4 (fr) 2001-08-14 2001-08-14 Systeme et procede de vcd sur demande

Country Status (7)

Country Link
EP (1) EP1417577A4 (fr)
JP (1) JP4102752B2 (fr)
KR (1) KR100928134B1 (fr)
CN (1) CN1308819C (fr)
CA (1) CA2420027C (fr)
IL (3) IL160392A0 (fr)
WO (1) WO2003017099A1 (fr)

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007528553A (ja) * 2004-03-09 2007-10-11 セヤン ヤン 検証性能と検証效率性を高める動的検証−基盤方式の検証装置及びこれを用いた検証方法論
JP2007305137A (ja) * 2006-05-12 2007-11-22 Samsung Electronics Co Ltd 分配された同時的シミュレーション
US9715325B1 (en) 2012-06-21 2017-07-25 Open Text Corporation Activity stream based interaction
JP5926807B2 (ja) * 2012-09-06 2016-05-25 株式会社日立製作所 協調シミュレーション用計算機システム、組込みシステムの検証システム及び組込みシステムの検証方法
US9208008B2 (en) 2013-07-24 2015-12-08 Qualcomm Incorporated Method and apparatus for multi-chip reduced pin cross triggering to enhance debug experience
KR101660580B1 (ko) 2014-04-02 2016-09-28 프레스티지 바이오파마 피티이. 엘티디. 항체의 당 함량 조절을 통한 항체의 제조 방법
CN109426518B (zh) * 2017-08-29 2021-02-19 杭州旗捷科技有限公司 单核处理器设备的并行写码方法、电子设备、存储介质
CN109710536B (zh) * 2018-12-29 2022-03-18 湖北航天技术研究院总体设计所 一种自动提取fpga软件验证结果仿真波形的系统及方法
CN109740250B (zh) * 2018-12-29 2022-03-18 湖北航天技术研究院总体设计所 基于uvm的fpga软件验证结果仿真波形的获取方法和系统
CN111125975B (zh) * 2019-12-09 2024-06-14 上海思尔芯技术股份有限公司 一种fpga时分复用多路数据传输的方法、存储介质及终端
CN112486076B (zh) * 2020-12-08 2022-02-15 长光卫星技术有限公司 一种多fpga间时钟同步与复位同步系统
CN113342697B (zh) * 2021-07-19 2022-08-26 英韧科技(上海)有限公司 闪存转换层仿真测试系统及方法
US20240070345A1 (en) * 2022-08-30 2024-02-29 Rockwell Automation Technologies, Inc. Parallel emulation for controls testing

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6009256A (en) * 1997-05-02 1999-12-28 Axis Systems, Inc. Simulation/emulation system and method
DE10053207A1 (de) * 1999-10-28 2001-05-03 Advantest Corp Verfahren und Vorrichtung zur Gültigkeitsprüfung bei Systemchip-Entwürfen

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3506202B2 (ja) * 1997-06-30 2004-03-15 住友電装株式会社 基板用コネクタ
US6083269A (en) 1997-08-19 2000-07-04 Lsi Logic Corporation Digital integrated circuit design system and methodology with hardware
US6249891B1 (en) 1998-07-02 2001-06-19 Advantest Corp. High speed test pattern evaluation apparatus
US6061283A (en) * 1998-10-23 2000-05-09 Advantest Corp. Semiconductor integrated circuit evaluation system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6009256A (en) * 1997-05-02 1999-12-28 Axis Systems, Inc. Simulation/emulation system and method
DE10053207A1 (de) * 1999-10-28 2001-05-03 Advantest Corp Verfahren und Vorrichtung zur Gültigkeitsprüfung bei Systemchip-Entwürfen

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of WO03017099A1 *

Also Published As

Publication number Publication date
EP1417577A4 (fr) 2009-08-26
JP2005500618A (ja) 2005-01-06
KR100928134B1 (ko) 2009-11-25
JP4102752B2 (ja) 2008-06-18
CN1491385A (zh) 2004-04-21
IL154481A (en) 2008-03-20
IL154481A0 (en) 2003-09-17
CA2420027C (fr) 2012-01-03
CA2420027A1 (fr) 2003-02-27
KR20040028598A (ko) 2004-04-03
WO2003017099A1 (fr) 2003-02-27
CN1308819C (zh) 2007-04-04
IL160392A0 (en) 2004-07-25

Similar Documents

Publication Publication Date Title
US7512728B2 (en) Inter-chip communication system
US7480606B2 (en) VCD-on-demand system and method
US6321366B1 (en) Timing-insensitive glitch-free logic system and method
JP4125675B2 (ja) タイミングに鈍感なグリッチのない論理システムおよび方法
US6785873B1 (en) Emulation system with multiple asynchronous clocks
US6810442B1 (en) Memory mapping system and method
US6651225B1 (en) Dynamic evaluation logic system and method
US9195784B2 (en) Common shared memory in a verification system
US6389379B1 (en) Converification system and method
US6754763B2 (en) Multi-board connection system for use in electronic design automation
US6026230A (en) Memory simulation system and method
US6134516A (en) Simulation server system and method
US6421251B1 (en) Array board interconnect system and method
US6009256A (en) Simulation/emulation system and method
US7366652B2 (en) Method of programming a co-verification system
KR100483636B1 (ko) 에뮬레이션및시뮬레이션을이용한설계검증방법및장치
CA2420027C (fr) Systeme et procede de vcd sur demande
Shabany et al. ECE1373 VLSI Systems Design project report MonteCarlo-based Channel Estimator (MCCE)

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20030224

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

AX Request for extension of the european patent

Extension state: AL LT LV MK RO SI

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: VERISITY DESIGN, INC.

A4 Supplementary search report drawn up and despatched

Effective date: 20090729

17Q First examination report despatched

Effective date: 20100302

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: CADENCE DESIGN SYSTEMS, INC.

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20150303