US20030121011A1 - Functional coverage analysis systems and methods for verification test suites - Google Patents

Functional coverage analysis systems and methods for verification test suites Download PDF

Info

Publication number
US20030121011A1
US20030121011A1 US10/352,591 US35259103A US2003121011A1 US 20030121011 A1 US20030121011 A1 US 20030121011A1 US 35259103 A US35259103 A US 35259103A US 2003121011 A1 US2003121011 A1 US 2003121011A1
Authority
US
United States
Prior art keywords
objects
coverage
event
analysis
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US10/352,591
Inventor
Hamilton Carter
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cirrus Logic Inc
Original Assignee
Cirrus Logic Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Cirrus Logic Inc filed Critical Cirrus Logic Inc
Priority to US10/352,591 priority Critical patent/US20030121011A1/en
Publication of US20030121011A1 publication Critical patent/US20030121011A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G01MEASURING; TESTING
    • G01RMEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G01R31/28Testing of electronic circuits, e.g. by signal tracer
    • G01R31/317Testing of digital circuits
    • G01R31/3181Functional testing
    • G01R31/3183Generation of test inputs, e.g. test vectors, patterns or sequences
    • G01R31/318314Tools, e.g. program interfaces, test suite, test bench, simulation hardware, test compiler, test program languages
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01RMEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G01R31/28Testing of electronic circuits, e.g. by signal tracer
    • G01R31/317Testing of digital circuits
    • G01R31/3181Functional testing
    • G01R31/3183Generation of test inputs, e.g. test vectors, patterns or sequences
    • G01R31/318342Generation of test inputs, e.g. test vectors, patterns or sequences by preliminary fault modelling, e.g. analysis, simulation
    • G01R31/31835Analysis of test coverage or failure detectability
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01RMEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G01R31/28Testing of electronic circuits, e.g. by signal tracer
    • G01R31/317Testing of digital circuits
    • G01R31/3181Functional testing
    • G01R31/3183Generation of test inputs, e.g. test vectors, patterns or sequences
    • G01R31/318342Generation of test inputs, e.g. test vectors, patterns or sequences by preliminary fault modelling, e.g. analysis, simulation
    • G01R31/318357Simulation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00Computer-aided design [CAD]
    • G06F30/30Circuit design
    • G06F30/32Circuit design at the digital level
    • G06F30/33Design verification, e.g. functional simulation or model checking

Definitions

  • patent application Ser. No. ______ patent application Ser. No. ______ and patent application Ser. No. ______, having respective titles “Systems And Methods For Generating Interchangable Device Event Description Databases For Use in a Functional Coverage Tool That Is Portable Between Designs”, “Systems And Methods For Manipulating Configuration Events in a Dataflow Functional Coverage Tool For a Verification Test Suite”, and “Systems and Methods for Allowing Graphical User Connection of Coverage Analyzer Operators in a Data Flow Functional Coverage Tool”, with the same filing date as the present patent application, and the same inventorship.
  • This application relates to digital design methods and systems and more particularly to methods and systems for gauging the completeness of simulation test suites for digital design operations.
  • the tool output in particular instances is no more than a simple bar graph of predetermined event counters.
  • particular tools require the user either to instrument the entire HDL description of the device under test, at a considerable risk of error, or to write coverage code in a separate language, which on occasion is non-intuitive and significantly time intensive.
  • One current tool containing a relational data base for analysis of event traces originating from a particular simulation resulted in each recorded event in the device under test being stored as a row in a database table.
  • the user is forced to evaluate device events through a database oriented filter utilizing, in one instance, an SQL query language process.
  • the user is limited in coverage ability by the limitations of SQL and the user familiarity with the language.
  • the use of the SQL tool is limited to post-processing of information.
  • coverage metrics are subject to user specification before or after simulation operation. Further, the coverage metrics are subject to user specification with an intuitive graphical interface based upon data flow, without specific or arcane program language knowledge being required for implementation. According to the present invention, additional coverage analysis and presentation objects are conveniently integrated into an expanded system functionality. No original system modifications are required for the implemented expansions. Further, instrumentation of device under test hardware design language code is not required for operation according to the present invention. Additionally, according to the present invention, coverage results are provided which enable device functionality to be understood. In particular under the present invention, events in a device under test are modeled as objects. Each event object has a name and a number of attributes that describe the event.
  • the event objects are introspected at run-time, allowing the user to determine the event object's attributes with specification of coverage metrics subject to a selected combination of the event object's attributes.
  • the event objects are serialized according to one embodiment of the present invention, to permanent storage, allowing the user to specify and execute new coverage metrics at any time after simulation.
  • Operations used to describe coverage metrics are modeled as analysis objects according to the present invention.
  • Such analysis objects accept event objects as inputs, using a predetermined, well-defined interface.
  • the combination of event objects and analysis objects according to the present invention allows coverage metrics to be specified in a simple data flow manner. Event objects are particularly treated as a data stream that is operated on by predetermined analysis objects.
  • the analysis objects are modeled after user-familiar objects such as for example, comparators, logic gates, counters, and analyzers. With such a coverage metric, the user attaches or wires (metaphorically) the analysis objects together in a visual builder environment according to the present invention.
  • desired coverage metrics which would otherwise be difficult or impossible otherwise to implement, such as coverage of sequences of events and/or coverage of events that occur during the same time window of a simulation. Because analysis objects accept and pass on event objects through a standardized interface, the addition of new analysis objects can be accomplished without modification of the coverage tool utilized.
  • the resulting output of specialized coverage metrics is displayed with a special class of analysis objects known currently as presentation objects.
  • the display functionality of the coverage tool is expandable because the presentation objects use the same event object interface as the analysis operator objects. Analysis objects and their interconnections are serialized into selected storage locations according to the present invention, permitting the user to save desired coverage metrics and to enable measurement of the completeness of future test suites. Further according to the present invention, coverage metrics are subject to specification either before or after device simulation occurrence. Additionally, the user specifies coverage metrics using an intuitive graphical interface based upon data flow, without any specific programming language skills being necessary. Moreover, the system according to the present invention is easily expandable, by adding new coverage analysis and presentation objects. Such expansions require no original system alterations. Implementation according to the present invention is architecturally independent, meaning that a change in a device under test does not require a change in the coverage tool.
  • the functional behavior of a given device under test is described in a series of functional events which indicate that a particular occurrence has transpired within the device in which a functional portion of the device has been exercised, and which indicate what the settings of the device related to a particular functionality were when the event occurred.
  • functional events in the device are treated as event objects.
  • Each functional event according to the present invention has a name and zero or more attributes with which it is associated.
  • An example functional event according to the present invention is a bus cycle on a digital device.
  • example attributes are read or write type, simulation time at initiation, bus cycle address, and data transferred by cycle operation.
  • the system according to the present invention is versatile and intuitive as a coverage tool in that it treats device events detected as an independent object with name and attributes. Each event object may be passed to selected analysis tools chosen by the user, such as analyzers, logic gates, and coincidence counters.
  • event objects are created by functions added to the original HDL code describing the device under test, despite possibility of errors and vulnerability to device architecture changes. Such a device change requires code adaptation.
  • a functional model of the device under test is constructed to verify the device behavior and to produce functional event objects. The functional model according to the present invention accordingly receives the states of inputs and outputs to the device and the modeled behavior determines when device output events occur in a simulation.
  • the functional model creates an event object and serializes it to selected storage, without any necessity to instrument HDL files describing the device.
  • the coverage of a selected test suite is thus capable of analysis for functional coverage, simply, intuitively, and powerfully, as well as dynamically at runtime by specification of coverage metrics.
  • the invention also has an extensible architecture that allows the expansion of the tool without core system modifications.
  • the tool according to the present invention is independent of the device under test, allowing it to be used for selected digital designs according to user selection. According to one embodiment, the tool of the present invention is unaffected by changes in the underlying device under test that do not directly change the functionality of the device, such as when architectural changes are made to enhance performance of the device under test, such as for example net name changes.
  • FIG. 1 is a block diagram of the architecture of a generalized wiring system for a data flow functional coverage tool, according to the present invention
  • FIG. 2 is a block diagram of a graphical user interface, including a toolbar and a specification area, according to one embodiment of the present invention
  • FIG. 3 is a block diagram of a particular test environment, according to the present invention.
  • FIG. 4 is a block diagram of a test environment having a selected device under test (DUT) within a simulation domain, and further including a coverage tool and a storage mechanism in a verification environment in communication with the simulation domain, according to another embodiment of the present invention
  • FIG. 5 is a data flow diagram describing operation of a coverage tool method, according to one embodiment of the present invention.
  • FIG. 6 is a block diagram of an EventInfo object architecture according to the present invention.
  • FIG. 7 is a block diagram of an analysis object used to define coverage metrics to be executed on the event object stream created by a given simulation or set of simulations, according to one embodiment of the present invention
  • FIG. 8 is a diagram of a registration step of event transmission, according to the present invention, operating in first and second domains, a user interface domain and an associated functional domain;
  • FIG. 9 is a diagram of a request storage step of event transmission according to the present invention.
  • FIG. 10 is a diagram of an event object transmission step with respect to event transmission operation 120 according to the present invention.
  • FIG. 11 is a block diagram of a computer system for implementing operation of a computer program product according to the present invention.
  • FIG. 1 there is shown a block diagram of the architecture of a generalized wiring system 19 for a data flow functional coverage tool, according to the present invention.
  • the wiring system 19 is used in the graphical user interface 3 to implement a data flow functional coverage tool according to the present invention, which is conveniently capable of adaptation or modification according to the needs of the design test engineer.
  • the wiring system 19 includes the following objects which are part of the architecture of the present invention.
  • the wiring system 19 includes a graphic analyzer manager 20 , a plurality of graphic analyzers 21 , any number of analyzers 22 , an analyzer manager 23 , any number of wire objects 24 , and any number of graphic wire objects 25 .
  • the graphic analyzer manager 23 includes a plurality of graphic analyzers 30 - 31 .
  • the graphic analyzer 21 includes a reference to a parent analyzer 34 .
  • the analyzer object 22 includes reference to a input wires 35 and output wires 36 .
  • the wire object 24 includes reference to a source analyzer 38 and a target analyzer 39 .
  • Analyzer objects 22 are the core of a functional coverage tool according to the present invention.
  • the analyzer objects 22 perform the operations that execute specified coverage metrics on a selected event stream specified by a functional coverage tool.
  • the graphic analyzer 21 is the representation of a selected analyzer object, that the user sees on a screen. Examples of analyzer objects are an event parser, a property analyzer, and a counter.
  • the analyzer manager 23 maintains references to all analyzer objects currently being used, to describe a coverage metric. Further, the analyzer manager 23 provides common services that are required by all analyzers.
  • the analyzer manager 23 is the object that facilitates and manages the wiring process overall.
  • the graphic analyzer manager 20 maintains references to all graphic analyzers that correspond to the analyzer objects currently being used to describe a selected coverage metric.
  • the graphic analyzer manager 20 determines which graphic analyzer has been clicked on by a user. Additionally, the graphic analyzer manager 20 initiates wiring activities according to the present invention.
  • the wire object 24 is responsible for the actual wiring of first and second analyzers selected for interconnection.
  • the wire object 24 maintains references to its source and target analyzers.
  • the wiring system 19 of the present invention is built as an object-oriented framework.
  • the interfaces between the objects described above define the wiring process according to the present invention.
  • the functional coverage tool can be expanded as required. All that is required of the descended classes is that they provide a common function call interface.
  • the descended classes may perform different operations on the event object stream, and they can customize the wiring process in different manners via the hook functions provided in the wiring process.
  • the wiring system according to the present invention is also modular. According to the present invention, wiring activity is isolated within the wire object. By modifying the wire object 24 according to the present invention, the system is reusable for other applications that are not related to the functional coverage application described herein.
  • a GUI system 3 is provided for a functional coverage tool using a plurality of analyzer objects and a generalized wiring mechanism, which allows a user to operatively interconnect selected analyzer components together in order to operate a user-friendly GUI functional coverage tool according to the present invention.
  • FIG. 2 there is shown a block diagram of a graphical user interface 103 including a toolbar 104 and a specification area 105 , according to one embodiment of the present invention.
  • the toolbar 104 includes a plurality of elements for use in connection with the present invention, including but not limited to an event source 106 , an attribute comparator 107 , an AND gate 108 , and a multi-digit counter 109 .
  • the specification area 105 of the graphical user interface 103 includes particular instances of an event source 116 , first and second attribute comparators 117 and 127 , an AND gate 118 , and a counter 119 . These instances are interconnected according to one embodiment of the present invention, as described below.
  • the event source instance 116 and the first and second attribute comparator instances 117 and 127 are linked with each other by a graphic wire 136 . Further, first and second attribute comparator instances 117 and 127 are linked with graphic wires 137 and 147 to the AND instance 118 , which in turn is linked by a graphic wire 148 to three-digit counter instance 119 .
  • the user has chosen an event type for creation of a specific event source 116 that detects exclusively a particular kind of event.
  • the output of the event source 116 is furthermore fed to the input of respective first and second attribute comparators, 117 and 127 .
  • attribute comparators 117 , 127 are configured, according to one embodiment of the present invention, to detect events having particular attributes that are equal to a user-specified value.
  • Each of the attribute comparators 117 , 127 is attached at its output connection to a common logical AND gate 118 at corresponding ones of its inputs.
  • an event is detected having attributes which satisfy the restrictions established by the first and second attribute comparators 117 , 127 .
  • the AND gate 118 passes the event object to the counter object 118 , causing the counter 118 to increment its count.
  • the user is able to determine the number of events detected which have attributes satisfying the particular user-defined coverage metric which is of interest.
  • FIG. 3 is a block diagram of a particular test environment 159 according to the present invention.
  • the test environment includes a coverage tool 163 in a selected verification environment, according to one embodiment of the present invention.
  • the test environment 159 includes a selected device under test (DUT) 161 , a functional model 162 of the DUT 161 , a coverage tool 163 designed to evaluate the DUT 161 based upon evaluations performed with the functional model 162 , and a storage structure 164 connected to communicate with the functional model 164 and the coverage tool 163 .
  • the DUT 161 is particularly configured to have respective inputs 171 and outputs 172 for operation in a simulation domain, according to the present invention.
  • the functional model 162 is further connected to the coverage tool 163 to enable reception of an event object stream, and the storage structure 164 is in turn also connected to the coverage tool 163 , in a verification domain.
  • An event object stream passes from the functional model 162 to a selected coverage tool 163 according to the present invention.
  • the coverage tool 163 is able to receive event objects in a stream from the functional model 162 or from event objects from storage structure 164 , according to the present invention.
  • the functional model 162 particularly receives information from inputs 171 and outputs 172 , which are disposed in the simulation domain. In this manner, the relationship of the coverage tool to the verification environment is set forth clearly and explicitly.
  • a functional model of the DUT 161 is established for functional verification and is able to communicate the outputs and inputs of the DUT 161 in simulation.
  • the functional model 162 provides for creation of an output stream of event objects, as it detects different events occurring in the DUT 161 .
  • This stream of event objects from the functional model 162 is used by the coverage tool 163 to analyze the completeness of particular tests being simulated.
  • the stream of events from the DUT 161 is fed directly to the coverage tool, according to one embodiment of the present invention.
  • the event object stream from the DUT 161 is provided not immediately, but at a later time to the coverage tool 163 , after the event objects have first been serialized into a storage location 164 . Then, at a later time, the event objects are reconstructed in a stream for transportation to the coverage tool, for subsequent processing by the coverage tool at a later time when it is desired to be done, by the user.
  • FIG. 4 is a block diagram of a test environment 169 having a selected device under test (DUT) 161 within a simulation domain, and further including a coverage tool 163 and a storage mechanism 164 in a verification environment in communication with the simulation domain, according to another embodiment of the present invention.
  • the test environment 169 includes the DUT 161 , a coverage tool 163 receiving elements of an event object stream from the DUT 161 , and a storage structure 164 configured to receive elements of an event object stream from the DUT 161 .
  • the DUT 161 is directly connected at its output to the coverage tool 163 and the storage structure 164 , but the coverage tool 163 is configured to receive events from either one or the other, but not from both of the event object stream sources, i.e., the DUT 161 and the storage structure 164 .
  • storage structure 164 is connected to the coverage tool 163 to provide event objects from storage to the coverage tool, in a verification domain.
  • An event object stream flows from the DUT 161 to the coverage tool 163 .
  • the coverage tool 163 either receives event objects from the DUT 161 , or it receives event objects from storage structure 164 , according to the present invention. In this manner, the role of the coverage tool 163 to the verification environment is set forth clearly.
  • the stream of event objects originating from the DUT 161 is used by selected coverage tools to permit analysis of the coverage and completeness of particular tests being simulated.
  • the stream of events from the DUT 161 is accordingly fed directly to the coverage tool 163 , according to one embodiment of the present invention.
  • the stream of event objects is provided at a later time after the events have first been serialized at a selected storage location, to enable reading of the objects by the coverage tool 163 at a later time.
  • the HDL code has been instrumented to create event objects directly.
  • FIG. 5 is a data flow diagram describing operation of a coverage tool method 170 according to one embodiment of the present invention.
  • the coverage tool method 170 according to the present invention includes receipt of an event object stream by an analysis object manager 171 which is configured to communicate elements of an event object stream to at least one of a plurality of event analyzers 172 - 175 , i.e., first through fourth event analyzers, either directly or indirectly.
  • the output of a first event analyzer 172 is connected to the input of second and third event analyzers, respectively 174 and 175 .
  • second and third event analyzers 174 , and 175 are provided to a AND gate 176 which is connected at its output to the input of a counter mechanism 177 .
  • the event objects of the event object stream according to the present invention are particularly fed into the analysis object manager 171 which in turn feeds these event objects into each of its associated top level event analyzers, i.e., first and fourth event analyzers 172 , 173 .
  • the top level event analyzers are responsible for searching for specific types of events and are called event sources. After detecting the correct event types according to predetermined criteria, these event sources pass the detected event objects down to one or more additional analysis objects, e.g., second and third event analyzers 174 and 175 .
  • analysis objects 174 , 175 may be connected in any manner specified by the user to create composite coverage analysis objects.
  • analyzers there are two additional analyzers, which are set-up to detect specific event attributes.
  • the analyzers sense the arrival of the specified attribute, they call a function in an attached AND block 176 . If the AND block 176 receives both of the indicated function calls on either of its inputs, it calls an applicable function which is preconfigured to increment the attached counter object 177 , as in indication for example of the level of coverage applied to a device under test.
  • An event object includes a predetermined code block.
  • An event object also includes a data block which includes an attribute array sub-block.
  • the block code of an event object is used to specify which portion of the design a particular event is from in a selected devices under test (DUT), where one type of event is detected in more than one block of a circuit in a DUT.
  • DUT devices under test
  • a particular device under test may include first and second independent serial ports for external input of data or signals. Because the serial ports of the DUT are in this case identical, the events being monitored for the indicated serial ports are substantially identical. If the user wishes to specify independent coverage metrics for each of the two ports, but not for both of the ports, particular block codes enable user specification of each applicable serial port.
  • the event code is thus used to distinguish particular types of events from each other, such as for example read bus cycle events, write bus cycle events, and interrupt events, from each other.
  • the attribute array thus contains applicable information for a range of particular event attributes, such as for example without limitation time of occurrence, bus cycle address, or interrupt vector number, as the case may be.
  • the structure of such event objects is generic, and this enables description of substantially all device events of interest.
  • the event objects can moreover be serialized for persistent storage according to the present invention, to enable post-simulation coverage analysis.
  • An interface is provided according to the present invention that permits user query of event type for information about the name of the event, the source block of the event, and the available attributes of the event, as desired.
  • the user uses this interface to determine what information is available for a particular device under test, to create coverage metrics.
  • English language descriptions of event objects are stored in a discrete object called an EventInfo object.
  • Such event objects are stored in the EventInfo object using numeric codes that reference various English description tables which are stored in the EventInfo object.
  • the storage intensive English language descriptions are stored only once in a coverage tool, and the event objects contain only compact numeric codes. Thus, storage space is considerably reduced with the present invention.
  • FIG. 6 is a block diagram of an EventInfo object architecture 180 according to the present invention.
  • the event architecture 180 includes first through sixth objects 181 - 186 .
  • first and second objects 181 and 182 respectively named EventHT and AttributeGroupHT.
  • the first object 181 EventHT is an event hashtable having the label EventHT.
  • the EventHT hashtable particularly holds the names of events that are detectable in the device under test, as well as containing associated event numberic codes for each detectable event name.
  • the second object 182 is a hashtable labled as the AttributeGroupHT. This object AttributeGroupHT is keyed by a given event object's English name.
  • the AttributeGroupHT contains one hashtable for each event that has one or more aattributes and can be detected by the device under test.
  • the contained hashtables are called AtributeSets and contain English language descriptions of the attributes described by particular event objects.
  • the AttibuteSet hashtable is subject to query for lists of English names of attributes corresponding to particular event objects.
  • the attribute object 183 contains three pieces of information about each attribute, according to one embodiment of the present invention. As a first piece of information, the attribute object 183 returns an index that the given attribute's value is located at in the event objects's attribute array.
  • the Attribute object holds a hashtable that relates the English description of an attribute's possible values to the numeric codes used to store each possible value in the event object.
  • the Attribute object contains minimum and maximum allowed values for the particular attribute.
  • the table below demonstrates the utility of the English language translation functionality according to the present invention, in terms of an attribute value coding example.
  • an audio serial input port takes an input from several different sources, based upon device settings with particular variable settings as shown in the table indicated. The user specifies the sources for which coverage metrics are desired, according to one embodiment of the present invention.
  • serial Port Source Numeric Code Compressed Data Interface 0 Digital Audio Interface 1 S/PDIF 2
  • FIG. 7 is a block diagram of the input structure of an analysis object 90 used to define coverage metrics to be executed on the event object stream created by a given simulation or set of simulations, according to one embodiment of the present invention.
  • the analysis object 90 has a multiple input mapping interface between first and second function groups respectively 91 and 92 .
  • the first function group includes a single function, which is an EventInput function
  • the second function group 92 includes a plurality of analysis functions.
  • the EventInput Function includes a switch function( ) and a plurality of numbered cases corresponding to ordered ones of the analysis functions in the second function group 92 .
  • Analysis objects receive event objects as inputs. Operations are performed according to the present invention using the event codes and attribute values of the event objects.
  • Event objects are either dropped, passed downstream to additional analysis objects, or a display is updated indicating the result of analysis.
  • Event objects are passed to analysis objects using an interface configured according to the present invention.
  • the event object itself and an associated numeric code are passed to the analysis object.
  • a function “EventInput” exists in each analysis object, according to one embodiment of the present invention.
  • an arbitrary number of inputs are added to an analysis object at run-time by using a dynamic data structure that allocates additional storage for each input. Further, each input can be operated on in a similar fashion by looping the inputs.
  • This type of mapping arrangement facilitates building of analysis objects with multi-inputs, such as such as AND or OR logical gates or structures, for example.
  • FIG. 8 is a diagram of a registration step of event transmission 1100 according to the present invention, operating in first and second domains, a user interface domain and an associated functional domain.
  • event transmission 1100 includes operation of first and second objects respectively 1101 and 1103 in the user interface domain, and first and second objects respectively 1102 and 1104 operating in the functional domain.
  • Objects 1101 and 1102 are the driving analysis objects and objects 1103 and 1104 are the driven analysis objects.
  • the user interface domain the user draws a connection from the driving to the driven analysis objects respectively 1101 and 1103 .
  • the driven analysis object 1104 sends a message RequestEvent(this,1); to the driving analysis object 1102 .
  • FIG. 9 is a diagram of a request storage step of event transmission 1110 according to the present invention.
  • event transmission 110 includes first and second objects respectively 1111 and 1112 .
  • Object 1101 is the driving analysis object which references an associated request storage list in object 1112 .
  • the driving analysis object 1111 responds to a message RequestEvent(this,1). The message is sent to the driving analysis object 1102 , and this causes it to request storage in object 1112 .
  • Object 1112 contains previous references, ref 1 associated with index 1 , and ref 2 associated with index 2 , as well as reference “this” associated with “1”.
  • FIG. 10 is a diagram of an event object transmission step with respect to event transmission operation 1120 according to the present invention.
  • event transmission operation 1120 includes. first and second objects respectively 1121 and 1122 .
  • Object 1121 is the driving analysis object referencing a request storage list in object 1122 .
  • the driving analysis object 1121 responds to a message InputEvent(evObj,iInd) from upstream.
  • the message is sent to the driving analysis object 1122 , which upon receipt causes it to request storage in object 1122 .
  • Object 1122 contains multiple references, ref 1 associated with index 1 , ref 2 associated with index 2 , as well as ref 3 associated with Index 3 .
  • the object 1122 produces messages to downstream analysis objects for example, InputEvent(evObj,iInd 1 ), InputEvent(evObj,iInd 2 ), and InputEvent(evObj,iInd 3 ).
  • the driving object uses this index when calling the EventInput function on the driving analysis object.
  • the driving analysis object stores this index and a pointer or reference to the driven analysis object in a list.
  • the driving analysis object receives an event object, it first performs operations using the event object's event code and attribute values, to determine whether the event object should be passed downstream. If the event object is to be passed downstream to further driven analysis objects, then the current analysis object calls the EventInput function of every analysis object stored in its list.
  • the driving analysis object passes both the event object and the requested input index stored, when the driven analysis objects are attached.
  • FIG. 11 is a block diagram of a computer system for implementing operation of a computer program product according to the present invention.
  • the computer system 210 which implements the computer program product according to the present invention includes a random access memory (RAM) 211 , monitor 212 , a printer 213 , a disk drive 214 , a compact disk (CD) read only memory (ROM) drive 215 , a microprocessor 216 , a read only memory (ROM) 217 , a keyboard controller 218 , a hard drive 219 , a network interface 220 , a keyboard 221 , and first and second buses 222 and 223 .
  • First bus 222 connects microprocessor 216 with RAM 211 and ROM 217 .
  • Second bus 223 connects microprocessor 216 with monitor 212 , printer 213 , disk drive 214 , CD ROM drive 215 , hard drive 219 , and network interface 220 .
  • Microprocessor 218 is additionally connected to a keyboard controller 218 which in turn is connected to a keyboard for communication with a user.
  • Disk drive 214 is configured to read and write information with respect to a disk medium 224 on which the information is stored, and from which it can be read.
  • CD ROM drive 215 is configured to read information with respect to a CD ROM medium 225 on which the information is stored, and from which it can be read.
  • Computer program products according to the present invention can be embodied on the disk medium 224 or the CD ROM medium 225 or the like (such as an optical or magnetic disk, for example).
  • a functional coverage tool includes without limitation, an intuitive user interface that allows the user to specify coverage metrics in terms of objects that they are already familiar with,(e.g. logic gates, comparators, counters, histograms, for example without limitation).
  • the tool is portable between different devices under test (i.e., DUTs).
  • DUTs devices under test
  • To port the tool a file is created that describes the events fired by the DUT, and the tool reads in the file, yielding the benefit of portability.
  • the tool further allows coverage metrics with respect to device configuration to be specified either before or after the simulation of the DUT is run. A unique treatment of DUT configuration changes allows this functionality.
  • the tool uses event history files that are significantly smaller that those used by other tools, because of the aforementioned unique treatment of DUT configuration changes.
  • the tool uses event history files that are significantly smaller that those used by other tools, because the functional events are stored as numeric values rather than text strings.
  • the tool is easily expandable (in analyzers and presentation objects), without modification because the interface between analyzers is well defined.
  • the tool allows users to specify coverage metrics after a simulation without rerunning the simulation, thus saving valuable simulation time.

Abstract

Coverage metrics are expressed with an intuitive graphical interface based upon data flow. Coverage analysis and presentation objects are integrated to produce coverage results which enable device functionality in a device under test to be modeled as objects, subject to event occurrence. Event objects are introspected at run-time, allowing the user to determine the event object's attributes with specification of coverage metrics subject to a selected combination of the event object's attributes. The event objects are serialized into permanent storage, allowing the user to specify and execute new coverage metrics at any time after simulation. Operations used to describe coverage metrics are modeled as analysis objects. Such analysis objects accept event objects as inputs, using a predetermined, well-defined interface. The combination of event objects and analysis objects allows coverage metrics to be specified in a simple data flow manner. With such a coverage metric, the user attaches or wires (metaphorically) the analysis objects together in a visual builder environment. Using the analysis objects, the user specifies desired coverage metrics, such as coverage of sequences of events and/or coverage of events that occur during the same time window of a simulation. The display functionality of the coverage tool is expandable because the presentation objects use the same event object interface as the analysis operator objects. Coverage metrics are subject to specification either before or after event occurrence. The user specifies coverage metrics using an intuitive graphical interface based upon data flow, without any specific programming language skills being necessary. Functional events in the device under test are treated as event objects. Each event object may be passed to selected analysis tools chosen by the user, such as analyzers, logic gates, and coincidence counters.

Description

    CROSS-REFERENCE TO RELATED PATENT APPLICATIONS
  • This patent application is related to the following additional patent applications which are hereby expressly referenced and incorporated hereinto in their entirety by reference: patent application Ser. No. ______, patent application Ser. No. ______ and patent application Ser. No. ______, having respective titles “Systems And Methods For Generating Interchangable Device Event Description Databases For Use in a Functional Coverage Tool That Is Portable Between Designs”, “Systems And Methods For Manipulating Configuration Events in a Dataflow Functional Coverage Tool For a Verification Test Suite”, and “Systems and Methods for Allowing Graphical User Connection of Coverage Analyzer Operators in a Data Flow Functional Coverage Tool”, with the same filing date as the present patent application, and the same inventorship.[0001]
  • BACKGROUND OF THE INVENTION
  • 1. Field of the Invention [0002]
  • This application relates to digital design methods and systems and more particularly to methods and systems for gauging the completeness of simulation test suites for digital design operations. [0003]
  • 2. Description of Related Art [0004]
  • A substantial portion of the digital semiconductor development cycle is devoted to pre-production verification. With increasingly complex circuits, it is more and more difficult to gauge when verification should be considered to be completed. Verification completeness is subject to assessment by measuring the amount of coverage provided by test suites run against a particular design. A number of tools now perform this daunting task. Currently, these tools include code coverage tools and functional coverage tools. Code coverage tools describe little about the device functionality. The code in an HDL description can be executed without simulation of substantial device functional features. Accordingly, only a coarse measure of test suite effectiveness is provided with the HDL description code. On the other side, while functional test tools provide a more complete view of the coverage of the actual device functionality, at present the functional tools are primitive and limited in many ways. The tool output in particular instances is no more than a simple bar graph of predetermined event counters. Further, particular tools require the user either to instrument the entire HDL description of the device under test, at a considerable risk of error, or to write coverage code in a separate language, which on occasion is non-intuitive and significantly time intensive. One current tool containing a relational data base for analysis of event traces originating from a particular simulation resulted in each recorded event in the device under test being stored as a row in a database table. Thus, the user is forced to evaluate device events through a database oriented filter utilizing, in one instance, an SQL query language process. As a result, the user is limited in coverage ability by the limitations of SQL and the user familiarity with the language. In any case, the use of the SQL tool is limited to post-processing of information. [0005]
  • Gauging the completeness of simulation test suites for digital designs is further a difficult technical and practical task. Current tools that measure the coverage provided by a particular test suite have proven inadequate for a variety of reasons. In particular, current code coverage tools do not accurately measure the amount of device functionality exercised. For example, HDL code lines executed do not correlate directly to functionality exercised. In some instances, current functional coverage tools are programmed with specific coverage metrics prior to simulation and are effective merely for providing simple counts of activity and simplistic bar graph outputs. New coverage metrics then require the user to reprogram and rerun the simulation. Further, some functional coverage tools currently do not allow post simulation specification of coverage metrics. In some tools, the user is required to know a specialized query language which is unsuited for describing desired coverage metrics. Accordingly, it is desirable to develop new coverage tool methodologies and systems which overcome the shortcomings of currently available tools. [0006]
  • SUMMARY OF THE INVENTION
  • According to one embodiment of the present invention, coverage metrics are subject to user specification before or after simulation operation. Further, the coverage metrics are subject to user specification with an intuitive graphical interface based upon data flow, without specific or arcane program language knowledge being required for implementation. According to the present invention, additional coverage analysis and presentation objects are conveniently integrated into an expanded system functionality. No original system modifications are required for the implemented expansions. Further, instrumentation of device under test hardware design language code is not required for operation according to the present invention. Additionally, according to the present invention, coverage results are provided which enable device functionality to be understood. In particular under the present invention, events in a device under test are modeled as objects. Each event object has a name and a number of attributes that describe the event. The event objects are introspected at run-time, allowing the user to determine the event object's attributes with specification of coverage metrics subject to a selected combination of the event object's attributes. The event objects are serialized according to one embodiment of the present invention, to permanent storage, allowing the user to specify and execute new coverage metrics at any time after simulation. Operations used to describe coverage metrics are modeled as analysis objects according to the present invention. Such analysis objects accept event objects as inputs, using a predetermined, well-defined interface. The combination of event objects and analysis objects according to the present invention allows coverage metrics to be specified in a simple data flow manner. Event objects are particularly treated as a data stream that is operated on by predetermined analysis objects. The analysis objects are modeled after user-familiar objects such as for example, comparators, logic gates, counters, and analyzers. With such a coverage metric, the user attaches or wires (metaphorically) the analysis objects together in a visual builder environment according to the present invention. Using the analysis objects, the user specifies desired coverage metrics which would otherwise be difficult or impossible otherwise to implement, such as coverage of sequences of events and/or coverage of events that occur during the same time window of a simulation. Because analysis objects accept and pass on event objects through a standardized interface, the addition of new analysis objects can be accomplished without modification of the coverage tool utilized. The resulting output of specialized coverage metrics is displayed with a special class of analysis objects known currently as presentation objects. The display functionality of the coverage tool is expandable because the presentation objects use the same event object interface as the analysis operator objects. Analysis objects and their interconnections are serialized into selected storage locations according to the present invention, permitting the user to save desired coverage metrics and to enable measurement of the completeness of future test suites. Further according to the present invention, coverage metrics are subject to specification either before or after device simulation occurrence. Additionally, the user specifies coverage metrics using an intuitive graphical interface based upon data flow, without any specific programming language skills being necessary. Moreover, the system according to the present invention is easily expandable, by adding new coverage analysis and presentation objects. Such expansions require no original system alterations. Implementation according to the present invention is architecturally independent, meaning that a change in a device under test does not require a change in the coverage tool. According to the present invention, the functional behavior of a given device under test is described in a series of functional events which indicate that a particular occurrence has transpired within the device in which a functional portion of the device has been exercised, and which indicate what the settings of the device related to a particular functionality were when the event occurred. According to the present invention, functional events in the device are treated as event objects. Each functional event according to the present invention has a name and zero or more attributes with which it is associated. An example functional event according to the present invention is a bus cycle on a digital device. For a bus cycle event example attributes are read or write type, simulation time at initiation, bus cycle address, and data transferred by cycle operation. The system according to the present invention is versatile and intuitive as a coverage tool in that it treats device events detected as an independent object with name and attributes. Each event object may be passed to selected analysis tools chosen by the user, such as analyzers, logic gates, and coincidence counters. According to one embodiment of the present invention, event objects are created by functions added to the original HDL code describing the device under test, despite possibility of errors and vulnerability to device architecture changes. Such a device change requires code adaptation. According to another embodiment of the present invention, a functional model of the device under test is constructed to verify the device behavior and to produce functional event objects. The functional model according to the present invention accordingly receives the states of inputs and outputs to the device and the modeled behavior determines when device output events occur in a simulation. The functional model creates an event object and serializes it to selected storage, without any necessity to instrument HDL files describing the device. The coverage of a selected test suite is thus capable of analysis for functional coverage, simply, intuitively, and powerfully, as well as dynamically at runtime by specification of coverage metrics. The invention also has an extensible architecture that allows the expansion of the tool without core system modifications. The tool according to the present invention is independent of the device under test, allowing it to be used for selected digital designs according to user selection. According to one embodiment, the tool of the present invention is unaffected by changes in the underlying device under test that do not directly change the functionality of the device, such as when architectural changes are made to enhance performance of the device under test, such as for example net name changes. [0007]
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a block diagram of the architecture of a generalized wiring system for a data flow functional coverage tool, according to the present invention; [0008]
  • FIG. 2 is a block diagram of a graphical user interface, including a toolbar and a specification area, according to one embodiment of the present invention; [0009]
  • FIG. 3 is a block diagram of a particular test environment, according to the present invention; [0010]
  • FIG. 4 is a block diagram of a test environment having a selected device under test (DUT) within a simulation domain, and further including a coverage tool and a storage mechanism in a verification environment in communication with the simulation domain, according to another embodiment of the present invention; [0011]
  • FIG. 5 is a data flow diagram describing operation of a coverage tool method, according to one embodiment of the present invention; [0012]
  • FIG. 6 is a block diagram of an EventInfo object architecture according to the present invention; [0013]
  • FIG. 7 is a block diagram of an analysis object used to define coverage metrics to be executed on the event object stream created by a given simulation or set of simulations, according to one embodiment of the present invention; [0014]
  • FIG. 8 is a diagram of a registration step of event transmission, according to the present invention, operating in first and second domains, a user interface domain and an associated functional domain; [0015]
  • FIG. 9 is a diagram of a request storage step of event transmission according to the present invention; [0016]
  • FIG. 10 is a diagram of an event object transmission step with respect to event transmission operation [0017] 120 according to the present invention; and
  • FIG. 11 is a block diagram of a computer system for implementing operation of a computer program product according to the present invention. [0018]
  • DETAILED DESCRIPTION OF A PREFERRED MODE
  • Referring now to FIG. 1, there is shown a block diagram of the architecture of a [0019] generalized wiring system 19 for a data flow functional coverage tool, according to the present invention. The wiring system 19 is used in the graphical user interface 3 to implement a data flow functional coverage tool according to the present invention, which is conveniently capable of adaptation or modification according to the needs of the design test engineer. The wiring system 19 includes the following objects which are part of the architecture of the present invention. In particular, the wiring system 19 includes a graphic analyzer manager 20, a plurality of graphic analyzers 21, any number of analyzers 22, an analyzer manager 23, any number of wire objects 24, and any number of graphic wire objects 25. The graphic analyzer manager 23 includes a plurality of graphic analyzers 30-31. The graphic analyzer 21 includes a reference to a parent analyzer 34. The analyzer object 22 includes reference to a input wires 35 and output wires 36. The wire object 24 includes reference to a source analyzer 38 and a target analyzer 39. Analyzer objects 22 are the core of a functional coverage tool according to the present invention. The analyzer objects 22 perform the operations that execute specified coverage metrics on a selected event stream specified by a functional coverage tool. The graphic analyzer 21 is the representation of a selected analyzer object, that the user sees on a screen. Examples of analyzer objects are an event parser, a property analyzer, and a counter. The analyzer manager 23 maintains references to all analyzer objects currently being used, to describe a coverage metric. Further, the analyzer manager 23 provides common services that are required by all analyzers. The analyzer manager 23 is the object that facilitates and manages the wiring process overall. The graphic analyzer manager 20 maintains references to all graphic analyzers that correspond to the analyzer objects currently being used to describe a selected coverage metric. The graphic analyzer manager 20 determines which graphic analyzer has been clicked on by a user. Additionally, the graphic analyzer manager 20 initiates wiring activities according to the present invention. The wire object 24 is responsible for the actual wiring of first and second analyzers selected for interconnection. The wire object 24 maintains references to its source and target analyzers. The wiring system 19 of the present invention is built as an object-oriented framework. The interfaces between the objects described above define the wiring process according to the present invention. By descending classes from the base objects described here, (specifically, the analyzer and graphic analyzer base classes), the functional coverage tool can be expanded as required. All that is required of the descended classes is that they provide a common function call interface. The descended classes may perform different operations on the event object stream, and they can customize the wiring process in different manners via the hook functions provided in the wiring process. The wiring system according to the present invention is also modular. According to the present invention, wiring activity is isolated within the wire object. By modifying the wire object 24 according to the present invention, the system is reusable for other applications that are not related to the functional coverage application described herein. Thus, a GUI system 3 is provided for a functional coverage tool using a plurality of analyzer objects and a generalized wiring mechanism, which allows a user to operatively interconnect selected analyzer components together in order to operate a user-friendly GUI functional coverage tool according to the present invention.
  • Referring now to FIG. 2, there is shown a block diagram of a [0020] graphical user interface 103 including a toolbar 104 and a specification area 105, according to one embodiment of the present invention. The toolbar 104 includes a plurality of elements for use in connection with the present invention, including but not limited to an event source 106, an attribute comparator 107, an AND gate 108, and a multi-digit counter 109. The specification area 105 of the graphical user interface 103 includes particular instances of an event source 116, first and second attribute comparators 117 and 127, an AND gate 118, and a counter 119. These instances are interconnected according to one embodiment of the present invention, as described below. In particular, the event source instance 116 and the first and second attribute comparator instances 117 and 127 are linked with each other by a graphic wire 136. Further, first and second attribute comparator instances 117 and 127 are linked with graphic wires 137 and 147 to the AND instance 118, which in turn is linked by a graphic wire 148 to three-digit counter instance 119. In the example shown, the user has chosen an event type for creation of a specific event source 116 that detects exclusively a particular kind of event. The output of the event source 116 is furthermore fed to the input of respective first and second attribute comparators, 117 and 127. These attribute comparators 117, 127 are configured, according to one embodiment of the present invention, to detect events having particular attributes that are equal to a user-specified value. Each of the attribute comparators 117, 127 is attached at its output connection to a common logical AND gate 118 at corresponding ones of its inputs. According to one embodiment of the present invention, an event is detected having attributes which satisfy the restrictions established by the first and second attribute comparators 117, 127. When such an event is detected, the AND gate 118 passes the event object to the counter object 118, causing the counter 118 to increment its count. Thus, according to the present invention, the user is able to determine the number of events detected which have attributes satisfying the particular user-defined coverage metric which is of interest.
  • FIG. 3 is a block diagram of a [0021] particular test environment 159 according to the present invention. The test environment includes a coverage tool 163 in a selected verification environment, according to one embodiment of the present invention. In particular, the test environment 159 includes a selected device under test (DUT) 161, a functional model 162 of the DUT 161, a coverage tool 163 designed to evaluate the DUT 161 based upon evaluations performed with the functional model 162, and a storage structure 164 connected to communicate with the functional model 164 and the coverage tool 163. The DUT 161 is particularly configured to have respective inputs 171 and outputs 172 for operation in a simulation domain, according to the present invention. The functional model 162 is further connected to the coverage tool 163 to enable reception of an event object stream, and the storage structure 164 is in turn also connected to the coverage tool 163, in a verification domain. An event object stream passes from the functional model 162 to a selected coverage tool 163 according to the present invention. The coverage tool 163 is able to receive event objects in a stream from the functional model 162 or from event objects from storage structure 164, according to the present invention. The functional model 162 particularly receives information from inputs 171 and outputs 172, which are disposed in the simulation domain. In this manner, the relationship of the coverage tool to the verification environment is set forth clearly and explicitly. Thus, a functional model of the DUT 161 is established for functional verification and is able to communicate the outputs and inputs of the DUT 161 in simulation. The functional model 162 provides for creation of an output stream of event objects, as it detects different events occurring in the DUT 161. This stream of event objects from the functional model 162 is used by the coverage tool 163 to analyze the completeness of particular tests being simulated. The stream of events from the DUT 161 is fed directly to the coverage tool, according to one embodiment of the present invention. However, according to another embodiment, the event object stream from the DUT 161 is provided not immediately, but at a later time to the coverage tool 163, after the event objects have first been serialized into a storage location 164. Then, at a later time, the event objects are reconstructed in a stream for transportation to the coverage tool, for subsequent processing by the coverage tool at a later time when it is desired to be done, by the user.
  • FIG. 4 is a block diagram of a [0022] test environment 169 having a selected device under test (DUT) 161 within a simulation domain, and further including a coverage tool 163 and a storage mechanism 164 in a verification environment in communication with the simulation domain, according to another embodiment of the present invention. In particular, the test environment 169 includes the DUT 161, a coverage tool 163 receiving elements of an event object stream from the DUT 161, and a storage structure 164 configured to receive elements of an event object stream from the DUT 161. Further, the DUT 161 is directly connected at its output to the coverage tool 163 and the storage structure 164, but the coverage tool 163 is configured to receive events from either one or the other, but not from both of the event object stream sources, i.e., the DUT 161 and the storage structure 164. In particular, storage structure 164 is connected to the coverage tool 163 to provide event objects from storage to the coverage tool, in a verification domain. An event object stream flows from the DUT 161 to the coverage tool 163. Moreover, the coverage tool 163 either receives event objects from the DUT 161, or it receives event objects from storage structure 164, according to the present invention. In this manner, the role of the coverage tool 163 to the verification environment is set forth clearly. The stream of event objects originating from the DUT 161 is used by selected coverage tools to permit analysis of the coverage and completeness of particular tests being simulated. The stream of events from the DUT 161 is accordingly fed directly to the coverage tool 163, according to one embodiment of the present invention. According to another embodiment, the stream of event objects is provided at a later time after the events have first been serialized at a selected storage location, to enable reading of the objects by the coverage tool 163 at a later time. According to this embodiment of the present invention, the HDL code has been instrumented to create event objects directly.
  • FIG. 5 is a data flow diagram describing operation of a [0023] coverage tool method 170 according to one embodiment of the present invention. In particular, the coverage tool method 170 according to the present invention includes receipt of an event object stream by an analysis object manager 171 which is configured to communicate elements of an event object stream to at least one of a plurality of event analyzers 172-175, i.e., first through fourth event analyzers, either directly or indirectly. According to one embodiment of the present invention, the output of a first event analyzer 172 is connected to the input of second and third event analyzers, respectively 174 and 175. Further, the outputs of second and third event analyzers 174, and 175, are provided to a AND gate 176 which is connected at its output to the input of a counter mechanism 177. The event objects of the event object stream according to the present invention are particularly fed into the analysis object manager 171 which in turn feeds these event objects into each of its associated top level event analyzers, i.e., first and fourth event analyzers 172, 173. The top level event analyzers are responsible for searching for specific types of events and are called event sources. After detecting the correct event types according to predetermined criteria, these event sources pass the detected event objects down to one or more additional analysis objects, e.g., second and third event analyzers 174 and 175. These analysis objects 174, 175 in turn may be connected in any manner specified by the user to create composite coverage analysis objects. As shown, there are two additional analyzers, which are set-up to detect specific event attributes. Thus, when the analyzers sense the arrival of the specified attribute, they call a function in an attached AND block 176. If the AND block 176 receives both of the indicated function calls on either of its inputs, it calls an applicable function which is preconfigured to increment the attached counter object 177, as in indication for example of the level of coverage applied to a device under test.
  • An event object according to the present invention includes a predetermined code block. An event object also includes a data block which includes an attribute array sub-block. The block code of an event object is used to specify which portion of the design a particular event is from in a selected devices under test (DUT), where one type of event is detected in more than one block of a circuit in a DUT. For example, a particular device under test may include first and second independent serial ports for external input of data or signals. Because the serial ports of the DUT are in this case identical, the events being monitored for the indicated serial ports are substantially identical. If the user wishes to specify independent coverage metrics for each of the two ports, but not for both of the ports, particular block codes enable user specification of each applicable serial port. The event code is thus used to distinguish particular types of events from each other, such as for example read bus cycle events, write bus cycle events, and interrupt events, from each other. The attribute array thus contains applicable information for a range of particular event attributes, such as for example without limitation time of occurrence, bus cycle address, or interrupt vector number, as the case may be. The structure of such event objects is generic, and this enables description of substantially all device events of interest. The event objects can moreover be serialized for persistent storage according to the present invention, to enable post-simulation coverage analysis. An interface is provided according to the present invention that permits user query of event type for information about the name of the event, the source block of the event, and the available attributes of the event, as desired. According to the present invention, the user uses this interface to determine what information is available for a particular device under test, to create coverage metrics. According to one embodiment of the present invention, English language descriptions of event objects are stored in a discrete object called an EventInfo object. Such event objects are stored in the EventInfo object using numeric codes that reference various English description tables which are stored in the EventInfo object. However, the storage intensive English language descriptions are stored only once in a coverage tool, and the event objects contain only compact numeric codes. Thus, storage space is considerably reduced with the present invention. [0024]
  • FIG. 6 is a block diagram of an EventInfo object architecture [0025] 180 according to the present invention. In particular, the event architecture 180 includes first through sixth objects 181-186. At the top level of the EventInfo object 180, there are first and second objects 181 and 182, respectively named EventHT and AttributeGroupHT. The first object 181 EventHT is an event hashtable having the label EventHT. The EventHT hashtable particularly holds the names of events that are detectable in the device under test, as well as containing associated event numberic codes for each detectable event name. The second object 182 is a hashtable labled as the AttributeGroupHT. This object AttributeGroupHT is keyed by a given event object's English name. The AttributeGroupHT contains one hashtable for each event that has one or more aattributes and can be detected by the device under test. The contained hashtables are called AtributeSets and contain English language descriptions of the attributes described by particular event objects. The AttibuteSet hashtable is subject to query for lists of English names of attributes corresponding to particular event objects. The attribute object 183 contains three pieces of information about each attribute, according to one embodiment of the present invention. As a first piece of information, the attribute object 183 returns an index that the given attribute's value is located at in the event objects's attribute array. Second, in the case of attributes that are not numerically described, e.g., the serial input port source described below, the Attribute object holds a hashtable that relates the English description of an attribute's possible values to the numeric codes used to store each possible value in the event object. Finally, in the event of attributes that are numerically representable, the Attribute object contains minimum and maximum allowed values for the particular attribute. The table below demonstrates the utility of the English language translation functionality according to the present invention, in terms of an attribute value coding example. According to this example, an audio serial input port takes an input from several different sources, based upon device settings with particular variable settings as shown in the table indicated. The user specifies the sources for which coverage metrics are desired, according to one embodiment of the present invention. Each communcation cycle that travels through the serial input port is effective for causing an event object to be created and to be serialized to storage. To conserve space in storage, the attribute value is stored as a number shown in the second column. Allowing the user to chose the numbers is useless according to the present invention, but by query of the EventInfo object, the user generates event object names in English definition form, enabling intelligent choices.
    Serial Port Source Numeric Code
    Compressed Data Interface 0
    Digital Audio Interface 1
    S/PDIF 2
  • FIG. 7 is a block diagram of the input structure of an [0026] analysis object 90 used to define coverage metrics to be executed on the event object stream created by a given simulation or set of simulations, according to one embodiment of the present invention. The analysis object 90 has a multiple input mapping interface between first and second function groups respectively 91 and 92. The first function group includes a single function, which is an EventInput function, and the second function group 92 includes a plurality of analysis functions. The EventInput Function includes a switch function( ) and a plurality of numbered cases corresponding to ordered ones of the analysis functions in the second function group 92. Analysis objects receive event objects as inputs. Operations are performed according to the present invention using the event codes and attribute values of the event objects. Dependent on the results of a particular operation, the event objects are either dropped, passed downstream to additional analysis objects, or a display is updated indicating the result of analysis. Event objects are passed to analysis objects using an interface configured according to the present invention. In particular, the event object itself and an associated numeric code are passed to the analysis object. A function “EventInput” exists in each analysis object, according to one embodiment of the present invention. According to yet another embodiment of the present invention, an arbitrary number of inputs are added to an analysis object at run-time by using a dynamic data structure that allocates additional storage for each input. Further, each input can be operated on in a similar fashion by looping the inputs. This type of mapping arrangement facilitates building of analysis objects with multi-inputs, such as such as AND or OR logical gates or structures, for example.
  • FIG. 8 is a diagram of a registration step of event transmission [0027] 1100 according to the present invention, operating in first and second domains, a user interface domain and an associated functional domain. As the user sets up each analysis object, the outputs of analysis objects are effective to drive the inputs of other objects as specified. According to the present invention, the driven analysis object internally passes an input index to the driving analysis object. More particularly, event transmission 1100 includes operation of first and second objects respectively 1101 and 1103 in the user interface domain, and first and second objects respectively 1102 and 1104 operating in the functional domain. Objects 1101 and 1102 are the driving analysis objects and objects 1103 and 1104 are the driven analysis objects. In the user interface domain, the user draws a connection from the driving to the driven analysis objects respectively 1101 and 1103. In the functional domain, the driven analysis object 1104 sends a message RequestEvent(this,1); to the driving analysis object 1102.
  • FIG. 9 is a diagram of a request storage step of [0028] event transmission 1110 according to the present invention. In particular, event transmission 110 includes first and second objects respectively 1111 and 1112. Object 1101 is the driving analysis object which references an associated request storage list in object 1112. In the functional domain, the driving analysis object 1111 responds to a message RequestEvent(this,1). The message is sent to the driving analysis object 1102, and this causes it to request storage in object 1112. Object 1112 contains previous references, ref1 associated with index1, and ref2 associated with index2, as well as reference “this” associated with “1”.
  • FIG. 10 is a diagram of an event object transmission step with respect to [0029] event transmission operation 1120 according to the present invention. In particular, event transmission operation 1120 includes. first and second objects respectively 1121 and 1122. Object 1121 is the driving analysis object referencing a request storage list in object 1122. In the functional domain, the driving analysis object 1121 responds to a message InputEvent(evObj,iInd) from upstream. The message is sent to the driving analysis object 1122, which upon receipt causes it to request storage in object 1122. Object 1122 contains multiple references, ref1 associated with index1, ref2 associated with index2, as well as ref3 associated with Index3. The object 1122 produces messages to downstream analysis objects for example, InputEvent(evObj,iInd1), InputEvent(evObj,iInd2), and InputEvent(evObj,iInd3). The driving object uses this index when calling the EventInput function on the driving analysis object. The driving analysis object stores this index and a pointer or reference to the driven analysis object in a list. When the driving analysis object receives an event object, it first performs operations using the event object's event code and attribute values, to determine whether the event object should be passed downstream. If the event object is to be passed downstream to further driven analysis objects, then the current analysis object calls the EventInput function of every analysis object stored in its list. The driving analysis object passes both the event object and the requested input index stored, when the driven analysis objects are attached.
  • FIG. 11 is a block diagram of a computer system for implementing operation of a computer program product according to the present invention. In particular, the computer system [0030] 210 which implements the computer program product according to the present invention includes a random access memory (RAM) 211, monitor 212, a printer 213, a disk drive 214, a compact disk (CD) read only memory (ROM) drive 215, a microprocessor 216, a read only memory (ROM) 217, a keyboard controller 218, a hard drive 219, a network interface 220, a keyboard 221, and first and second buses 222 and 223. First bus 222 connects microprocessor 216 with RAM 211 and ROM 217. Second bus 223 connects microprocessor 216 with monitor 212, printer 213, disk drive 214, CD ROM drive 215, hard drive 219, and network interface 220. Microprocessor 218 is additionally connected to a keyboard controller 218 which in turn is connected to a keyboard for communication with a user. Disk drive 214 is configured to read and write information with respect to a disk medium 224 on which the information is stored, and from which it can be read. CD ROM drive 215 is configured to read information with respect to a CD ROM medium 225 on which the information is stored, and from which it can be read. Computer program products according to the present invention can be embodied on the disk medium 224 or the CD ROM medium 225 or the like (such as an optical or magnetic disk, for example).
  • In summary, a functional coverage tool according to the present invention includes without limitation, an intuitive user interface that allows the user to specify coverage metrics in terms of objects that they are already familiar with,(e.g. logic gates, comparators, counters, histograms, for example without limitation). The tool is portable between different devices under test (i.e., DUTs). To port the tool, a file is created that describes the events fired by the DUT, and the tool reads in the file, yielding the benefit of portability. The tool further allows coverage metrics with respect to device configuration to be specified either before or after the simulation of the DUT is run. A unique treatment of DUT configuration changes allows this functionality. The tool uses event history files that are significantly smaller that those used by other tools, because of the aforementioned unique treatment of DUT configuration changes. The tool uses event history files that are significantly smaller that those used by other tools, because the functional events are stored as numeric values rather than text strings. The tool is easily expandable (in analyzers and presentation objects), without modification because the interface between analyzers is well defined. The tool allows users to specify coverage metrics after a simulation without rerunning the simulation, thus saving valuable simulation time. [0031]

Claims (15)

What is claimed is:
1. A functional coverage tool comprising:
an analyzer manager configured to produce analyzers upon user request.
2. The functional coverage tool according to claim 1 further including graphical analyzer manager configured to communicate with the analyzer manager, to enable display of graphic analyzers.
3. The functional coverage tool according to claim 2 further including at least a single analyzer.
4. The functional coverage tool according to claim 2 further including at least a single graphic analyzer handling display functions for analyzers which have been created.
5. A method of constructing analyzers comprising:
engaging the analyzer manager with communication of an analyzer class name; and
using the class name to construct an analyzer of that class.
6. A method of constructing a wire object comprising:
constructing first and second analyzers,
creating wire objects to connect said first and second analyzers, and
passing references to enable attachment of the two analyzers to each other.
7. A graphical user interface (GUI) for expressing device under test coverage metrics based upon data flow, comprising:
an expression mechanism for at least a single presentation object which expresses the functionality of a selected device under test, said expression mechanism adapted for receipt of event objects; and
an expression mechanism for at least a single analysis object for operation with the expression mechanism for said at least a single presentation object, said expression mechanism adapted to enable the user to determine event object attributes with specification of coverage metrics subject to a selected combination of the event object's attributes.
8. The GUI according to claim 7 wherein said event objects are serialized into permanent storage, allowing the user to specify and execute new coverage metrics at any time after simulation.
9. The GUI according to claim 7 analysis objects are modeled to describe coverage metrics.
10. The GUI according to claim 7 including combination of event objects and analysis objects, permitting coverage metrics to be specified in a simple data flow manner.
11. The GUI according to claim 7 wherein said coverage metric permits the user to connect analysis objects together in a visual builder environment.
12. The GUI according to claim 7 wherein the user specifies desired coverage metrics, such as coverage of sequences of events and/or coverage of events that occur during the same time window of a simulation, using analysis objects.
13. The GUI according to claim 7 wherein the display functionality of the coverage tool is expandable because the presentation objects use the same event object interface as the analysis operator objects.
14. The GUI according to claim 7 wherein said coverage metrics are subject to specification either before or after device simulation events as object.
15. The GUI according to claim 7 wherein functional events in a device under test are configured as event objects, and event objects are passed to selected analysis tools chosen by the user, such as analyzers, logic gates, and coincidence counters.
US10/352,591 1999-06-30 2003-01-28 Functional coverage analysis systems and methods for verification test suites Abandoned US20030121011A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US10/352,591 US20030121011A1 (en) 1999-06-30 2003-01-28 Functional coverage analysis systems and methods for verification test suites

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US34517199A 1999-06-30 1999-06-30
US10/352,591 US20030121011A1 (en) 1999-06-30 2003-01-28 Functional coverage analysis systems and methods for verification test suites

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US34517199A Division 1999-06-30 1999-06-30

Publications (1)

Publication Number Publication Date
US20030121011A1 true US20030121011A1 (en) 2003-06-26

Family

ID=23353861

Family Applications (1)

Application Number Title Priority Date Filing Date
US10/352,591 Abandoned US20030121011A1 (en) 1999-06-30 2003-01-28 Functional coverage analysis systems and methods for verification test suites

Country Status (1)

Country Link
US (1) US20030121011A1 (en)

Cited By (53)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030131325A1 (en) * 1999-11-30 2003-07-10 Bridges2Silicon, Inc. Method and user interface for debugging an electronic system
US20030191617A1 (en) * 2002-04-04 2003-10-09 Gabele Carol Ivash Method and system for selectively storing and retrieving simulation data utilizing keywords
US20030191621A1 (en) * 2002-04-04 2003-10-09 International Business Machines Corporation Method and system for reducing storage and transmission requirements for simulation results
US20030191869A1 (en) * 2002-04-04 2003-10-09 International Business Machines Corp. C-API instrumentation for HDL models
US20030191620A1 (en) * 2002-04-04 2003-10-09 International Business Machines Corp. Dynamic loading of C-API HDL model instrumentation
US20040025122A1 (en) * 2000-11-28 2004-02-05 Schubert Nils Endric Hardware-based HDL code coverage and design analysis
US20040225973A1 (en) * 2003-05-10 2004-11-11 Johnson Tyler James Post-silicon test coverage verification
US20040243880A1 (en) * 2002-03-28 2004-12-02 International Business Machines Corporation System for facilitating coverage feedback testcase generation reproducibility
US20050010880A1 (en) * 1999-11-30 2005-01-13 Bridges2Silicon, Inc. Method and user interface for debugging an electronic system
US20050149313A1 (en) * 2003-12-31 2005-07-07 International Business Machines Corp. Method and system for selective compilation of instrumentation entities into a simulation model of a digital design
US20060089826A1 (en) * 2004-10-21 2006-04-27 International Business Machines Corporation Method, system and program product for defining and recording minimum and maximum count events of a simulation
US20060089827A1 (en) * 2004-10-21 2006-04-27 International Business Machines Corporation Method, system and program product for defining and recording minium and maximum event counts of a simulation utilizing a high level language
US20060122818A1 (en) * 2004-12-07 2006-06-08 International Business Machines Corporation Method, system and program product for defining and recording threshold-qualified count events of a simulation by testcases
US20060122820A1 (en) * 2004-12-03 2006-06-08 The Mitre Corporation Scripting language for domain-specific modification of a simulation model
US7181708B1 (en) * 2004-08-10 2007-02-20 Cadence Design Systems, Inc. Coverage metric and coverage computation for verification based on design partitions
US20070061121A1 (en) * 2005-09-15 2007-03-15 Gabor Bobok Method, system and program product for selectively removing instrumentation logic from a simulation model
US20070118827A1 (en) * 2005-11-21 2007-05-24 Asifur Rahman Method and apparatus for integrated circuit fault isolation and failure analysis using linked tools cockpit
US20070260441A1 (en) * 2006-05-08 2007-11-08 Gabor Bobok Method, system and program product supporting phase events in a simulation model of a digital system
US20070260443A1 (en) * 2006-05-03 2007-11-08 Gabor Bobok Method, system and program product supporting specification of signals for simulation result viewing
US7373290B2 (en) 2002-04-04 2008-05-13 International Business Machines Corporation Method and system for reducing storage requirements of simulation data via keyword restrictions
US20080172652A1 (en) * 2007-01-15 2008-07-17 Microsoft Corporation Identifying Redundant Test Cases
US20080172580A1 (en) * 2007-01-15 2008-07-17 Microsoft Corporation Collecting and Reporting Code Coverage Data
US20080172655A1 (en) * 2007-01-15 2008-07-17 Microsoft Corporation Saving Code Coverage Data for Analysis
US20080172651A1 (en) * 2007-01-15 2008-07-17 Microsoft Corporation Applying Function Level Ownership to Test Metrics
US20080184204A1 (en) * 2007-01-31 2008-07-31 Microsoft Corporation Dynamic validation using reflection
US20080183458A1 (en) * 2007-01-30 2008-07-31 Gabor Bobok Method, system and program product supporting print events in the simulation of a digital system
CN100428252C (en) * 2006-09-14 2008-10-22 华为技术有限公司 System and method for testing and measuring percentage of coverage of function
US20090112561A1 (en) * 2007-10-31 2009-04-30 Behm Michael L Method, System and Program Product for Defining and Recording Threshold-Qualified Count Events of a Simulation By Testcases
US7536288B2 (en) 2003-12-31 2009-05-19 International Business Machines Corporation Method, system and program product supporting user tracing in a simulator
US20090210746A1 (en) * 2008-02-19 2009-08-20 Ditmyer Bruce J Generating test coverage bin based on simulation result
US20090313607A1 (en) * 2008-06-16 2009-12-17 International Business Machines Corporation Code Coverage Tool
US20100153898A1 (en) * 2008-12-16 2010-06-17 International Business Machines Corporation Model build in the presence of a non-binding reference
US7779374B1 (en) * 2006-09-29 2010-08-17 Breker Verification Systems, Inc. Generating self-checking test cases from reduced case analysis graphs
US7890902B1 (en) * 2007-06-07 2011-02-15 Cadence Design Systems, Inc. Methods and apparatus for merging coverage for multiple verification and design scenarios
CN102073590A (en) * 2011-01-11 2011-05-25 百度在线网络技术(北京)有限公司 Event simulation-based user interface test method and device
US8050902B2 (en) 2007-10-31 2011-11-01 International Business Machines Corporation Reporting temporal information regarding count events of a simulation
US8160857B2 (en) 2008-12-16 2012-04-17 International Business Machines Corporation Selective compilation of a simulation model in view of unavailable higher level signals
US8413088B1 (en) 2007-06-07 2013-04-02 Cadence Design Systems, Inc. Verification plans to merging design verification metrics
US8839203B2 (en) 2011-05-25 2014-09-16 Microsoft Corporation Code coverage-based taint perimeter detection
US9063809B2 (en) 2013-01-15 2015-06-23 International Business Machines Corporation Content space environment representation
US9069647B2 (en) 2013-01-15 2015-06-30 International Business Machines Corporation Logging and profiling content space data and coverage metric self-reporting
US9075544B2 (en) 2013-01-15 2015-07-07 International Business Machines Corporation Integration and user story generation and requirements management
US9081645B2 (en) 2013-01-15 2015-07-14 International Business Machines Corporation Software product licensing based on a content space
US9087155B2 (en) 2013-01-15 2015-07-21 International Business Machines Corporation Automated data collection, computation and reporting of content space coverage metrics for software products
US9111040B2 (en) 2013-01-15 2015-08-18 International Business Machines Corporation Integration of a software content space with test planning and test case generation
US9141379B2 (en) 2013-01-15 2015-09-22 International Business Machines Corporation Automated code coverage measurement and tracking per user story and requirement
US9182945B2 (en) 2011-03-24 2015-11-10 International Business Machines Corporation Automatic generation of user stories for software products via a product content space
US9218161B2 (en) 2013-01-15 2015-12-22 International Business Machines Corporation Embedding a software content space for run-time implementation
US9396342B2 (en) 2013-01-15 2016-07-19 International Business Machines Corporation Role based authorization based on product content space
US9659053B2 (en) 2013-01-15 2017-05-23 International Business Machines Corporation Graphical user interface streamlining implementing a content space
US20170308450A1 (en) * 2016-04-25 2017-10-26 EMC IP Holding Company, LLC Computer-implemented method, computer program product and computing system
US10635766B2 (en) 2016-12-12 2020-04-28 International Business Machines Corporation Simulation employing level-dependent multitype events
CN113075911A (en) * 2021-03-24 2021-07-06 中广核工程有限公司 Automatic verification method and system for nuclear power plant safety-level digital instrument control system software

Citations (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5475695A (en) * 1993-03-19 1995-12-12 Semiconductor Diagnosis & Test Corporation Automatic failure analysis system
US5870088A (en) * 1996-05-09 1999-02-09 National Instruments Corporation System and method for editing a control via direct graphical user interaction
US5905649A (en) * 1996-09-23 1999-05-18 National Instruments Corporation System and method for performing type checking and class propagation of attributes in a graphical data flow program
US5910895A (en) * 1997-06-13 1999-06-08 Teradyne, Inc. Low cost, easy to use automatic test system software
US5963724A (en) * 1996-02-16 1999-10-05 Analogy, Inc. Component-based analog and mixed-signal simulation model development
US5963726A (en) * 1998-03-20 1999-10-05 National Instruments Corporation Instrumentation system and method including an improved driver software architecture
US5987246A (en) * 1997-02-14 1999-11-16 National Instruments Corp. Graphical programming system and method including three-dimensional nodes with pre-defined input and output capabilities
US6064409A (en) * 1993-09-22 2000-05-16 National Instruments Corporation System and method for providing audio probe and debugging features in a graphical data flow program
US6064816A (en) * 1996-09-23 2000-05-16 National Instruments Corporation System and method for performing class propagation and type checking in a graphical automation client
US6108717A (en) * 1995-09-06 2000-08-22 Seiko Epson Corporation Control system using plural objects, a programming method therefor, and a peripheral devices control system
US6236956B1 (en) * 1996-02-16 2001-05-22 Avant! Corporation Component-based analog and mixed-signal simulation model development including newton step manager
US6356858B1 (en) * 1998-04-03 2002-03-12 International Business Machines Corp. Coverage measurement tool for user defined coverage models
US6362839B1 (en) * 1998-09-29 2002-03-26 Rockwell Software Inc. Method and apparatus for displaying mechanical emulation with graphical objects in an object oriented computing environment
US6427223B1 (en) * 1999-04-30 2002-07-30 Synopsys, Inc. Method and apparatus for adaptive verification of circuit designs
US6437805B1 (en) * 1996-09-23 2002-08-20 National Instruments Corporation System and method for accessing object capabilities in a graphical program
US6542166B1 (en) * 1996-05-09 2003-04-01 National Instruments Corporation System and method for editing a control
US6549199B1 (en) * 1999-03-19 2003-04-15 Corel Inc. System and method for adjusting a graphical object
US6560572B1 (en) * 1999-04-15 2003-05-06 Interactive Image Technologies, Ltd. Multi-simulator co-simulation
US6625804B1 (en) * 2000-07-06 2003-09-23 Microsoft Corporation Unified event programming model

Patent Citations (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5475695A (en) * 1993-03-19 1995-12-12 Semiconductor Diagnosis & Test Corporation Automatic failure analysis system
US6064409A (en) * 1993-09-22 2000-05-16 National Instruments Corporation System and method for providing audio probe and debugging features in a graphical data flow program
US6108717A (en) * 1995-09-06 2000-08-22 Seiko Epson Corporation Control system using plural objects, a programming method therefor, and a peripheral devices control system
US6236956B1 (en) * 1996-02-16 2001-05-22 Avant! Corporation Component-based analog and mixed-signal simulation model development including newton step manager
US5963724A (en) * 1996-02-16 1999-10-05 Analogy, Inc. Component-based analog and mixed-signal simulation model development
US5870088A (en) * 1996-05-09 1999-02-09 National Instruments Corporation System and method for editing a control via direct graphical user interaction
US6542166B1 (en) * 1996-05-09 2003-04-01 National Instruments Corporation System and method for editing a control
US6064816A (en) * 1996-09-23 2000-05-16 National Instruments Corporation System and method for performing class propagation and type checking in a graphical automation client
US6437805B1 (en) * 1996-09-23 2002-08-20 National Instruments Corporation System and method for accessing object capabilities in a graphical program
US5905649A (en) * 1996-09-23 1999-05-18 National Instruments Corporation System and method for performing type checking and class propagation of attributes in a graphical data flow program
US5987246A (en) * 1997-02-14 1999-11-16 National Instruments Corp. Graphical programming system and method including three-dimensional nodes with pre-defined input and output capabilities
US5910895A (en) * 1997-06-13 1999-06-08 Teradyne, Inc. Low cost, easy to use automatic test system software
US5963726A (en) * 1998-03-20 1999-10-05 National Instruments Corporation Instrumentation system and method including an improved driver software architecture
US6356858B1 (en) * 1998-04-03 2002-03-12 International Business Machines Corp. Coverage measurement tool for user defined coverage models
US6362839B1 (en) * 1998-09-29 2002-03-26 Rockwell Software Inc. Method and apparatus for displaying mechanical emulation with graphical objects in an object oriented computing environment
US6549199B1 (en) * 1999-03-19 2003-04-15 Corel Inc. System and method for adjusting a graphical object
US6560572B1 (en) * 1999-04-15 2003-05-06 Interactive Image Technologies, Ltd. Multi-simulator co-simulation
US6427223B1 (en) * 1999-04-30 2002-07-30 Synopsys, Inc. Method and apparatus for adaptive verification of circuit designs
US6625804B1 (en) * 2000-07-06 2003-09-23 Microsoft Corporation Unified event programming model

Cited By (84)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050010880A1 (en) * 1999-11-30 2005-01-13 Bridges2Silicon, Inc. Method and user interface for debugging an electronic system
US20030131325A1 (en) * 1999-11-30 2003-07-10 Bridges2Silicon, Inc. Method and user interface for debugging an electronic system
US7356786B2 (en) 1999-11-30 2008-04-08 Synplicity, Inc. Method and user interface for debugging an electronic system
US6823497B2 (en) * 1999-11-30 2004-11-23 Synplicity, Inc. Method and user interface for debugging an electronic system
US7836416B2 (en) 2000-11-28 2010-11-16 Synopsys, Inc. Hardware-based HDL code coverage and design analysis
US20070198959A1 (en) * 2000-11-28 2007-08-23 Schubert Nils E Hardware-based HDL code coverage and design analysis
US20040025122A1 (en) * 2000-11-28 2004-02-05 Schubert Nils Endric Hardware-based HDL code coverage and design analysis
US7409592B2 (en) * 2002-03-28 2008-08-05 International Business Machines Corporation System for facilitating coverage feedback testcase generation reproducibility
US20040243880A1 (en) * 2002-03-28 2004-12-02 International Business Machines Corporation System for facilitating coverage feedback testcase generation reproducibility
US20030191620A1 (en) * 2002-04-04 2003-10-09 International Business Machines Corp. Dynamic loading of C-API HDL model instrumentation
US7194400B2 (en) * 2002-04-04 2007-03-20 International Business Machines Corporation Method and system for reducing storage and transmission requirements for simulation results
US20030191869A1 (en) * 2002-04-04 2003-10-09 International Business Machines Corp. C-API instrumentation for HDL models
US7373290B2 (en) 2002-04-04 2008-05-13 International Business Machines Corporation Method and system for reducing storage requirements of simulation data via keyword restrictions
US7206732B2 (en) * 2002-04-04 2007-04-17 International Business Machines Corporation C-API instrumentation for HDL models
US20030191621A1 (en) * 2002-04-04 2003-10-09 International Business Machines Corporation Method and system for reducing storage and transmission requirements for simulation results
US7158924B2 (en) * 2002-04-04 2007-01-02 International Business Machines Corporation Dynamic loading of C-API HDL model instrumentation
US7203633B2 (en) * 2002-04-04 2007-04-10 International Business Machines Corporation Method and system for selectively storing and retrieving simulation data utilizing keywords
US20030191617A1 (en) * 2002-04-04 2003-10-09 Gabele Carol Ivash Method and system for selectively storing and retrieving simulation data utilizing keywords
US20040225973A1 (en) * 2003-05-10 2004-11-11 Johnson Tyler James Post-silicon test coverage verification
US7373619B2 (en) * 2003-05-10 2008-05-13 Hewlett-Packard Development Company, L.P. Post-silicon test coverage verification
US7536288B2 (en) 2003-12-31 2009-05-19 International Business Machines Corporation Method, system and program product supporting user tracing in a simulator
US7236918B2 (en) 2003-12-31 2007-06-26 International Business Machines Corporation Method and system for selective compilation of instrumentation entities into a simulation model of a digital design
US20050149313A1 (en) * 2003-12-31 2005-07-07 International Business Machines Corp. Method and system for selective compilation of instrumentation entities into a simulation model of a digital design
US7181708B1 (en) * 2004-08-10 2007-02-20 Cadence Design Systems, Inc. Coverage metric and coverage computation for verification based on design partitions
US7712059B1 (en) 2004-08-10 2010-05-04 Cadence Design Systems, Inc. Coverage metric and coverage computation for verification based on design partitions
US7529655B2 (en) * 2004-10-21 2009-05-05 International Business Machines Corporation Program product for defining and recording minimum and maximum event counts of a simulation utilizing a high level language
US20060089826A1 (en) * 2004-10-21 2006-04-27 International Business Machines Corporation Method, system and program product for defining and recording minimum and maximum count events of a simulation
US20060089827A1 (en) * 2004-10-21 2006-04-27 International Business Machines Corporation Method, system and program product for defining and recording minium and maximum event counts of a simulation utilizing a high level language
US7392169B2 (en) * 2004-10-21 2008-06-24 International Business Machines Corporation Method, system and program product for defining and recording minimum and maximum event counts of a simulation utilizing a high level language
US20060122820A1 (en) * 2004-12-03 2006-06-08 The Mitre Corporation Scripting language for domain-specific modification of a simulation model
US7454325B2 (en) 2004-12-07 2008-11-18 International Business Machines Corporation Method, system and program product for defining and recording threshold-qualified count events of a simulation by testcases
US20060122818A1 (en) * 2004-12-07 2006-06-08 International Business Machines Corporation Method, system and program product for defining and recording threshold-qualified count events of a simulation by testcases
US20070061121A1 (en) * 2005-09-15 2007-03-15 Gabor Bobok Method, system and program product for selectively removing instrumentation logic from a simulation model
US7552043B2 (en) 2005-09-15 2009-06-23 International Business Machines Corporation Method, system and program product for selectively removing instrumentation logic from a simulation model
US20070118827A1 (en) * 2005-11-21 2007-05-24 Asifur Rahman Method and apparatus for integrated circuit fault isolation and failure analysis using linked tools cockpit
US7711537B2 (en) 2006-05-03 2010-05-04 International Business Machines Corporation Signals for simulation result viewing
US20070260443A1 (en) * 2006-05-03 2007-11-08 Gabor Bobok Method, system and program product supporting specification of signals for simulation result viewing
US20070260441A1 (en) * 2006-05-08 2007-11-08 Gabor Bobok Method, system and program product supporting phase events in a simulation model of a digital system
US7493248B2 (en) 2006-05-08 2009-02-17 International Business Machines Corporation Method, system and program product supporting phase events in a simulation model of a digital system
CN100428252C (en) * 2006-09-14 2008-10-22 华为技术有限公司 System and method for testing and measuring percentage of coverage of function
US7849425B1 (en) 2006-09-29 2010-12-07 Breker Verification Systems, Inc. Generating self-checking test cases from a reduced case analysis graph using path inheritance
US7779374B1 (en) * 2006-09-29 2010-08-17 Breker Verification Systems, Inc. Generating self-checking test cases from reduced case analysis graphs
US20080172580A1 (en) * 2007-01-15 2008-07-17 Microsoft Corporation Collecting and Reporting Code Coverage Data
US20080172652A1 (en) * 2007-01-15 2008-07-17 Microsoft Corporation Identifying Redundant Test Cases
US20080172655A1 (en) * 2007-01-15 2008-07-17 Microsoft Corporation Saving Code Coverage Data for Analysis
US20080172651A1 (en) * 2007-01-15 2008-07-17 Microsoft Corporation Applying Function Level Ownership to Test Metrics
US7912694B2 (en) 2007-01-30 2011-03-22 International Business Machines Corporation Print events in the simulation of a digital system
US20080183458A1 (en) * 2007-01-30 2008-07-31 Gabor Bobok Method, system and program product supporting print events in the simulation of a digital system
US20080184204A1 (en) * 2007-01-31 2008-07-31 Microsoft Corporation Dynamic validation using reflection
US7890902B1 (en) * 2007-06-07 2011-02-15 Cadence Design Systems, Inc. Methods and apparatus for merging coverage for multiple verification and design scenarios
US8413088B1 (en) 2007-06-07 2013-04-02 Cadence Design Systems, Inc. Verification plans to merging design verification metrics
US20090112561A1 (en) * 2007-10-31 2009-04-30 Behm Michael L Method, System and Program Product for Defining and Recording Threshold-Qualified Count Events of a Simulation By Testcases
US8050902B2 (en) 2007-10-31 2011-11-01 International Business Machines Corporation Reporting temporal information regarding count events of a simulation
US7925489B2 (en) 2007-10-31 2011-04-12 International Business Machines Corporation Defining and recording threshold-qualified count events of a simulation by testcases
US20090210746A1 (en) * 2008-02-19 2009-08-20 Ditmyer Bruce J Generating test coverage bin based on simulation result
US7831879B2 (en) 2008-02-19 2010-11-09 International Business Machines Corporation Generating test coverage bin based on simulation result
US20090313607A1 (en) * 2008-06-16 2009-12-17 International Business Machines Corporation Code Coverage Tool
US8495574B2 (en) 2008-06-16 2013-07-23 International Business Machines Corporation Code coverage tool
US8160857B2 (en) 2008-12-16 2012-04-17 International Business Machines Corporation Selective compilation of a simulation model in view of unavailable higher level signals
US20100153898A1 (en) * 2008-12-16 2010-06-17 International Business Machines Corporation Model build in the presence of a non-binding reference
US8453080B2 (en) 2008-12-16 2013-05-28 International Business Machines Corporation Model build in the presence of a non-binding reference
CN102073590A (en) * 2011-01-11 2011-05-25 百度在线网络技术(北京)有限公司 Event simulation-based user interface test method and device
US9182945B2 (en) 2011-03-24 2015-11-10 International Business Machines Corporation Automatic generation of user stories for software products via a product content space
US8839203B2 (en) 2011-05-25 2014-09-16 Microsoft Corporation Code coverage-based taint perimeter detection
US9081645B2 (en) 2013-01-15 2015-07-14 International Business Machines Corporation Software product licensing based on a content space
US9256423B2 (en) 2013-01-15 2016-02-09 International Business Machines Corporation Software product licensing based on a content space
US9069647B2 (en) 2013-01-15 2015-06-30 International Business Machines Corporation Logging and profiling content space data and coverage metric self-reporting
US9087155B2 (en) 2013-01-15 2015-07-21 International Business Machines Corporation Automated data collection, computation and reporting of content space coverage metrics for software products
US9111040B2 (en) 2013-01-15 2015-08-18 International Business Machines Corporation Integration of a software content space with test planning and test case generation
US9141379B2 (en) 2013-01-15 2015-09-22 International Business Machines Corporation Automated code coverage measurement and tracking per user story and requirement
US9170796B2 (en) 2013-01-15 2015-10-27 International Business Machines Corporation Content space environment representation
US9063809B2 (en) 2013-01-15 2015-06-23 International Business Machines Corporation Content space environment representation
US9218161B2 (en) 2013-01-15 2015-12-22 International Business Machines Corporation Embedding a software content space for run-time implementation
US9075544B2 (en) 2013-01-15 2015-07-07 International Business Machines Corporation Integration and user story generation and requirements management
US9256518B2 (en) 2013-01-15 2016-02-09 International Business Machines Corporation Automated data collection, computation and reporting of content space coverage metrics for software products
US9396342B2 (en) 2013-01-15 2016-07-19 International Business Machines Corporation Role based authorization based on product content space
US9513902B2 (en) 2013-01-15 2016-12-06 International Business Machines Corporation Automated code coverage measurement and tracking per user story and requirement
US9569343B2 (en) 2013-01-15 2017-02-14 International Business Machines Corporation Integration of a software content space with test planning and test case generation
US9612828B2 (en) 2013-01-15 2017-04-04 International Business Machines Corporation Logging and profiling content space data and coverage metric self-reporting
US9659053B2 (en) 2013-01-15 2017-05-23 International Business Machines Corporation Graphical user interface streamlining implementing a content space
US20170308450A1 (en) * 2016-04-25 2017-10-26 EMC IP Holding Company, LLC Computer-implemented method, computer program product and computing system
US11003562B2 (en) * 2016-04-25 2021-05-11 EMC IP Holding Company, LLC Computer-implemented method, computer program product and computing system
US10635766B2 (en) 2016-12-12 2020-04-28 International Business Machines Corporation Simulation employing level-dependent multitype events
CN113075911A (en) * 2021-03-24 2021-07-06 中广核工程有限公司 Automatic verification method and system for nuclear power plant safety-level digital instrument control system software

Similar Documents

Publication Publication Date Title
US20030121011A1 (en) Functional coverage analysis systems and methods for verification test suites
Gargantini et al. Using model checking to generate tests from requirements specifications
US6132109A (en) Architecture and methods for a hardware description language source level debugging system
US5673387A (en) System and method for selecting test units to be re-run in software regression testing
US7912694B2 (en) Print events in the simulation of a digital system
US6598183B1 (en) Software tool for automated diagnosis and resolution of problems of voice, data and VoIP communications networks
US6163763A (en) Method and apparatus for recording and viewing error data generated from a computer simulation of an integrated circuit
US6263301B1 (en) Method and apparatus for storing and viewing data generated from a computer simulation of an integrated circuit
US20030035008A1 (en) Method and apparatus for controlling an instrumentation system
US7610578B1 (en) Test manager for integrated test environments
Marchok et al. Complexity of sequential ATPG
KR20010108043A (en) Method and apparatus for executing pl/sql stored code objects in a database
US7146572B2 (en) System and method for configuring database result logging for a test executive sequence
US6449750B1 (en) Design verification device, method and memory media for integrated circuits
Yannakakis Hierarchical state machines
US7451358B2 (en) Test executive system with automatic expression logging and parameter logging
US20020123875A1 (en) Hierarchical processing of simulation model events
US20030140326A1 (en) Race condition detection and expression
Amla et al. Model checking synchronous timing diagrams
US7506279B2 (en) Design supporting apparatus capable of checking functional description of large-scale integrated circuit to detect fault in said circuit
CN115168392A (en) Structural rule description method and system for building specification index query
CA2447163A1 (en) A visual debugging interface
US7243319B2 (en) Race condition detection and expression
Stillger et al. Testing the quality of a query optimizer
CN111008113A (en) SAS-Expander test method and tool

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION