CN112948193B - FPGA comprehensive tool defect detection method based on difference test - Google Patents

FPGA comprehensive tool defect detection method based on difference test Download PDF

Info

Publication number
CN112948193B
CN112948193B CN202110251720.5A CN202110251720A CN112948193B CN 112948193 B CN112948193 B CN 112948193B CN 202110251720 A CN202110251720 A CN 202110251720A CN 112948193 B CN112948193 B CN 112948193B
Authority
CN
China
Prior art keywords
test
tool
fpga
test case
cases
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110251720.5A
Other languages
Chinese (zh)
Other versions
CN112948193A (en
Inventor
江贺
张漪�
施重阳
刘辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Institute of Technology BIT
Original Assignee
Beijing Institute of Technology BIT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Institute of Technology BIT filed Critical Beijing Institute of Technology BIT
Priority to CN202110251720.5A priority Critical patent/CN112948193B/en
Publication of CN112948193A publication Critical patent/CN112948193A/en
Application granted granted Critical
Publication of CN112948193B publication Critical patent/CN112948193B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
    • G06F11/2273—Test methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3684—Test management for test design, e.g. generating new test cases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3688—Test management for test execution, e.g. scheduling of test suites

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Test And Diagnosis Of Digital Computers (AREA)

Abstract

The invention relates to a method for detecting defects of an FPGA (field programmable gate array) comprehensive tool based on a difference test, and belongs to the technical field of computer software testing. The invention gives a test code by using a reference FPGA comprehensive tool for comparison, generates a variant code equivalent to the test code, and compiles the reference FPGA comprehensive tool and the tool to be tested to obtain respective execution results. Since the reference synthesis tool and the FPGA synthesis tool to be tested follow the same protocol in the differential test, the execution results are the same. Therefore, by comparing the execution results of the tool to be tested with those of the reference tool, whether the tool to be tested has defects can be effectively judged. Compared with the prior art, the method provided by the invention can be used for effectively detecting the defects in the comprehensive tool by generating the test seed cases for testing the effective FPGA comprehensive tool through mutation, and storing the defect reports of the defects, so that developers can quickly repair the Bug.

Description

FPGA comprehensive tool defect detection method based on difference test
Technical Field
The invention relates to a defect detection method of an FPGA (field programmable gate array) comprehensive tool, in particular to a defect detection method of the FPGA comprehensive tool based on a difference test, and belongs to the technical field of computer software testing.
Background
In recent years, the normal operation of software has increasingly relied on hardware. Almost all digital computing relies on logic synthesis tools in this or that way, and more fields use FPGA (Field Programmable Gate Array) to accelerate applications on servers. Furthermore, the increasing complexity of the hardware also results in a great demand for automation, which means that the correctness of the synthesis tool is critical to the reliability of the hardware. Therefore, it is necessary to strictly test the FPGA integration tool.
To test the correctness of FPGA synthesis tools, it has been proposed to generate random, efficient, deterministic or indeterminate Verilog designs by construction and pass them to the synthesis tool, which will produce an equivalent nestlist netlist that can be compared to the original design, and then determine if the synthesis tool is defective based on the principle that the output of the Verilog design and the synthesized netlist should always be equal at the clock edge.
While existing methods are well able to detect FPGA integrated tool defects, random build behaving Verilog is unlikely to be exhaustive of testing many different Verilog build combinations. Meanwhile, if an uncertain Verilog design is used, the cost of testing needs to be considered because generating an uncertain Verilog requires performing additional simulation steps to avoid false positives. These would result in poor FPGA integration tool bug detection efficiency.
Disclosure of Invention
The invention aims to provide a method for detecting defects of an FPGA integrated tool based on a difference test, aiming at the technical problem that the existing method for detecting defects of the FPGA integrated tool is poor in effect.
The innovation point of the method is that: by using two or more reference FPGA synthesis tools for comparison, one test code is given, the test code is generated into variant codes equivalent to the test code, and the reference FPGA synthesis tools and the tools to be tested are compiled to obtain respective execution results. Since the reference synthesis tool and the FPGA synthesis tool to be tested follow the same protocol in the differential test, the execution results are the same. Therefore, by comparing the execution results of the tool to be tested with those of the reference tool, whether the tool to be tested has defects can be effectively judged.
The invention is realized by adopting the following technical scheme.
A method for detecting defects of an FPGA integrated tool based on a difference test comprises the following steps:
step 1: and constructing a seed library of the FPGA comprehensive tool test case.
Specifically, the required test cases can be obtained from the comprehensive design cases input by the testers. For example, the high-level language file can be obtained by crawling and analyzing the existing high-level comprehensive tool official network and the effective links of the open source communities.
Step 2: based on a genetic algorithm, sequencing the test seed cases according to the priority of the test effect from high to low, and executing the test seed cases sequentially. The test time can be reduced, and the test efficiency can be improved.
Specifically, the general description of the problem of prioritizing test cases is: test case set T, full permutation set PT of test cases, ordering objective function f (x): pt→r. Searching for T '∈PT so that for any T' ∈PT (T '+.T'), there is f (T '). Gtoreq.f (T'). Wherein T ', T' represents a sequence of test cases arbitrarily selected in the full permutation set of test cases, T 'noteqT'.
In the testing of the FPGA and the software system, when HDL is detected, logic is detected according to a signal flow diagram, and consistency of time-varying signals is detected according to a time sequence diagram. Test points are designed according to the test requirements, and each test case can correspond to one or a plurality of test points in the test case set.
Firstly, an evaluation index APTC (Average Percentage of Test-point Coverage) based on test point Coverage is defined, namely, the relation between the number of test case use and the number of detection errors is quantified. The test case set T is provided with n test cases, and m defects can be detected. For a given test case execution order, TF i The calculation formula of the APTC is given by the order in which the test cases where the ith defect can be detected are located in the execution order:
wherein, APTC epsilon (0, 100%) and the higher the value, the faster the defect detection speed.
As shown in Table 1 for the example, there are 5 test cases and 10 test points. For the test case sequences A-B-C-D-E and E-D-C-B-A, the APTC values are respectively 50% and 64%, so that the effect of the execution sequence 2 is preferential.
TABLE 1 example of test case to test point correspondence
Then, the genetic algorithm is guided to search based on the fitness function of the genetic algorithm (the value of the fitness function is between 0 and 1), namely, the evaluation index APTC.
Step 3: and selecting a test case P.
Wherein, selecting test cases follows the following conditions:
if the initial test is performed, randomly selecting a test case from the seed library in the step 2. Otherwise, selecting from large to small according to the value of the test case APTC.
Step 4: and mutating the test case to obtain a new test case P'.
Wherein, generating a mutation code equivalent to the test case, the original AST can be used for analyzing the test case and performing equivalent mutation. Analyzing the sentences of the AST of the test case, if a certain sentence is not executed and the child of the sentence is not executed, marking the sentence as the non-executed sentence, and deleting or inserting the sentence, when deleting the sentence, removing all related sentences in the AST subtree, including the child sentence, and when inserting the code segment, inserting the sentence into the marked non-executed sentence without limit.
Step 5: and (3) using an FPGA synthesis tool to be tested, selecting at least 2 existing reference tools, and converting the variation test code P' generated in the step (4) to obtain a synthesis netlist of the test code HDL.
The selected reference tool and the FPGA comprehensive tool to be tested have the same protocol, namely, the tool to be tested and the reference tool are the FPGA comprehensive tools capable of being compared, and the implementation of the tools is in accordance with the same protocol principle.
Step 6: judging whether a tool has a breakdown phenomenon or not. If the test running situation occurs, the bug report is directly generated and stored. If both the reference tool and the tool under test are successful in converting the test case, step 7 is performed.
Specifically, in the FPGA synthesis tool, an equivalent netlist is generated for a design test case and compared with the test case, that is, the output of the Verilog design of the synthesis tool and the netlist generated by conversion are always equal at clock edges, and if the output of the Verilog design and the netlist generated by conversion are not equal, the execution breakdown of the test case is judged.
Step 7: and comparing the conversion results of the existing reference tool and the FPGA comprehensive tool to be tested.
Specifically, when the test cases are synthesized by the synthesis tool to form a netlist file, edf, the synthesis tool calls a self-contained simulation tool to perform functional simulation to obtain a result.
Alternatively, the integration tool may create a new project to simulate it with the help of the integrated 3. V files, one is the testbench file, one is the tool-generated xxx_synthesis. V, and the other is the glbl. V file.
Step 8: comparing whether the simulation results of the previous step are the same or not, if so, indicating that the selected test case is not suitable, returning to the step 3, and if the results are inconsistent, generating and storing bug report results.
Advantageous effects
Compared with the prior art, the method provided by the invention can be used for effectively detecting the defects in the comprehensive tool and storing the defect report thereof by generating the test seed case for testing the effective FPGA comprehensive tool through mutation, so that developers can quickly repair the Bug.
Drawings
FIG. 1 is a schematic flow chart of the method of the present invention.
Detailed Description
The process according to the invention is described in further detail below with reference to the figures and examples.
Examples
The method is deployed in a hardware environment shown in a table 2, corresponding software to be tested is installed according to experimental requirements, 2 existing reference tools 1-Yosys 0.9 and reference tools 2-Vivado 18.2 selected in the experiment are used for detecting a Field Programmable Gate Array (FPGA) comprehensive tool Prime 19.2, wherein Yosys 0.9 is open-source. The embodiment is composed of programs such as a test case seed pool framework, test code variation, test seed case sequencing and the like.
Table 2: hardware environment configuration information table
Step 1: collecting design examples of existing FPGA comprehensive tools, constructing a test case seed library, initializing test cases in the seed pool, and setting an initial priority parameter value c for each test case 0 So that the chance that each test case is randomly selected is equalized.
The method comprises the following steps: using crawler technology, high-level language files are crawled and parsed from the active links of the open source communities and the official networks of some high-level comprehensive tools.
Step 2: the test seed cases are prioritized based on a genetic algorithm. The test cases are sequenced from high to low according to the priority of the test effect and are sequentially executed, so that the test time can be reduced, and the test efficiency is improved.
The method comprises the following steps:
a general description of the test case prioritization problem is as follows:
test case set T, full permutation set PT of test cases, ordering objective function f (x): pt→r. Searching for T '∈PT so that f (T') ∈PT (T '+.T') is greater than or equal to f (T ') for any T' ∈PT (T '+.T'). Wherein T 'and T' represent a sequence of test cases arbitrarily selected in the full permutation set of test cases, T 'noteqT'.
In the testing of the FPGA and the software system, when HDL is detected, logic is detected according to a signal flow diagram, and consistency of time-varying signals is detected according to a time sequence diagram. Test points are designed according to the test requirements, and each test case can correspond to one or a plurality of test points in the test case set.
An evaluation index APTC based on test point coverage is defined, namely the relation between the number of test case use and the number of detection errors is quantified. The test case set T is set, n test cases are provided, m defects can be detected, and the test cases are executed in a given order, TF i The calculation formula of the APTC is given by the order in which the test cases where the ith defect can be detected are located in the execution order:
wherein, APTC epsilon (0, 100%) is higher, and the defect detection speed is faster.
Then, the genetic algorithm is guided to search based on the fitness function of the genetic algorithm, wherein the fitness function is 0-1, namely, the evaluation index APTC is used for guiding the genetic algorithm to search.
Step 3: and selecting a test case P.
The test case is selected to follow the following rules:
and (3) if the initial test is executed, randomly selecting the test cases from the seed library in the step (2), otherwise, selecting the test cases with high priority according to the value of the test case APTC.
Step 4: and mutating the test case to obtain a new test case P'.
The method comprises the steps of generating variant codes equivalent to test cases, analyzing test cases by using original AST and carrying out equivalent variants, analyzing sentences of the AST of the test cases, marking a sentence as an unexecuted sentence if one sentence is unexecuted and children of the sentence are unexecuted, and deleting or inserting the sentence. When the statement is deleted, all relevant phrases in the AST subtree, including its child statements, are removed; when inserting code segments, statements may be inserted indefinitely all the way to these marked unexecuted statements. The specific process is shown in fig. 1.
Step 5: and (3) converting the variation test code P' generated in the step (4) by using an FPGA (field programmable gate array) synthesis tool to be tested, a reference tool 1 and a reference tool 2 to obtain a synthesis netlist of the test code HDL, wherein the reference tool 1 and the reference tool 2 are realized by the same specifications as the FPGA synthesis tool to be tested, namely the target tool to be tested and the reference target tool are the FPGA synthesis tools capable of being compared, and the realization of the tools is in accordance with the same rules of the specifications.
Step 6: judging whether a tool has a breakdown phenomenon or not; if the test running situation occurs, the bug report is directly generated and stored, and if the reference tool and the tool to be tested are successful in converting the test case, the next step is executed.
The method comprises the following steps:
in the FPGA synthesis tool, for a test case of a design, an equivalent netlist is generated for comparison with the test case, that is, the output of the Verilog design of the synthesis tool and the netlist generated by conversion should be always equal at clock edges, and when the output of the Verilog design and the netlist generated by conversion are not equal, it is determined that the test case is in execution breakdown.
Step 7: and comparing conversion results of the reference tool 1, the reference tool 2 and the FPGA integrated tool to be tested.
The method comprises the following steps:
with the aid of the integrated 3. One is the testbench file, one is the tool generated xxx_synthis.v, and the other is the glbl.v file, simulated using the ModelSim.
Step 8: comparing whether the simulation results of the previous step are the same or not, if so, indicating that the selected test case is not suitable, returning to the step 3, and if the results are inconsistent, generating and storing bug report results.
The results are shown below;
run time of tools under the same resource:
Yosys 0.9：200 CPU hours
Vivado 18.2：180 CPU hours
Quartus Prime 19.2:180 CPU hours
number of detected bug:
Quartus Prime 19.2：2
the results show that: by using the method of the invention, the bug existing in the comprehensive tool can be effectively detected.

Claims (4)

1. The method for detecting the defects of the FPGA comprehensive tool based on the difference test is characterized by comprising the following steps of:
step 1: constructing an FPGA comprehensive tool test case seed library;
step 2: based on a genetic algorithm, sequencing the test seed cases according to the priority of the test effect from high to low, and sequentially executing the test seed cases;
the general description of the test case prioritization problem is: test case set T, full permutation set PT of test cases, ordering objective function f (x): PT-R;
searching T 'E PT so that f (T') ∈PT is greater than or equal to f (T ') for any T', wherein T ', T' represent a test case sequence arbitrarily selected in a full permutation set of test cases, and T 'noteqT';
firstly, defining an evaluation index APTC based on test point coverage, namely quantifying the relation between the number of test case use and the number of detection errors; the test case set T is provided with n test cases, and m defects can be detected; for a given test case execution order, TF i The calculation formula of the APTC is given by the order in which the test cases where the ith defect can be detected are located in the execution order:
wherein, APTC epsilon (0, 100%) and the higher the value is, the faster the defect detection speed is;
then, based on the value of the evaluation index APTC, selecting the execution sequence of the test case with the highest APTC value;
step 3: selecting a test case P, wherein the selected test case follows the test case execution sequence corresponding to the highest APTC value in the step 2;
step 4: the test case is mutated to obtain a new test case P';
step 5: using an FPGA synthesis tool to be tested, selecting at least 2 existing reference tools, and converting the variation test code P' generated in the step 4 to obtain a synthesis netlist of the test code HDL;
step 6: judging whether a tool has a breakdown phenomenon or not; if the test tool crashes, directly generating and storing a bug report; if the reference tool and the FPGA comprehensive tool to be tested are successful in converting the test case, executing the step 7;
step 7: comparing conversion results of the existing reference tool and the FPGA comprehensive tool to be tested; when the test cases are synthesized by the synthesis tool, a netlist file edf is formed, and the synthesis tool calls a self-contained simulation tool to perform functional simulation to obtain a result;
step 8: comparing whether the simulation results of the previous step are the same, if so, indicating that the selected test cases are not suitable, returning to the step 3, and if not, generating and storing bug report results.
2. The method for detecting defects of an FPGA integrated tool based on a difference test as claimed in claim 1, wherein in step 4, variant codes equivalent to test cases are generated, and the original AST is used to analyze the test cases and to perform equivalent variants;
analyzing the sentences of the AST of the test case, if a certain sentence is not executed and the child of the sentence is not executed, marking the sentence as the non-executed sentence, and deleting or inserting the sentence, when deleting the sentence, removing all related sentences in the AST subtree, including the child sentences, and when inserting the code segment, inserting the sentence into the marked non-executed sentences without limit.
3. The method for detecting defects of an FPGA integrated tool based on a differential test as claimed in claim 1, wherein when comparing the conversion results in step 7, the FPGA integrated tool creates a new project to simulate it by means of the 3 x.v files after integration, one is a testbench file, the other is a xxx_synchronization.v file generated by the tool, and the other is a glbl.v file.
4. The method for detecting defects of an FPGA integrated tool based on a difference test as set forth in claim 1, wherein 2 existing reference tools are selected, wherein reference tool 1 is Yosys 0.9 and reference tool 2 is Vivado 18.2.
CN202110251720.5A 2021-03-08 2021-03-08 FPGA comprehensive tool defect detection method based on difference test Active CN112948193B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110251720.5A CN112948193B (en) 2021-03-08 2021-03-08 FPGA comprehensive tool defect detection method based on difference test

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110251720.5A CN112948193B (en) 2021-03-08 2021-03-08 FPGA comprehensive tool defect detection method based on difference test

Publications (2)

Publication Number Publication Date
CN112948193A CN112948193A (en) 2021-06-11
CN112948193B true CN112948193B (en) 2023-10-17

Family

ID=76228755

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110251720.5A Active CN112948193B (en) 2021-03-08 2021-03-08 FPGA comprehensive tool defect detection method based on difference test

Country Status (1)

Country Link
CN (1) CN112948193B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113434390B (en) * 2021-06-21 2022-11-08 北京理工大学 FPGA logic comprehensive tool fuzzy test method based on variation
CN114385492A (en) * 2021-12-30 2022-04-22 大连理工大学 Advanced comprehensive tool optimization option defect detection method based on differential test

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8621125B2 (en) * 2009-10-13 2013-12-31 Intellitech Corporation System and method of sending and receiving data and commands using the TCK and TMS of IEEE 1149.1
CN111400169B (en) * 2020-02-25 2023-04-18 中科亿海微电子科技（苏州）有限公司 Method and system for automatically generating netlist file for testing software and hardware
CN111737120B (en) * 2020-06-15 2022-09-23 河南工业大学 Software defect repairing method and device
CN112034331B (en) * 2020-08-17 2023-04-18 北京时代民芯科技有限公司 Circuit module testing method based on FPGA

Also Published As

Publication number Publication date
CN112948193A (en) 2021-06-11

Similar Documents

Publication Publication Date Title
CN111459799B (en) Software defect detection model establishing and detecting method and system based on Github
JP3858000B2 (en) Verification method of combinational circuit using filtering type approach
Luo et al. ICS protocol fuzzing: Coverage guided packet crack and generation
CN112948193B (en) FPGA comprehensive tool defect detection method based on difference test
US7188061B2 (en) Simulation monitors based on temporal formulas
US6553514B1 (en) Digital circuit verification
CN112035359B (en) Program testing method and device, electronic equipment and storage medium
US20140258987A1 (en) Determining correctness of an application
CN115687115B (en) Automatic testing method and system for mobile application program
CN109542783B (en) Extended finite-state machine test data generation method based on variable segmentation
CN115098292B (en) Method and device for identifying root cause of application program crash and electronic equipment
CN102141959B (en) Test case generation method restrained by context-free grammar
CN108563561B (en) Program implicit constraint extraction method and system
CN114936158A (en) Software defect positioning method based on graph convolution neural network
Letychevskyi et al. Modeling method for development of digital system algorithms based on programmable logic devices
US20110022906A1 (en) Method and system for test point insertion
US6099577A (en) Logic circuit conversion method and logic circuit design support device
CN111625448B (en) Protocol packet generation method, device, equipment and storage medium
CN115185920B (en) Method, device and equipment for detecting log type
CN101833505A (en) Method for detecting security bugs of software system
JP2011048785A (en) Multiple event definition device, multiple event verification device, multiple event definition method, and multiple event definition program
Imtiaz et al. Predicting vulnerability for requirements
CN110750449A (en) Test-driven webpage component function extraction method
CN107678734B (en) CPU benchmark test program set construction method based on genetic algorithm
CN111782553A (en) Software reaction defect analysis method based on fault injection

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant