CN117851270B

CN117851270B - Method and device for testing system-on-chip compiler, electronic equipment and storage medium

Info

Publication number: CN117851270B
Application number: CN202410259794.7A
Authority: CN
Inventors: 陈相宇; 吴春春; 王士成; 李征陆; 晁麒雅; 李佳桐; 刘佳兴; 杨亦力
Original assignee: CETC 15 Research Institute
Current assignee: CETC 15 Research Institute
Priority date: 2024-03-07
Filing date: 2024-03-07
Publication date: 2024-05-03
Anticipated expiration: 2044-03-07
Also published as: CN117851270A

Abstract

The invention relates to a method and a device for testing a system-on-chip compiler, electronic equipment and a storage medium, wherein the method comprises the following steps: and acquiring first test data, and calling a cipher algorithm library according to the cipher test algorithm to process the first test data so as to generate an expected output result. The source code is obtained based on MLIR automatically generated data formats of the first interface or the compiler custom second interface. The source code is analyzed by the compiler front end to generate a first intermediate representation, and operators satisfying the corresponding compiler hardware operating characteristics are extracted from the first intermediate table. Optimizing the extracted operator to obtain a target operator, and replacing the source operator in the source code with the target operator to obtain an optimized code. And compiling the optimized code through the back end of the compiler, generating a target instruction corresponding to the running characteristics of the hardware of the compiler, and sending the target instruction to the corresponding hardware of the compiler to perform collaborative operation. And calling FILECHECK to verify the collaborative operation result according to the expected output result to generate a compiler test report.

Description

Method and device for testing system-on-chip compiler, electronic equipment and storage medium

Technical Field

The present invention relates to the field of system-on-chip compilers, and in particular, to a method and apparatus for testing a system-on-chip compiler, an electronic device, and a storage medium.

Background

The system on a chip is a novel chip system based on wafer-level integration and constructed in a software and hardware integrated design mode. In addition, the on-chip system adopts a complete wafer substrate to serve as an interconnection base of each module in the system, and a wafer densely covered with various core grains is used for realizing a complete system, so that the connection among the modules supports software definition and the functions of the modules, and can also support the software definition. The system on a chip breaks through the limitation of the technology on improving the chip integration level from the system angle, reduces the energy consumption of communication between the core grains, and has extremely strong flexibility and expandability.

However, the implementation of high flexibility and software definition of the on-chip system needs to rely on a reconfigurable computing unit at the bottom layer, and on the system level, the partitioning, compiling and scheduling of application tasks need to be performed by means of a powerful compiler, so that the performance of the whole system can be fully exerted. The compiler is a key ring of the system on a chip from theory to practical application and from hardware to software and hardware integration. Aiming at the compiler design of the on-chip system, the following links are mainly needed to be paid attention to:

(1) Hardware awareness

The on-chip system has the characteristics of reconfigurable computing units and reconfigurable interconnection modes, and the compiler can acquire the information such as the layout, interconnection modes and the like of the computing units at the current moment, so that the on-chip system is a precondition for maximizing the utilization of the system performance and reducing the system overhead.

(2) Task scheduling

According to the known system hardware information, hardware resource state and application task characteristics, a compiler reasonably decomposes and distributes tasks, and the tasks are disassembled to different computing units for execution.

(3) Heterogeneous compilation

The on-chip system is provided with various computing units based on different computing architectures, and a compiler compiles the divided source operators into machine instructions of target hardware according to scheduling.

Thus, compilers of systems on die need to support heterogeneous and cross-platform compilation of multiple architectures. Based on the three points, the design and development of the on-chip system compiler have extremely high complexity, the result correctness of the on-chip system compiler is evaluated, the on-chip system compiler is verified to be capable of fully playing the system performance, and the method also becomes a core link in the compiler design.

One of the more significant features of the on-die system is that the on-die system has heterogeneous computing resources at a higher level, finer granularity, than the on-die system. Heterogeneous computing, namely hardware computing resources with different instruction sets and different architectures in the same system, is successfully realized by reasonably allocating the resources with different characteristics. The heterogeneous computing resources of the system on a chip mainly comprise CPU, GPU, FPGA, ASIC, DSP and the like, and have the characteristics of richer heterogeneous resources, stronger resource expansion reconfigurability, finer heterogeneous granularity and the like.

Currently, the structure of a conventional compiler can be divided into three parts: front-end interface representation, intermediate representation layer and back-end target instruction generation. Wherein the compiler front-end interface represents an application program responsible for receiving the representation in a high-level language form, performing syntax analysis, lexical analysis, semantic analysis, and then converting to an intermediate representation IR (Intermediate Representation) in a unified format. The intermediate representation layer performs unified optimization work on the IR independent of the source code, mainly converting various source operators in the source code into target operators for intermediate representation. The back-end target instruction then generates machine instructions that translate the intermediate representation into underlying hardware for execution. Most of traditional compilers perform compiling optimization aiming at isomorphic hardware, and unified compiling optimization is difficult to realize aiming at various heterogeneous devices.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a method, an apparatus, an electronic device, and a storage medium for testing a system-on-chip compiler, which can implement unified compilation optimization for a plurality of heterogeneous devices.

The invention provides a system-on-chip compiler testing method, which comprises the following steps:

Acquiring first test data, and calling a cipher algorithm library according to a cipher test algorithm to process the first test data so as to generate an expected output result, wherein the first test data is obtained by automatically generating random numbers or inputting the random numbers by a compiler test framework in a self-defining way;

acquiring source codes based on data formats of first interfaces or second interfaces customized by compilers automatically generated MLIR, wherein the source codes are codes meeting the data formats of the first interfaces or the second interfaces customized by compilers automatically generated MLIR;

Analyzing the source code through a front end of a compiler to generate a first intermediate representation, and extracting a first operator and a second operator from the first intermediate representation, wherein the first operator and the second operator are operators meeting the running characteristics of different compiler hardware respectively;

Optimizing the first operator and the second operator respectively based on the different compiler hardware operation characteristics to obtain a first target operator and a second target operator, and replacing the first operator and the second operator in the source code with the first target operator and the second target operator respectively to obtain an optimized code;

Compiling the optimized code through the back end of a compiler to generate a corresponding first target instruction and a corresponding second target instruction, and respectively sending the first target instruction and the second target instruction to compiler hardware with corresponding operation characteristics to perform collaborative operation to obtain a collaborative operation result;

And calling FILECHECK to verify the collaborative operation result based on the expected output result to obtain a verification result, and generating a compiler test report based on the verification result, wherein the verification result is used for representing whether the collaborative operation result is consistent with the expected output result or not.

In one embodiment, the acquiring the first test data and calling a cryptographic algorithm library according to the cryptographic test algorithm to process the first test data to generate the expected output result includes:

Building a compiler testing framework based on llvm-lit testing suite, and sorting the testing paths of the first testing data, wherein the compiler testing framework is used for supporting automatic operation deployment of compiler testing;

a compiler testing environment is built based on the compiler testing framework, wherein the compiler testing environment is obtained by writing litcfg and litlocal cfg.

In one embodiment, the step of taking the first test data and calling a cryptographic algorithm library according to the cryptographic test algorithm to process the first test data to generate an expected output result further includes:

And acquiring a compiler test execution range according to the test content of the first test data, wherein the compiler test execution range at least comprises a root directory for executing the test of the first test data, a subdirectory under the root directory, which does not need to execute the test, and a file suffix for executing the test.

In one embodiment, the cryptographic test algorithm at least includes SHA1 and SM4 cryptographic test algorithms, and the cryptographic algorithm library at least includes OpenSSL and GmSSL cryptographic algorithm libraries;

the obtaining the first test data, and calling a cryptographic algorithm library according to a cryptographic test algorithm to process the first test data to generate an expected output result, including:

automatically generating data or custom input data according to the random number to generate batch test data so as to acquire the first test data;

And calling an OpenSSL or GmSSL cryptographic algorithm library through the SHA1 or SM4 cryptographic test algorithm to generate the expected output result.

In one embodiment, the first interface is ciface interfaces, and the second interface is a #pragma interface;

The method comprises the steps of acquiring source codes based on the data format of a first interface or a second interface customized by a compiler which is automatically generated by MLIR, and then further comprising:

Compiling the source code according to a ciface interface automatically generated by MLIR or a data format of a # pragma interface customized by a compiler, and judging whether the first intermediate representation generated by the front end of the compiler has equivalent semantics with the source code; and

Further judging whether a source operator of the password test algorithm is optimized to generate a target operator meeting the operation characteristics of corresponding compiler hardware after being processed through intermediate representation, and obtaining a first judgment result;

wherein the source operator includes at least a message population and a hash map, and the compiler test report further includes the first determination result.

In one embodiment, the calling FILECHECK performs verification on the collaborative operation result based on the expected output result to obtain a verification result, and generates a compiler test report based on the verification result, which further includes:

selecting an object code from a different cipher algorithm library, compiling the object code into an object instruction, and transmitting the object instruction to compiler hardware which is the same as the optimized code for operation to obtain an operation result of the object code;

Judging whether the running result of the target code is consistent with the collaborative running result or not based on the running result of the target code, and obtaining a second judging result;

Wherein the target code and the optimized code have the same compiler hardware operation characteristics, and the compiler test report further includes the second determination result.

In one embodiment, the method further comprises:

Taking the same password test algorithm as a test task for a plurality of times, and obtaining the cooperative operation result corresponding to the password test algorithm each time;

judging whether the collaborative operation results corresponding to the multiple password test algorithms are consistent or not so as to obtain a third judgment result for verifying the test stability of the compiler;

wherein the compiler test report further includes the third determination result.

The invention also provides a system-on-chip compiler testing device, which comprises:

The expected result generation module is used for acquiring first test data, calling a password algorithm library according to a password test algorithm to process the first test data so as to generate an expected output result, wherein the first test data is obtained by automatically generating random numbers or custom inputting a compiler test framework;

The source code obtaining module is used for obtaining source codes based on the data format of the first interface or the second interface customized by the compiler which is automatically generated MLIR, wherein the source codes are codes meeting the data format of the first interface or the second interface customized by the compiler which is automatically generated MLIR;

The operator extraction module is used for analyzing the source code through the front end of the compiler to generate a first intermediate representation, and extracting a first operator and a second operator from the first intermediate representation, wherein the first operator and the second operator are operators meeting the running characteristics of different compiler hardware respectively;

The operator optimizing module is used for optimizing the first operator and the second operator respectively based on the different compiler hardware operation characteristics to obtain a first target operator and a second target operator, and replacing the first operator and the second operator in the source code with the first target operator and the second target operator respectively to obtain an optimized code;

the collaborative operation module is used for compiling the optimized code through the rear end of the compiler to generate a corresponding first target instruction and a corresponding second target instruction, and respectively sending the first target instruction and the second target instruction to compiler hardware corresponding to operation characteristics to perform collaborative operation to obtain a collaborative operation result;

And the test report generation module is used for calling FILECHECK to verify the collaborative operation result based on the expected output result to obtain a verification result, and generating a compiler test report based on the verification result, wherein the verification result is used for representing whether the collaborative operation result is consistent with the expected output result or not.

The invention also provides an electronic device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the system-on-chip compiler testing method according to any one of the above when executing the computer program.

The invention also provides a computer storage medium storing a computer program which when executed by a processor implements a system-on-chip compiler testing method as described in any one of the above.

According to the method, the device, the electronic equipment and the storage medium for testing the system on chip compiler, the compiler testing framework automatically generates or manually inputs the random numbers to generate batch test data, and the cryptographic algorithm library is called according to the cryptographic testing algorithm to process the test data to generate the expected output result. Then, based on the ciface interface or the data format of the compiler-customized #pragma interface automatically generated MLIR, the source code meeting the data format of the ciface interface or the compiler-customized #pragma interface is acquired. And then analyzing the source code through the front end of the compiler to generate a corresponding intermediate representation, extracting operators from the intermediate representation for optimization to obtain target operators meeting the operation characteristics of the corresponding compiler hardware, and replacing the corresponding target operators with the source operators in the source code to obtain the optimized code. Compiling the obtained optimized code representation through the back end of the compiler to generate a plurality of target instructions, and sending the generated target instructions to compiler hardware corresponding to the operation characteristics to perform cooperative operation to obtain a cooperative operation result. And finally, calling FILECHECK to check the collaborative operation result to obtain a check result used for representing whether the collaborative operation result is consistent with the expected output result or not, and generating a compiler test report based on the check result. The method realizes the multi-dimensional test verification of the on-chip system by evaluating the correctness of the test result of the on-chip system compiler and verifying whether the on-chip system compiler can fully exert the system performance. In addition, the method also supports one-key deployment of users, can effectively reduce the testing cost of the compiler and improves the development efficiency of the system-on-chip compiler.

Drawings

In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a testing method of a system-on-chip compiler according to the present invention;

FIG. 2 is a schematic diagram of a system-on-chip compiler testing framework of a method for testing a system-on-chip compiler according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a testing flow of a system-on-chip compiler according to a testing method of a system-on-chip compiler according to an embodiment of the present invention;

FIG. 4 is a second flowchart of a testing method of a system-on-chip compiler according to the present invention;

FIG. 5 is a third flowchart illustrating a testing method of a system on chip compiler according to the present invention;

FIG. 6 is a flowchart illustrating a testing method of a system-on-chip compiler according to the present invention;

FIG. 7 is a flowchart of a testing method of a system-on-chip compiler according to the present invention;

FIG. 8 is a flowchart illustrating a testing method of a system on chip compiler according to the present invention;

FIG. 9 is a schematic diagram of a system-on-chip compiler testing apparatus according to the present invention;

fig. 10 is an internal structural diagram of a computer device provided by the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The system-on-chip compiler testing method, apparatus, electronic device and storage medium of the present invention are described below with reference to fig. 1 to 10.

As shown in fig. 1, in one embodiment, a system-on-chip compiler testing method includes the steps of:

step S110, first test data are obtained, and a password algorithm library is called according to a password test algorithm to process the first test data so as to generate an expected output result, wherein the first test data are obtained by automatically generating random numbers or inputting the random numbers by a compiler test framework.

Specifically, batch test data, namely first test data, is generated by automatically generating random numbers or manually inputting data, and a password algorithm library is called according to a password test algorithm to process the test data, so that an expected output result is generated.

In a specific embodiment, as shown in fig. 2 and fig. 3, batch test data is generated by automatically generating random numbers or manually inputting data, cryptographic test algorithms such as SHA1, SM4 and the like are selected, and expected output results are generated by calling a cryptographic algorithm library such as OpenSSL, gmSSL and the like, wherein the size and the use case generation number of single input data can be specified.

It should be noted that the test case of the compiler consists of an open source algorithm library and a custom program segment. The open source algorithm library comprises an OpenSSL algorithm library and a GmSSL cryptographic algorithm library, and the PolyBench algorithm library comprises thirty typical numerical calculation programs for static control. Some of the programs in the algorithm library need to be input with custom data, and the input data can be automatically generated or manually defined based on a random algorithm according to test requirements and targets.

The OpenSSL is an open source security tool kit with rich functions and self-contained, provides a rich high-performance cryptography algorithm library, is widely applied, and can be used as a performance test comparison standard of encryption and decryption tasks. GmSSL is a domestic commercial cipher open source library, which realizes the comprehensive functional coverage of cryptographic algorithms, standards and secure communication protocols, supports the main stream operating system and processor including mobile terminals, supports typical domestic cipher hardware such as cipher keys, cipher cards and the like, and provides command line tools with rich functions and programming interfaces of multiple compiling languages.

In step S120, based on the data format of the first interface or the second interface customized by the compiler automatically generated MLIR, a source code is obtained, where the source code is a code that satisfies the data format of the first interface or the second interface customized by the compiler automatically generated MLIR.

Specifically, the compiler testing framework takes the password testing algorithm as a testing task, and obtains the corresponding source code meeting the data format of the first interface automatically generated by MLIR or the second interface customized by the compiler according to the data format of the first interface automatically generated by MLIR (such as ciface interface) or the second interface customized by the compiler (such as #pragma interface).

In a specific embodiment, taking the SHA1 password test algorithm as a test task, the compiler test framework firstly writes source codes according to a ciface interface which is automatically generated by MLIR or a # pragma interface which is self-defined by a compiler, and whether the front end of a system on chip compiler can analyze the source codes to generate equivalent Intermediate Representation (IR) or not is judged. Then, whether the target operator of CPU, GPU, DSP devices is successfully generated after the source operators such as message filling, hash mapping and the like of the cryptographic algorithm are optimized through intermediate representation is judged.

Wherein the multi-level intermediate Representation MLIR (Muti-LEVEL INTERMEDIATE presentation) is an extensible reusable compiler framework for domain-specific languages, which can be thought of as LLVM in the machine learning era, which aims to solve the transformation problem of intermediate representations in many different domain languages. By incorporating various intermediate representations into the dialect under MLIR's own framework (dialect), the syntax of the various intermediate representations is unified, facilitating the construction of domain-specific compilers, and integrating existing compiler technology, suitable for constructing the compilation framework of a system on a chip.

It should be noted that the testing of the compiler is divided into a functional testing and a performance testing. Wherein the functional test can be divided into according to the test object and the test object: front-end interface testing, intermediate representation optimization testing, heterogeneous instruction generation testing, consistency testing and stability testing. Because the on-chip system integrates CPU, GPU, DSP and other heterogeneous devices, the on-chip system compiler designs based on the MLIR multi-level heterogeneous compiling framework, and therefore the supporting condition of the on-chip system compiler on optimizing and compiling the heterogeneous devices is needed to be considered during testing.

The front-end interface test mainly tests whether the system-on-chip compiler supports ciface interfaces automatically generated by MLIR or inputs of two forms of # pragma interfaces defined by the compiler. The intermediate representation optimization test is to test whether a system on chip compiler can support a set of intermediate representation methods irrelevant to heterogeneous devices, and can effectively realize the correct generation from a source operator to different heterogeneous device target operators. The heterogeneous instruction generation test is verification of an end-to-end compiling function, and mainly comprises the step of judging whether an optimized intermediate representation can be correctly compiled into a target machine instruction or judging whether an operation result of the intermediate representation on a heterogeneous device is correct. The consistency test and the stability test are repeated tests on the first three types of tests in different dimensions to verify the stability and consistency of the compiling functions of the compiler.

The repetition of the consistency test is the repetition of the output result, the equivalent and same source code program segments are input to the compiler, and whether the same result can be output or not is observed. The repetition of the stability test refers to the repetition of the test flow, the whole test case is automatically circulated and operated for a plurality of times, and whether the compiler can stably output the same test result report is verified.

The performance test is mainly to obtain the running time and other indexes of the executable file after compiling the task-level test cases by a compiler, and the running time and other indexes can be monitored by predefining an operating system-level timer and a google-benchmark.

The Google-benchmark is a C++ program benchmark test platform built based on a Google-test framework and is used for measuring the performance, execution time and resource use condition of codes, a relatively automatic test running scheme and a flexible calling interface are provided, and rich and customizable performance statistics data can be provided.

In step S130, the source code is analyzed by the front end of the compiler to generate a first intermediate representation, and a first operator and a second operator, which are operators satisfying the running characteristics of different hardware of the compiler, are extracted from the first intermediate representation.

Specifically, the compiler testing framework analyzes the source code through the front end of the compiler, and further generates a corresponding intermediate representation, namely a first intermediate representation, extracts the first operator and the second operator from the first intermediate representation, namely one operator or a plurality of operators from the first intermediate representation, and each extracted operator corresponds to compiler hardware with corresponding operation characteristics.

Step S140, based on different compiler hardware operation characteristics, the first operator and the second operator are respectively optimized to obtain a first target operator and a second target operator, and the first operator and the second operator in the source code are respectively replaced by the first target operator and the second target operator to obtain an optimized code.

Specifically, the compiler testing framework performs optimization processing on the first operator and the second operator extracted in the step S130 based on the operation characteristics of different compiler hardware, so as to obtain a first target operator and a second target operator corresponding to the first operator and the second operator respectively, and replaces the source operators corresponding to the first operator and the second operator in the source code with the first target operator and the second target operator, so as to obtain a final optimized code.

And step S150, compiling the optimized code through the rear end of the compiler to generate a corresponding first target instruction and a corresponding second target instruction, and respectively sending the first target instruction and the second target instruction to compiler hardware corresponding to the operation characteristics to perform cooperative operation to obtain a cooperative operation result.

Specifically, the compiler testing framework compiles the optimized code through the back end of the compiler to generate a first target instruction and a second target instruction which respectively correspond to the first target operator and the second target operator, and then the first target instruction and the second target instruction are respectively sent to compiler hardware (such as CPU, GPU, DSP devices) with corresponding operation characteristics to perform cooperative operation, so that a cooperative operation result is obtained.

It should be noted that, the number of target instructions generated by compiling the optimized code by the compiler back end corresponds to the number of target operators in the optimized code, that is, when the number of target operators replaced in the optimized code is one, only one target instruction of the compiler hardware corresponding to the operation feature is generated, when the number of target operators replaced in the optimized code is multiple (more than two arbitrary numbers), the target instructions of the compiler hardware corresponding to the operation feature are generated, and each generated target instruction is distributed to the corresponding compiler hardware to perform cooperative operation.

In a specific embodiment, the optimized intermediate representation is compiled by the back end of the system-on-chip compiler, and the target instructions formed by compiling are respectively cooperated with each other in CPU, GPU, DSP of the system-on-chip.

In this embodiment, through the form of script running, the compiler is called for multiple times with different cryptographic algorithms to complete the compiler front-end interface test, the intermediate representation optimization test and whether the compiler back-end object code is generated to verify the consistency of the system-on-chip compiler. Meanwhile, the target instruction generated by compiling is circularly operated on the on-chip system so as to verify the stability of the on-chip system compiler. In order to verify the performance of the system on chip compiler, the compiling can be completed by calling OpenSSL, gmSSL a cryptographic algorithm library by adopting a traditional isomorphic compiler aiming at the SHA1 cryptographic algorithm, the compiled target instruction runs on the CPU, and the running time of the cryptographic algorithm on the CPU platform is compared with the running time of the cryptographic algorithm on the system on chip.

In addition, the execution performance test order/SHA 1CoreBenchmark.out, the compiler test framework calls a MLIR-based system-on-chip compiler to optimally compile SHA1 algorithm tasks, and after compiling, the target code and OpenSSL, gmSSL target code run on the same hardware platform.

And step S160, calling FILECHECK to verify the collaborative operation result based on the expected output result to obtain a verification result, and generating a compiler test report based on the verification result, wherein the verification result is used for representing whether the collaborative operation result is consistent with the expected output result or not.

Specifically, the compiler framework verifies the collaborative operation result by calling FILECHECK to obtain a verification result for representing whether the collaborative operation result is consistent with the expected output result or not, and generates a compiler test report based on the verification result.

It should be noted that, the verification result for indicating whether the co-operation result is consistent with the expected output result is only a part of the compiler test report, and other parts of the compiler test report are described in detail in the embodiment, which is not repeated herein.

In a specific embodiment, the test result check is divided into a functional check and a performance check. The functional verification mainly focuses on the correctness of test operation results, the correctness is verified by comparing the compiling operation results of main stream C/C++ compilers such as GCC, clang and the like, and the verification results can be automatically obtained by defining scripts through preset assertions of labels such as FILECHECK CHECK, CHECK-NOT, CHECK-NEXT and the like. The performance verification is judged by comparing the time consumed by compiling and running the same test case under the wafer system compiler, GCC and Clang.

Wherein, GCC is a compiling suite developed by GNU, which supports compiling of C, C ++, fortran, java, objective-C and other multiple programming languages. Meanwhile, GCC is the most widely used c/c++ compiler at present, and has stable and wide cross-operating system compiling support.

In addition, the compiler framework is driven by llvm-lites, so that automatic operation deployment of compiler testing is realized, and parameters such as a testing coverage range, a testing using tool, a testing environment and the like are configured through the lites cfg file. The global configuration can be covered by a litlocal cfg file in the subfolder, so that the local parameters are specially set, and the test result of the test framework automatically generates a test result log through script.

It should be noted that, it is a portable and extensible automatic testing framework for compilers, which supports the configuration of testing environment, management of testing cases and execution of testing cases for compilers in llvm form, and by combining with FILECHECK text pattern matching tools, it can verify testing results and give out prompt information of corresponding testing results.

In a specific embodiment, functional test result verification and performance test result verification are performed after the test is completed. In the functional test result, the front-end interface test and the intermediate representation optimization test are mainly verified based on FILECHECK matching, and whether the source operators of the cryptographic algorithm, such as message filling, hash mapping and the like, succeed in generating the target operator is judged through preset assertions of labels, such as CHECK, CHECK-NOT, CHECK-NEXT and the like, of FILECHECK. The heterogeneous instruction generation test, the consistency test and the stability test are mainly to judge whether the test result is correct by comparing the target code operation result of the system-on-chip compiler with the compiling results of other compilers and comparing the repeated cycle operation results. And the performance test result is that the running time of different compiling target codes of the same algorithm is read for comparison, and whether the performance test index meets the requirement is confirmed. After the test result is checked, test results such as test names, failure test numbers, success test numbers, unsupported tests, test time, application running speed and the like are automatically reserved in a test log, and finally a test report is formed and output.

Wherein FILECHECK is one of LLVM developer tools for verifying whether the output result of LLVM IR code meets expectations, which is commonly used in LLVM regression testing, the correctness of the code can be verified by checking a specific text pattern in the output file.

According to the system on chip compiler testing method, the compiler testing framework automatically generates or manually inputs the random numbers to generate batch testing data, and the cryptographic algorithm library is called according to the cryptographic testing algorithm to process the testing data to generate expected output results. Then, based on the ciface interface or the data format of the compiler-customized #pragma interface automatically generated MLIR, the source code meeting the data format of the ciface interface or the compiler-customized #pragma interface is acquired. And then analyzing the source code through the front end of the compiler to generate a corresponding intermediate representation, extracting operators from the intermediate representation for optimization to obtain target operators meeting the operation characteristics of the corresponding compiler hardware, and replacing the corresponding target operators with the source operators in the source code to obtain the optimized code. Compiling the obtained optimized code representation through the back end of the compiler to generate a plurality of target instructions, and sending the generated target instructions to compiler hardware corresponding to the operation characteristics to perform cooperative operation to obtain a cooperative operation result. And finally, calling FILECHECK to check the collaborative operation result to obtain a check result used for representing whether the collaborative operation result is consistent with the expected output result or not, and generating a compiler test report based on the check result. The method realizes the multi-dimensional test verification of the on-chip system by evaluating the correctness of the test result of the on-chip system compiler and verifying whether the on-chip system compiler can fully exert the system performance. In addition, the method also supports one-key deployment of users, can effectively reduce the testing cost of the compiler and improves the development efficiency of the system-on-chip compiler.

As shown in fig. 4, in one embodiment, the method for testing a system on a chip compiler provided by the present invention obtains first test data, and calls a cryptographic algorithm library according to a cryptographic test algorithm to process the first test data to generate an expected output result, which includes the following steps:

Step S410, a compiler testing framework is built based on llvm-lite testing suite, testing paths of the first testing data are sorted, and the compiler testing framework is used for supporting automatic operation deployment of compiler testing.

Specifically, before performing the compiler test, the compiler test framework needs to be built and the test environment is configured, firstly, the compiler test framework is built based on llvm-lite test suite, the test paths of the test data are arranged, and the compiler test framework is used for supporting the automatic operation deployment of the compiler test.

In a specific embodiment, as shown in fig. 2 and 3, the wafer level compiler testing framework is built based on llvm-lite testing suite, and needed software and corresponding versions thereof are built before testing, and corresponding paths of the testing catalogue and the testing code are sorted.

In step S420, a compiler testing environment is built based on the compiler testing framework, the compiler testing environment being obtained by writing litcfg and litlocal cfg.

Specifically, the configuration of the test environment is mainly realized by writing the lit.cfg and the lit.local.cfg.

Step S430, according to the test content of the first test data, obtaining a compiler test execution range, wherein the compiler test execution range at least comprises a root directory of the first test data for executing the test, a subdirectory under the root directory which does not need to execute the test and a file suffix of the executing test.

Specifically, according to the test content of the test data determined previously, the execution range of the compiler test is obtained, and the execution range of the compiler test at least comprises a root directory of the test data for executing the test, a subdirectory under the root directory which does not need to execute the test, and a file suffix for executing the test.

In a specific embodiment, the content to be configured includes: the corresponding path, test name, operating system environment, support component and the like of the test tool, after the configuration of the test environment is completed, the test content is selected, and the test execution range is set according to the test target, wherein the test execution range comprises a root directory for executing the test, a directory under the root directory which does not need to execute the test, a file suffix for executing the test and the like.

As shown in fig. 5, in one embodiment, the method for testing a system on a chip compiler provided by the present invention obtains first test data, and calls a cryptographic algorithm library according to a cryptographic test algorithm to process the first test data to generate an expected output result, and specifically includes the following steps:

Step S112, generating batch test data according to the random number automatic generation data or the custom input data to obtain first test data.

Specifically, the compiler testing framework automatically generates data according to random numbers or manually and custom-inputs data to generate batch testing data, so as to obtain required testing data, namely first testing data.

Step S114, the OpenSSL or GmSSL cryptographic algorithm library is called by the SHA1 or SM4 cryptographic test algorithm to generate the expected output result.

Specifically, the compiler testing framework calls the OpenSSL or GmSSL cryptographic algorithm library to generate the expected output result through the SHA1 or SM4 cryptographic testing algorithm.

As shown in fig. 6, in one embodiment, the method for testing a system on chip compiler according to the present invention obtains source code based on MLIR automatically generated data formats of a first interface or a second interface customized by the compiler, and then further includes the following steps:

in step S610, the source code is compiled according to the ciface interface automatically generated MLIR or the data format of the #pragma interface customized by the compiler, and it is determined whether the first intermediate representation generated by the front end of the compiler has equivalent semantics with the source code.

Specifically, the compiler testing framework writes the source code according to the ciface interface automatically generated by MLIR or the data format of the # pragma interface customized by the compiler, and judges whether the first intermediate representation generated by the front end of the compiler has equivalent semantics with the source code or not, so as to obtain a test result whether the intermediate representation is successfully generated or not.

Step S620, further judging whether the source operator of the password test algorithm is optimized to generate a target operator meeting the operation characteristics of the corresponding compiler hardware after being processed through intermediate representation, and obtaining a first judgment result.

Specifically, based on the judgment in step S610, it is further judged whether the source operators such as message filling and hash mapping of the cryptographic test algorithm are optimized to generate the target operator corresponding to the compiler hardware operation feature after the intermediate representation processing, for example, the first target operator and the second target operator, so as to obtain the test result whether the target operator is successfully generated.

The first judgment result is composed of whether the intermediate representation is successfully generated or not and whether the target operator is successfully generated or not, and is part of a compiler test report finally formed.

As shown in fig. 7, in one embodiment, the method for testing a system on chip compiler provided by the present invention calls FILECHECK to verify the collaborative operation result based on the expected output result, to obtain a verification result, and generates a compiler test report based on the verification result, and further includes the following steps:

and step S710, selecting the target code from different cipher algorithm libraries, compiling the target code into a target instruction, and transmitting the target instruction to the same compiler hardware as the optimized code for operation to obtain an operation result of the target code.

Specifically, the compiler testing framework selects target codes with the same compiler hardware operation characteristics as the optimized codes from different cipher algorithm libraries, compiles the selected target codes into corresponding target instructions, and sends the corresponding target instructions to the compiler hardware which is the same as the optimized codes for operation, so that operation results of the selected target codes are obtained.

Step S720, based on the running result of the target code, judging whether the running result of the target code is consistent with the collaborative running result, and obtaining a second judging result.

Specifically, the compiler testing framework determines, based on the running result of the object code obtained in step S710, whether the running result of the object code is consistent with the obtained collaborative running result, to obtain a corresponding determination result, that is, a second determination result.

The second determination result is a part of the finally formed compiler test report.

As shown in fig. 8, in one embodiment, the method for testing a system-on-chip compiler provided by the present invention further includes the following steps:

And step 810, taking the same password test algorithm as a test task for a plurality of times, and obtaining a collaborative operation result corresponding to each password test algorithm.

Specifically, the compiler testing framework loops for a plurality of times, takes the same password testing algorithm as a testing task, and obtains a cooperative operation result corresponding to each password testing algorithm.

Step S820, judging whether the collaborative operation results corresponding to the multiple password test algorithms are consistent or not to obtain a third judgment result for verifying the test stability of the compiler.

Specifically, the compiler testing framework determines, based on the cooperative operation result obtained in step S810, whether the cooperative operation results corresponding to the multiple-cycle cryptographic testing algorithm are consistent, so as to obtain a determination result for verifying the stability of the compiler test, that is, a third determination result, so as to implement verification of the stability of the compiler.

It should be noted that the third determination result is part of the finally formed compiler test report.

The system-on-chip compiler testing device provided by the invention is described below, and the system-on-chip compiler testing device described below and the system-on-chip compiler testing method described above can be referred to correspondingly.

As shown in FIG. 9, in one embodiment, a system-on-chip compiler testing apparatus includes an expected result generation module 910, a source code acquisition module 920, an operator extraction module 930, an operator optimization module 940, a co-operation module 950, and a test report generation module 960.

The expected result generating module 910 is configured to obtain first test data, and call a cryptographic algorithm library according to a cryptographic test algorithm to process the first test data to generate an expected output result, where the first test data is obtained by automatically generating a random number or performing custom input on the first test data by using a compiler test framework.

The source code obtaining module 920 is configured to obtain source code based on a data format of the first interface or the second interface customized by the compiler, which is automatically generated MLIR, where the source code is a code that satisfies the data format of the first interface or the second interface customized by the compiler, which is automatically generated MLIR.

The operator extraction module 930 is configured to analyze the source code through the front end of the compiler to generate a first intermediate representation, and extract, from the first intermediate representation, a first operator and a second operator, which are operators that satisfy different hardware operation characteristics of the compiler, respectively.

The operator optimization module 940 is configured to optimize the first operator and the second operator based on different compiler hardware operation features, to obtain a first target operator and a second target operator, and replace the first operator and the second operator in the source code with the first target operator and the second target operator, respectively, to obtain an optimized code.

The co-operation module 950 is configured to compile the optimized code through a back end of the compiler to generate a corresponding first target instruction and a corresponding second target instruction, and send the first target instruction and the second target instruction to the compiler hardware corresponding to the operation feature to perform co-operation, so as to obtain a co-operation result.

The test report generating module 960 is configured to invoke FILECHECK to verify the collaborative operation result based on the expected output result, obtain a verification result, and generate a compiler test report based on the verification result, where the verification result is used to characterize whether the collaborative operation result is consistent with the expected output result.

In this embodiment, the system on chip compiler testing device provided by the invention further includes a testing frame building module for:

and building a compiler testing framework based on llvm-lite testing suite, and sorting the testing paths of the first testing data, wherein the compiler testing framework is used for supporting automatic operation deployment of compiler testing.

A compiler testing environment is built based on the compiler testing framework, the compiler testing environment being obtained by writing lit.cfg and litlocal.cfg.

In this embodiment, the system on chip compiler testing device provided by the present invention further includes a testing range determining module, configured to:

In this embodiment, the system-on-chip compiler testing device provided by the present invention, the expected result generating module is specifically configured to:

and automatically generating data according to the random number or custom input data to generate batch test data so as to acquire first test data.

And (5) calling an OpenSSL or GmSSL cryptographic algorithm library through the SHA1 or SM4 cryptographic test algorithm to generate an expected output result.

In this embodiment, the system-on-chip compiler testing device provided by the present invention further includes a consistency testing module, configured to:

Compiling the source code according to the ciface interface automatically generated MLIR or the data format of the # pragma interface customized by the compiler, and judging whether the first intermediate representation generated by the front end of the compiler has equivalent semantics with the source code. And

And further judging whether a source operator of the password test algorithm is optimized to generate a target operator meeting the operation characteristics of corresponding compiler hardware after being processed through intermediate representation, and obtaining a first judgment result.

The source operator at least comprises message filling and hash mapping, and the compiler test report further comprises a first judgment result.

In this embodiment, the system on chip compiler testing device provided by the present invention further includes a performance testing module, configured to:

And selecting the target code from the different cipher algorithm libraries, compiling the target code into a target instruction, and transmitting the target instruction to the same compiler hardware as the optimized code for operation to obtain an operation result of the target code.

And judging whether the running result of the target code is consistent with the collaborative running result or not based on the running result of the target code, and obtaining a second judging result.

The target code and the optimized code have the same compiler hardware operation characteristics, and the compiler test report further comprises a second judgment result.

In this embodiment, the system on chip compiler testing device provided by the present invention further includes a stability testing module, configured to:

And taking the same password test algorithm as a test task for a plurality of times, and obtaining a cooperative operation result corresponding to each password test algorithm.

And judging whether the cooperative operation results corresponding to the multiple password test algorithms are consistent or not so as to obtain a third judgment result for verifying the test stability of the compiler.

The compiler test report further includes a third judgment result.

Fig. 10 illustrates a physical structure diagram of an electronic device, which may be an intelligent terminal, and an internal structure diagram thereof may be as shown in fig. 10. The electronic device includes a processor, an internal memory, and a network interface connected by a system bus. Wherein the processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device includes a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the electronic device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a system-on-a-chip compiler testing method, the method comprising:

acquiring first test data, and calling a cryptographic algorithm library according to a cryptographic test algorithm to process the first test data so as to generate an expected output result, wherein the first test data is obtained by automatically generating random numbers or inputting the random numbers by a compiler test framework;

Acquiring source codes based on the data format of the first interface or the second interface customized by the compiler automatically generated MLIR, wherein the source codes are codes meeting the data format of the first interface or the second interface customized by the compiler automatically generated MLIR;

Analyzing the source code through the front end of the compiler to generate a first intermediate representation, and extracting a first operator and a second operator from the first intermediate representation, wherein the first operator and the second operator are operators meeting the running characteristics of different compiler hardware respectively;

optimizing the first operator and the second operator respectively based on different compiler hardware operation characteristics to obtain a first target operator and a second target operator, and replacing the first operator and the second operator in the source code with the first target operator and the second target operator respectively to obtain an optimized code;

Compiling the optimized code through the back end of the compiler to generate a corresponding first target instruction and a corresponding second target instruction, and respectively sending the first target instruction and the second target instruction to compiler hardware corresponding to the operation characteristics to perform cooperative operation to obtain a cooperative operation result;

And calling FILECHECK to verify the collaborative operation result based on the expected output result to obtain a verification result, generating a compiler test report based on the verification result, wherein the verification result is used for representing whether the collaborative operation result is consistent with the expected output result or not.

In another aspect, the present invention also provides a computer storage medium storing a computer program, which when executed by a processor, implements a method for testing a system-on-chip compiler, the method comprising:

In yet another aspect, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from a computer readable storage medium, the processor when executing the computer instructions implementing a system-on-a-chip compiler testing method, the method comprising:

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory.

By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SYNCHLINK) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples illustrate only a few embodiments of the invention and are described in detail herein without thereby limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.

Claims

1. A system-on-chip compiler testing method, the method comprising:

2. The system-on-a-chip compiler testing method of claim 1, wherein the acquiring the first test data and invoking a cryptographic algorithm library to process the first test data according to a cryptographic test algorithm to generate the expected output result comprises:

3. The system-on-a-chip compiler testing method of claim 2, wherein the fetching the first test data and invoking a cryptographic algorithm library to process the first test data according to a cryptographic test algorithm to generate the expected output result, further comprising:

4. The system on a chip compiler testing method according to claim 1, wherein the cryptographic testing algorithm at least comprises SHA1 and SM4 cryptographic testing algorithms, and the cryptographic algorithm library at least comprises OpenSSL and GmSSL cryptographic algorithm libraries;

5. The system on a chip compiler testing method of claim 1, wherein the first interface is ciface interface and the second interface is #pragma interface;

6. The system-on-a-chip compiler testing method of claim 5, wherein said invoking FILECHECK verifies said co-operating results based on said expected output results to obtain verification results and generates a compiler test report based on said verification results, further comprising:

7. The system-on-a-chip compiler testing method according to any one of claims 1 to 6, further comprising:

8. A system-on-chip compiler testing apparatus, the apparatus comprising:

9. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 7 when the computer program is executed.

10. A computer storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method of any one of claims 1 to 7.