CN111338874B

CN111338874B - Method and system for testing GPU (graphics processing Unit) rate switching

Info

Publication number: CN111338874B
Application number: CN202010132969.XA
Authority: CN
Inventors: 孔祥源
Original assignee: Suzhou Inspur Intelligent Technology Co Ltd
Current assignee: Suzhou Inspur Intelligent Technology Co Ltd
Priority date: 2020-02-29
Filing date: 2020-02-29
Publication date: 2022-05-31
Anticipated expiration: 2040-02-29
Also published as: CN111338874A

Abstract

The application discloses a method and a system for testing GPU rate switching, wherein the method comprises the following steps: acquiring the bus numbers of all GPUs in the current system, storing the bus numbers into a first file, acquiring the total testing times, defining a testing time variable, sequentially reading the bus numbers of all GPUs in the first file, and judging whether the reading is successful or not; if the reading is successful, sequentially carrying out gen1, gen2 and gen3 switching tests on all GPUs in the current system; after testing of all GPUs in the current system is finished, adding 1 to the testing time variable, and reading the bus numbers of all GPUs in the first file again in sequence until the testing time variable reaches the total testing time; if the read fails, the test ends. The system comprises: the bus number acquisition module, the test frequency counting module, the reading module, the judgment module, the test module and the circulation module. Through the method and the device, the testing efficiency and the accuracy of the testing result can be effectively improved.

Description

Method and system for testing GPU (graphics processing Unit) rate switching

Technical Field

The present disclosure relates to the field of GPU (Graphics Processing Unit) testing technologies, and in particular, to a method and a system for testing GPU rate switching.

Background

Along with the development of artificial intelligence technology, AI calculation puts higher demands on the performance and stability of the server. The AI server widely adopts a CPU + GPU heterogeneous form, and in the development and design process of the AI server, a plurality of tests are usually adopted to ensure the stability of the server, wherein the test of the GPU stability in the AI server is particularly important. In the test of the stability of the GPU, the GPU rate switching test is an effective test means. Therefore, how to perform rate switching test on the GPU so as to verify the stability of the GPU and further verify the stability of the AI server is an important issue.

At present, a method for performing rate switching test on GPUs generally includes that a tester inputs instructions one by one to obtain PCIE bus number of a certain GPU, and then performs rate switching test on a current GPU through the instructions.

However, in the current method for performing rate switching test on GPUs, because one system includes several GPUs, there may be differences in the number of PCIE buses of the GPUs in each system. The testing personnel need to determine the PCIE bus numbers of all the GPUs in the system one by one and test the GPUs one by one, the test occupies too much time, and test errors such as tested and untested confusion easily occur, so that the test efficiency of the current test method is low.

Disclosure of Invention

The application provides a method and a system for testing GPU rate switching, which aim to solve the problem of low testing efficiency in the prior art.

In order to solve the technical problem, the embodiment of the application discloses the following technical scheme:

a method of testing GPU rate switching, the method comprising:

the method comprises the steps of obtaining bus numbers of all GPUs in a current system, and storing the bus numbers into a first file, wherein any GPU is matched with one bus number;

acquiring the total test times, and defining a test time variable, wherein the test time variable is used for counting the test times;

sequentially reading the bus numbers of all the GPUs in the first file;

judging whether the reading is successful;

if the reading is successful, sequentially carrying out gen1, gen2 and gen3 switching tests on all GPUs in the current system;

after testing of all GPUs in the current system is finished, adding 1 to the testing time variable, and reading the bus numbers of all GPUs in the first file again in sequence until the testing time variable reaches the total testing time;

if the read fails, the test ends.

Optionally, the sequentially performing gen1, gen2, and gen3 switching tests on all GPUs in the current system includes:

performing gen1, gen2 and gen3 switching tests on the first GPU respectively;

judging whether the gen1, gen2 and gen3 switching tests of the first GPU are all tested to be qualified;

if so, judging that the rate switching test of the first GPU is qualified;

if not, judging that the rate switching test of the first GPU is unqualified;

saving error information related to unqualified test in a second file, and deleting the current GPU from the first file, wherein the error information comprises: the bus number of the current GPU, the error reporting time and the sequence of the current error reporting in the total error reporting amount;

and respectively carrying out gen1, gen2 and gen3 switching tests on the second GPU until all GPUs in the current system are respectively subjected to gen1, gen2 and gen3 switching tests.

Optionally, the test results are stored in a local test log.

Optionally, the test results are displayed.

A system to test GPU rate switching, the system comprising:

the system comprises a Bus number acquisition module, a Bus number acquisition module and a Bus number storage module, wherein the Bus number acquisition module is used for acquiring Bus numbers of all GPUs in the current system and storing the Bus numbers into a first file, and any GPU is matched with one Bus number;

the test frequency counting module is used for acquiring the total test frequency and defining a test frequency variable, and the test frequency variable is used for counting the test frequency;

the reading module is used for sequentially reading the bus numbers of all the GPUs in the first file;

the judging module is used for judging whether the reading is successful or not, and when the reading is failed, the test is finished;

the test module is used for sequentially carrying out gen1, gen2 and gen3 switching tests on all GPUs in the current system when reading is successful;

and the circulating module is used for adding 1 to the testing time variable after all the GPUs in the current system are tested, and reading the bus numbers of all the GPUs in the first file again in sequence until the testing time variable reaches the total testing time.

Optionally, the test module comprises:

the test unit is used for respectively carrying out gen1, gen2 and gen3 switching tests on the first GPU;

the judging unit is used for judging whether the gen1, gen2 and gen3 switching tests of the first GPU are qualified or not, judging that the rate switching test of the first GPU is qualified if the switching tests of the gen1, the gen2 and the gen3 of the first GPU are qualified, and judging that the rate switching test of the first GPU is unqualified if the switching tests of the first GPU are not qualified;

the processing unit is used for storing error information related to unqualified test into a second file and deleting the current GPU from the first file, wherein the error information comprises: the bus number of the current GPU, the error reporting time and the sequence of the current error reporting in the total error reporting amount;

the test unit is further configured to perform gen1, gen2, and gen3 switching tests on the second GPU, respectively, until the gen1, gen2, and gen3 switching tests on all GPUs in the current system are completed.

Optionally, the system further includes a storage module for storing the test result.

Optionally, the system further includes a display module for displaying the test result.

The technical scheme provided by the embodiment of the application can have the following beneficial effects:

the method can ensure that all the GPUs in the current system execute the rate switching test by acquiring the bus numbers of all the GPUs in the current system, storing the bus numbers into a first file and defining a test time variable to prepare for sequentially carrying out the rate switching test on all the GPUs subsequently, is favorable for improving the accuracy of test results, is convenient for carrying out the test according to the sequence and improves the test efficiency. In the embodiment, the bus numbers of the GPUs in the first file are read in sequence, and the test is performed step by step according to the sequence of the bus numbers from large to small or from small to large, so that the improvement of the test efficiency is facilitated, the omission of any test can be avoided, and the improvement of the accuracy of the test result is facilitated. By sequentially performing gen1, gen2 and gen3 switching tests on all GPUs in the current system and taking the GPUs as units, three speed switching tests are respectively executed on each GPU, and the improvement of the testing efficiency is facilitated. In addition, in the embodiment, all the GPUs in the current system can be tested in one terminal, and the number of the GPUs is not limited by the number of the terminals, so that the method is favorable for popularization and application.

The present application further provides a system for testing GPU rate switching, which mainly includes: the bus number acquisition module, the test frequency counting module, the reading module, the judgment module, the test module and the circulation module. The setting of the Bus number acquisition module and the test frequency counting module in the embodiment can provide basis for subsequent tests, so that the test frequency can be counted conveniently, and the test efficiency is improved. The test module is arranged, so that any GPU in the current system can be ensured to execute gen1, gen2 and gen3 switching tests, and the accuracy of test results is improved. And the circulation module is arranged, so that the speed switching test of the next GPU is started in sequence, and the test efficiency is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flowchart of a method for testing GPU rate switching according to an embodiment of the present disclosure;

FIG. 2 is a diagram of an error reporting GPU log output format;

FIG. 3 is a schematic flow chart illustrating GPU rate switching in practical applications according to an embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of a system for testing GPU rate switching according to an embodiment of the present disclosure.

Detailed Description

In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

For a better understanding of the present application, embodiments of the present application are explained in detail below with reference to the accompanying drawings.

Example one

Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a method for testing GPU rate switching according to an embodiment of the present disclosure. As shown in fig. 1, the method for testing GPU rate switching in this embodiment mainly includes the following processes:

s1: and acquiring the bus numbers of all the GPUs in the current system, and storing the bus numbers into a first file, wherein any one GPU is matched with one bus number.

The server system of this embodiment is an AI server, and the current server system includes a plurality of GPUs, one GPU for each GPU. Firstly, obtaining the bus numbers of all the GPUs in the current system is equivalent to obtaining the information of each GPU, and then storing the bus numbers of all the GPUs in a first file. A folder GPU _ SPEEDTEST _ LOG may be created under the current working directory under which subsequent related files are stored, e.g. here the first file is stored under the folder GPU _ SPEEDTEST _ LOG, the first file may be named: gpubus. The first file is established, so that the GPU information can be managed conveniently, and the testing efficiency and the accuracy of the testing result are improved.

S2: and acquiring the total test times, and defining a test time variable which is used for counting the test times.

The total number of tests is determined, typically ten thousand and more, based on the instructions fetched. A variable may be denoted as loopcnt, which is incremented after all GPUs in the first file gpubus. The embodiment utilizes a variable of the test times, so that the test times can be accurately recorded according to the test result.

S3: and sequentially reading the bus numbers of all the GPUs in the first file.

S4: and judging whether the reading is successful.

If the reading is successful, go to step S5: and sequentially carrying out gen1, gen2 and gen3 switching tests on all GPUs in the current system. All GPUs in the system may be tested in a certain order, for example: the GPU can be sequentially tested according to the sequence of bus number of the GPU from small to large. If the reading is unsuccessful, all the GPUs in the current first file fail to be tested after a certain loopcnt cycle is shown, and the test is finished.

In the present example, the gen1, gen2, and gen3 switching tests were performed at a gen1 rate of 2.5GT/s, a gen2 rate of 5GT/s, and a gen3 rate of 8 GT/s. When the speed is switched to a certain speed, for example, when the speed is switched to 8GT/s, an lspci instruction is adopted to carry out information scanning on the current GPU, the speed state keyword of the current GPU is intercepted, if the state keyword is 8GT/s, the switching is successful, otherwise, the switching is failed.

Specifically, step S5 includes the following processes:

s51: the first GPU was tested for gen1, gen2, and gen3 switching, respectively.

S52: a determination is made as to whether the gen1, gen2, and gen3 toggle tests for the first GPU all passed.

If all the gen1, gen2, and gen3 switching tests of the first GPU pass, execute step S53: and judging that the rate switching test of the first GPU is qualified.

If any of the gen1, gen2, and gen3 switching tests of the first GPU fail to meet the test condition, step S54 is executed: and judging that the rate switching test of the first GPU is not qualified.

When the test fails, step S55 is executed: and storing error information related to unqualified test into a second file, and deleting the current GPU from the first file. Wherein the error information includes: the bus number of the current GPU, the error reporting time, and the order of the current error reporting in the total number of errors. That is, information such as the bus number, the error reporting time, and the number of times of error reporting of the current GPU is saved in the second folder. The second file and the first file are stored in the same folder, so that the search is convenient, the second file and the first file can be stored in a GPU _ SPEEDTEST _ LOG folder, and the second file can be named as failgpullog. In this embodiment, taking the test of the GPU0 as an example, if the rate switching of the GPU0 fails in a certain test, according to step S55, the error information is saved in the second file failgpubl.txt in the GPU _ SPEEDTEST _ LOG folder, and the GPU0 is deleted from the first file gpubus.tx. Through step S55, in the next cycle of testing, the current GPU will not be tested any more, and only the GPU qualified in the testing executes the next cycle of testing, so that an accurate testing result can be obtained, and the testing efficiency is improved.

After the first GPU is tested, returning to step S51, performing gen1, gen2, and gen3 switching tests on the second GPU respectively until all GPUs in the current system are subjected to gen1, gen2, and gen3 switching tests respectively.

Namely: steps S51-S55 are re-executed for the second GPU. Steps S51-S55 are then re-executed for the third GPU. The test of the first cycle ends until the last GPU in the current system performs steps S51-S55.

Further, step S56 may also be included: judging whether all GPUs in the current first file are completely tested;

if all the GPUs in the current first file are tested, the next step S6 is executed. And if the test of part of the GPUs in the current first file is not completed, returning to the step S51, and continuously testing the rest GPUs.

With continued reference to fig. 1, after the gen1 test, the gen2 test and the gen3 test are performed on all GPUs in the current system in sequence, step S6 is performed: add 1 to the variable of the number of tests, and then return to step S5: and sequentially reading the bus numbers of all the GPUs in the first file until the variable of the testing times reaches the total testing times.

After the test of the first loop is finished, testing the variable loopcnt +1 of the number of times, and setting the current reading position as the first line in the first file, that is: and re-reading the bus numbers of all the GPUs in the first file.

Further, after the test in this embodiment is completed, the method further includes step S7: and storing the test result into a local test log. And the subsequent analysis of the test result by the staff is facilitated. The GPU log output format for error reporting in this embodiment can be seen in fig. 2.

Further, the present embodiment further includes step S8: the test result is displayed, so that the test result can be conveniently and visually checked by the staff, and the debugging efficiency is improved.

Fig. 3 is a schematic flowchart of the GPU rate switching in practical application according to the embodiment of the present application. In fig. 3, Y indicates yes, N indicates no, gpubs.txt is the first file, and failgpoulg.txt is the second file.

Carry out two

Based on the embodiments shown in fig. 1-3, referring to fig. 4, fig. 4 is a schematic structural diagram of a system for testing GPU rate switching according to the embodiments of the present disclosure. As can be seen from fig. 4, the system for testing GPU rate switching in this embodiment mainly includes: the bus number acquisition module, the test frequency counting module, the reading module, the judgment module, the test module and the circulation module.

The Bus number obtaining module is used for obtaining Bus numbers of all GPUs in the current system and storing the Bus numbers into a first file, wherein any GPU is matched with one Bus number. And the test frequency counting module is used for acquiring the total test frequency and defining a test frequency variable, and the test frequency variable is used for counting the test frequency. And the reading module is used for sequentially reading the bus numbers of all the GPUs in the first file. And the judging module is used for judging whether the reading is successful or not, and when the reading is failed, the test is finished. And the test module is used for sequentially carrying out gen1, gen2 and gen3 switching tests on all GPUs in the current system when the reading is successful. And the circulating module is used for adding 1 to the testing time variable after all the GPUs in the current system are tested, and reading the bus numbers of all the GPUs in the first file again in sequence until the testing time variable reaches the total testing time.

Further, the test module includes: the device comprises a test unit, a judgment unit and a processing unit. The test unit is used for respectively performing gen1, gen2 and gen3 switching tests on the first GPU. And the judging unit is used for judging whether the gen1 test, the gen2 test and the gen3 test of the first GPU are qualified or not, judging that the rate switching test of the first GPU is qualified if the test is qualified, and judging that the rate switching test of the first GPU is unqualified if the test is not qualified. The processing unit is used for storing error information related to unqualified test into a second file and deleting the current GPU from the first file, wherein the error information comprises: the bus number of the current GPU, the error reporting time, and the order of the current error reporting in the total number of errors. And the test unit is further used for respectively carrying out gen1, gen2 and gen3 switching tests on the second GPU until the gen1, gen2 and gen3 switching tests on all GPUs in the current system are finished.

Further, the system of this embodiment further includes a storage module for storing the test result. The device also comprises a display module used for displaying the test result.

The working principle and the working method for testing the GPU rate switching system in this embodiment have been described in detail in the embodiments shown in fig. 1 to fig. 3, and are not described herein again.

The above description is merely exemplary of the present application and is presented to enable those skilled in the art to understand and practice the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for testing GPU rate switching, the method comprising:

sequentially reading the bus numbers of all the GPUs in the first file;

judging whether the reading is successful;

if the read fails, the test ends.

2. The method for testing GPU rate switching according to claim 1, wherein the sequential gen1, gen2 and gen3 switching tests on all GPUs in the current system comprise:

performing gen1, gen2 and gen3 switching tests on the first GPU respectively;

if so, judging that the rate switching test of the first GPU is qualified;

if not, judging that the rate switching test of the first GPU is unqualified;

3. The method of claim 1, wherein the test results are stored in a local test log.

4. A method for testing GPU rating switching according to any of claims 1-3, wherein the test results are displayed.

5. A system for testing GPU rate switching, the system comprising:

the Bus number acquisition module is used for acquiring the Bus numbers of all GPUs in the current system and storing the Bus numbers into a first file, wherein any GPU is matched with one Bus number;

6. The system for testing GPU rate switching according to claim 5, wherein the testing module comprises:

the processing unit is used for storing error information related to unqualified tests into a second file and deleting the current GPU from the first file, wherein the error information comprises: the bus number of the current GPU, the error reporting time and the sequence of the current error reporting in the total error reporting amount;

7. The system according to claim 5, further comprising a storage module configured to store the test results.

8. A system for testing GPU rate switching according to any of claims 5-7, further comprising a display module for displaying the test results.