TW202328905A - General-purpose graphics processor core function configuration setting prediction system and method having a set screening module, a resource occupancy screening module, an execution module, a training data extraction module, and a modeling module - Google Patents

Abstract

The present invention provides a general-purpose graphics processor core function configuration setting prediction system and method, which is suitable for the general-purpose graphics processor, wherein the system includes: a set screening module to perform preliminary screening on the parameter set of the configuration setting of the general-purpose graphics processor core function, the screened parameter set being used as an input parameter; a resource occupancy screening module to perform secondary screening on the input parameters with the resource occupancy rate value of the Warp and registers and the shared memory in the stream composite processor, so as to obtain the first execution configuration setting; an execution module executing the first execution configuration setting by the general-purpose graphics processor, so as to obtain the execution time; a training data extraction module extracting a second execution configuration setting as the training data from the first execution configuration setting according to the execution time; and a modeling module using the training data to establish a configuration setting prediction model of a core function of the general-purpose graphics processor. Therefore, the present invention can quickly and accurately predict the configuration setting of the core function.

Description

System and method for configuration setting prediction of general-purpose graphics processor core function

The present invention relates to a general-purpose graphics processor core function configuration setting prediction system and method, in particular to a general-purpose graphics processor core function configuration setting prediction system and method using machine learning to establish a supervised learning model .

With the development of artificial intelligence, machine learning, and deep learning technologies, related technologies require a lot of computing power, and the development of models is becoming more and more complex. If only a general processor (Central Processing Unit; hereinafter referred to as CPU) is used for computing , it will take more time to train the model; on the other hand, since the operations of each element in the neural network layer have no dependencies and are independent, and will not affect each other's calculation results, it can be achieved by using general-purpose graphics processing The GPU (General-Purpose Graphics Processing Unit, GPGPU, for convenience, hereinafter referred to as GPU) hardware acceleration shortens the time required for the training process. The GPU has a highly parallel computing architecture. Compared with only a few or dozens of cores in the CPU, the GPU has thousands of CUDA (Compute Unified Device Architecture, unified computing architecture) cores, and these CUDA cores can be parallelized. operate. Although its individual cores are not as powerful as CPU cores, which can execute more complex instructions and perform various forms of task loads, GPUs can greatly improve the work efficiency of specific operations by virtue of their number advantages.

However, configuration settings will affect the use of the underlying resources of the GPU during execution. Although it is possible to make the core function execute by randomly setting a configuration, it is not easy to achieve the best execution performance. If you want to get the best configuration setting, you may need to test all possible configuration setting combinations before you can determine which one is the best configuration setting; in addition, the programs of different core functions operate differently, and the resources and logic used The difference is large, so the optimal configuration settings are also different. Even if the same core function is used in different GPUs or the amount of data of different sizes is calculated, the optimal configuration settings may be different. Although the best configuration settings can be found in all combinations by using the brute force method or by testing all the optimization methods and factors, but the time and space costs are too large, so the optimal configuration can be obtained quickly and immediately in the execution stage. Optimum configuration settings are impossible.

In this case, in order to effectively improve the computing performance of the GPU, the existing technology can use the resource occupancy of the estimated GPU hardware data in the execution of the core function, through the three types of thread group (Warp), temporary register, and shared memory. The amount of limited resources used in the GPU is used to calculate the usage ratio of Warp in a Streaming Multiprocessor: SM. When the GPU executes core functions, the above three limited resources will affect the existence of Warp in the SM at the same time. The number of warps that can be executed at the same time in SM has a certain upper limit, so the resource occupancy can be calculated by "estimating the number of warps/the upper limit of warps", which can roughly show the parallel efficiency.

However, considering the utilization rate of the three limited resources of Warp, register, and shared memory in each configuration setting, if a single resource has a high occupancy rate, it does not necessarily mean better performance. Therefore, it is difficult to truly reflect the performance only by considering the Warp utilization rate. Better configuration settings.

On the other hand, a core function will process and operate data of various possible sizes, and there may be many combinations of configuration settings that it can adopt. Therefore, it will be a challenge to completely test data of all sizes and all corresponding configuration settings time consuming and impossible. In addition, in the prior art, the author proposes a way to optimize the performance and energy consumption of single and multiple core functions, select a configuration with a higher occupancy rate for testing, and avoid testing all possible configurations, but it still uses violence Moreover, when using occupancy screening, misjudgments may occur, and configurations with better performance but lower occupancy are eliminated first. [Prior Technical Literature] [Non-patent literature]

[Non-Patent Document 1] K. Iliakis, S. Xydis, and D. Soudris, “LOOG: Improving GPU Efficiency With Light-weight Out-Of-Order Execution,” IEEE Computer Architecture Letters, Vol. 18, No. 2, pp. 166-169, 1 July-Dec. 2019. [Non-Patent Document 2] J. Guerreiro, A. Ilic, N. Roma and P. Tomás, "Multi-kernel Auto-Tuning on GPUs: Performance and Energy-Aware Optimization," 2015 23rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, 2015.

[Technical problem to be solved by the invention]

Accordingly, the present invention proposes a two-stage mechanism: offline and online, which can quickly and accurately suggest a set of configuration settings for the core function to complete operations with less execution time when the core function is to be executed. [Technical means]

In the off-line stage, the present invention firstly screens the parameter set, then comprehensively considers the resource occupancy rate of each configuration setting, leaves configuration settings with better execution performance, and then uses these configuration settings to execute the core functions respectively. And record the time it takes to execute. Afterwards, the training data needed to build the model is extracted. In the online stage, the present invention chooses machine learning methods to establish a predictive model to recommend a set of configuration settings with better performance.

Specifically, the present invention provides a general-purpose graphics processor core function configuration setting prediction system, which is suitable for the general-purpose graphics processor, which includes: A set screening module, which preliminarily screens the parameter set of the configuration setting of the general-purpose graphics processor core function, and takes the filtered parameter set as an input parameter; A resource occupancy rate screening module, which performs secondary screening on the input parameters by the resource occupancy values of Warp, temporary register and shared memory in the stream composite processor to obtain the first execution configuration setting; An execution module, the first execution configuration is set to be executed by the general-purpose graphics processor, and the execution time is obtained; A training data extraction module, extracting a second execution configuration from the first execution configuration according to the execution time as training data; and A modeling module is used to establish a configuration setting prediction model of the general-purpose graphics processor core function by using the training data.

Further, the parameter set may include a data volume set, a necessary configuration set and an optional configuration set.

Further, the training data extraction module can extract from the necessary configuration set and the optional configuration set according to the execution time for each data volume set, and combine the data volume set, the extracted The required configuration set and the optional configuration set are used as training data.

Furthermore, the modeling module can use K-neighbor algorithm or logistic regression algorithm to establish a model.

Further, the set screening module can preliminarily filter the data set by setting the interval of element attributes in the data set.

Further, the set screening module can preliminarily screen the necessary configuration set by selecting the number of threads in the thread block with a multiple of 32.

Furthermore, the set screening module can further optimize the optional configuration set, and the optimization method can be tiled or shared memory patches.

Further, the preliminary screening method of the set screening module for the optional configuration set may be: select several sets of the data volume, and select the front, middle, and back of these ranges within the range of the several sets of data volumes Segment sub-ranges, test the sub-ranges, all the required configuration sets and all the optional configuration sets, and filter out the optional configuration sets from the execution results with most execution performances.

The present invention also provides a general-purpose graphics processor core function configuration setting prediction method, which is suitable for the general-purpose graphics processor, which includes: Preliminary screening is performed on the parameter set of the configuration setting of the general-purpose graphics processor core function, and the parameter set that has been screened is used as an input parameter; The input parameter is secondarily screened by the resource occupancy values of Warp, temporary register and shared memory in the stream composite processor to obtain the first execution configuration setting; Setting the first execution configuration to be executed by the general-purpose graphics processor to obtain an execution time; Extracting a second execution configuration from the first execution configuration according to the execution time as training data; Establishing a configuration setting prediction model of the general-purpose GPU kernel function by using the training data; and A set of configuration settings is predicted by the prediction model.

Further, the parameter set may include a data volume set, a necessary configuration set and an optional configuration set. [Efficacy of the invention]

With the general-purpose graphics processor core function configuration setting prediction system and method of the present invention, comprehensively considering the resource occupancy values of Warp, temporary register and shared memory in the stream composite processor as screening criteria, it can be more Accurately calculate the performance of configuration settings.

In addition, the preliminary screening of the parameter set in the offline stage can effectively reduce the cost in the predictive configuration setting process according to the user parameters.

Moreover, by using the prediction model in the online stage, the present invention can reduce the number of accesses to the global memory, greatly reduce the time for accessing data from the memory, and then improve the efficiency of the core function when it is running, and quickly recommend appropriate necessary configuration settings.

Moreover, the present invention obtains execution time by first executing different amounts of data under related configuration settings, and then importing these existing data into the prediction model in the online stage, so that the prediction model can recommend a shorter time effectively and quickly. Configuration settings for execution time.

Moreover, if a variety of parameter sets are further adopted, the present invention can reduce the amount of data to be executed in the offline stage, and can also recommend appropriate execution configurations according to the amount of data in the online stage.

To sum up, the present invention proposes a system and method capable of rapidly and accurately predicting the configuration setting, the predicted configuration setting error value is small, and the execution performance is better.

The configuration and setting prediction system and method of the general-purpose graphics processor kernel function of the present invention are described below by means of exemplary implementations. It should be noted that the following exemplary embodiments are only used to illustrate the present invention, but not to limit the scope of the present invention.

FIG. 1 is a schematic block diagram of a configuration setting prediction system for a general-purpose graphics processor core function, and FIG. 2 is a schematic flowchart of a configuration setting prediction method for a general-purpose graphics processor core function. The prediction method of the present invention is mainly divided into two stages: off-line and online. The off-line stage analyzes the execution time under different data volumes using individual configuration setting combinations, and screens the source of training data for building models in the on-line stage; the on-line stage Extracting the features and labels of the training data required for building a model, and establishing a prediction model based on the training data. [Offline stage]

First, the offline phase will be described. In the offline phase, as shown in Figure 2, the offline phase is divided into three parts: (1) Preliminary screening of parameter sets; (2) Resource occupancy screening; (3) Execute configuration setting.

In part (1), since a core function will process and operate data of various possible sizes, there may be many combinations of configuration settings it can adopt; for example, the parameter set of a set of configuration settings can include There are three parameter settings of state and optional configuration. In this case, if these parameter sets are not preliminarily screened, assuming that the data set to be processed in the offline stage is X = {𝑥 ₁ , 𝑥 ₂ , 𝑥 ₃ , … , 𝑥 _𝑝 }, the necessary configuration set is M = {𝑚 ₁ , 𝑚 ₂ , 𝑚 ₃ , … , 𝑚 _𝑞 }, the set of optional configurations is O = {𝑜 ₁ , 𝑜 ₂ , 𝑜 ₃ , … , 𝑜 _𝑟 }, then you will have to test |X| ∗ | M| ∗ |O| combinations, the number of combinations is extremely large, and it will take a lot of time and space costs. Therefore, the set screening module of the present invention filters out parameters that are unnecessary or impossible to set for proper configuration according to the relevant characteristics of each parameter set, so as to limit the possible scope of proper configuration.

In the following, the three sets of data volume, required configuration and optional configuration are taken as examples to illustrate the mechanism of preliminary screening. [Data collection screening mechanism]

The size of the data set X = {𝑥 ₁ , 𝑥 ₂ , 𝑥 ₃ , … , 𝑥 _𝑝 } is |X| = 𝑝, assuming that the number of attributes of the element is 𝑛, and an element in the set is 𝑥 _𝑖 = (𝑥 _{𝑖 ,1} , 𝑥 _{𝑖 ,2} , … , 𝑥 _{𝑖 , 𝑛} ), the range of the upper and lower limits of each attribute of an element is infinite, and the relationship between them may be independent or dependent, depending on the actual application The core functions made vary.

Exemplarily, the maximum and minimum limit values of the attribute 𝑗 of the elements in the data set are set to be _𝑗 and _𝑗 , therefore, _𝑗 ≤ 𝑥 _{𝑖 , 𝑗} ≤ _𝑗 . In order to reduce the number of combinations, the value of the attribute 𝑗 of the element 𝑥 _𝑖 will be limited to _{𝑗to 𝑗} range, and because it represents the number of data, so 𝑥 _{𝑖 , 𝑗} are positive integers.

In order to reduce the amount of data and make the amount of data regular, the present invention selects the amount of data to be executed at a fixed interval. Assuming that the attribute 𝑗 interval is 𝑑 _𝑗 , the number of attributes 𝑗 will change from becomes ⌈ ⌉, the data collection size p will change from reduced to . [Required Configuration Filtering Mechanism]

Necessary configuration set M = {𝑚 ₁ , 𝑚 ₂ , 𝑚 ₃ , … , 𝑚 _𝑞 }, elements in the set 𝑚 _𝑘 = (𝑚 _{𝑘 ,1} , 𝑚 _{𝑘 ,2} ), attributes 𝑚 _{𝑘 ,1} and 𝑚 _{𝑘 , 2} respectively represent the number of thread blocks (TB Number) and the number of threads in the thread block (TB Size).

Assume that the combination of the amount of data to be processed is 𝑥 _𝑖 = (𝑥 _{𝑖 ,1} , 𝑥 _{𝑖 ,2} , … , 𝑥 _{𝑖 , 𝑛} ), the total number of operations is 𝑂𝑝 _Total , which is the core function for the data The number of operations required to perform calculations based on the data combined by the quantity, and the definition of an operation consists of two operands and an operator. The operator does not have to be an inseparable operation (Atomic Operation). Different cores Because of the different program logic and behavior of the function, the algorithm for calculating the total will also be different.

Taking the core function of convolution operation as an example, the calculation amount of a single convolution is the multiplication of the elements in the filter array (Filter) by the corresponding elements in the input image array (Input_Image) and summing up. Each element in the filter array will perform a multiplication and an addition operation with the corresponding input array element. The calculation required for a single convolution is the size of the filter array multiplied by 2, and the direction to the right and down in sequence Slide until the feature map array (Feature_Map) is completed, and the total number of operations is the number of slides multiplied by the calculation amount of the single convolution.

Assume here that 𝑊𝑖𝑑𝑡ℎ _{𝐼𝑛𝑝𝑢𝑡} and 𝐻𝑒𝑖𝑔ℎ𝑡 _{𝐼𝑛𝑝𝑢𝑡} are the width and _height of the input image respectively, and 𝑊𝑖𝑑𝑡ℎ _{𝐹 𝑖𝑙𝑡𝑒𝑟} and 𝐻𝑒𝑖𝑔ℎ𝑡 _{𝐹𝑖𝑙𝑡𝑒𝑟} are the width and height of the filter respectively. 𝑢𝑡 − 𝑊𝑖𝑑𝑡ℎ _{𝐹𝑖𝑙𝑡𝑒𝑟} + 1) ∗ (𝐻𝑒𝑖𝑔ℎ𝑡 _{𝐼𝑛𝑝𝑢𝑡} − 𝐻𝑒𝑖𝑔ℎ𝑡 _{𝐹𝑖𝑙𝑡𝑒𝑟} + 1) ∗ (𝑊𝑖𝑑 𝑡ℎ _{𝐹𝑖𝑙𝑡𝑒𝑟} ∗ 𝐻𝑒𝑖𝑔ℎ𝑡 _{𝐹𝑖𝑙𝑡𝑒𝑟} ).

The total number of operations of the core function is 𝑂𝑝 _{𝑇𝑜𝑡𝑎𝑙} , the amount of operations that a thread block needs to process is 𝑂𝑝 _{𝑝𝑒𝑟 _ 𝑇𝐵} , and the amount of operations that each thread needs to process is 𝑂𝑝 _{𝑝𝑒𝑟 _ 𝑡 ℎ 𝑟𝑒𝑎𝑑} , execute The number of thread blocks 𝑚 _{𝑘 , 1} is . The amount of computation required by a thread in a thread block is 𝑂𝑝 _{𝑝𝑒𝑟 _ 𝑡 ℎ 𝑟𝑒𝑎𝑑} × 𝑚 _{𝑘 ,2} , 𝑂𝑝 _{𝑝𝑒𝑟 _ 𝑡 ℎ 𝑟𝑒 𝑎𝑑} is the amount of computation required to be processed by a thread, and 𝑚 _{𝑘 , 2} is the number of threads in the thread block, so the relationship can be deduced as 𝑚 _{𝑘 , 1} = .

If you use other optimization methods, such as tiling, it will increase the workload of each execution thread, which will affect 𝑚 _{𝑘 ,1} . Assuming that the tiling setting value is 𝑡, it means that when implementing the core function, the workload of each thread needs to be increased by 𝑡 times of the original, then the relationship will become 𝑚 _{𝑘 ,1} = .

Exemplarily, the present invention defines 𝑚 _{𝑘 , and the upper limit of 1} is ₁ , the lower limit is ₁ ; while the upper limit of 𝑚 ₂ is ₂ , the lower limit is ₂ . According to the specifications of NVIDIA GPU (CUDA Toolkit Documentation v11.3.0), there can only be a maximum of 1024 threads in a TB, so ₂ with The values of ₂ are 1024 and 1 respectively; while the number of TB 𝑚 _{𝑘 , the number of 1 1} with ₁ is 2 ³¹ − 1 and 1 respectively.

According to the characteristics of the core function, the workload of each thread 𝑂𝑝 _{𝑝𝑒𝑟 _ 𝑡 ℎ 𝑟𝑒𝑎𝑑} can be divided into two cases: fixed amount and non-fixed amount. Taking the core function of convolution operation as an example, each thread must be based on its own The index value of is used to perform a convolution operation on the input image array element with its corresponding filter array element, so the amount of operation allocated to each thread is a fixed amount. Taking the parallel reduction core function as an example, the amount of calculations required to be processed by different execution threads is different during the operation process, so 𝑂𝑝 _{𝑝𝑒𝑟 _ 𝑡 ℎ 𝑟𝑒𝑎𝑑} is not fixed.

Take the core function with fixed computation as an example. If 𝑂𝑝 _{𝑝𝑒𝑟 _ 𝑡 ℎ 𝑟𝑒𝑎𝑑} is a fixed amount, without using other optimization methods, when a core function processes a fixed size of data, the total number of operations required 𝑂𝑝 _{𝑇𝑜𝑡𝑎𝑙} will be fixed, so 𝑚 _{𝑘 ,} The relationship between ₁ and 𝑚 _{𝑘 , 2} is 𝑚 _{𝑘 , 1} = , and 𝑂𝑝 _{𝑝𝑒𝑟 _ 𝑡 ℎ 𝑟𝑒𝑎𝑑} and 𝑂𝑝 _{𝑇𝑜𝑡𝑎𝑙} are known numbers. Given 𝑚 _{𝑘 ,1} or 𝑚 _{𝑘 ,2} , the other value can be calculated, so the necessary configuration set size 𝑞 is ₂ – ₂ +1.

In order to reduce time and space costs, for example, the number of necessary configuration sets M is reduced in the following manner. Warp is used as a basic unit for GPU execution. A Warp can contain up to 32 execution threads. If the number of execution threads in a Warp is less than 32, the performance may be poor due to the idle Streaming Processor (hereinafter referred to as SP). Therefore, this The invention selects 𝑚 ₂ in multiples of 32. With this selection step, when 𝑂𝑝 _{𝑝𝑒𝑟 _ 𝑡 ℎ 𝑟𝑒𝑎𝑑} is a fixed amount, the size of the set M can be obtained from ( ₂ – ₂ + 1) reduces to ⌈ ; and in the case of an unfixed amount, the size of the set M ( _k – _k +1) becomes ⌈ ₁ – ₁ +1). [Optional Configuration Filtering Mechanism]

Optional configuration set O = {𝑜 ₁ , 𝑜 ₂ , 𝑜 ₃ , … , 𝑜 _𝑟 }, assuming that there are 𝑠 kinds of optimization methods that can be selected, the number of attributes of elements is 𝑠, and an element in the set 𝑜 _𝑙 Defined as 𝑜 _𝑙 = (𝑜 _{𝑙 ,1} , 𝑜 _{𝑙 ,2} , … , 𝑜 _{𝑙 , 𝑠} ), the attributes 𝑢 of the element 𝑜 _𝑙 will vary due to the optimization methods used by different core functions.

Taking the convolution operation as an example, the following optimization methods can be used: such as tiling and shared memory patching (Padding). The optional configuration is very flexible, and it can be divided into two forms: use or not use and set the value; assuming 𝑜 _{𝑙 , 1} represents the shared memory patch, 𝑜 _{𝑙 , 2} represents the tiling workload, and the shared memory patch is selected to use or Not used, attribute 𝑜 _{𝑙 , the number of values of 1} is 2; and tiling is to set the thread workload as a multiple of the original basic workload, the basic workload is defined as not using tiling technology, or using tiling technology But when the parameter is set to 1.

In the following exemplary implementations, the present invention will not define the upper and lower ranges of 𝑜 _{𝑙 , 𝑢} because the range of their attributes will vary greatly according to different core functions.

Exemplarily, several data volume combination tests can be selected for screening, for example, within the data volume range to be tested, respectively select the front, middle, and back segment sub-ranges in this range as representatives of the observed trend, and then test these selected All necessary configuration combinations and all optional configuration set combinations corresponding to the data volume combination, respectively observe the trend of the execution results, and select the optional configurations from the execution results with the majority of better execution performance as follow-up Optional configuration setting at test time, and reduces the size r of the optional configuration set O from |O| to 1.

The way to select the optional configuration representative used as the data volume will change according to different core functions. For example, it is not expected that all data volumes of the convolution core function will perform well with the same optional configuration. During the operation of the core function, the filter array will be highly reused, so the filter array is placed in the shared memory resource, and the low-latency feature of the shared memory is used to reduce execution time and improve performance. Therefore, the basic usage of the shared memory will vary with the size of the filter array. The larger the filter array, the more basic usage of the shared memory. If you use the same tiling workload setting as the smaller filter array , may make the demand for shared memory resources too high, resulting in insufficient shared memory for core functions to execute. Therefore, the way of selecting the optional configuration as a representative must be adjusted according to the combination of data volume.

For example, when using the parallel reduction core function, because the optimization method used by it will not affect the usage of shared memory under different data volume combinations, using the same optional configuration as a representative will not cause different The combination of the amount of data results in insufficient amount of specific hardware resources, so this optional configuration can be used as a representative to apply to all execution combinations. [Resource occupancy screening]

Part (2) of the offline stage is the resource occupancy rate screening. The resource occupancy rate screening module calculates the resource occupancy rate of each combination of the filtered data volume set, necessary configuration set, and optional configuration set elements, using A high resource occupancy rate may have better performance characteristics, and further eliminate combinations that may have poor execution performance to reduce the number of combinations.

First of all, nouns and their corresponding symbols are used to define the calculation resource utilization rate; because each SM has an upper limit on resources, such as: the number of temporary registers, the size of shared memory, the number of warps that can be executed at the same time, and The number of threads that can be executed at the same time is represented by 𝑀𝑎𝑥𝑅𝑒𝑔 _𝑆𝑀 , 𝑀𝑎𝑥𝑆𝑚𝑒𝑚 _𝑆𝑀 , 𝑀𝑎𝑥𝑊𝑎𝑟𝑝 _{𝑆 𝑀} , and 𝑀𝑎𝑥𝑇ℎ𝑟𝑒𝑎𝑑 _𝑆𝑀 as these limit values.

At the same time, it also defines the number of threads in each thread block when the core function is executed as 𝑁𝑢𝑚 _{𝑇 ℎ _ 𝑝𝑒𝑟𝑇𝐵} , and the number of scratchpads used by each thread is 𝑁𝑢𝑚 _{𝑅𝑒𝑔 _ 𝑝𝑒𝑟 𝑇 ℎ} , per The shared memory size used by each thread block is 𝑁𝑢𝑚 _{𝑆𝑚𝑒𝑚 _ 𝑝𝑒𝑟𝑇𝐵} , and the thread blocks that can be executed simultaneously in a single SM are 𝑁𝑢𝑚 _{𝑇𝐵 _ 𝑝𝑒𝑟𝑆𝑀} .

As an example, consider the different resource occupancy rates of each SM when only a single core function is executed, and all the thread blocks in the core function cannot be simultaneously executed by the SMs in the GPU: (a) Warp Occupancy Rate (𝑂𝑐𝑐𝑢𝑝𝑎𝑛𝑐𝑦𝑊𝑎𝑟𝑝): the proportion of Warp resources used in SM; (b) Temporary register occupancy rate (𝑂𝑐𝑐𝑢𝑝𝑎𝑛𝑐𝑦𝑅𝑒𝑔): the proportion of register usage in SM; (c) Shared memory occupancy rate (𝑂𝑐𝑐𝑢𝑝𝑎𝑛𝑐𝑦𝑆𝑚𝑒𝑚): Shared memory usage ratio in SM. Based on these occupancy information, it is possible to estimate the possible state of performance when running with this configuration setting.

Calculate the Warp, scratchpad, and shared memory occupancy in an SM separately, because in an SM, the number of Warp, scratchpad, and shared memory that can be used is limited, and the calculation is based on these three resources. In this case, the number of thread blocks that can be run at the same time is obtained, and the minimum value among the three is used as the number of thread blocks that can be run in one SM (𝑁𝑢𝑚 _{𝑇𝐵 _ 𝑝𝑒𝑟𝑆𝑀} ).

The calculation methods for the number of concurrently running thread blocks are explained as follows: (1) The number of concurrently running thread blocks obtained under the limit of Warp is 𝑁𝑢𝑚 _{𝑇𝐵 _ 𝑊𝑎𝑟𝑝𝐿𝑖𝑚𝑖𝑡𝑒𝑑} , the calculation method is: 𝑁𝑢 𝑚 _{𝑇𝐵 _ 𝑊𝑎𝑟𝑝𝐿𝑖𝑚𝑖𝑡𝑒𝑑} = ; (2) The number of blocks that can run simultaneously under the limit of the scratchpad is 𝑁𝑢𝑚𝑇𝐵_𝑅𝑒𝑔𝐿𝑖𝑚𝑖𝑡𝑒𝑑, the calculation method is: 𝑁𝑢𝑚 _{𝑇𝐵 _ 𝑅𝑒 𝑔𝐿𝑖𝑚𝑖𝑡𝑒𝑑} = ; (3) The number of concurrently running thread blocks obtained under the limitation of shared memory is 𝑁𝑢𝑚 _{𝑇𝐵 _ 𝑆𝑚𝑒𝑚𝐿𝑖𝑚𝑖𝑡𝑒𝑑} , the calculation method is: 𝑁𝑢𝑚 _{𝑇𝐵 _ 𝑆𝑚𝑒𝑚𝐿𝑖𝑚𝑖𝑡𝑒𝑑} = .

Among them, if the number of scratchpads allocated to a Warp is not a multiple of 256 when allocating scratchpad resources, it will be automatically filled to a multiple of 256 when allocating. Therefore, when calculating the scratchpad usage in a Warp, you must first Divide by 256 and then take the Gaussian upper limit, which is the allocated scratchpad usage during real runtime.

There are three ways to obtain the value of the number of registers used by each thread 𝑁𝑢𝑚 _{𝑅𝑒𝑔 _ 𝑝𝑒𝑟𝑇ℎ} . The first is to estimate the amount of registers used by each thread by observing the code; the second is The method is to use the CUDA compiler nvcc and add the parameter -res-usage to estimate the number of scratchpad usage; the third method is to use the CUDA built-in resource monitoring tool nvprof during the execution of the core function , plus the parameter --print-gpu-trace, feedback the scratchpad usage of each execution thread.

Only the third way can accurately know the value of 𝑁𝑢𝑚 _{𝑅𝑒𝑔 _ 𝑝𝑒𝑟𝑇ℎ} , but it is also the most cost-intensive way. Therefore, if you want to reduce the cost as much as possible, you can use the first and second methods; on the contrary, if you need high accuracy, you can choose the third method.

Similarly, if the number of bytes used by a thread block is not a multiple of 256 when allocating shared memory resources, the insufficient number of bytes will be automatically filled to make it a multiple of 256, so in calculation You need to divide by 256 and then take the upper limit of Gaussian to get the shared memory usage allocated during real runtime.

After calculating the number of thread blocks that can run simultaneously with the above three resources as the limit, take the minimum value as the number of thread blocks that can run in one SM, and then calculate the different resources of the three resources in one SM at runtime. Divide the usage amount by the upper limit of each resource in an SM to obtain the following formula to calculate the resource occupancy rate: (1) 𝑂𝑐𝑐𝑢𝑝𝑎𝑛𝑐𝑦 _{𝑊𝑎𝑟𝑝} = (2) 𝑂𝑐𝑐𝑢𝑝𝑎𝑛𝑐𝑦 _𝑅𝑒𝑔 = (3) 𝑂𝑐𝑐𝑢𝑝𝑎𝑛𝑐𝑦 _{𝑆𝑚𝑒𝑚} =

For example, assuming that the number of execution threads 𝑁𝑢𝑚 _{𝑇 ℎ _ 𝑝𝑒𝑟𝑇𝐵} in each execution block in a certain configuration setting is 128, the number of registers required by each execution thread 𝑁𝑢𝑚 _{𝑅𝑒𝑔 _ 𝑝𝑒𝑟 𝑇 ℎ} is 50 , the number of shared memory bytes 𝑁𝑢𝑚 _{𝑆𝑚𝑒𝑚 _ 𝑝𝑒𝑟𝑇𝐵} required for each thread block is 16,384 Bytes.

Taking the RTX 2060 graphics card as an example, one SM can execute the maximum number of warps 𝑀𝑎𝑥𝑊𝑎𝑟𝑝 _𝑆𝑀 , the number of scratchpads 𝑀𝑎𝑥𝑅𝑒𝑔 _𝑆𝑀 , the number of shared memory bytes 𝑀 𝑎𝑥𝑆𝑚𝑒𝑚 _𝑆𝑀 and the maximum number of concurrently executing threads The numbers 𝑀𝑎𝑥𝑇ℎ𝑟𝑒𝑎𝑑 _𝑆𝑀 are 32 bytes, 65,536 bytes, 65,536 bytes and 1,024 bytes respectively.

Under the resource constraints of Warp, scratchpad, and shared memory, the calculated number of executable thread blocks 𝑁𝑢𝑚 _{𝑇𝐵 _ 𝑊𝑎𝑟𝑝𝐿𝑖𝑚𝑖𝑡𝑒𝑑} is 8 (= ), 𝑁𝑢𝑚 _{𝑇𝐵 _ 𝑅𝑒𝑔𝐿𝑖𝑚𝑖𝑡𝑒𝑑} is 9.14 (= ) and 𝑁𝑢𝑚 _{𝑇𝐵 _ 𝑆𝑚𝑒𝑚𝐿𝑖𝑚𝑖𝑡𝑒𝑑} is 4 (= ). Take the smallest value among the three as 𝑁𝑢𝑚 _{𝑇𝐵 _ 𝑝𝑒𝑟𝑆𝑀} , and 𝑁𝑢𝑚 _{𝑇𝐵 _ 𝑝𝑒𝑟𝑆𝑀} is 4.

Finally, the above computing resource occupancy calculation formula can be used to calculate the three occupancy _ratios : ), 𝑂𝑐𝑐𝑢𝑝𝑎𝑛𝑐𝑦 _𝑅𝑒𝑔 is 0.4 (= ), 𝑂𝑐𝑐𝑢𝑝𝑎𝑛𝑐𝑦 _{𝑆𝑚𝑒𝑚} is 1 (= ).

The above three values are used for comprehensive consideration, which may be the average or weighted average of the three values, and the value after comprehensive consideration is used as the screening standard. There are two screening methods. The first is to select a limit value and delete combinations whose resource occupancy is less than this value; the second is to sort the resource occupancy of all combinations after comprehensive consideration from large to small, and select a ratio value (eg: 70%), delete the combination after the ratio value (eg: delete the combination of 70%). The larger the above two values are, the more time and space costs can be reduced, but the error deletion rate will also increase accordingly.

The elimination method of selecting the limit value can reduce the chance of mistaken deletion than the method of selecting the proportional value. If the fixed value is not properly selected when the selected proportional value is used, too many combinations may be deleted at one time. By choosing the limit value, it will be easier to control the number of combinations to be deleted. [Execute Configuration Settings]

Part (3) of the offline stage is to execute the module to make the GPU actually execute the filtered configuration settings, and record the execution time, which is used as the source of training data required for building the model in the online stage.

An example is used to further illustrate. Taking the convolution operation as an example, assuming that the size of the input image array is 1000*1000, and the size of the filter array is 3*3, the resource occupancy and execution time of some combinations will be shown in Table 1.

Table 1 thread block size Warp share register occupancy Shared Memory Occupancy execution time 160 93.75% 41.02% 9.38% 6.5446*10 ⁻² ms 512 100.0 % 43.75% 7.81% 7.5167*10 ⁻² ms 544 53.12% 23.24% 4.3% 9.7753*10 ⁻² ms 992 96.88% 42.38% 7.03% 7.8713*10 ⁻² ms

It can be found that when the thread block size is 160, although the Warp occupancy rate of 93.5% is lower than 100% and 96.88% of the thread block size of 512 and 992, the shared memory occupancy rate of 9.38% is higher than both 7.81 % and 7.03 %, it can be seen that a high Warp share does not necessarily mean better execution performance. Therefore, taking other occupancy ratios into consideration in the present invention will improve the accuracy of the screening criteria.

When the thread block size is 544, its three resource occupancy rates are smaller than the other three, and its execution time is also the longest. It can be seen that the execution performance will be poor if the occupancy rate is too low, so the present invention screens resources according to the resource occupancy rate The standard deletes combinations with too low occupancy.

Assuming that after the element screening process for the three sets, the number of combinations left to be executed is y, and the resource occupancy elimination rate is z, after the resource occupancy screening, the combinations that must be tested will be reduced from the original y to 𝑦 × (1 – z). Assume that the number of combinations that must be tested is |X| × |M| × |O|, and the formula after expansion is × ( ₂ – ₂ + 1) ∗ |O|; After the above 4 screening and combination processes, the amount of computation required to be executed in each thread 𝑂𝑝 _{𝑝𝑒𝑟 _ 𝑡 ℎ 𝑟𝑒𝑎𝑑} is a fixed amount, and the elimination rate of resource occupancy is z In the case of × ( ₂ – ₂ + 1) ∗ |O| reduces to × ⌈ × 1 × (1 - 𝑧).

The present invention uses a screening mechanism to greatly reduce the number of execution combinations that are originally extremely large, reducing time and space costs, and leaving relatively good feature data as training data required for model building in the online stage. [Online stage]

Next, in the online stage, the training data is extracted and adjusted from the remaining combinations screened in the aforementioned offline stage, and the training data is imported into the model to build the model. Finally, the established model can be used to predict the configuration settings.

Referring to Figure 2, the online stage can be divided into two parts: (1) Extract training data; (2) Build a model and make predictions. [Extract training data]

In the process of building a predictive model for machine learning, feature data and corresponding label data are required; accordingly, the training data extraction module needs to extract suitable data from the execution results of the execution combination obtained in the offline stage as training data. It is divided into features and label data.

Exemplarily, the combination of data volumes can be used as feature data, and the corresponding necessary configurations and optional configurations can be extracted as label data. Originally, each data volume will be tested with all necessary configuration combinations and optional configuration combinations. In this step, the combination with the best execution result among these execution combinations can be used as the representative of the data volume combination.

When the workload of each thread 𝑂𝑝 _{𝑝𝑒𝑟 _ 𝑡 ℎ 𝑟𝑒𝑎𝑑} is fixed, this step will be performed at ⌈ Select an execution combination with the best execution result from the combinations, and finally reduce the training data to Pen. The purpose is to enable the predictive model in machine learning to be able to clearly distinguish under a specific data volume combination, which category to judge the data volume combination as, and reduce the probability of misjudgment. [Modeling]

Exemplarily, the modeling module adopts a supervised learning method in machine learning, and can select K-neighborhood algorithm (KNN) or logistic regression (Logistic Regression: LR) algorithm as a model for application.

In this section, the modeling module may have to adjust the dimensions of the training data. In the feature data part, the KNN and LR models must have at least two dimensions; in the label data part, the KNN model has no restriction on the dimension, and the LR model needs to be one-dimensional; however, different core functions will make the execution of the execution combination The dimensions of the training data extracted from the results are different, so the dimensions of the training data may need to be adjusted before they can be imported into the model. If the training data extracted by a core function, feature data and label data are both two-dimensional, then importing into the K-nearest algorithm does not need to adjust the dimension; if the extracted feature data is one-dimensional, it needs to increase the dimension. In order not to affect the characteristic data when dimensioning, the value can be uniformly filled with 1 in the added dimension.

Accordingly, a general-purpose graphics processor core function configuration setting prediction model is established; and a group of configuration settings with better performance can be suggested by using the model.

In summary, the model established by the general-purpose graphics processor core function configuration setting prediction system and method of the present invention can quickly and accurately predict the configuration setting with better performance; and the system and method can also Effectively screen the amount of data for building a forecast model, and build a forecast model with high accuracy in core function configuration settings at a relatively low time and space cost. [Example 1]

Use KNN as the prediction model, and use the core function of convolution operation. R(a, b)_KNN represents the prediction mechanism, R represents the screening in the offline stage, and a represents the selection interval in the data collection screening mechanism b is the standard value set in the resource occupancy rate screening mechanism. There are two types of standard values: (1) P represents the elimination method of the use ratio value; (2) T represents the use limit value elimination method.

The minimum value and maximum value of the input image array width (height) in the data volume collection of the convolution operation core function are 1,000 to 2,000, and the selected interval 5, 10, 15, and 20; the minimum and maximum values of the width (height) of the filter array are 3 and 74 respectively; the value of the proportional value elimination mechanism in the resource occupancy screening mechanism is set to 20%, 40% And 60%, when the setting ratio is 20%, it means deleting 20% of the configuration settings with low resource occupancy, leaving 80% of the configuration settings with high occupancy; the limit value elimination mechanism value is set to 50% , 55% and 60%, when the set limit value is 50%, it means that the configuration settings whose average resource occupancy rate is less than 50% will be deleted.

by As the number of pens of the width (height) degree size of each filter array, there will be test data, and the width (height) of the input image array in the test data is randomly selected.

Table 2 shows the actual space cost comparison table of the core function of the convolution operation; it can be known that the larger the used , P value or T value, the more time cost can be reduced, so for The choice of , P value or T value is very important, because it will have a close relationship with space cost and execution performance.

Table 2

FIG. 3 shows the ratio of the time spent in prediction to the execution time of the predicted configuration setting. The smaller the ratio, the lower the cost of time spent in the prediction phase in this embodiment. It can be seen in the figure that at different screening parameters , P value or T value, their proportions are all less than 0.7%, which means that the prediction mechanism we proposed has a very small time cost in the prediction stage, so a set of configuration settings can be recommended very quickly. [Example 2]

Use the logistic regression algorithm as the predictive model with a parallel reduction core function. R(a, b)_LR represents the prediction mechanism, R represents the screening in the offline stage, and a represents the selection interval in the data collection screening mechanism b is the standard value set in the resource occupancy rate screening mechanism. There are two types of standard values: (1) P represents the elimination method of the use ratio value; (2) T represents the use limit value elimination method.

The length of the input array in the data set of the parallel reduction core function is 8 million to 12 million, and the selection interval is 10,000, 20,000, 30,000 and 40,000. The proportion of resource occupancy is set to 0%, 70%, 80% and 90%, and the limit value is set to 50%, 55% and 60%.

There are a total of 500 test data for analyzing the execution performance of the parallel reduction core function, and the size of the input array in the test data is randomly selected.

Figure 4 shows the comparison of the time spent in prediction with the proportion of execution time set in the predicted configuration. The smaller the proportion, the lower the cost of time spent in the prediction stage in this embodiment. It can be seen in the figure that different filter parameters , P value or T value, their proportions are all less than 1.2%, which means that the prediction mechanism we proposed has a very small time cost in the prediction stage, so a set of configuration settings can be recommended very quickly.

1: Configuration setting prediction system for general-purpose graphics processor core functions 11:Collection filter module 12: Resource occupancy screening module 13: Execute the module 14: Training data extraction module 15:Modeling module

[FIG. 1] A block diagram of a configuration setting prediction system for general-purpose graphics processor core functions. [FIG. 2] Schematic diagram of the flow chart of the configuration setting prediction method for the core function of a general-purpose graphics processor. [Figure 3] Comparison of the time spent in prediction with the execution time of the predicted configuration settings. [Figure 4] Comparison of the time spent in prediction with the execution time of the predicted configuration settings.

1: Configuration setting prediction system for general-purpose graphics processor core functions

11:Collection filter module

12: Resource occupancy screening module

13: Execute the module

14: Training data extraction module

15:Modeling module

Claims (10)

A general-purpose graphics processor core function configuration setting prediction system is applicable to the general-purpose graphics processor, and its features include: A set screening module, which preliminarily screens the parameter set of the configuration setting of the general-purpose graphics processor core function, and takes the filtered parameter set as an input parameter; A resource occupancy rate screening module, which performs secondary screening on the input parameters by the resource occupancy values of Warp, temporary register and shared memory in the stream composite processor to obtain the first execution configuration setting; An execution module, the first execution configuration is set to be executed by the general-purpose graphics processor, and the execution time is obtained; a training data extraction module, extracting a second execution configuration from the first execution configuration according to the execution time as training data; and A modeling module is used to establish a configuration setting prediction model of the general-purpose graphics processor core function by using the training data. The general-purpose graphic processor core function configuration setting prediction system as described in Claim 1, wherein the parameter set includes a data volume set, a necessary configuration set, and an optional configuration set. The general-purpose graphics processor core function configuration setting prediction system as described in claim 2, wherein, the training data extraction module is for each data volume set, according to the execution time from the necessary configuration set and the The optional configuration set is extracted, and the data volume set, the extracted necessary configuration set and the optional configuration set are used as training data. The general-purpose graphics processor core function configuration setting prediction system as described in claim 1 or 2, wherein the modeling module uses K-neighbor algorithm or logistic regression algorithm to establish a model. The general-purpose graphic processor core function configuration setting prediction system as described in claim 2, wherein the set screening module preliminarily screens the data set by setting intervals of element attributes in the data set. The general-purpose graphics processor core function configuration setting prediction system as described in claim 2, wherein, the set screening module preliminarily screens the necessary by selecting the number of execution threads of the execution thread block with a multiple of 32 Configuration collection. The general-purpose graphics processor core function configuration setting prediction system as described in claim 2, wherein the set screening module further optimizes the optional configuration set, and the optimization method is tiling or shared memory patching. The general-purpose graphic processor core function configuration setting prediction system as described in claim 2, wherein, the preliminary screening method of the set screening module for the optional configuration set is: select several sets of the data volume, Select the front, middle, and back sub-ranges of these ranges within the range of several data sets, and test these sub-ranges, all of the necessary configuration sets and all of the optional configuration sets, which will have the majority of execution performance. This set of optional configurations is filtered out from the best execution results. A configuration setting prediction method for the core function of a general-purpose graphics processor, which is suitable for the general-purpose graphics processor, and its characteristics include: Preliminary screening is performed on the parameter set of the configuration setting of the general-purpose graphics processor core function, and the parameter set that has been screened is used as an input parameter; The input parameter is secondarily screened by the resource occupancy values of Warp, temporary register and shared memory in the stream composite processor to obtain the first execution configuration setting; Setting the first execution configuration to be executed by the general-purpose graphics processor to obtain an execution time; Extracting a second execution configuration from the first execution configuration according to the execution time as training data; Establishing a configuration setting prediction model of the general-purpose GPU kernel function by using the training data; and A set of configuration settings is predicted by the prediction model. The method for predicting the configuration setting of a general-purpose graphics processor core function as described in Claim 9, wherein the parameter set includes a data volume set, a necessary configuration set, and an optional configuration set.