CN109816107A - A BFGS Quasi-Newton Neural Network Training Algorithm Based on Heterogeneous Computing Platform - Google Patents

Abstract

The invention discloses a BFGS quasi-Newton neural network training algorithm based on a heterogeneous computing platform. (3) Calculate the search direction dir; (4) Calculate the step size λ and update the weight w based on the golden section method; (5) Calculate the neural network training Gradient g of the error evaluation function; (6) Calculate the Hessian matrix H; (7) Parallel reduction. The invention adopts CPU and GPU heterogeneous computing platform as neural network training computing equipment, uses GPU to accelerate the BFGS quasi-Newton algorithm, and greatly improves the running speed compared with the traditional CPU-based implementation; High convergence efficiency and global search capability.

Description

A BFGS Quasi-Newton Neural Network Training Algorithm Based on Heterogeneous Computing Platform

technical field

The invention relates to the fields of high-performance computing and machine learning, in particular to a BFGS quasi-Newton neural network training algorithm based on a heterogeneous computing platform.

Background technique

An artificial neural network is an information processing system that can learn any input-output relationship through a series of data and build an accurate model. Currently, one of the main challenges facing artificial neural networks is training. Before training, the neural network does not carry any information; after training, the weights of the neural network are determined, so as to build an accurate model based on the training data. The process of determining the weight value of the neural network is actually an optimization process, that is, the neural network uses various optimization algorithms, such as gradient descent, quasi-Newton algorithm (QN), particle swarm optimization (PSO) and conjugate gradient algorithm (CG). ), etc., after repeated iterative calculation, a more accurate fitting weight value is obtained. Therefore, neural network training involves multiple iterative calculations of a large amount of training data, which is a very time-consuming process.

With the rapid development of GPU technology, the current GPU has strong parallel computing capabilities and data processing capabilities, but the logical processing capabilities of GPUs are still far behind that of CPUs. Algorithms for scalability and design cost.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the deficiencies in the prior art, solve the problem of low efficiency in the training process of traditional artificial neural networks, and provide a BFGS quasi-Newton neural network training algorithm based on a heterogeneous computing platform. The computing platform is used as a computing device for neural network training, and the GPU is used to accelerate the BFGS quasi-Newton algorithm to obtain a method that can quickly complete the neural network training and build a high-precision model.

The purpose of this invention is to realize through the following technical solutions:

A BFGS quasi-Newton neural network training algorithm based on a heterogeneous computing platform, comprising the following steps:

(1) Division of tasks: The BFGS quasi-Newton neural network training algorithm includes computing tasks and control tasks. The control tasks are completed by the CPU, and the computing tasks are completed by the GPU. The computing tasks are divided into five kernel functions (kernel);

(2) Divide the degree of parallelism, the total number of work-item required to complete the computing task and the number of work-groups that organize the work-item into;

(3) Find the training error of the neural network;

(4) Calculate the search direction dir;

(5) Calculate the step size λ and update the weight w based on the golden section method;

(6) Calculate the gradient g of the neural network training error evaluation function;

(7) Calculate the Hessian matrix H;

(8) Parallel reduction.

Further, the five kernel functions in step (1) are respectively: kernel1 represents the calculation of the training error of the neural network, kernel2 represents the calculation of the search direction, kernel3 represents the calculation of the search step size and the update of the neural network connection weight, and kernel4 represents the neural network. The calculation of the gradient of the training error evaluation function, kernel5 represents the calculation of the Hessian matrix.

Further, in step (1), the control task includes the transmission control of the initial data from the host to the computing device, the transmission of the calculation result from the computing device to the host, the conditional judgment of whether the upper limit of the number of iterations is reached, and the judgment of whether the training error meets the requirements. And the control of whether the computing task is terminated.

Compared with the prior art, the beneficial effects brought by the technical solution of the present invention are:

1. The present invention uses CPU and GPU heterogeneous computing platform as computing equipment, and compared with the traditional realization based on CPU, the running speed is greatly improved;

2. The present invention uses OpenCL as a programming language to implement, has higher portability compared to the implementation using CUDA language, and can be used on GPUs and FPGAs of different manufacturers;

3. The present invention adopts the BFGS quasi-Newton algorithm as the optimization algorithm for neural network training, which has higher convergence efficiency and global search capability than other optimization algorithms.

Description of drawings

Figure 1 is a schematic diagram of the neural network structure.

Figure 2 is a schematic diagram of a CPU and GPU heterogeneous computing platform.

Figure 3 is a schematic diagram of parallel reduction.

Figure 4 is a flow chart of the design of the parallel quasi-Newton neural network training algorithm.

Figure 5 is a schematic diagram of data transmission between various modules.

FIG. 6 is a flow chart of a specific operation.

Figure 7 is a schematic diagram of the results

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings.

As shown in Figure 1, it is a schematic diagram of the neural network structure, including: input layer, hidden layer and output layer. The two adjacent neurons are all connected together, and each connection corresponds to a weight. The number of neurons in the input layer corresponds to the number of inputs of a set of training data, the number of neurons in the output layer corresponds to the number of outputs of a set of training data, and the number of neurons in the hidden layer is set as required, generally greater than the number of neurons in the input layer quantity.

As shown in Figure 2, it is a schematic diagram of a CPU and GPU heterogeneous computing platform. The CPU and GPU communicate through the PCIE bus. The part where the CPU is located is the host side, which is in charge of control, the part where the GPU is located is the device side, which is in charge of computing, and the storage space on the host side is the host memory, which can read and write data with the global memory on the GPU side. When operating, just put the data into the corresponding directory, and the software will read it by itself.

As shown in FIG. 4 , it is the design flow chart of the algorithm of the present invention, and the overall tasks are divided according to the characteristics of the CPU and GPU. The judgment part and the data initialization part are completed by the CPU, and the neural network training error function and the BFGS quasi-Newton algorithm are implemented on the GPU. The division of parallelism of each module takes into account both the characteristics of the algorithm itself and the structural characteristics of the GPU.

details as follows:

1) Task division: The algorithm generally includes two types of tasks, namely computing tasks and control tasks. The control task is completed by the CPU, including the transmission control of the initial data from the host to the computing device, the transmission of the calculation result from the computing device to the host, the conditional judgment of whether the upper limit of the number of iterations has been reached, the judgment of whether the training error meets the requirements, and the calculation task. Whether to terminate the control. The computing task is completed by the GPU. In order to facilitate task scheduling and optimization, the computing task is divided into five kernel functions, as shown in Figure 5, including the calculation of the training error of the neural network of kernel1, the calculation of the search direction of kernel2, and the search step of kernel3. Long calculation and update of neural network connection weight, calculation of gradient of kernel4 neural network training error evaluation function, calculation of kernel5 Hessian matrix.

2) Parallelism division: For the five kernel functions in 1), parallel implementation on the GPU first needs to divide the parallelism, that is, the total number of work-items required to complete these computing tasks and the organization of these work-items into The number of work-groups. Among the five kernel function kernels, kernel1 and kernel4 divide the degree of parallelism according to the scale of training data. For example, if the training data consists of 1024 groups, then these two kernels need 1024 work-items to participate in the calculation, and each work-item performs a set of training data. related calculations. kernel2, kernel3 and kernel5 are divided into parallelism according to the number of neural network weights. As shown in Figure 1, the number of connection weights of the neural network is 32, then the number of work-items required by the three kernels under the neural network structure is 32 .

3) Neural network training error evaluation function: This function corresponds to kernel1 in Figure 5. The kernel reads the training data and the connection weight w from the memory for calculation, and finally obtains the training error E _T (w) corresponding to w. In this function, the calculation content of the sth work-item is shown in formula (1)(2)(3), where the value of the input neuron is the training data, which is represented by x _i ^s , which is the same as the i-th work-item. The value of the sth group of training data corresponding to the input neuron; the value of the hidden neuron is calculated by formulas (1) and (2), where hj represents the accumulation of the input neuron and the weight corresponding to the jth hidden neuron, f (h _j ) represents the value of the jth hidden neuron, w _ij represents the weight from the ith input neuron to the jth hidden neuron, n represents the number of input neurons; the value of the output neuron y _m is determined by the formula ( 3) Calculated, where m represents the mth output neuron, and N represents the number of hidden neurons. Finally, the neural network training error _ET is obtained according to formula (4) using the reduction technique. In formula (4), S _T represents the scale of the training data set, N _y represents the number of output neurons, y _m represents the calculation result of the m-th output neuron, and d _km represents the k-th group of training data and the m-th output neuron. Meta corresponds to the ideal output result. d _max,m represents the maximum ideal output value in the given training data, and d _min,m represents the smallest ideal output value in the given training data.

4) Calculate the search direction dir: This function corresponds to kernel2 in Figure 5. The calculation process requires the use of the Hessian matrix H and the gradient g. Each work-item reads a row of data in the matrix H and performs a multiply-accumulate operation with the gradient g to generate a search. An element of the direction dir.

5) Calculate the step size λ and update the weight w based on the golden section method: This function corresponds to kernel3 in Figure 5. The process is to first initialize a step size interval and calculate the two golden section points of the interval, and then use these two points. Update w for the step size, and then call the _ET function to find the corresponding values of the two groups of w and compare the sizes, leaving the smaller step size and deleting a section outside the larger step size. Repeat this until the remaining step size interval is less than a fixed value, stop the iteration and determine the step size. Since the main purpose of the kernel is to update the weight w, the division of the internal parallelism of the function is based on the dimension of w. That is, each work-item computes one element of the weight w. This function needs to repeatedly call the neural network training error evaluation function, that is, it needs to call another kernel inside one kernel. This functionality can be implemented with OpenCL versions higher than 1.1. However, since the parallelism of the two kernels is not the same, although the overall number of work-item is the same, the active work-item is not the same. To avoid the resulting work-item conflict, a synchronous operation is enforced on all work-items before each invocation of the training error function.

6) Calculate the gradient g of the neural network training error evaluation function: this function corresponds to kernel4 in Figure 5. The process is mainly to calculate the partial derivatives of each element in each group of weights w layer by layer, and then use the parallel reduction method to find the partial derivatives of each element. The sum of the derivatives, the gradient of the neural network training error function can be obtained. In this function, each work-item calculates the partial derivatives of all elements in the weight w based on a set of training data, and finally reduces and sums all work-items

7) Calculate the Hessian matrix H: the subscript k in all formulas represents the kth iteration, this function corresponds to kernel5 in Figure 5, and the weight w and gradient g calculated by steps 5) and 6) are obtained through formula (5) (6) ) to obtain s and z, and then calculate x from these two vectors according to formula (7), where s is the change in weight, z is the change in gradient, and x is the correction term. Finally, the Hessian matrix is calculated according to formula (8). Each work-item in this part is calculated to obtain the elements of one column of the Hessian matrix H. H _k is the Hessian matrix H of the k-th iteration.

s _k =w _k+1 -w _k (5)

z _k =g _k+1 -g _k (6)

8) Parallel reduction: In the kernel1 and kernel4 of the BFGS quasi-Newton neural network training parallel algorithm, the parallel reduction technology needs to be used to process some vectors to obtain the cumulative sum of each element in the vector. The parallel reduction technique is shown in Figure 3. The first work-item calculates the sum of the first variable and the last variable, the second work-item calculates the sum of the second variable and the penultimate variable, and so on By analogy, the final result is calculated by the first work-item.

Specifically, FIG. 5 is a schematic diagram of data transmission between various modules. Each module corresponds to a kernel. First, kernel2 uses the initial Hessian matrix H and gradient g to calculate the search direction d; then, kernel3 determines the step size according to the golden section algorithm, and updates the weight w by the search direction d and step size λ and passes it into kernel1 to calculate the training error error and Evaluate until the conditions are met to determine the final step size and determine w; kernel4 calculates the partial derivative of E _T (w) according to some intermediate variables in kernel1 and determines the gradient of the function; finally, in kernel5, use the previous several kernels to generate The weights w and gradients g compute the Hessian matrix H. After judgment on the external CPU, if the requirements are met, the results will be output, otherwise, the five kernels will continue to be run in a loop.

Its specific operation is shown in Figure 6:

(1) Neural network parameter setting

As shown in Figure 1, it is the structure diagram of the neural network; in this embodiment, a single hidden layer neural network is selected, the number of neurons in the input layer is set according to the number of input variables of the model to be fitted, and the number of neurons in the output layer is set according to the required number of input variables. The number of output variables of the fitted model is set, and the number of neurons in the hidden layer is set according to its own needs, which is generally larger than the number of neurons in the input layer. The number of neural network weights will be calculated and changed by the formula w=(input+output)*hidden according to the number of neurons set earlier.

(2) GPU side parameter setting

The GPU-side parameter settings mainly include the setting of the number of work-items and the setting of the scale of the work-group. Work-item is the smallest unit running on the GPU, and its number can be set according to the size of the training data. For example, the training data scale is 4096 groups, then the number of work-items can be set to 4096. A certain number of work-items can be organized into work-groups, which facilitates data transfer and management between work-items in the same group. The number of work-items in a work-group can be set according to the number of neural network weights. As shown in Figure 1, the number of weights of the neural network is (3+1)*8=32, then the number of work-items in a work-group It is set to 32.

(3) Import training data

The software can only read files in csv format, so you need to store the training data in the csv file first, then store it in the software directory, and change the corresponding file name to training-data.

(4) Termination condition setting

After the above steps are completed, it is also necessary to set the termination conditions of the software, which generally include the setting of the maximum number of iterations and the setting of the training error boundary conditions. After the setting is completed, the software will terminate the program and output the result when the software reaches the maximum number of iterations or the training error is less than the training error boundary condition.

(5) Result record

The output result is shown in Figure 7. The result contains four pieces of information. The first line is the training error, which represents the accuracy of the neural network fitting model, as shown in the figure, which is 37.5297; the second line is the number of iterations, which represents the number of iterations from running to termination of the program, as shown in the figure. is 20; the third row starts with the obtained neural network weights; the last row is the running time, in seconds.

The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solutions of the present invention, and the above-mentioned specific embodiments are only illustrative and not restrictive. Without departing from the spirit of the present invention and the protection scope of the claims, those of ordinary skill in the art can also make many specific transformations under the inspiration of the present invention, which all fall within the protection scope of the present invention.

Claims (3)

1. a kind of BFGS based on heterogeneous computing platforms intends newton neural network BP training algorithm, which is characterized in that including following step It is rapid:

(1) divide task: it includes calculating task and control task that BFGS, which intends newton neural network BP training algorithm, control task by CPU is completed, and calculating task is completed by GPU, and calculating task is divided into five kernel function kernel；

(2) degree of parallelism is divided, the sum of work-item needed for completing calculating task and is organized into work-item The quantity of work-group；

(3) neural metwork training error is found out；

(4) direction of search dir is calculated；

(5) based on Fibonacci method material calculation λ and update weight w；

(6) the gradient g of neural metwork training error evaluation function is calculated；

(7) Hessian matrix H is calculated；

(8) parallel reduction.

2. a kind of BFGS based on heterogeneous computing platforms intends newton neural network BP training algorithm according to claim 1, feature exists In five kernel functions are respectively as follows: the calculating of kernel1 expression neural metwork training error in step (1), and kernel2 is indicated The calculating of the direction of search, kernel3 indicate the calculating of step-size in search and the update of neural network connection weight, and kernel4 is indicated The calculating of the gradient of neural metwork training error evaluation function, kernel5 indicate the calculating of Hessian matrix.

3. a kind of BFGS based on heterogeneous computing platforms intends newton neural network BP training algorithm according to claim 1, feature exists In control task includes primary data by host side to the transmission control for calculating equipment end in step (1), calculated result is by calculating Equipment end to host side transmission, whether reach the number of iterations upper limit condition judgement, whether training error satisfactory sentences The control whether disconnected and calculating task terminates.