CN106875002A - Complex value neural network training method based on gradient descent method Yu generalized inverse - Google Patents

Abstract

The invention relates to a complex-valued neural network training method based on gradient descent method and generalized inverse. Step 1 is to select a single hidden layer complex-valued neural network model; Step 2 is to use gradient descent method and generalized inverse to calculate single hidden layer complex-valued neural network. The weight matrix and weight vector in the network, step 3, according to the weight matrix and weight vector, obtain complex-valued neural network parameters, and calculate the mean square error; add 1 to the number of iterations, and return to step 2. The hidden layer input weight of the present invention is iteratively generated by the gradient descent method, and the output weight is always solved by generalized inversion. The number of iterations of this method is small, the corresponding training time is short, the convergence speed is fast, and the learning efficiency is high, and the number of hidden layer nodes required is small. Therefore, the present invention can more accurately reflect the performance of the complex-valued neural network model than the BSCBP method and the CELM method.

Description

Complex-valued neural network training method based on gradient descent method and generalized inverse

technical field

The invention belongs to the technical fields of image processing, pattern recognition and communication transmission, and in particular relates to a complex-valued neural network training method based on gradient descent method and generalized inverse.

Background technique

In image processing, pattern recognition and communication transmission, the method of using neural network modeling for sample training and testing has a wide range of applications. In the neural network model modeling in the training samples, the neural network signals (input signal, output signal and weight parameter) can be real-valued or complex-valued, so the neural network is divided into real-valued neural network and complex-valued neural network. Most of the existing neural network modeling methods are real-valued neural network models, but with the rapid development of electronic information science, complex-valued signals appear more and more frequently in engineering practice, and calculations that only consider real values cannot be performed well. To solve practical problems, complex-valued neural networks can solve some problems that real-valued neural networks cannot solve. The complex-valued neural network is a neural network that processes complex information through complex parameters and variables (that is, the input, output, and network weight of the signal are all complex). Therefore, a series of complex-valued neural network models have been proposed and studied in depth.

A BSCBP method is proposed in Batch Split-Complex Backpropagation Algorithm for training complex-valued neural networks. The activation function selects the activation function of the real and imaginary parts, and activates the real and imaginary parts of the hidden layer input separately to avoid the appearance of singular points; the BSCBP method first randomly assigns values to the input weight matrix and output weight matrix, and then uses the gradient descent method. The gradient is updated, and finally the accuracy on the test samples is calculated. However, the BSCBP model based on the gradient descent method requires multiple iterations of training, which consumes a long time and has low learning efficiency.

In Fully Complex Extreme Learning Machine, a CELM method is proposed to extend the ELM method from the real number domain to the complex number domain, and apply it to nonlinear channel equalization. CELM only needs to set the appropriate number of network hidden layer nodes, and randomly assign the input weights of the network. The optimal solution of the output layer weights is obtained by the least square method. The activation function can be selected from sigmoid function, (inverse) trigonometric function and The (inverse) hyperbolic function, unlike BSCBP, activates the input matrix of the hidden layer directly. The whole process is completed once without iteration, so it has the advantages of easy parameter selection and extremely fast learning speed. However, in order to make up for the arbitrary selection of hidden layer node parameters, the CELM method often requires a large number of hidden layer nodes, and the training accuracy needs to be further improved.

In summary, BSCBP is slow in training and has low precision. Although the CELM method is fast, it requires too many nodes in the hidden layer of the network, and the accuracy needs to be improved. In the prior art, how to solve complex value There is still no effective solution to the problems of slow training speed, low precision and too many nodes in the network hidden layer in the neural network training method.

Contents of the invention

In order to solve the above problems, the present invention overcomes the problems that the traditional complex-valued neural network training method cannot simultaneously solve the problems of slow training speed, low precision and too many nodes in the network hidden layer, and provides a complex-valued neural network based on gradient descent method and generalized inverse Neural network training method (Gradient based Generalized Complex Neural Networks, referred to as GGCNN).

In order to achieve the above object, the present invention adopts the following technical solutions:

A complex-valued neural network training method based on gradient descent method and generalized inverse, said method steps comprising:

(1) Select a single hidden layer complex-valued neural network model to model the sample data set;

(2) According to the single-hidden-layer complex-valued neural network model selected in step (1), use the generalized inverse to calculate the weight matrix in the single-hidden-layer complex-valued neural network, set the initial value of the number of iterations to 1, and use the gradient Descent method calculates the weight vector in the single hidden layer complex-valued neural network;

(3) According to the weight matrix calculated in step (2) and the weight vector, obtain the complex-valued neural network parameters, calculate the mean square error of the current sample data; judge whether the current number of iterations is equal to the maximum number of iterations , if yes, end the training; if not, add 1 to the current iteration number, and return to step (2).

Preferably, the sample data set in the step (1) includes a training sample data set or a test data set.

Preferably, the single hidden layer complex-valued neural network model in the step (1) is:

The number of neurons in the input layer, hidden layer and output layer in the single hidden layer complex-valued neural network model is L, M and 1 respectively;

Given Q input samples, its sample matrix is Z=(z _ij ) _L×Q =Z ^R +iZ ^I , where Z ^R is the real part of Z, and Z ^I is the imaginary part of Z;

The input for the qth sample is Among them, i=1,2...L;

The ideal output matrix corresponding to the input samples is D=(d ₁ ,d ₂ ...d _Q ) ^T ＝D ^R +iD ^I , where D ^R is the real part of D, and D ^I is the imaginary part of D;

The ideal output of the qth sample is d ^q ∈ C.

Preferably, the activation function of the hidden layer in the single hidden layer complex-valued neural network model is g _c : C→C;

The weight matrix connecting the input layer and the hidden layer is W=(w _ij ) _M×L =W ^R +iW ^I , wherein, W ^R is the real part of W, and W ^I is the imaginary part of W;

The connection weight between the input layer and the i-th hidden node is recorded as w _i =(w _i1 ,w _i2 ...w _iL ) ^∈CL , where i=1,2...M;

The weight vector connecting the hidden layer and the output layer is V=(v ₁ ,v ₂ ...v _M ) ^T =V ^{R +} iV ^I , where VR is the real part of V, and V ^I is the imaginary part of V;

The connection weight between the kth hidden node and the output layer is denoted as v _k ∈ C, where k=1,2...M.

Preferably, the specific steps in the step (2) are:

(2-1) Initialize the weight matrix from the input layer to the hidden layer, and obtain the initial weight matrix W ⁰ , where W ⁰ is randomly assigned within a given interval;

(2-2) Using gradient descent method and generalized inverse to calculate the weight matrix and weight vector in the single hidden layer complex-valued neural network.

Preferably, the specific steps of calculating the weight matrix V from the hidden layer to the output layer by generalized inversion in the step (2-2) are:

(2-2a-1) Calculate the input matrix U=(u _ij ) _M×Q of the hidden layer according to the initial weight matrix W ⁰ in step (2-1) and the sample matrix Z in step (1),

(2-2a-2) Activate the real and imaginary parts of the input matrix U in the matrix step (2-2a-1) respectively, to obtain the output matrix H=(h _ij ) _M×Q of the hidden layer, H =g _c (U ^R ))+ig _c (U ^I ))=H ^R +iH ^I , wherein, ^HR is the real part of H, and H ^I is the imaginary part of H;

(2-2a-3): Calculate the weight matrix V from the hidden layer to the output layer by generalized inverse,

Wherein, H is the output matrix of the hidden layer in step (2-2a-2), and D is the ideal output matrix in step (1).

Preferably, the specific steps of optimizing the initial weight matrix W0 in the step (2-2 ⁾ are:

(2-2b-1): Set the initial number of iterations k=1, and the maximum number of iterations is K;

(2-2b-2): Calculate the gradient of the mean square error E with respect to the hidden layer weight W;

(2-2b-3): The weight update formula is Among them, n=1,2,...,η is the learning rate.

Preferably, in the step (2-2b-2), the gradient of the mean square error E about the hidden layer weight W is divided into two parts for calculation, and the gradient of E to W ^R is first obtained, and then the gradient of E to W ^I is obtained, wherein , W ^R is the real part of W, and W ^I is the imaginary part of W;

In the formula, z ^q is the input vector of the qth sample, z ^{q, R} is the real part of the input vector of the qth sample, z ^{q, I} is the imaginary part of the input vector of the qth sample, g _c is the implicit The activation function of the layer, is the input of the qth sample at the mth hidden node, is the real part of the input of the qth sample at the mth hidden node, is the imaginary part of the input of the qth sample to the mth hidden node.

Preferably, the specific steps in the step (3) are:

(3-1) The activation function of the output layer chooses a linear function, then the input of the output layer is equal to the output of the output layer, the actual output of the network is O=(o ₁ ,o ₂ …o _Q ) ^T , the actual qth sample The output is o _q ∈ C, q=1,2...Q, the matrix O is divided into two parts O ^R (real part) and O ^I (imaginary part), O=H ^T V=O ^R +iO ^I the qth Actual output of the sample:

Among them, v ^R is the real part of the weight vector from the hidden layer to the output layer, v ^I is the imaginary part of the weight vector from the hidden layer to the output layer, h ^{q, R} is the real part of the qth sample hidden layer output vector, h ^{q, I} is the imaginary part of the qth sample hidden layer output vector;

(3-2) Calculate the mean square error of the current sample data, judge whether the current iteration number k is equal to the maximum iteration number K, if so, end the training; if not, add 1 to the current iteration number, and return to step (2-2b-2) .

Preferably, the mean square error of calculating the current sample data in the step (3) adopts:

Among them, o ^q is the actual output of the qth sample, and d ^q is the ideal output of the qth sample.

Beneficial effects of the present invention:

1. A complex-valued neural network training method based on the gradient descent method and generalized inverse of the present invention, the hidden layer input weight is iteratively generated by the gradient descent method, and the output weight is always solved by the generalized inverse. Therefore, compared with BSCBP, the number of iterations of this method is less, the corresponding training time is shorter, the convergence speed is faster, and the learning efficiency is higher. Compared with CELM, the required number of hidden layer nodes is small, and the learning efficiency is high. Therefore, the present invention has higher precision than the BSCBP method and the CELM method, and can more accurately reflect the performance of the complex-valued neural network model.

2, a kind of complex-valued neural network training method based on gradient descent method and generalized inverse of the present invention, when solving the weight value from hidden layer to output layer, has adopted the method for generalized inverse, does not need to iterate, and one step obtains output weight The minimum norm least squares solution of , compared with related methods based on the gradient descent method (such as the CBP method), the training speed is faster.

Description of drawings

Fig. 1 is a schematic flow sheet of the method of the present invention;

Fig. 2 is a graph comparing the present invention with BSCBP and CELM modeling methods.

detailed description:

It should be pointed out that the following detailed description is exemplary and intended to provide further explanation to the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terminology used here is only for describing specific implementations, and is not intended to limit the exemplary implementations according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that when the terms "comprising" and/or "comprising" are used in this specification, they mean There are features, steps, operations, means, components and/or combinations thereof.

In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

Example 1:

In this embodiment, the three-dimensional equalizer model of the nonlinear distortion of the 4-QAM signal in the document "Channel Equalization Using Adaptive Complex Radial Basis Function Networks" is cited. Among them, the input of the equalizer is

The ideal output of the equalizer is 0.7+0.7i, 0.7-0.7i, -0.7+0.7i, -0.7-0.7i.

In this embodiment, the training data set and the testing data set take 70% and 30% of the overall sample data set respectively.

First, the data set is modeled by a complex-valued neural network training method based on the gradient descent method and the generalized inverse of the present invention.

A kind of complex-valued neural network training method based on gradient descent method and generalized inverse, its method flowchart as shown in Figure 1, described method step comprises:

(1) Select a single hidden layer complex-valued neural network model to model the sample training data set or the sample test data set;

The single hidden layer complex-valued neural network model in the described step (1) is:

The number of neurons in the input layer, hidden layer and output layer in the single hidden layer complex-valued neural network model is L, M and 1 respectively;

Given Q input samples, its sample matrix is Z=(z _ij ) _L×Q ＝Z ^R +iZ ^I , where Z ^R is

The real part of Z, Z ^I is the imaginary part of Z;

The input for the qth sample is Among them, i=1,2...L;

The ideal output matrix corresponding to the input samples is D=(d ₁ ,d ₂ ...d _Q ) ^T ＝D ^R +iD ^I , where D ^R is the real part of D, and D ^I is the imaginary part of D;

The ideal output of the qth sample is d ^q ∈ C.

The activation function of the hidden layer in the single hidden layer complex-valued neural network model is g _c : C→C;

The weight matrix connecting the input layer and the hidden layer is W=(w _ij ) _M×L =W ^R +iW ^I , wherein, W ^R is the real part of W, and W ^I is the imaginary part of W;

The connection weight between the input layer and the i-th hidden node is recorded as w _i =(w _i1 ,w _i2 ...w _iL ) ^∈CL , where i=1,2...M;

The weight vector connecting the hidden layer and the output layer is V=(v ₁ ,v ₂ ...v _M ) ^T =V ^{R +} iV ^I , where VR is the real part of V, and V ^I is the imaginary part of V;

The connection weight between the kth hidden node and the output layer is denoted as v _k ∈ C, where k=1,2...M.

(2) According to the single-hidden-layer complex-valued neural network model selected in step (1), use the generalized inverse to calculate the weight matrix in the single-hidden-layer complex-valued neural network, set the initial value of the number of iterations to 1, and use the gradient Descent method calculates the weight vector in the single hidden layer complex-valued neural network;

Step S21: Initialize the weight matrix from the input layer to the hidden layer, and obtain the initial weight matrix W ⁰ , where W ⁰ is randomly assigned in a given interval;

Step S22: The input matrix of the hidden layer is U=(u _ij ) _M×Q , the matrix U is divided into U ^R (real part) and U ^I (imaginary part), U=WZ=U ^R +iU ^I , the first The input of the mth hidden layer node of q samples:

In the formula, x ^q is the real part of the qth input sample, y ^q is the imaginary part of the qth input sample, is the real part of the input weight vector of the mth hidden node, is the imaginary part of the input weight vector of the mth hidden node;

Step S23: Activate the real part and the imaginary part of the matrix U respectively, and obtain the output matrix of the hidden layer as H=(h _ij ) _M×Q , and the matrix H is divided into two parts: H ^R (real part) and H ^I (imaginary part). part, H=g _c (U ^R )+ig _c (U ^I )=H ^R +iH ^I

Step S24: Calculate the weight matrix V from the hidden layer to the output layer by generalized inversion,

Step S25 ^: Optimizing the initial weight matrix W0, the optimization step S25 is as follows:

Step S251: set the initial number of iterations k=1; (the maximum number of iterations is K)

Step S252: Calculate the gradient of the mean square error E with respect to the hidden layer weight W, and divide the calculation into two parts. First, calculate the gradient of E to W ^R , and then calculate the gradient of E to W ^I ,

Step S27: The weight update formula is W ⁿ⁺¹ =W ⁿ +ΔW ⁿ n is the number of iterations n=1,2,..., where and That is, the gradient of the mth hidden node at the nth iteration; the learning rate η is a constant;

(3) According to the weight matrix calculated in step (2) and the weight vector, obtain the complex-valued neural network parameters, calculate the mean square error of the current sample data; judge whether the current number of iterations is equal to the maximum number of iterations , if yes, end the training; if not, add 1 to the current iteration number, and return to step (2).

Step S31: The activation function of the output layer is a linear function, then the input of the output layer is equal to the output of the output layer, the actual output of the network is O=(o ₁ ,o ₂ ...o _Q ) ^T , and the actual output of the qth sample is o _q ∈ C, q=1,2...Q, divide the matrix O into two parts, O ^R (real part) and O ^I (imaginary part), O=H ^T V=O ^R +iO ^I The qth sample Actual output:

In the formula, v ^R is the real part of the weight vector from the hidden layer to the output layer, v ^I is the imaginary part of the weight vector from the hidden layer to the output layer, h ^q,R is the real part of the hidden layer output vector of the qth sample, h ^{q, I} is the imaginary part of the qth sample hidden layer output vector;

Step S32: Calculate the error function of the training samples:

Let k=k+1, return to step S22 (the weight matrix V from the hidden layer to the output layer is always solved by generalized inversion).

This embodiment includes two comparative modeling methods, the BSCBP method and the CELM method. The CELM method is a method in the document "Fully complex extreme learning machine". This method randomly assigns values to the input weight matrix, and the output weight matrix is solved by generalized inversion. The three-dimensional equalizer model of the nonlinear distortion of the 4-QAM signal in the data set (document "Channel Equalization Using Adaptive Complex Radial Basis Function Networks" in the literature "Channel Equalization Using Adaptive Complex Radial Basis Function Networks" is processed by the BSCBP method and the CELM method respectively. Wherein, the input of the equalizer is

The ideal output of the equalizer is 0.7+0.7i, 0.7-0.7i, -0.7+0.7i, -0.7-0.7i) for modeling. The experimental results are shown in Figure 2.

It can be seen from Figure 2 that under the same network structure, the training errors of this method are lower than those of the BSCBP method and the CELM method, indicating that the method of the present invention effectively optimizes the training weights and achieves very high training accuracy.

In the method of the present invention, the input weight matrix of the hidden layer is updated by the gradient descent method. Therefore, compared with CELM, under the condition of the same initial weight value, the number of hidden nodes required by the GGCNN model is far less than that required by CELM The number of hidden nodes is smaller than the mean square error produced by the CELM model;

In the complex-valued neural network modeling method (Gradient based Generalized Complex Neural Networks, referred to as the GGCNN method) based on the gradient descent method and the generalized inverse of the present invention, the output weight matrix of the hidden layer is solved through the generalized inverse, without iteration, and can be obtained in one step. Output the minimum norm least squares solution of the output weight matrix. Therefore, under the condition that the initial weight value and the number of hidden nodes are the same, the method of the present invention is not only faster than the BSCBP training speed, but also greatly reduces the training error and test error. The detailed comparison results are shown in Table 1.

Table 1

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, there may be various modifications and changes in the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims (10)

1. A complex-valued neural network training method based on gradient descent method and generalized inverse, it is characterized in that: the method steps include: (1) Select a single hidden layer complex-valued neural network model to model the sample data set; (2) According to the single-hidden-layer complex-valued neural network model selected in step (1), use the generalized inverse to calculate the weight matrix in the single-hidden-layer complex-valued neural network, set the initial value of the number of iterations to 1, and use the gradient Descent method calculates the weight vector in the single hidden layer complex-valued neural network; (3) According to the weight matrix calculated in step (2) and the weight vector, obtain the complex-valued neural network parameters, calculate the mean square error of the current sample data; judge whether the current number of iterations is equal to the maximum number of iterations , if yes, end the training; if not, add 1 to the current iteration number, and return to step (2). 2. a kind of complex-valued neural network training method based on gradient descent method and generalized inverse as claimed in claim 1, is characterized in that: described single hidden layer complex-valued neural network model in described step (1) is: The number of neurons in the input layer, hidden layer and output layer in the single hidden layer complex-valued neural network model is L, M and 1 respectively; given Q input samples; its sample matrix is Z=(z _ij ) _{L ×Q} , the ideal output matrix corresponding to the input sample is D=(d ₁ ,d ₂ …d _Q ) ^T are all complex numbers; the input of the qth sample is Among them, i=1,2...L; the ideal output of the qth sample is d ^q ∈C. 3. a kind of complex-valued neural network training method based on gradient descent method and generalized inverse as claimed in claim 2, is characterized in that: in the described single hidden layer complex-valued neural network model in described step (1) The activation function of the hidden layer is g _c :C→C; the weight matrix connecting the input layer and the hidden layer is W=(w _ij ) _M×L =W ^R +iW ^I , where W ^R is the real part, W ^I is the imaginary part of W; the connection weight between the input layer and the i-th hidden node is recorded as w _i =(w _i1 ,w _i2 …w _iL ) ^∈CL , where i=1,2…M ;The weight vector connecting the hidden layer and the output layer is V=(v ₁ , v ₂ ...v _M ) ^T =V ^R +iV ^I , ^wherein , VR is the real part of V, and V ^I is the imaginary part of V; The connection weight between the kth hidden node and the output layer is denoted as v _k ∈ C, where k=1,2...M. 4. a kind of complex-valued neural network training method based on gradient descent method and generalized inverse as claimed in claim 3, is characterized in that: the concrete steps in described step (2) are: (2-1) Initialize the weight matrix from the input layer to the hidden layer, and obtain the initial weight matrix W ⁰ , where W ⁰ is randomly assigned within a given interval; (2-2) Using gradient descent method and generalized inverse to calculate the weight matrix and weight vector in the single hidden layer complex-valued neural network. 5. a kind of complex-valued neural network training method based on gradient descent method and generalized inverse as claimed in claim 4, is characterized in that: described in the described step (2-2) by generalized inverse calculation hidden layer to output The specific steps of the weight matrix V of the layer are: (2-2a-1) Calculate the input matrix U=(u _ij ) _M×Q of the hidden layer according to the initial weight matrix W ⁰ in step (2-1) and the sample matrix Z in step (1), (2-2a-2) Activate the real and imaginary parts of the input matrix U in the matrix step (2-2a-1) respectively, to obtain the output matrix H=(h _ij ) _M×Q of the hidden layer, H =g _c (U ^R )+ig _c (U ^I )=HR ⁺ iH ^I , where HR is the real part of H, and ^{H I} is the imaginary part of H; (2-2a-3): Calculate the weight matrix V from the hidden layer to the output layer by generalized inverse, Wherein, H is the output matrix of the hidden layer in step (2-2a-2), and D is the ideal output matrix in step (1). 6. a kind of complex-valued neural network training method based on gradient descent method and generalized inverse as claimed in claim 4, is characterized in that: initial weight matrix W in described step (2-2 ⁾ is optimized specifically The steps are: (2-2b-1): Set the initial number of iterations k=1, and the maximum number of iterations is K; (2-2b-2): Calculate the gradient of the mean square error E with respect to the hidden layer weight W; (2-2b-3): The weight update formula is Among them, n=1,2,...,η is the learning rate. 7. a kind of complex-valued neural network training method based on gradient descent method and generalized inverse as claimed in claim 6, is characterized in that: in the described step (2-2b-2), the mean square error E is about hidden layer weights The gradient of W is calculated in two parts. First, calculate the gradient of E to W ^R , and then calculate the gradient of E to W ^I , where W ^R is the real part of W, and W ^I is the imaginary part of W; 8. a kind of complex-valued neural network training method based on gradient descent method and generalized inverse as claimed in claim 7, is characterized in that: In the formula, z ^q is the input vector of the qth sample, z ^{q, R} is the real part of the input vector of the qth sample, z ^{q, I} is the imaginary part of the input vector of the qth sample, g _c is the implicit The activation function of the layer, is the input of the qth sample at the mth hidden node, is the real part of the input of the qth sample at the mth hidden node, is the imaginary part of the input of the qth sample to the mth hidden node. 9. a kind of complex-valued neural network training method based on gradient descent method and generalized inverse as claimed in claim 1, is characterized in that: the concrete steps in described step (3) are: (3-1) The activation function of the output layer chooses a linear function, then the input of the output layer is equal to the output of the output layer, the actual output of the network is O=(o ₁ ,o ₂ …o _Q ) ^T , the actual qth sample The output is o _q ∈ C, q=1,2...Q, the matrix O is divided into two parts O ^R (real part) and O ^I (imaginary part), O=H ^T V=O ^R +iO ^I the qth Actual output of the sample: Among them, v ^R is the real part of the weight vector from the hidden layer to the output layer, v ^I is the imaginary part of the weight vector from the hidden layer to the output layer, h ^{q, R} is the real part of the qth sample hidden layer output vector, h ^{q, I} is the imaginary part of the qth sample hidden layer output vector; (3-2) Calculate the mean square error of the current sample data, judge whether the current iteration number k is equal to the maximum iteration number K, if so, end the training; if not, add 1 to the current iteration number, and return to step (2-2b-2) . 10. a kind of complex-valued neural network training method based on gradient descent method and generalized inverse as claimed in claim 9, is characterized in that: the mean square error of calculating current sample data in the described step (3) adopts: Among them, o ^q is the actual output of the qth sample, and d ^q is the ideal output of the qth sample.