CN113159299B - Fractional order depth BP neural network optimization method based on extremum optimization - Google Patents
Fractional order depth BP neural network optimization method based on extremum optimization Download PDFInfo
- Publication number
- CN113159299B CN113159299B CN202110484178.8A CN202110484178A CN113159299B CN 113159299 B CN113159299 B CN 113159299B CN 202110484178 A CN202110484178 A CN 202110484178A CN 113159299 B CN113159299 B CN 113159299B
- Authority
- CN
- China
- Prior art keywords
- neural network
- fractional order
- optimization
- population
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 88
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000005457 optimization Methods 0.000 title claims abstract description 43
- 238000012549 training Methods 0.000 claims abstract description 24
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 14
- 230000008569 process Effects 0.000 claims abstract description 9
- 238000003062 neural network model Methods 0.000 claims abstract description 6
- 230000006870 function Effects 0.000 claims description 16
- 230000035772 mutation Effects 0.000 claims description 6
- 210000002569 neuron Anatomy 0.000 claims description 6
- 230000004913 activation Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000012937 correction Methods 0.000 claims description 2
- 230000001351 cycling effect Effects 0.000 claims description 2
- 238000011423 initialization method Methods 0.000 claims description 2
- 239000011159 matrix material Substances 0.000 claims description 2
- 238000011160 research Methods 0.000 abstract description 4
- 230000002411 adverse Effects 0.000 abstract description 2
- 230000000694 effects Effects 0.000 abstract description 2
- 238000004088 simulation Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Abstract
The invention discloses a fractional order depth BP neural network optimization method based on extremum optimization. The fractional order system has the advantages of high convergence speed and high convergence accuracy, the method utilizes a population extremum optimization algorithm to perform preferential selection on the initial weight of the established fractional order depth BP (PEO-FODBP) neural network, and the optimal individuals in the population and the optimal fitness value thereof are also subjected to iterative optimization while the network level weight is corrected in the iterative training process of the network, so that adverse effects of the initial weight on the neural network are improved. The problems of easy sinking into local minimum value, long time consumption and low convergence speed in the prior art are solved. By utilizing the optimization method, the application performance of the fractional order depth BP neural network in various fields can be improved. In addition, the population extremum optimizing method and the fractional derivative calculating method in the method can be popularized to other more neural network models, and the optimization research of the neural network is promoted.
Description
Technical Field
The invention belongs to the technical field of artificial intelligence, relates to a fractional order depth BP neural network optimization method based on extremum optimization, and particularly relates to modeling of a depth BP neural network and design of an extremum optimization algorithm.
Background
Neural networks have received considerable attention from researchers in the fields of machine learning, statistics and computer vision as a powerful data regression and classification tool. The deep BP neural network is a developed network system composed of a large number of processing units, which is inspired by the working principle of biological neural tissue, has the basic characteristics of the biological neural system, and has the advantages of large-scale parallel operation, distributed processing, self-adaption and self-learning.
Fractional calculus theory has been a classical concept in mathematics for hundreds of years, and is based on differentiation and integration of any fractional order, and is a popularization of integer calculus popular at present. Compared with the traditional integer-order system, the fractional-order system has the advantages of high convergence speed and high convergence accuracy, so that the fractional-order system is widely applied to the fields of image processing, machine learning and neural networks.
Extremum optimization is a new optimization method developed by being inspired by equilibrium dynamics far from the criticality of self-organization, and has been successfully applied to various combinatorial optimization problems. The basic principle of the extremum optimizing algorithm is that the individual with the lowest fitness value in the current solving range and the related variable are selected for variation, so that the system is continuously improved to the optimal solution. Therefore, the extremum optimizing algorithm does not terminate in an equilibrium state, but continuously fluctuates to improve the searching ability of the algorithm in the solution domain.
Many research works related to improving neural network training exist at home and abroad, but most of the research works are iterative optimization on training parameters of the network, but important influences of initial weights of the neural network on the performance of the neural network are ignored, and the problems of easy local minimum, long time consumption and slow convergence speed of the methods generally exist.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a fractional order depth BP neural network optimization method based on extremum optimization, which utilizes a Population extremum optimization algorithm (PEO-based extremal optimization) to preferentially select the initial weight of a fractional order depth BP (PEO-FODBP) neural network and improves the adverse effect of the initial weight on the neural network.
The fractional order depth BP neural network optimization method based on extremum optimization specifically comprises the following steps:
step one, establishing a fractional order depth BP neural network model;
establishing an L-layer deep BP neural network model, wherein the number of the neuron nodes of the first layer is n l ，l＝1,2,...,L，I=1, 2, n., is a weight between the neural network layer i and layer (l+1) l ,j＝1,2,...,n l +1 。f l (. Cndot.) is the activation function of the first layer, X is the input sample of the neural network, O is the ideal output of the input sample X,for input of the layer I neural network, A l Is the output of the layer-1 neural network. />Consists of two-part nodes->Is an external node, and is not connected with the neuron node of the upper layer; />Is an internal node, and is fully connected with the neuron node of the upper layer. Thus, the forward propagation process of the neural network can be expressed as:
the neural network has a loss function ofWherein, I is Euclidean norm, and Sigma is sum symbol;
the back propagation process of the fractional order depth BP neural network can be divided into two parts of gradient propagation and gradient calculation. In the gradient propagation process, the gradient propagation term of the first layer of the neural network can be defined as:
in the method, in the process of the invention,representing the partial derivative;
delta according to the chain law l And delta l+1 The relationship of (2) can be expressed as:
wherein f' (. Cndot.) represents the first derivative of f (. Cndot.);
thus, the fractional gradient computation under the Caputo definition can be expressed as:
where v is a fraction, representing the order of the fractional order,representing the weight W of the loss function E under the definition of the fractional derivative of Caputo l Solving fractional derivatives, wherein Γ (·) is a gamma function;
the weight correction expression of each layer of the neural network is as follows:
wherein t is a natural number, represents the current training algebra of the neural network, and mu is the learning rate.
Initializing parameters;
various parameters in the initialization method comprise: neural network training times I max The learning rate mu, the order v of fractional order, the population size T, the population iteration times G, and the neural network training times I of individual fitness values are calculated f And a mutation factor b.
Step three, generating an initial population;
P＝x min +(x max -x min )×rand(T,c) (6)
randomly generating an initial population p= { S according to formula (6) 1 ,S 2 ,...,S T S, where S p ＝{x 1 ,x 2 ,...,x c }，p＝1,2,...,T，Representing the total number of weights contained in the neural network, x max And x min Representing the upper and lower bounds of the weights, respectively, rand (T, c) represents the generation of a T C-dimensional random matrix ranging between (0, 1).
Step four, calculating individual fitness;
individuals S of the population p After decoding, substituting the decoded data into the FODBP neural network as an initial weight, inputting a training sample into the neural network for training, correcting the weight of the neural network according to the step one, and enabling the training frequency to reach I f Then obtaining training accuracy and using the training accuracy as fitness function value f of the individual p 。
Step five, selecting individuals;
the individuals of the T populations are subjected to the fitness function value f p Ordering is performed such that f Π1 >f Π2 >...>f ΠT Selecting pi 1 to piIs replaced by pi->To pi T, and obtaining an optimal individual S best ＝S Π1 Optimal fitness value f best ＝f Π1 。
Step six, generating a new population;
performing mutation operation on the population selected in the fifth step according to Non-uniform mutation (NUM) rules to generate a new population P N ＝{S N1 ,S N2 ,…,S NT -a }; the non-uniform variation rule is as follows:
wherein e is a random number between (0, 1), x mid ＝(x max +x min )/2，The positive integer g is the current iteration number of the population extremum optimizing algorithm.
Step seven, iteration circulation;
repeating the fourth to sixth steps to obtain a new optimal individual S Nbest Fitness value f Nbest If f Nbest >f best Then the new optimal individual and the optimal fitness value are saved, namely S best ＝S Nbest ,f best ＝f Nbest Cycling for multiple times until the number of times of group iterative optimization reaches G, and obtaining a global optimal solution S best And after decoding, substituting the obtained product into the FODBP neural network to serve as an initial weight, and completing optimization of the fractional order depth BP neural network.
The invention has the following beneficial effects:
1. and the initial weight of the fractional order depth BP neural network is preferentially selected by using a population extremum optimization algorithm, so that the aim of improving the performance of the neural network is fulfilled.
2. The problem that the fractional order depth BP neural network is easy to be trapped in a local extremum is remarkably improved, and the performance of the fractional order depth BP neural network is improved, so that the fractional order depth BP neural network has stronger applicability.
3. The population extremum optimizing method and the fractional derivative calculating method can be popularized to other more neural network models, and the optimization research of the neural network is promoted.
Drawings
FIG. 1 is a conceptual diagram of a fractional order deep BP neural network;
FIG. 2 is a flow chart of the present optimization method;
FIG. 3 is a block diagram of a fractional order depth BP neural network built in an embodiment;
FIG. 4 is a chart showing error convergence of the embodiment and other algorithms.
Detailed Description
The invention is further explained below with reference to the drawings;
a fractional order depth BP neural network optimization method based on extremum optimization, wherein the structure of the fractional order depth BP neural network is shown in figure 1, and figure 2 is a flow chart of the optimization method.
Establishing a fractional order depth BP neural network shown in fig. 3, wherein the network has 8 layers in total, 196 external nodes in the first four layers, 32 internal nodes in the first four layers, and 1 input of the internal nodes in the first layer of the neural network; the latter four layers of neural networks have no external nodes and 64 internal nodes. The activation function of each layer of the neural network is a sigmoid function. In the identification of handwriting, MATLAB is adopted for the simulation experiment to carry out software programming, and MNIST handwriting digital data is adopted for identification simulation. MNIST handwriting digital data consists of 60000 training data and 10000 test data, each set of data is normalized by handwriting digital images. Wherein, a digital image has 28×28 gray data, and the gray value is between 0 and 255. In the simulation experiment, a digital image is divided into 4 parts, each part has 14×14 gray data, the data are arranged into 196×1 vectors, and finally 4 groups of data are respectively input into a neural network for training.
The parameters of the simulation are shown in table 1.
TABLE 1
Wherein, the Batch size is the number of the input samples at one time. The generated neural network initial weights obey a standard normal distribution.
Table 2 shows the training accuracy and the testing accuracy of the FODBP neural network. Table 3 shows the training accuracy and testing accuracy of the PEO-FODBP neural network. From the data in the two tables, it can be found that the training precision and the testing precision of the fractional order depth BP neural network are obviously improved by using the group extremum optimization algorithm, and the performance of the fractional order depth BP neural network is obviously improved.
TABLE 2
TABLE 3 Table 3
As shown in FIG. 4, the error convergence diagrams of the PEO-FODBP algorithm and the FODBP algorithm along with the training process of the invention show that the convergence speed of the fractional depth BP neural network after the optimization of the PEO algorithm is faster and the convergence accuracy is higher. From the simulation results, it can be concluded that: the invention provides a fractional order depth BP neural network method based on population extremum optimization, and compared with a common fractional order depth BP neural network, the performance is obviously improved. The method provided by the invention can improve the application performance of the fractional order depth BP neural network in various fields.
Claims (6)
1. A fractional order depth BP neural network optimization method based on extremum optimization is characterized in that: the method specifically comprises the following steps:
step one, establishing a fractional order depth BP neural network model;
establishing an L-layer deep BP neural network model, wherein the number of the neuron nodes of the first layer is n l ，l＝1,2,2,L，I=1, 2, n, which is the weight between the first layer and the (l+1) th layer of the neural network l ,j＝1,2,2,n l+1 The method comprises the steps of carrying out a first treatment on the surface of the X is the input sample of the neural network, O is the ideal output of the input sample X, ++>For input of the layer I neural network, A l Is the output of the layer I neural network; the neural network has a loss function of +.>Wherein, I is Euclidean norm, and Sigma is sum symbol;
fractional order gradient under the Caputo definitionE is:
wherein delta l The gradient propagation term of the first layer of the neural network, v is a fraction, representing the order of the fractional order,e represents the weight W of the loss function E under the definition of the fractional derivative of Caputo l Solving fractional derivatives, wherein Γ (·) is a gamma function;
the weight correction expression of each layer of the neural network is as follows:
wherein t is a natural number, represents the current training algebra of the neural network, and mu is a learning rate;
initializing parameters;
various parameters in the initialization method comprise: neural network training times I max The learning rate mu, the order v of fractional order, the population size T, the population iteration times G, and the neural network training times I of individual fitness values are calculated f A mutation factor b;
step three, generating an initial population;
randomly generating an initial population p= { S 1 ,S 2 ,2,S T S, where S p ＝{x 1 ,x 2 ,...,x c }，p＝1,2,...,T，Representing the total number of weights contained by the neural network;
step four, calculating individual fitness;
individuals S of the population p After decoding, substituting the decoded data into the FODBP neural network as an initial weight, inputting MNIST handwriting digital data into the neural network for training, correcting the weight of the neural network according to the step one, and enabling the training frequency to reach I f Then obtaining training accuracy and using the training accuracy as fitness function value f of the individual p ；
Step five, selecting individuals;
sorting the individuals of the T populations according to their fitness function values so that f Π1 >f Π2 >...>f ΠT Selecting pi 1 toIndividual replacement of->To pi T, and obtaining an optimal individual S best ＝S Π1 Optimal fitness value f best ＝f Π1 ；
Step six, generating a new population;
performing mutation operation on the population selected in the fifth step according to the non-uniform mutation rule to generate a new population P N ＝{S N1 ,S N2 ,…,S NT }；
Step seven, iteration circulation;
repeating the fourth to sixth steps to obtain a new optimal individual S Nbest Fitness value f Nbest If f Nbest >f best Then the new optimal individual and the optimal fitness value are saved, namely S best ＝S Nbest ,f best ＝f Nbest Cycling for multiple times until the number of times of group iterative optimization reaches G, and obtaining a global optimal solution S best After decoding, substituting the obtained product into the FODBP neural network as an initial weight to finish fractional order depthOptimizing a BP neural network;
step eight, character identification
Inputting handwriting numbers into the fractional order depth BP neural network after the optimization in the step seven, and outputting the identification result by the network.
2. The method for optimizing the fractional order depth BP neural network based on extremum optimization as defined in claim 1, wherein the method is characterized by comprising the following steps of: output of layer 1 neural networkConsists of two-part nodes->Is an external node, and is not connected with the neuron node of the upper layer; />Is an internal node, and is fully connected with the neuron node of the upper layer; f (f) l (. Cndot.) is the activation function of layer i, the forward propagation process of the network is:
3. the method for optimizing the fractional order depth BP neural network based on extremum optimization as defined in claim 1 or 2, wherein the method comprises the following steps of: activation function f of the first layer of the neural network l (. Cndot.) is a sigmoid function.
4. The method for optimizing the fractional order depth BP neural network based on extremum optimization as defined in claim 1, wherein the method is characterized by comprising the following steps of: the back propagation process of the fractional order depth BP neural network is divided into two parts of gradient propagation and gradient calculation; in the gradient propagation process, the gradient propagation term delta of the first layer of the neural network l The method comprises the following steps:
delta according to the chain law l And delta l+1 The relation of (2) is:
wherein f' (. Cndot.) represents the first derivative of f (. Cndot.).
5. The method for optimizing the fractional order depth BP neural network based on extremum optimization as defined in claim 1, wherein the method is characterized by comprising the following steps of: the random generation method of the initial population comprises the following steps:
P＝x min +(x max -x min )×rand(T,c) (6)
wherein x is max And x min Representing the upper and lower bounds of the weights, respectively, rand (T, c) represents the generation of a T C-dimensional random matrix ranging between (0, 1).
6. The method for optimizing the fractional order depth BP neural network based on extremum optimization as defined in claim 1, wherein the method is characterized by comprising the following steps of: the non-uniform variation rule of the population in the step six is as follows:
wherein e is a random number between (0, 1), x mid ＝(x max +x min )/2，The positive integer g is the current iteration number of the population extremum optimizing algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110484178.8A CN113159299B (en) | 2021-04-30 | 2021-04-30 | Fractional order depth BP neural network optimization method based on extremum optimization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110484178.8A CN113159299B (en) | 2021-04-30 | 2021-04-30 | Fractional order depth BP neural network optimization method based on extremum optimization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113159299A CN113159299A (en) | 2021-07-23 |
| CN113159299B true CN113159299B (en) | 2024-02-06 |
Family
ID=76873052
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110484178.8A Active CN113159299B (en) | 2021-04-30 | 2021-04-30 | Fractional order depth BP neural network optimization method based on extremum optimization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113159299B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110046710A (en) * | 2019-04-11 | 2019-07-23 | 山东师范大学 | A kind of the nonlinear function Extremal optimization method and system of neural network |
| CN111160520A (en) * | 2019-12-06 | 2020-05-15 | 南京理工大学 | BP neural network wind speed prediction method based on genetic algorithm optimization |
-
2021
- 2021-04-30 CN CN202110484178.8A patent/CN113159299B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110046710A (en) * | 2019-04-11 | 2019-07-23 | 山东师范大学 | A kind of the nonlinear function Extremal optimization method and system of neural network |
| CN111160520A (en) * | 2019-12-06 | 2020-05-15 | 南京理工大学 | BP neural network wind speed prediction method based on genetic algorithm optimization |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113159299A (en) | 2021-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhu et al. | Multi-objective evolutionary federated learning | |
| Kim et al. | SplitNet: Learning to semantically split deep networks for parameter reduction and model parallelization | |
| Zhang et al. | AS-NAS: Adaptive scalable neural architecture search with reinforced evolutionary algorithm for deep learning | |
| CN109948029A (en) | Based on the adaptive depth hashing image searching method of neural network | |
| CN111898689B (en) | Image classification method based on neural network architecture search | |
| CN107330902B (en) | Chaotic genetic BP neural network image segmentation method based on Arnold transformation | |
| CN108268947A (en) | For improving the device and method of the processing speed of neural network and its application | |
| Zou et al. | Bare-bones teaching-learning-based optimization | |
| Li et al. | Automatic design of convolutional neural network architectures under resource constraints | |
| CN103914527B (en) | Graphic image recognition and matching method based on genetic programming algorithms of novel coding modes | |
| CN114118369A (en) | Image classification convolution neural network design method based on group intelligent optimization | |
| CN113157919A (en) | Sentence text aspect level emotion classification method and system | |
| Hao et al. | Annealing genetic GAN for imbalanced web data learning | |
| CN111126560A (en) | Method for optimizing BP neural network based on cloud genetic algorithm | |
| CN113159299B (en) | Fractional order depth BP neural network optimization method based on extremum optimization | |
| Phan et al. | Efficiency enhancement of evolutionary neural architecture search via training-free initialization | |
| CN107590538B (en) | Danger source identification method based on online sequence learning machine | |
| CN113971367A (en) | Automatic design method of convolutional neural network framework based on shuffled frog-leaping algorithm | |
| CN111695689B (en) | Natural language processing method, device, equipment and readable storage medium | |
| CN112819161B (en) | Neural network construction system, method and storage medium for variable-length gene genetic algorithm | |
| CN110059806B (en) | Multi-stage weighted network community structure detection method based on power law function | |
| Hu et al. | Apenas: An asynchronous parallel evolution based multi-objective neural architecture search | |
| Brouwer | A hybrid neural network for input that is both categorical and quantitative | |
| CN111639797A (en) | Gumbel-softmax technology-based combined optimization method | |
| CN109918659A (en) | A method of based on not retaining optimum individual genetic algorithm optimization term vector |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |