CN108764472A - Convolutional neural networks fractional order error back propagation method - Google Patents
Convolutional neural networks fractional order error back propagation method Download PDFInfo
- Publication number
- CN108764472A CN108764472A CN201810490436.1A CN201810490436A CN108764472A CN 108764472 A CN108764472 A CN 108764472A CN 201810490436 A CN201810490436 A CN 201810490436A CN 108764472 A CN108764472 A CN 108764472A
- Authority
- CN
- China
- Prior art keywords
- convolutional neural
- neural networks
- back propagation
- network
- fractional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
Convolutional neural networks fractional order error back propagation method of the present invention, includes the following steps：Step 1）Select convolutional neural networks model；Step 2）Network weight is updated using fractional order error back propagation method；Step 3）According to the network weight, convolutional neural networks parameter is obtained.Advantageous effect：The integer order derivative of activation primitive during error back propagation is become into Fractional Derivative, fractional order derivation model advantage more accurate than integer rank derivation model is played, " gradient disappearance " problem during error back propagation has been effectively relieved, improve the training effectiveness of convolutional neural networks, so that convolutional neural networks is realized preferably fitting for training data, improves the generalization ability of network.
Description
Technical field
The invention belongs to intelligent algorithm fields, more particularly to a kind of convolutional neural networks fractional order error reversely passes
Broadcasting method.
Background technology
Convolutional neural networks are the algorithms of artificial intelligence field, and the image from the automobile of automatic Pilot to Google is searched,
Convolutional neural networks can be used to realize.Training process is a very important part in convolutional neural networks, in the mistake
Generally network weight (filter) is set to constantly update adjustment by back-propagation method in journey, until the output of network becomes with target
In consistent.The process of back-propagation method is divided into 4 basic steps：Propagated forward, costing bio disturbance, backpropagation, weights are more
Newly.However, in " backpropagation " step, usually will appear because " gradient disappearance " phenomenon causes network to be difficult to trained problem.
Therefore, it is that convolutional neural networks are able to normal training to be effectively improved " gradient disappearance ", to obtain the important of precision net parameter
Ensure.
Currently, the combination of fractional calculus and neural network has become the research hotspot of artificial intelligence field, fractional order
The application of neural network shows the advantage for having bigger than traditional neural network.《Voice based on fractional order convolutional neural networks
Recognizer research》(Zhang Lei Harbin University of Science and Technologys, 2016.) is directly handled activation primitive using fractional order method,
Network training is participated in using fractional order activation primitive, improves the training speed of convolutional neural networks.《Fractional order theory is in BP god
Through the application in network》(Chen little Lei Nanjing Forestry University, 2013.) convolutional neural networks train back-propagation process in, directly
It connects and fractional order partial derivative is asked to error, and become the iterative learning of order, realize the switching of fractional-order and integer order
With the adaptive adjustment of fractional-order so that the receipts precision that network training not only has convergence rate but also had.Patent application《It is a kind of
The method for improving convolutional neural networks structure》(publication number：106960243 A of CN) fractional order maximum value pond principle is utilized, it will
Maximum value pond layer Change All in traditional convolution neural network structure is fractional order, reaches the down-sampling of the arbitrary dimension of image
Dimensionality reduction improves network training efficiency.Although above-mentioned algorithm has some superiority compared with traditional algorithm, the combination side of " loose "
Formula does not probe into fractional order theory and convolutional neural networks theory combination the name of the game.Targetedly improve convolutional Neural
The training process of network can just further increase network parameter precision, improve network performance.
Invention content
The purpose of the present invention is overcoming the shortcomings of above-mentioned background technology, fractional order and convolutional neural networks theory are realized
It is " close " to combine, a kind of convolutional neural networks fractional order error back propagation method is provided, is specifically realized by following scheme：
The convolutional neural networks fractional order error back propagation method, includes the following steps：
Step 1) selects convolutional neural networks model；
Step 2) updates network weight using fractional order error back propagation method；
Step 3) obtains convolutional neural networks parameter according to the network weight.
The further design of the convolutional neural networks fractional order error back propagation method is that the step 1) is specific
Including：
Step 1-1) setting neural network model, the neural network model include one layer of input layer, n-layer hidden layer and
One layer of output layer, the input layer, hidden layer, output layer node number correspond to n respectively1, n2, 1；
Step 1-2) a large amount of high-definition picture { X of settingiAnd its corresponding low-resolution image { YiComposition training
Collection, wherein：X indicates that high-definition picture, Y indicate that low-resolution image, i indicate image index；Step 1-3) setting convolution god
Weights through network and it is biased to Θ={ W, B }, it is respectively L and σ to set error function and activation primitive, and wherein W indicates weights,
B is biasing.
The further design of the convolutional neural networks fractional order error back propagation method is that the step 2 includes
Following sub-step：
Step 2-1) netinit：Initialize network weight and biasing Θ={ W1,B1, maximum iteration is set；
Step 2-2) input training data { Xi,Yi}；
Step 2-3) setting primary iteration number k=1；
Step 2-4) network weight is optimized；
Step 2-5) according to k=k+1 update iterations, and jump to step 2-4).
The convolutional neural networks fractional order error back propagation method it is further design be, step 2-4) to network
Weights, which optimize, specifically includes following steps：
Step 2-4-1) calculate network output valve；
Step 2-4-2) calculate error：Using the error function, the error between network output valve and actual value is calculated,
Whether error in judgement meets the requirements, if met the requirements, network training terminates to be transferred to step 3)；If being unsatisfactory for requiring,
Modified weight is carried out, 2-4-3 is entered step)；
Step 2-4-3) Fractional Derivative of activation primitive in error back propagation gradient is calculated, and calculate each neuron
The Fractional Derivative of node；
Step 2-4-4) calculate error gradient；
Step 2-4-5) correct weights：Gradient according to error about weight, corrective networks weights.
The convolutional neural networks fractional order error back propagation method it is further design be, the step 2-4-3)
The middle Fractional Derivative that activation primitive in error back propagation gradient is calculated according to formula (1)：
Wherein, GradlIndicate l layers of error back propagation gradient,For the input of activation primitive, i.e.,
The output of network node,Indicate the of (l-1) layerA neuron is connected toThe weight of j-th of neuron of layer,Indicate that the biasing of l layers of j-th of neuron, μ indicate that the order of Fractional Derivative, ⊙ are matrix corresponding position product, table
Show point-to-point multiplying between matrix or vector；
The Fractional Derivative of each neuron node is calculated according to formula (2)：
Wherein,For fractional order differential operator, D is 1 rank differential operator, and x indicates the output valve of neural network node, σ
(x) indicate that activation primitive, σ ' (x) indicate that 1 order derivative of activation primitive, μ indicate that the order of fractional order differential, Γ (x) are Gamma
Function is defined asAnd Re (x) > 0.
The convolutional neural networks fractional order error back propagation method it is further design be, step 2-4-5) use
Classical stochastic gradient descent method corrects weights according to formula (3),
Wherein, i indicates that the network number of plies, η indicate learning rate, obtained by way of line search,Indicate activation
The Fractional Derivative of function.
The convolutional neural networks fractional order error back propagation method it is further design be, step 1-3) in swash
Function σ expression formulas such as formula (4) living：
Wherein, x is network node output valve.
Beneficial effects of the present invention are：
The convolutional neural networks fractional order error back propagation method of the present invention will activate letter during error back propagation
Several integer order derivatives become Fractional Derivative, and it is more accurately more excellent than integer rank derivation model to have played fractional order derivation model
Gesture has been effectively relieved " gradient disappearance " problem during error back propagation, has improved the training effectiveness of convolutional neural networks,
So that convolutional neural networks is realized preferably fitting for training data, improves the generalization ability of network.
Description of the drawings
Fig. 1 is convolutional neural networks fractional order error back propagation method flow diagram provided by the invention.
Fig. 2 is that fractional order error backward propagation method model provided by the invention (every layer only there are one neurons) shows
It is intended to.
Specific implementation mode
Below with reference to attached drawing, technical scheme of the present invention is described in detail.
As shown in Figure 1, the basic step of convolutional neural networks fractional order error back propagation method is as follows：
Step 1) selects convolutional neural networks model.
Step 2) updates network weight using fractional order error back propagation method.
Step 3) obtains convolutional neural networks parameter according to the network weight.
Preferably, step 1) the convolutional neural networks model of the present embodiment is as follows：
Step 1-1) four layers of neural network model of selection, including 1 layer of input layer, n-layer hidden layer, 1 layer of output layer be described defeated
It is respectively n to enter layer, hidden layer, output layer node number1, n2, 1.
Step 1-2) a large amount of high-definition picture { X of selectioniAnd its corresponding low-resolution image { YiComposition training
Collection.
Step 1-3) convolutional neural networks weights and biasing be denoted as Θ={ W1,W2,...,Wn+1,B1,B2,...,Bn+1,
Error loses function and activation primitive is denoted as L and σ respectively.
It is further preferred that error function L expression formulas such as formula (1) in the step 1) of the present embodiment：
N is the quantity of training sample in formula, and F is high-low resolution image mapping function end to end,
F(Yi, Θ) and indicate that neural network forecast value, X indicate desired value.
It is further preferred that the activation primitive σ expression formulas in the step 1) of the present embodiment are：
Wherein, x is network node output valve.
Preferably, the step 2) of the present embodiment includes following sub-step：
Step 2-1) netinit.Network weight and biasing are initialized, it is 0 that wherein weight initialization, which is mean value, standard
The Gaussian Profile that difference is 0.001, biasing are initialized as 0；Maximum iteration K is set；
Step 2-2) input training data { Xi,Yi}；
Step 2-3) setting primary iteration number k=1；
Step 2-4) network weight is optimized；
Preferably, step 2-4) network weight is optimized includes the following steps：
Step 2-4-1) calculate network output valve；
Step 2-4-2) calculate error.Using the error function, the error between network output valve and actual value is calculated,
Whether error in judgement meets the requirements.If met the requirements, network training terminates to be transferred to step 3；If being unsatisfactory for requiring, and repeatedly
Generation number not up to sets maximum times K, then carries out modified weight, enter step 2-4-3；
Step 2-4-3) calculate error back propagation gradient in activation primitive Fractional Derivative.To each neuron node
Using the Fractional Derivative of identical order；
Error back propagation gradient calculation expression is：
Wherein, GradlIndicate l layers of error back propagation gradient,For the input of activation primitive, i.e.,
The output of network node,Indicate the of (l-1) layerA neuron is connected toThe weight of j-th of neuron of layer,Indicate that the biasing of l layers of j-th of neuron, μ indicate that the order of Fractional Derivative, ⊙ are Hadamard product (matrixes pair
Answer position product), point-to-point multiplying between representing matrix or vector；
Fractional Derivative calculation formula is：
Wherein,For fractional order differential operator, D is 1 rank differential operator, and x indicates the output valve of neural network node, σ
(x) indicate that activation primitive, σ ' (x) indicate that 1 order derivative of activation primitive, μ indicate that the order of fractional order differential, Γ (x) are Gamma
Function is defined asAnd Re (x) > 0.
Step 2-4-4) calculate error gradient.
Step 2-4-5) correct weights.Gradient according to error about weight, using classical stochastic gradient descent method amendment
Network weight.Right value update mode is：
Wherein, i indicates that the network number of plies, η indicate learning rate, obtained by way of line search.
Step 2-5) k=k+1, go to step 2-4).
The foregoing is only a preferred embodiment of the present invention, but scope of protection of the present invention is not limited thereto,
Any one skilled in the art in the technical scope disclosed by the present invention, the change or replacement that can be readily occurred in,
It should be covered by the protection scope of the present invention.Therefore, protection scope of the present invention should be with scope of the claims
Subject to.
Claims (7)
1. a kind of convolutional neural networks fractional order error back propagation method, it is characterised in that include the following steps：
Step 1) selects convolutional neural networks model；
Step 2) updates network weight using fractional order error back propagation method；
Step 3) obtains convolutional neural networks parameter according to the network weight.
2. convolutional neural networks fractional order error back propagation method according to claim 1, it is characterised in that the step
It is rapid 1) to specifically include：
Step 1-1) neural network model is set, the neural network model includes one layer of input layer, n-layer hidden layer and one layer
Output layer, the input layer, hidden layer, output layer node number correspond to n respectively1, n2, 1；
Step 1-2) a large amount of high-definition picture { X of settingiAnd its corresponding low-resolution image { YiComposition training set,
In：X indicates that high-definition picture, Y indicate that low-resolution image, i indicate image index；
Step 1-3) it sets the weights of convolutional neural networks and is biased to Θ={ W, B }, set error function and activation primitive point
Not Wei L and σ, wherein W indicate weights, B be biasing.
3. convolutional neural networks fractional order error back propagation method according to claim 1, it is characterised in that the step
Rapid 2 include following sub-step：
Step 2-1) netinit：Initialize network weight and biasing Θ={ W1,B1, maximum iteration is set；
Step 2-2) input training data { Xi,Yi}；
Step 2-3) setting primary iteration number k=1；
Step 2-4) network weight is optimized；
Step 2-5) according to k=k+1 update iterations, and jump to step 2-4).
4. convolutional neural networks fractional order error back propagation method according to claim 3, which is characterized in that
Step 2-4) it network weight is optimized specifically includes following steps：
Step 2-4-1) calculate network output valve；
Step 2-4-2) calculate error：Using the error function L, the error between network output valve and actual value is calculated, is sentenced
Whether disconnected error meets the requirements, if met the requirements, network training terminates to be transferred to step 3)；
If being unsatisfactory for requiring, modified weight is carried out, enters step 2-4-3)；
Step 2-4-3) Fractional Derivative of activation primitive in error back propagation gradient is calculated, and calculate each neuron node
Fractional Derivative；
Step 2-4-4) calculate error gradient；
Step 2-4-5) correct weights：Gradient according to error about weight, corrective networks weights.
5. convolutional neural networks fractional order error back propagation method according to claim 4, which is characterized in that the step
Rapid 2-4-3) according to formula (1) calculate error back propagation gradient in activation primitive Fractional Derivative：
Wherein, GradlIndicate l layers of error back propagation gradient,For the input of activation primitive, i.e. network section
The output of point,Indicate the of (l-1) layerA neuron is connected toThe weight of j-th of neuron of layer,It indicates
The biasing of l layers of j-th of neuron, μ indicate that the order of Fractional Derivative, ⊙ are matrix corresponding position product, representing matrix
Or point-to-point multiplying between vector；
The Fractional Derivative of each neuron node is calculated according to formula (2)：
Wherein,For fractional order differential operator, D is 1 rank differential operator, and x indicates the output valve of neural network node, σ (x) tables
Show that activation primitive, σ ' (x) indicate that 1 order derivative of activation primitive, μ indicate that the order of fractional order differential, Γ (x) are Gamma functions,
It is defined asAnd Re (x) > 0.
6. convolutional neural networks fractional order error back propagation method according to claim 4, which is characterized in that step 2-
Weights 4-5) are corrected according to formula (3) using classical stochastic gradient descent method,
Wherein, i indicates that the network number of plies, η indicate learning rate, obtained by way of line search,Indicate activation primitive
Fractional Derivative.
7. according to claim 2 be based on convolutional neural networks fractional order error back propagation method, which is characterized in that step
Rapid 1-3) in activation primitive σ expression formulas such as formula (4)：
Wherein, x is network node output valve.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810490436.1A CN108764472A (en) | 2018-05-18 | 2018-05-18 | Convolutional neural networks fractional order error back propagation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810490436.1A CN108764472A (en) | 2018-05-18 | 2018-05-18 | Convolutional neural networks fractional order error back propagation method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108764472A true CN108764472A (en) | 2018-11-06 |
Family
ID=64007660
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810490436.1A Pending CN108764472A (en) | 2018-05-18 | 2018-05-18 | Convolutional neural networks fractional order error back propagation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108764472A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109766993A (en) * | 2018-12-13 | 2019-05-17 | 浙江大学 | A kind of convolutional neural networks compression method of suitable hardware |
| CN111353534A (en) * | 2020-02-27 | 2020-06-30 | 电子科技大学 | Graph data category prediction method based on adaptive fractional order gradient |
| CN111382837A (en) * | 2020-02-05 | 2020-07-07 | 鹏城实验室 | Countermeasure sample generation method based on depth product quantization |
| CN111770437A (en) * | 2020-06-09 | 2020-10-13 | Oppo广东移动通信有限公司 | Method and device for determining data card, terminal and storage medium |
| CN112200139A (en) * | 2020-10-30 | 2021-01-08 | 杭州泰一指尚科技有限公司 | User image identification method based on variable-order-fraction multilayer convolutional neural network |
| CN112597519A (en) * | 2020-12-28 | 2021-04-02 | 杭州电子科技大学 | Non-key decryption method based on convolutional neural network in OFDM (orthogonal frequency division multiplexing) encryption system |
| CN112766414A (en) * | 2021-02-07 | 2021-05-07 | 辽宁大学 | Image identification method based on Hausdorff-RMSprop algorithm |
| CN113392696A (en) * | 2021-04-06 | 2021-09-14 | 四川大学 | Intelligent court monitoring face recognition system and method based on fractional calculus |
| CN115878968A (en) * | 2023-02-27 | 2023-03-31 | 山东交通学院 | Signal noise reduction method based on extreme value characteristic neural network |
| CN111461229B (en) * | 2020-04-01 | 2023-10-31 | 北京工业大学 | Deep neural network optimization and image classification method based on target transfer and line search |
-
2018
- 2018-05-18 CN CN201810490436.1A patent/CN108764472A/en active Pending
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109766993A (en) * | 2018-12-13 | 2019-05-17 | 浙江大学 | A kind of convolutional neural networks compression method of suitable hardware |
| CN111382837B (en) * | 2020-02-05 | 2023-07-18 | 鹏城实验室 | Antagonistic sample generation method based on depth product quantization |
| CN111382837A (en) * | 2020-02-05 | 2020-07-07 | 鹏城实验室 | Countermeasure sample generation method based on depth product quantization |
| CN111353534A (en) * | 2020-02-27 | 2020-06-30 | 电子科技大学 | Graph data category prediction method based on adaptive fractional order gradient |
| CN111461229B (en) * | 2020-04-01 | 2023-10-31 | 北京工业大学 | Deep neural network optimization and image classification method based on target transfer and line search |
| CN111770437A (en) * | 2020-06-09 | 2020-10-13 | Oppo广东移动通信有限公司 | Method and device for determining data card, terminal and storage medium |
| CN112200139A (en) * | 2020-10-30 | 2021-01-08 | 杭州泰一指尚科技有限公司 | User image identification method based on variable-order-fraction multilayer convolutional neural network |
| CN112200139B (en) * | 2020-10-30 | 2022-05-03 | 杭州泰一指尚科技有限公司 | User image identification method based on variable-order fractional multilayer convolutional neural network |
| CN112597519A (en) * | 2020-12-28 | 2021-04-02 | 杭州电子科技大学 | Non-key decryption method based on convolutional neural network in OFDM (orthogonal frequency division multiplexing) encryption system |
| CN112597519B (en) * | 2020-12-28 | 2024-02-13 | 杭州电子科技大学 | Non-key decryption method based on convolutional neural network in OFDM encryption system |
| CN112766414A (en) * | 2021-02-07 | 2021-05-07 | 辽宁大学 | Image identification method based on Hausdorff-RMSprop algorithm |
| CN113392696A (en) * | 2021-04-06 | 2021-09-14 | 四川大学 | Intelligent court monitoring face recognition system and method based on fractional calculus |
| CN115878968A (en) * | 2023-02-27 | 2023-03-31 | 山东交通学院 | Signal noise reduction method based on extreme value characteristic neural network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108764472A (en) | Convolutional neural networks fractional order error back propagation method | |
| Grathwohl et al. | Scalable reversible generative models with free-form continuous dynamics | |
| CN108171323B (en) | Artificial neural network computing device and method | |
| CN106815782A (en) | A kind of real estate estimation method and system based on neutral net statistical models | |
| CN109635936A (en) | A kind of neural networks pruning quantization method based on retraining | |
| CN107092960A (en) | A kind of improved parallel channel convolutional neural networks training method | |
| CN106529818B (en) | Water quality assessment Forecasting Methodology based on Fuzzy Wavelet Network | |
| CN110515303A (en) | A kind of adaptive dynamic path planning method based on DDQN | |
| CN106960243A (en) | A kind of method for improving convolutional neural networks structure | |
| CN108537335A (en) | A kind of BP neural network algorithm of autoadapted learning rate | |
| CN107275801A (en) | A kind of array element arrangement method based on the inheritance of acquired characters of L-type array antenna | |
| CN109948717B (en) | Self-growth training method for generating countermeasure network | |
| CN110414565A (en) | A kind of neural network method of cutting out based on Group Lasso for power amplifier | |
| CN109975250B (en) | Leaf area index inversion method and device | |
| CN106022471A (en) | Wavelet neural network model ship rolling real-time prediction method based on particle swarm optimization algorithm | |
| CN110211684A (en) | The electrocardiogram classification method of BP neural network based on genetic algorithm optimization | |
| CN109472105A (en) | Semiconductor product yield Upper bound analysis method | |
| CN109064493A (en) | A kind of method for tracking target and device based on meta learning | |
| CN114372438A (en) | Chip macro-unit layout method and system based on lightweight deep reinforcement learning | |
| CN108594793A (en) | A kind of improved RBF flight control systems fault diagnosis network training method | |
| CN110414115A (en) | A kind of wavelet neural network tomato yield prediction technique based on genetic algorithm | |
| Burse et al. | Improved back propagation algorithm to avoid local minima in multiplicative neuron model | |
| CN106296434A (en) | A kind of Grain Crop Yield Prediction method based on PSO LSSVM algorithm | |
| CN110245742A (en) | A kind of adaptive deep learning model optimization method based on Keras platform | |
| Wang et al. | Global dynamics of a cholera model with time delay |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181106 |