CN109886405A - It is a kind of inhibit noise based on artificial neural network structure's optimization method - Google Patents
It is a kind of inhibit noise based on artificial neural network structure's optimization method Download PDFInfo
- Publication number
- CN109886405A CN109886405A CN201910127126.8A CN201910127126A CN109886405A CN 109886405 A CN109886405 A CN 109886405A CN 201910127126 A CN201910127126 A CN 201910127126A CN 109886405 A CN109886405 A CN 109886405A
- Authority
- CN
- China
- Prior art keywords
- opt
- vector
- beta pruning
- optimal
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Best initial weights vector w is obtained including the optimal neuron number searching method based on growth based on artificial neural network structure's optimization method the invention discloses a kind of inhibition noiseopt, optimal activated matrix QoptAnd output net training time t1And the network structure based on rear beta pruning simplifies method and finds optimum structure.The present invention is based on Gegenbauer orthogonal polynomial neural networks, network optimum structure is found using growth-Pruning Optimization Algorithm, facts have proved the neural network approach, test, in terms of denoising have very high superiority and validity, thus have very high practical value.
Description
Technical field
The present invention relates to artificial neural networks, and in particular to a kind of to inhibit optimizing based on artificial neural network structure for noise
Method.
Background technique
Artificial neural network (ANN) be a kind of simulation human brain neural network working mechanism, by a large amount of artificial neuron by
It is formed by connecting according to certain rule, with the non-linear of height, is able to carry out complicated logical operation and non-linear relation and realizes
System, be a branch of artificial intelligence, have the characteristics that good concurrency, fault tolerance and adaptive learning.
For different neural networks, the excitation function of neuron, the topological structure of network and learning algorithm are all different.
There are artificial neural network good data to be fitted and generalization ability, can be the case where not knowing goal systems definite information
Under, the computation model comprising system features is established by training.In recent years, artificial neural network constantly provides new neural network
Model and method are simulated and are solved the problems, such as complicated in actual life and be difficult to define.It is superior due to neural network performance
Property, it is widely used in the fields such as pattern-recognition, artificial intelligence, signal processing, computer science, control engineering.
BP neural network based on backpropagation is the artificial neural network that current research is the most mature, application range is most wide
One of model, structure is simple, is widely used in the approximation problem for solving non-linear objective function, is artificial neural network
One of supervised learning algorithm.Energy can be approached to reach raising network by adjusting the connection weight between neuron
The purpose of power.But there is also the disadvantages of some inherences for BP neural network, such as: convergence rate is relatively slow, is easily trapped into local minimum
The problems such as value, not high study precision.In addition, neural network does not have one in the selection and optimization problem of hidden layer neuron number
A deterministic theoretical direction, it is difficult to obtain the optimum topology structure of neural network.
For defect existing for BP neural network, there has been proposed many modified hydrothermal process and method, but the above problem
It still has.These improved methods can be roughly divided into two types: one is modifying on the basis of standard BP algorithm, separately
One is by optimizing numerical value come innovatory algorithm.But these algorithms all cannot fundamentally solve lacking for BP network training method
It falls into.
Summary of the invention
In order to overcome the shortcomings of the prior art, the present invention provide it is a kind of inhibit noise based on artificial neural network knot
Structure optimization method.
This optimization method includes two parts, and a part is the optimal neuron number searching method based on growth, should
Method can calculate hidden layer neuron to the direct weight determination of output layer connection weight, search with a step
The optimum structure of Gegenbauer neural network, so that network obtains best approximation capability；Another part is cut after being based on
The network structure of branch simplifies method, and this method can delete hidden layer neuron extra in network, simplifies network structure, thus
Obtain better network performance.
The present invention adopts the following technical scheme:
It is a kind of inhibit noise based on artificial neural network structure's optimization method, comprising:
Optimal neuron number searching method based on growth, obtains best initial weights vector wopt, optimal activated matrix Qopt
And output net training time t1；
Network structure based on rear beta pruning simplifies method.
Preferably, the optimal neuron number searching method based on growth, the specific steps are as follows:
S1.1 constructs one two and inputs the Gegenbauer neural network singly exported, by input layer, hidden layer and output layer
It constitutes, connection weight is 1 between input layer and hidden layer, and wherein the excitation function of input layer and output layer is all made of linear identical
Function, the excitation function of hidden layer neuron are basic function composed by the product of two Gegenbauer orthogonal polynomials, k-th
Hidden layer neuron excitation function is qk, k=0,1,2 ..., K；
S1.2 initialization network parameter searches for hidden neuron number, and optimal hidden layer neuron is denoted as M+1, and M initial value is
0, enable training least mean-square error initial valueCounter c, initial value 0, maximum value c are setmax；
S1.3 judges whether k≤(d+1) (d+2)/2 be true, skips to step S1.5 if setting up, otherwise skips to step
S1.4, d are the sum of Gegenbauer order of a polynomial time of binary input variable, and d=i+j, i, j respectively indicate orthogonal polynomial
Order；
S1.4 judges c≤cmaxWhether true, d=d+1 skips to step S1.5 if setting up, and otherwise skips to step S1.8；
S1.5 sets the excitation function of the M+1 hidden neuron as qk, corresponding activated matrix Q is generated, and calculate hidden layer
Weight vector w, and calculate mean square error at this time
S1.6 Rule of judgment mean square errorWhether meet, if satisfied, then retaining the M+1 hidden neuron, enables meter
Number device c zero, M=M+1,The currently active matrix Q is saved as current optimal activated matrix Qopt, current weight to
Amount w saves as current best initial weights vector wopt；
If conditions are not met, the M+1 neuron corresponding element in weight vector w is deleted, the M+1 mind in activated matrix Q
Through the corresponding vector of member, and enable c=c+1；K=k+1 is enabled, S1.3 is returned；
S1.7 completes search, exports training least mean-square errorOptimal hidden layer neuron number M+1 and corresponding
Best initial weights vector wopt, optimal activated matrix Qopt, and export net training time t1。
Preferably, the Topological expansion method based on rear beta pruning, includes the following steps:
Beta pruning parameter after S2.1 initialization: beta pruning training error after orderEqual to training least mean-square errorBeta pruning
Weight vector wdelEqual to best initial weights vector wopt, beta pruning activated matrix QdelEqual to optimal activated matrix Qopt, i.e.,
wdel=wopt, Qdel=Qopt；
An initial pruning threshold g, specially best initial weights weight vector w is arranged in S2.2optThe smallest first number of middle absolute value
Twice of absolute value；
S2.3 is current beta pruning training errorSave as optimal mean square errorCurrent beta pruning weight vector wdel
Save as best initial weights vector wopt, current beta pruning activated matrix QdelSave as optimal activated matrix Qopt, i.e.,
wopt=wdel, Qopt=Qdel；
S2.4 deletes current beta pruning weight vector wdelMiddle weight is less than the element of pruning threshold g, and its activates square in beta pruning
Battle array QdelCorresponding vector calculates network beta pruning training error at this time
S2.5 obtains network beta pruning training error by S2.3Rule of judgmentIt is whether true, if setting up
Five times, i.e. g=5g that pruning threshold g is original are then updated, and skips to step S2.3；Otherwise, step S2.6 is skipped to；
S2.6 exits beta pruning process, the optimal mean square error of output trainingFinal hidden layer neuron number Mopt, and
Corresponding best initial weights vector wopt, optimal activated matrix Qopt, take best initial weights vector woptLength be final hidden layer nerve
First number Mopt, i.e. Mopt=length (wopt), and export the total training time T of network.Preferably, basic function is pressed in the S1.1
It sorts according to ascending order dictionary method.
Preferably, the S1.5 calculates mean square error at this timeDetailed process are as follows:
S1.5.1 directly determines method using the weight of principle of least square puppet inverse form, to directly be calculated hiding
Connection weight between layer neuron and output layer, the calculation formula of the connection weight by directly giving as follows:
Wherein, subscriptTIt is expressed as the transposition of matrix-vector, (QTQ)-1QTFor the pseudoinverse for inputting excited target matrix Q, it is denoted as Q+；
Wherein, weight column vector w, input excited target matrix Q and target export column vectorIt respectively indicates are as follows:
W=[w0, w1..., wm ..., wM]T∈RM
In above formula,Indicate correspond to s-th of training sample the m+1 hidden neuron excited target response s=1,2,
3 ..., S, M+1 are hidden layer neuron sum, and S is sample total, when input variable u sampling number is su, input variable u adopts
Number of samples is svWhen, S=su×sv,For training sample pair；
The mean square error of S1.5.2 calculating networkIt is specifically defined are as follows:
Wherein,For square of two norm of vector or matrix.
Beneficial effects of the present invention:
The present invention by determining the optimum structure of network based on the optimal neuron number searching method of growth, secondly,
Simplify method using the network structure based on rear beta pruning and delete extra neuron, simplifies the structure of neural network, Numerical Validation
Experiment shows that trained network has and good approach, predicts and noise removal capability.
Detailed description of the invention
Fig. 1 is work flow diagram of the invention；
Fig. 2 is Gegenbauer orthogonal polynomial neural network model figure of the invention；
Fig. 3 (a) is the hidden layer neuron using the network of the optimal neuron number searching method training based on growth
Weight distribution map；
Fig. 3 (b) is the hidden layer neuron for simplifying the simplified network of method using the network structure based on rear beta pruning
Weight distribution map；
Fig. 4 (a) is the surface chart for the binary objective function A to be approached；
Fig. 4 (b) is using a kind of inhibition noise based on growth-beta pruning Gegenbauer neural network structure optimization side
The network of method training approaches output surface chart；
Fig. 4 (c) is using a kind of inhibition noise based on growth-beta pruning Gegenbauer neural network structure optimization side
The network of method training approaches the relative error surface chart obtained after output and objective function A comparison；
Fig. 4 (d) is the surface chart for adding the binary objective function A after making an uproar；
Fig. 4 (e) is using a kind of inhibition noise based on growth-beta pruning Gegenbauer neural network structure optimization side
The network of method training denoises effect picture to the network for adding the objective function A after making an uproar；
Fig. 4 (f) is using a kind of inhibition noise based on growth-beta pruning Gegenbauer neural network structure optimization side
What the network of method training obtained after comparing to the network denoising output and non-plus noise objective function A that add the objective function A after making an uproar
Relative error surface chart；
Fig. 4 (g) is using a kind of inhibition noise based on growth-beta pruning Gegenbauer neural network structure optimization side
The network test of method training exports surface chart；
Fig. 4 (h) is using a kind of inhibition noise based on growth-beta pruning Gegenbauer neural network structure optimization side
The relative error surface chart obtained after the network test output of method training and objective function A comparison；
Fig. 5 is the learning curve figure that network of the invention approaches objective function A.
Specific embodiment
Below with reference to examples and drawings, the present invention is described in further detail, but embodiments of the present invention are not
It is limited to this.
Embodiment
As shown in Figure 1, it is a kind of inhibit noise based on artificial neural network structure's optimization method, include the following steps:
S1 constructs one two and inputs the Gegenbauer neural network that singly exports, and input layer, output layer neuron swash
Function living uses linear identity function, and the connection weight between input layer and hidden layer is fixed as 1, enables hidden layer neuron
Excitation function is basic function, i.e. q composed by the product of two Gegenbauer orthogonal polynomialsk(u, v)=Gi(u)Gj(v), k=
0,1,2 ... K；I, j=0,1,2 ...；Wherein, basic function qkSortord according to ascending order dictionary method, and binary input variable
The sum of Gegenbauer order of a polynomial time be denoted as d (d=i+j), initial value is set as 0；
S2, initialization network parameter search for hidden neuron number since 1, and optimal hidden neuron number scale is M+1, initial
Value is set as 0, and enables the trained least mean-square error initial value beAnd a counter c is set, initial value 0 is maximum
Value is cmax；
Whether S3, Rule of judgment k≤(d+1) (d+2)/2 are true, skip to step S5 if setting up, otherwise skip to step S4；
S4, Rule of judgment c≤cmaxWhether true, d=d+1 skips to step S5 if setting up, and otherwise skips to step S8；
K-th S5, calculating of basic function generate corresponding activated matrix Q at this time as the M+1 hidden neuron, and by weighing
One step of value Weigh Direct Determination calculates the weight vector w of network concealed layer (neuron number M+1) at this time, and calculates at this time
Mean square error
S6, Rule of judgmentWhether meet, if satisfied, retaining the M+1 hidden neuron, counter c enabled to be zeroed,
Updating preservation parameter makes M=M+1,And the currently active matrix Q is saved as current optimal activated matrix Qopt, working as
Preceding weight vector w saves as current best initial weights vector wopt；Otherwise, the M+1 neuron corresponding element in w is deleted, square is activated
The M+1 neuron corresponds to vector in battle array Q, and enables c=c+1；
S7, k=k+1, return step S3 are enabled；
S8, search is exited, exports training least mean-square errorOptimal hidden layer neuron number M+1 and corresponding
Best initial weights vector wopt, optimal activated matrix Qopt, and export net training time t1。
Beta pruning parameter after S9, initialization, beta pruning training error after orderIt is minimum equal equal to the training that first part obtains
Square errorBeta pruning weight vector wdelEqual to the w obtained before thisopt, beta pruning activated matrix QdelEqual to the Q obtained before thisopt,
I.e.wdel=wopt, Qdel=Qopt；
One S10, setting pruning threshold g, and it is set as best initial weights vector woptThe absolute value of the smallest first number of middle absolute value
Twice, i.e. g=2min (| wopt|)；
S11, current beta pruning training errorSave as optimal mean square errorCurrent beta pruning weight vector wdel
Save as best initial weights vector wopt, current beta pruning activated matrix QdelSave as optimal activated matrix Qopt, i.e.,
wopt=wdel, Qopt=Qdel；
S12, current beta pruning weight vector w is deleteddelMiddle weight is less than the element of pruning threshold g, and its activates square in beta pruning
Battle array QdelCorresponding vector；
S13, the network beta pruning training error of calculating at this time
S14, the network beta pruning training error obtained by S5Rule of judgmentIt is whether true, if setting up
Five times, i.e. g=5g that pruning threshold g is original are then updated, and skips to step S11；Otherwise, step S15 is skipped to；
S15, beta pruning process, the optimal mean square error of output training are exitedFinal hidden layer neuron number Mopt, and
Corresponding best initial weights vector wopt, optimal activated matrix Qopt, take best initial weights vector woptLength be final hidden layer nerve
First number Mopt, i.e. Mopt=length (wopt) and export the total training time T of network.
Fig. 2 show the present invention is based on Gegenbauer orthogonal polynomial neural network model figure, the neural network by
Input layer, hidden layer, output layer up of three-layer, wherein the excitation function of input layer and output layer is all made of linear identical excitation letter
Number, hidden layer neuron select the Gegenbauer orthogonal polynomial that gradually increases of one group of order as its excitation function, and k-th
Hidden neuron excitation function is qk(k=0,1,2 ..., K)；The neuron number of input layer is 2, the neuron number of output layer
It is 1, hidden layer neuron sum is M+1, and the weight between input layer and hidden layer is set as 1, and the threshold value of all neurons is set
It is set to 0.
Fig. 3 (a) is the hidden layer mind only with the network of the optimal neuron number searching method training based on growth
Through first weight distribution map, Fig. 3 (b) is the hidden layer for simplifying the simplified network of method using the network structure based on rear beta pruning
Neuron weight distribution map.It can be seen that many minimum neurons of weight are deleted after beta pruning, the number of hidden layer neuron
Greatly reduce.
Fig. 4 (a)-Fig. 4 (c) is shown using of the present invention based on growth-beta pruning Gegenbauer neural network
Network after structural optimization method is to the Approximation effect figure of binary objective function A, and wherein Fig. 4 (a) is the binary target to be approached
The surface chart of function A, Fig. 4 (b) are using a kind of inhibition noise based on growth-beta pruning Gegenbauer neural network structure
The network of optimization method training approaches output surface chart, and Fig. 4 (c) is using a kind of inhibition noise based on growth-beta pruning
The network of Gegenbauer Architecture Optimization for Neural Networks training approach after output and objective function A comparison obtain it is opposite accidentally
Poor surface chart.Comparison diagram 4 (a) and Fig. 4 (b) can see, and the curved surface of network output curved surface and realistic objective function A is kissed substantially
It closes；Fig. 4 (c) is it can be seen that network relative approximation margin of error magnitude reaches 10-11.In conjunction with emulating image and above-mentioned analysis it is found that
The Gegenbauer orthogonal polynomial neural network that the present invention is constructed can accurately approach binary objective function A.
Fig. 4 (d)-Fig. 4 (f) is shown using of the present invention based on growth-beta pruning Gegenbauer neural network
Network after structural optimization method is to the denoising effect picture for adding the binary objective function A after making an uproar, and wherein Fig. 4 (d) is to binary mesh
Scalar functions A carries out adding the surface chart after making an uproar, and Fig. 4 (e) is using a kind of inhibition noise based on growth-beta pruning Gegenbauer
The network of Architecture Optimization for Neural Networks training denoises effect picture to the network for adding the objective function A after making an uproar, and Fig. 4 (f) is to adopt
It is made an uproar with a kind of network based on the training of growth-beta pruning Gegenbauer Architecture Optimization for Neural Networks for inhibiting noise to adding
The relative error surface chart obtained after the network denoising output of objective function A afterwards and non-plus noise objective function A comparison.Comparison
Fig. 4 (d) and Fig. 4 (e) is it is found that network denoising output is almost the same with the curved surface without noise targets function A；From Fig. 4 (f)
Network output after denoising is very small with the relative approximation error of the network without noise, and the order of magnitude reaches 10-3.In conjunction with analogous diagram
As and above-mentioned analysis it is found that the present invention constructed based on growth-beta pruning Gegenbauer neural network, even if in binary
In the case where random noise of the objective function A with higher magnitude, network denoising output still is able to accurately approach binary mesh
Scalar functions A, denoising work well.
Fig. 4 (g) is using a kind of inhibition noise based on growth-beta pruning Gegenbauer neural network structure optimization side
The network test of method training exports surface chart, and Fig. 4 (h) is using a kind of inhibition noise based on growth-beta pruning
It is obtained after the network test output of Gegenbauer Architecture Optimization for Neural Networks training and objective function A comparison opposite
The curved surface of network test output surface chart and realistic objective function A is basic known to error surface figure, comparison diagram 4 (g) and Fig. 4 (a)
Unanimously, network test output and the relative error of former objective function are very small known to Fig. 4 (h), and the order of magnitude reaches 10-8.In conjunction with
Emulating image and above-mentioned analysis are it is found that a kind of based on growth-beta pruning Gegenbauer neural network structure using the present invention
The Gegenbauer orthogonal polynomial neural network of optimization method training can still obtain good Approximation effect in non-learning region,
Thus there is very strong extensive approximation capability.
Fig. 5 is binary objective function A based on the learning curve in growth-beta pruning Gegenbauer neural network.It is bent
Line upward arrow marked the optimal hidden neuron number M determined by the optimal neuron number searching method based on growth and by
Simplify the optimal hidden neuron number M after method is deleted based on rear beta pruning network structureopt, by learning curve it can be seen that rear
During beta pruning Topological expansion, a large amount of neuron is deleted in network, but the order of magnitude of training approximate error is kept not
Become.In addition, the testing time of network greatly shortens, but measuring accuracy errorBut it is not decreased obviously.
According to the correlation step of design flow diagram, detailed parsing now is carried out to algorithm involved in the present invention.Firstly, right
The Gegenbauer orthogonal polynomial neural network that two inputs constructed in the present invention singly export is described in detail.
First provide the theoretical basis and network struction of two input Gegenbauer orthogonal polynomial neural network approximation capabilities
Reasonability.By probability theory it is found that for two continuous independent independent variable u and v, broad sense joint probability density function (PDFs)
It can be written as:
F (u, v)=fu(u)fv(v)
Wherein, fu(u) and fv(v) the broad sense marginal probability density function on u and v is respectively indicated.
The polynomial general formula of Gegenbauer are as follows:
Wherein, (θ)k=θ (θ+1) ... (θ+k+1) (θ represents 2 λ, 2 λ+n, λ+1/2), λ=2/3.
By polynomial interopolation and approximation theory it is found that having the unknown object function of a variableIt can be by more than one
Item formulaIt approaches, further proves multinomial below according to least square approximation principleApproximation capability:
Indicate the successive objective function on closed interval [a, b], each polynomial sequenceConnect on [a, b]
It is continuous, definitionFor one group of linear independence (or orthogonal) polynomial sequence on closed interval [a, b], polynomial functionIt can indicate as follows:
According to differential polynomial and approximation theory, there is always one group of best initial weights [w0, w1..., wn-1] can be minimized closely
It is minimized like error, that is, by the value of following equation:
It is considered as unknown object functionLeast square approximation function.
Based on discussed above, we enter on approaching for binary function here:
The objective function of two independent variables u and v are had for oneSimultaneous approximation function can be passed throughTo approach objective functionBased on above-mentioned probability theory, objective functionIt can define
Are as follows:
According to least square fitting theorem and Gegenbauer multinomial general formula it is found that least square fitting functionCan be by the Gegenbauer orthonormal polynomial approximation of best initial weights, therefore objective function can be further
It rewrites are as follows:
Wherein, wk=wij=αiβj(i=0,1,2 ..., N1- 1, j=0,1,2 ..., N2- 1) corresponding basic function { G is indicatedi
(u)Gj(v) } optimal connection weight.It follows that one group of Gegenbauer orthogonal basis function with best initial weights can be most
Approach objective function to small errorIn this way, using the product of two Gegenbauer orthogonal polynomials, successfully
To two variable objective functionsLeast square approximation function.Therefore,It can be further rewritten as:
Wherein, wk=wijFor k-th of basic function qk(u, v)=Gi(u)Gj(v) weight, k=((i+j) (i+j+1))/2+
I+1, K=N1×N2=(d+1) (d+2)/2 is indicated for approaching objective functionBasic function sum.It may be noted that
To be basic function sort according to ascending order dictionary method, if d is Gi(u) and Gj(v) the sum of order.
So far, the theoretical basis and network struction of the two inputs Gegenbauer orthogonal polynomial neural network approximation capability
Reasonability proof terminates.
According to above-mentioned Gegenbauer orthogonal polynomial neural network, provides the weight based on the neural network and directly determine
The theoretical basis of method.Under normal circumstances, the connection weight between hidden layer neuron and output layer can be learnt to calculate by traditional BP
Method obtains.In order to avoid BP algorithm fit time is long, easily fall into local minimum point, is difficult to the disadvantages of obtaining optimum structure, this hair
Gegenbauer neural network in bright directly determines method using weight, thus be directly calculated hidden layer neuron with it is defeated
Connection weight between layer out.The calculation formula that weight directly determines directly gives as follows:
Wherein, subscriptTIt is expressed as the transposition of matrix-vector, (QTQ)-1QTFor the pseudoinverse for inputting excited target matrix Q, it is denoted as Q+。
In above-mentioned formula, weight column vector w, input excited target matrix Q and target export column vectorIt respectively indicates are as follows:
W=[w1, w2..., wm..., wM]T∈RM
In above formula,Indicate correspond to s-th of training sample m-th of hidden neuron excited target response (s=1,2,
3 ..., S), M+1 is hidden layer neuron sum, and S is sample total, when input variable u sampling number is su, input variable u adopts
Number of samples is svWhen, S=su×sv,For training sample pair.
In order to improve network performance, the present invention propose it is a kind of inhibit noise based on growth-beta pruning Gegenbauer mind
Through Topological expansion method.It is carried out specifically below for growth in flow chart-beta pruning optimization part specific algorithm process
It is bright.
Firstly, the single Gegenbauer neural network for exporting single hidden layer of one two input of building, enables hidden layer neuron
Excitation function be two Gegenbauer orthogonal polynomials product composed by basic function, i.e. qk(u, v)=Gi(u)Gj(v), k
=0,1,2 ... K；I, j=0,1,2 ...；Wherein, basic function qkIt sorts according to ascending order dictionary method.
Optimal hidden neuron number is set as M+1, initial value is set as 0, searches for hidden neuron number since 1, enables training
Least mean-square error initial value isAnd a counter c, initial value 0, maximum value c are setmax。
Whether Rule of judgment k≤(d+1) (d+2)/2 be true, and it is hidden as M+1 that k-th of basic function is calculated if setting up
Neuron generates corresponding activated matrix Q at this time, and calculates network concealed layer (neuron at this time by direct weight determination
Number is M+1) weight vector w, and calculate mean square error at this timeOtherwise continue Rule of judgment c≤cmaxIt is whether true,
The d=d+1 if setting up calculates k-th of basic function as the M+1 hidden neuron, generates corresponding activated matrix Q at this time, and
The weight vector w of network concealed layer (neuron number M+1) at this time is calculated by direct weight determination, and is calculated at this time
Mean square errorOtherwise search is exited, training least mean-square error is exportedOptimal hidden layer neuron number M+1 and right
The best initial weights vector w answeredopt, optimal activated matrix Qopt, and export net training time t1。
Rule of judgmentWhether meet, if satisfied, retaining the M+1 hidden neuron, counter c is enabled to be zeroed, updated
Saving parameter makes M=M+1,And the currently active matrix Q is saved as current optimal activated matrix Qopt, current power
Value vector w saves as current best initial weights vector wopt；Otherwise, the M+1 neuron corresponding element in w, activated matrix Q are deleted
In the M+1 neuron correspond to vector, and enable c=c+1.K=k+1 is enabled again, is returned and is continued network training.
Beta pruning parameter after initialization, beta pruning training error after orderThe training lowest mean square obtained equal to first part misses
DifferenceBeta pruning weight vector wdelEqual to the w obtained before thisopt, beta pruning activated matrix QdelEqual to the Q obtained before thisopt, i.e.,wdel=wopt, Qdel=Qopt。
One pruning threshold g is set, and is set as best initial weights vector woptThe two of the absolute value of the smallest first number of middle absolute value
Times, i.e. g=2min (| wopt|).Current beta pruning training errorSave as optimal mean square errorCurrent beta pruning power
It is worth vector wdelSave as best initial weights vector wopt, current beta pruning activated matrix QdelSave as optimal activated matrix Qopt, i.e.,wopt=wdel, Qopt=Qdel.Delete current beta pruning weight vector wdelMiddle weight is less than the member of pruning threshold g
Element, and its in beta pruning activated matrix QdelCorresponding vector, and calculate network beta pruning training error at this timeRule of judgmentIt is whether true, five times, i.e. g=5g that pruning threshold g is original are updated if setting up, and return to continuation network
Training.Otherwise, best initial weights vector w is takenoptLength be final hidden layer neuron number Mopt, i.e. Mopt=length (wopt),
Exit beta pruning process, the optimal mean square error of output trainingFinal hidden layer neuron number MoptAnd corresponding optimal power
It is worth vector wopt, optimal activated matrix Qopt, and export the total training time T of network.
So far, the introduction of neural network structure optimization algorithm finishes, and the network structure after growth-beta pruning optimization is this
A kind of invention inhibition noise is obtained most based on growth-beta pruning Gegenbauer Architecture Optimization for Neural Networks
Excellent network structure.
Herein, in order to show it is proposed by the present invention it is a kind of inhibit noise based on growth-beta pruning Gegenbauer nerve
The superiority of Topological expansion method is chosen the binary objective function A progress computer that one needs to approach as experiment and is imitated
True experiment, and approximation capability and noise removal capability based on this verifying network:
Firstly, in u ∈ [- 0.9,0.9], v ∈ [- 0.9,0.9] uniformly adopts objective function A in emulation experiment
Sample forms sample, sampling interval 0.04.Then a kind of inhibition noise proposed through the invention based on growth-beta pruning
Gegenbauer Architecture Optimization for Neural Networks carries out network training, obtains network best weight value and optimum structure.Emulation knot
Fruit such as Fig. 4 (a), Fig. 4 (b) and Fig. 4 (c) are shown.From Fig. 4 (a), Fig. 4 (b) and Fig. 4 (c) as can be seen that network output curved surface with
The curved surface of realistic objective function A coincide substantially, and network relative approximation error is smaller, and the order of magnitude reaches 10-11.The above analytical table
It is bright the present invention gained network have very strong approximation capability, that is, demonstrate a kind of inhibition noise proposed by the invention based on increasing
The superiority of the Gegenbauer Architecture Optimization for Neural Networks of length-beta pruning.
Secondly, superiority in order to further illustrate the present invention, also verifies the noise removal capability of network.In target letter
It joined the random noise of Gaussian distributed on number A, the average value of random noise is 0, noise range are as follows:
[- 10% (max Y-min Y), 10% (max Y-min Y)]
Simulation result is shown in figure and Fig. 4 (d), Fig. 4 (e) and Fig. 4 (f).By Fig. 4 (d) and Fig. 4 (e) it is found that network denoising is defeated
Almost the same with the curved surface without noise targets function A out, the network after the denoising known to Fig. 4 (f) exports and without noise
The relative approximation error of network is very small, and the order of magnitude reaches 10-3.Analysis shows, a kind of inhibition proposed by the invention is made an uproar above
The network of sound obtained based on growth-beta pruning Gegenbauer Architecture Optimization for Neural Networks, even if in objective function band
In the case where the random noise for having higher magnitude, network denoising output still is able to accurately approach binary objective function A, that is, demonstrate,proves
Superiority of the network in terms of approaching and denoising is illustrated.In practical applications, the noise removal capability of network can be used for reducing data and adopt
Bring error when sample, good denoising performance, which makes network in practical applications, has bigger value.
Finally, test verifying is carried out to network performance, in sampling interval u, v ∈ [- 1,1]2(wherein contain trained region
Region { [- 1,1] is not trained2[- 0.9,0.9]2) on uniform sampling generate training sample set(s
=1,2,3 ..., 4489), the sampling interval 0.03, come test neural network obtained by the present invention approach and predictive ability, imitate
True result is shown in Fig. 4 (g), Fig. 4 (a).Comparison diagram 4 (g) and Fig. 4 (a) know network test output surface chart and realistic objective function A
Curved surface it is almost the same, very small by network test output known to Fig. 4 (h) and the relative error of original objective function, the order of magnitude reaches
To 10-8.Above analysis shows, using the present invention it is a kind of inhibit noise based on growth-beta pruning Gegenbauer neural network
The resulting network of structural optimization method can still obtain good Approximation effect in non-learning region, thus have very strong extensive force
Nearly ability.
The above emulation experiment show a kind of inhibition noise proposed by the invention based on growth-beta pruning
Property of the neural network in terms of function approximation, prediction and denoising can be improved in Gegenbauer Architecture Optimization for Neural Networks
Can, there is high use value.
Present invention algorithm improvement different from the past, primary study is by the activation of each hidden neuron of feedforward neural network
Function is chosen for completely new activation primitive, such as linear independence or orthogonal basis function, proposes neural network power on this basis
Value directly determines method (weights direct determination, WDD).Reason is approached based on probability theory and polynomial interopolation
By constructing the single Gegenbauer orthogonal polynomial neural network for exporting single hidden layer of one two input in the present invention, pass through
Weight directly determines one step of method and calculates hidden layer neuron to the connection weight between output layer.It facts have proved this method
Can it is slow to avoid convergence rate existing for traditional BP iterative algorithm, be easily trapped into local minimum, the problems such as study precision is not high.
Further, since the performance of network and network structure are highly relevant, the present invention devises the optimal neuron searcher based on growth
Method can make the smallest optimal hidden layer neuron number of network approximate error to find, and beta pruning network structure, which simplifies, after recycling calculates
Method deletes the neuron of redundancy, to simplify neural network structure, to obtain optimum network performance.
The above embodiment is a preferred embodiment of the present invention, but embodiments of the present invention are not by the embodiment
Limitation, other any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present invention,
It should be equivalent substitute mode, be included within the scope of the present invention.
Claims (5)
1. it is a kind of inhibit noise based on artificial neural network structure's optimization method characterized by comprising
Optimal neuron number searching method based on growth, obtains best initial weights vector wopt, optimal activated matrix QoptAnd it is defeated
Net training time t out1；
Network structure based on rear beta pruning simplifies method.
2. according to claim 1 be based on artificial neural network structure's optimization method, which is characterized in that described based on growth
Optimal neuron number searching method, the specific steps are as follows:
S1.1 constructs one two and inputs the Gegenbauer neural network singly exported, is made of input layer, hidden layer and output layer,
Connection weight is 1 between input layer and hidden layer, and wherein the excitation function of input layer and output layer is all made of linear identity function,
The excitation function of hidden layer neuron is basic function composed by the product of two Gegenbauer orthogonal polynomials, and k-th hiding
Layer neuron excitation function is qk, k=0,1,2 ..., K；
S1.2 initialization network parameter searches for hidden neuron number, and optimal hidden layer neuron is denoted as M+1, and M initial value is 0, enables
Training least mean-square error initial valueCounter c, initial value 0, maximum value c are setmax；
S1.3 judges whether k≤(d+1) (d+2)/2 be true, skips to step S1.5 if setting up, otherwise skips to step S1.4, d is
The sum of Gegenbauer order of a polynomial time of binary input variable, d=i+j, i, j respectively indicate the order of orthogonal polynomial；
S1.4 judges c≤cmaxWhether true, d=d+1 skips to step S1.5 if setting up, and otherwise skips to step S1.8；
S1.5 sets the excitation function of the M+1 hidden neuron as qk, corresponding activated matrix Q is generated, and calculate the weight of hidden layer
Vector w, and calculate mean square error at this time
S1.6 Rule of judgment mean square errorWhether meet, if satisfied, then retaining the M+1 hidden neuron, enables counter
C zero, M=M+1,The currently active matrix Q is saved as current optimal activated matrix Qopt, current weight vector w is protected
Save as current best initial weights vector wopt,
If conditions are not met, the M+1 neuron corresponding element in weight vector w is deleted, the M+1 neuron in activated matrix Q
Corresponding vector, and enable c=c+1；K=k+1 is enabled, S1.3 is returned；
S1.7 completes search, exports training least mean-square errorOptimal hidden layer neuron number M+1 and corresponding optimal
Weight vector wopt, optimal activated matrix Qopt, and export net training time t1。
3. it is according to claim 1 be based on artificial neural network structure's optimization method, which is characterized in that it is described be based on after cut
The Topological expansion method of branch, includes the following steps:
Beta pruning parameter after S2.1 initialization: beta pruning training error after orderEqual to training least mean-square errorBeta pruning weight
Vector wdelEqual to best initial weights vector wopt, beta pruning activated matrix QdelEqual to optimal activated matrix Qopt, i.e.,
wdel=wopt, Qdel=Qopt；
An initial pruning threshold g, specially best initial weights weight vector w is arranged in S2.2optThe smallest first number of middle absolute value it is exhausted
To twice of value；
S2.3 is current beta pruning training errorSave as optimal mean square errorCurrent beta pruning weight vector wdelIt saves
For best initial weights vector wopt, current beta pruning activated matrix QdelSave as optimal activated matrix Qopt, i.e.,wopt
=wdel, Qopt=Qdel；
S2.4 deletes current beta pruning weight vector wdelMiddle weight is less than the element of pruning threshold g, and its in beta pruning activated matrix
QdelCorresponding vector calculates network beta pruning training error at this time
S2.5 obtains network beta pruning training error by S2.3Rule of judgmentIt is whether true, if setting up more
New pruning threshold g is five times, i.e. g=5g originally, and skips to step S2.3；Otherwise, step S2.6 is skipped to；
S2.6 exits beta pruning process, the optimal mean square error of output trainingFinal hidden layer neuron number Mopt, and it is corresponding
Best initial weights vector wopt, optimal activated matrix Qopt, take best initial weights vector woptLength be final hidden layer neuron number
Mopt, i.e. Mopt=length (wopt), and export the total training time T of network.
4. according to claim 2 be based on artificial neural network structure's optimization method, which is characterized in that will in the S1.1
Basic function sorts according to ascending order dictionary method.
5. according to claim 2 be based on artificial neural network structure's optimization method, which is characterized in that the S1.5 is calculated
Mean square error at this timeDetailed process are as follows:
S1.5.1 directly determines method using the weight of principle of least square puppet inverse form, so that hidden layer mind directly be calculated
Through the connection weight between member and output layer, the calculation formula of the connection weight by directly giving as follows:
Wherein, subscriptTIt is expressed as the transposition of matrix-vector, (QTQ)-1QTFor the pseudoinverse for inputting excited target matrix Q, it is denoted as Q+；
Wherein, weight column vector w, input excited target matrix Q and target export column vectorIt respectively indicates are as follows:
W=[w0, w1..., wm..., wM]T∈RM
In above formula,Indicate correspond to s-th of training sample the m+1 hidden neuron excited target response s=1,2,3 ...,
S, M+1 are hidden layer neuron sum, and S is sample total, when input variable u sampling number is su, input variable u sampling number
For svWhen,For training sample pair；
The mean square error of S1.5.2 calculating networkIt is specifically defined are as follows:
Wherein,For square of two norm of vector or matrix.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910127126.8A CN109886405A (en) | 2019-02-20 | 2019-02-20 | It is a kind of inhibit noise based on artificial neural network structure's optimization method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910127126.8A CN109886405A (en) | 2019-02-20 | 2019-02-20 | It is a kind of inhibit noise based on artificial neural network structure's optimization method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109886405A true CN109886405A (en) | 2019-06-14 |
Family
ID=66928656
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910127126.8A Pending CN109886405A (en) | 2019-02-20 | 2019-02-20 | It is a kind of inhibit noise based on artificial neural network structure's optimization method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109886405A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110766138A (en) * | 2019-10-21 | 2020-02-07 | 中国科学院自动化研究所 | Method and system for constructing self-adaptive neural network model based on brain development mechanism |
| CN111461229A (en) * | 2020-04-01 | 2020-07-28 | 北京工业大学 | Deep neural network optimization and image classification method based on target transfer and line search |
| CN117052380A (en) * | 2023-10-10 | 2023-11-14 | 四川宏大安全技术服务有限公司 | Wireless pressure measurement device and method |
-
2019
- 2019-02-20 CN CN201910127126.8A patent/CN109886405A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110766138A (en) * | 2019-10-21 | 2020-02-07 | 中国科学院自动化研究所 | Method and system for constructing self-adaptive neural network model based on brain development mechanism |
| CN111461229A (en) * | 2020-04-01 | 2020-07-28 | 北京工业大学 | Deep neural network optimization and image classification method based on target transfer and line search |
| CN111461229B (en) * | 2020-04-01 | 2023-10-31 | 北京工业大学 | Deep neural network optimization and image classification method based on target transfer and line search |
| CN117052380A (en) * | 2023-10-10 | 2023-11-14 | 四川宏大安全技术服务有限公司 | Wireless pressure measurement device and method |
| CN117052380B (en) * | 2023-10-10 | 2024-01-02 | 四川宏大安全技术服务有限公司 | Wireless pressure measurement device and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10762426B2 (en) | Multi-iteration compression for deep neural networks | |
| US10832123B2 (en) | Compression of deep neural networks with proper use of mask | |
| CN110909926A (en) | TCN-LSTM-based solar photovoltaic power generation prediction method | |
| Lu et al. | Generalized radial basis function neural network based on an improved dynamic particle swarm optimization and AdaBoost algorithm | |
| CN107679617A (en) | The deep neural network compression method of successive ignition | |
| CN107729999A (en) | Consider the deep neural network compression method of matrix correlation | |
| CN110046710A (en) | A kind of the nonlinear function Extremal optimization method and system of neural network | |
| CN109886405A (en) | It is a kind of inhibit noise based on artificial neural network structure's optimization method | |
| CN104636985A (en) | Method for predicting radio disturbance of electric transmission line by using improved BP (back propagation) neural network | |
| CN114330644B (en) | Neural network model compression method based on structure search and channel pruning | |
| CN103559537B (en) | Based on the template matching method of error back propagation in a kind of out of order data stream | |
| Suryo et al. | Improved time series prediction using LSTM neural network for smart agriculture application | |
| CN111382840B (en) | HTM design method based on cyclic learning unit and oriented to natural language processing | |
| CN108983180A (en) | A kind of high-precision radar sea clutter forecast system of colony intelligence | |
| CN113052373A (en) | Monthly runoff change trend prediction method based on improved ELM model | |
| CN111292121A (en) | Garden load prediction method and system based on garden image | |
| CN106295677A (en) | A kind of current image cluster-dividing method combining Lars regular terms and feature self study | |
| CN114154401A (en) | Soil erosion modulus calculation method and system based on machine learning and observation data | |
| Cheng et al. | Research on prediction method based on ARIMA-BP combination model | |
| CN117034060A (en) | AE-RCNN-based flood classification intelligent forecasting method | |
| KR102624710B1 (en) | Structural response estimation method using gated recurrent unit | |
| McLoone et al. | A variable memory quasi-Newton training algorithm | |
| CN115526308A (en) | Time sequence prediction method based on width pruning echo state network | |
| CN114048837A (en) | Deep neural network model reinforcement method based on distributed brain-like map | |
| Lu et al. | Apso-based optimization algorithm of lstm neural network model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190614 |