CN109146000A - A kind of method and device for improving convolutional neural networks based on frost weight - Google Patents
A kind of method and device for improving convolutional neural networks based on frost weight Download PDFInfo
- Publication number
- CN109146000A CN109146000A CN201811044605.5A CN201811044605A CN109146000A CN 109146000 A CN109146000 A CN 109146000A CN 201811044605 A CN201811044605 A CN 201811044605A CN 109146000 A CN109146000 A CN 109146000A
- Authority
- CN
- China
- Prior art keywords
- hidden layer
- layer node
- weight
- convolutional neural
- neural networks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/32—Digital ink
Abstract
The invention discloses a kind of method and device for improving convolutional neural networks based on frost weight, this method improves traditional BP convolutional neural networks by the theoretical of frost weight.This method acquires enough training and test sample data first, it is pre-processed, and establish convolutional neural networks model, processing is optimized to convolutional neural networks according to frost weight theory, the hidden layer node in convolutional neural networks is analyzed by being introduced into entropy weight method, freeze to contribute little hidden layer node to network output in convolutional neural networks training process, the renewal process of hidden layer node in reasonably optimizing convolutional neural networks, it is effectively reduced the computational complexity of convolutional neural networks, shortens the training duration of convolutional neural networks.This method has the advantages that efficient, reliable, precision is higher.
Description
Technical field
The invention belongs to nerual network technique algorithm fields, and in particular to one kind improves convolutional Neural net based on frost weight
The method and device of network.
Background technique
Convolutional neural networks (Convolutional Neural Network, CNN) are to be applied to deep learning in recent years
A kind of neural network be also widely used for image in recent years by convolutional neural networks to the high-performance of image procossing
The fields such as identification.The concept of convolutional neural networks is proposed by LeCun earliest, and LeCun et al. is proposed in Fukushima
Neocognitron (Neocognitron) on the basis of, using simple cell layer (S-layer) and complex cell layer (C-
Layer) the structure being alternately superimposed, and gone out warp using BP (Back Propagation back-propagation algorithm) algorithm successful design
The character identification system LeNet-5 model of allusion quotation.As the convolutional neural networks of first appearance, which has stronger Shandong
Stick, convolutional neural networks are reached by amplifying the space characteristics between data using the spatial coherence of data to each other
To the purpose for shortening the training time.
But convolutional neural networks equally also have many problems, and main problem therein is made from convolutional neural networks
Conventional counter propagation algorithm (Back Propagation, BP).Since the error function in traditional BP neural network is phase
For the first derivative of hidden layer node weight each in network, it can not accomplish that carrying out unlimited number of minimum asks in reality
Topic just will appear the problem of step-length is chosen.Secondly, traditional BP neural network objective function to be optimized be it is extremely complex,
Therefore it necessarily will appear " zigzag phenomenon ", this makes traditional BP neural network computing inefficient；Again due to traditional BP neural network
The objective function of middle optimization is very complicated, it some flat regions will necessarily occur in the case where neuron is exported close to 0 or 1,
In these regions, node weight changes very little, and weight error changes very little, network training process is made almost to pause.These problems
The convolutional neural networks training time will be caused longer, for complicated nerve network system of construction itself, this, which is undoubtedly, is avenged
Add frost.
In a kind of " the parametric source modeling method based on improved BP " of Publication No. CN103077267B
In state's patent, a kind of improvement BP optimized based on genetic algorithm to structure and parameter in convolutional neural networks model is disclosed
Neural network, by find preferably convolutional neural networks imply initial weight between the number of plies, hidden layer node and threshold value come
Optimize convolutional neural networks algorithm.This method optimizes convolutional neural networks to a certain extent, but this method introduces heredity
Algorithm increases the training duration of convolutional neural networks actually, and there is no the training durations for really shortening convolutional neural networks.
Summary of the invention
It is an object of the invention to overcome the above-mentioned deficiency in the presence of the prior art, a kind of introducing frost weight reason is provided
By the method for improving traditional BP convolutional neural networks with entropy weight data processing method, while this method can effectively promote network fortune
Defeated efficiency reduces the training time, guarantees that convolutional neural networks have stronger robustness.
In order to achieve the above-mentioned object of the invention, the present invention provides following technical schemes:
A method of convolutional neural networks are improved based on frost weight, comprising the following steps:
Step 101, pretreatment image obtains multiple batch training image data；
Step 102, convolutional neural networks are constructed, and the hidden layer node weight amount in convolutional neural networks are carried out initial
Change assignment；
Step 103, the training image data for inputting a batch carry out convolutional neural networks by forward conduction algorithm
It calculates, obtains the activation weight of each hidden layer node in convolutional neural networks, each implied according to the activation weight computing
The weight difference of node layer；
Step 104, the weight difference is analyzed based on entropy weight method, to obtain the entropy weight of each hidden layer node, and
The evaluation of estimate that hidden layer node is determined with this screens the hidden layer node based on the evaluation of estimate of hidden layer node, freezes
Evaluation of estimate is tied in evaluation of estimate threshold value hidden layer node below to maintain its weight amount as current weight amount, and transmits and is not frozen
Hidden layer node；
Step 105, by reverse conduction algorithm update transmitting come be not frozen hidden layer node weight amount, obtain
It is not frozen the updated weight amount of hidden layer node, to get the weight amount of current each hidden layer node；
Step 106, error judgment is carried out according to current each hidden layer node weight amount, if error is not default first
In range, then repeatedly step 103~105, until error stops the training of this batch in the first preset range；
Step 107, it uses multiple batch training image data instead to be trained convolutional neural networks, so that the error of network
Index is in the second preset range.
Further, the method for improving convolutional neural networks, the pretreatment image are normalized from hand-written
The a certain number of image datas got in volume data identification library, by all image data scaleds to unified scale.
Further, in the method for improving convolutional neural networks, the convolutional neural networks include an input layer,
At least two convolutional layers and at least two sample levels, an output layer, wherein output layer merges full articulamentum.
Further, in the method for improving convolutional neural networks, the weight difference is gained hidden layer node
Activate the absolute value of the difference of weight and hidden layer node current weight amount.
Further, in the method for improving convolutional neural networks, institute is analyzed based on entropy weight method described in step 104
Weight difference is stated, to obtain the entropy weight of each hidden layer node, and with this step of this evaluation of estimate for determining hidden layer node
It specifically includes:
According to hidden layer node number and its weight difference, data matrix is established；The normalized data matrix, obtains
Normalization matrix；According to normalization matrix, the entropy of hidden layer node weight difference is calculated；According to the entropy meter of gained hidden layer node
Calculate coefficient of variation；The entropy weight of hidden layer node is calculated according to gained coefficient of variation；Hidden layer is calculated according to the entropy weight acquired
The evaluation of estimate of node.
Further, the data matrix are as follows:
Wherein, m is hidden layer node number, and n is hidden layer node weight difference.
Further, in the method for improving convolutional neural networks, the weight amount of current each hidden layer node
Initial weight amount and the not frozen updated weight amount of hidden layer node including frozen hidden layer node.
Further, the error judgment formula are as follows:
Wherein, E (w, v) is the current error amount of convolutional neural networks, and wherein k is the quantity of mode, and C is hidden layer node
Quantity, tpiIt is the output target value of hidden layer node p, spiIt is reality output of the hidden layer node p in convolutional neural networks
Value.
Further, the method for improving convolutional neural networks, step 107 includes, by the training image of multiple batches
Data are separately input in convolutional neural networks, are repeated step 103- step 106, are executed multiple batches of training image data to network
Training.
Preferably, a kind of device improving convolutional neural networks based on frost weight, including at least one processor, and
The memory being connect at least one described processor communication；The memory, which is stored with, to be held by least one described processor
Capable instruction, described instruction are executed by least one described processor, so that at least one described processor is able to carry out right
It is required that method described in any one of 1 to 6.
Compared with prior art, beneficial effects of the present invention:
1, the structure for optimizing convolutional neural networks incorporates full articulamentum and output layer in convolutional neural networks, mentions
High convolutional neural networks operation efficiency.
2, it is theoretical to introduce frost weight, improves original BP convolutional neural networks model.In BP convolutional neural networks
In every batch of training process, the hidden layer node of frost and output result degree of correlation difference, in subsequent reverse conduction no longer
The weight for the hidden layer node being frozen is updated, so as to improve the complexity of convolutional neural networks training process, promotes network instruction
Practice efficiency, simplifies the calculating process that each reverse conduction updates hidden layer node weight, network training duration is reduced with this.
3, the destination node that entropy weight data processing method filters out required frost in every batch of training is introduced, is utilized
The entropy weight between node weight value difference value in every batch of training is in the frost theory of original Relative Fuzzy as assessment reference
The selection of node proposes better solution.
4, the experimental results showed that, training process in network not only can be effectively reduced in the improved convolutional neural networks of the present invention
Complexity, shorten network training duration, moreover it is possible to avoid falling into step-length problem and training time too long problem in network training.
Detailed description of the invention:
Fig. 1 is the flow chart of convolutional neural networks improved method according to an exemplary embodiment of the present invention；
Fig. 2 is LeNet-5 convolutional neural networks structure chart；
Fig. 3 is the convolutional neural networks structure chart after the optimization according to an exemplary embodiment of the present invention based on LeNet-5；
Fig. 4 is the flow chart of entropy weight method according to an exemplary embodiment of the present invention；
Fig. 5 is convolutional neural networks improved method device according to an exemplary embodiment of the present invention.
Specific embodiment
Below with reference to test example and specific embodiment, the present invention is described in further detail.But this should not be understood
It is all that this is belonged to based on the technology that the content of present invention is realized for the scope of the above subject matter of the present invention is limited to the following embodiments
The range of invention.
Embodiment 1
A method of convolutional neural networks are improved based on frost weight, comprising the following steps:
Step 101, pretreatment image obtains multiple batch training image data；
70000 image datas are obtained in library specifically, identifying first from the hand-written volume data of MNIST, wherein training image
Data 60000 are opened, and test image data 10000 are opened.To all image datas (including training image data and test image number
According to) processing that is normalized, it is the input matrix of 28 × 28 pixels by all image real time transfers, every batch of training is schemed
As data sample size is set as 50, to obtain the training image data of 1200 trained batches.
Step 102, convolutional neural networks are constructed, and the hidden layer node weight amount in convolutional neural networks are carried out initial
Change assignment；
Specifically, we are based on character identification system LeNet-5 building convolutional neural networks in this example, and optimize
The network structure of LeNet-5 merges the full articulamentum and output layer of the network, to reduce convolutional neural networks computational complexity.
As shown in Fig. 2, the LeNet-5 network architecture on basis is by 1 input layer, 3 convolutional layers, 2 pond layers, 1 full articulamentum and 1
A output layer is constituted, and the convolutional neural networks after optimizing in the present invention are as shown in figure 3, comprising 1 input layer, 2 convolutional layers, and 2
A sample level (also referred to as pond layer), 1 output layer, sequence are as follows: the first convolutional layer, first are sequentially connected after input layer
Pond layer, the second convolutional layer, the second pond layer, output layer (output layer incorporates full articulamentum).
Further, the input layer of the convolutional neural networks constructed in this example is according to test diagram data picture and training image
Designed by data, 50, while one in each hidden layer node processing image are set by hidden layer node number in input layer
A data point, thus the structure of input layer is 28 × 28 × 50.There are 65 × 5 volumes in subsequent first convolutional layer of input layer
Product core, the data matrix that the image data transmitted from input layer is 24 × 24 by the processing of the first convolutional layer, and at each
A biasing is added after convolution kernel.There are 61 × 12 convolution kernels in the second convolutional layer, is 12 8 × 8 by image procossing
Image, it is same that 12 biasings are added.
The design of first sample level and the second sample level will match the framework of previous convolutional layer, first sample level
In, the size that setting sampling window is 6 12 × 12 is biased to 0.Second sample level setting window is 12 4 × 4 big
It is small, equally it is biased to 0.Each sample level both for previous convolutional layer data structure, by the result of convolution at each
Dimension all reduces one times.A last output layer is similarly full articulamentum.
In this example, 150 hidden layer nodes are arranged in the first convolutional layer of the convolutional neural networks in we, second
1800 hidden layer nodes are arranged in convolutional layer.Then we carry out the weight amount of hidden layer nodes all in convolutional neural networks
Assignment is initialized, all hidden layer nodes are input layer in convolutional neural networks, convolutional layer, pond layer, output layer (containing complete
Articulamentum) in all hidden layer node.Here assignment is to determine in range (generally 0-100) to all hidden layer sections
Point carries out random assignment.In addition, the setting of hidden layer node all in input layer, output layer all after initializing assignment not
Become, and the hidden layer node in the layer of pond is not the target of this algorithm training, therefore, in this example, only in convolutional layer
1950 hidden layer nodes can be by innovatory algorithm of the invention training.
Step 103, the training image data for inputting a batch carry out convolutional neural networks by forward conduction algorithm
It calculates, obtains the activation weight of the hidden layer node, believed according to the weight difference of the activation weight computing hidden layer node
Breath；
Specifically, by training image data (50) the input convolutional neural networks of a batch, according to BP volumes
Forward conduction training method in product neural network calculates the convolutional neural networks, available in this calculating
The activation weight of each hidden layer node (activates the hidden layer node, the hidden layer node is made to obtain optimum Working institute
The weight amount needed).Then with the initial power of the activation weight and each hidden layer node of obtained each hidden layer node
Value amount does subtraction, and obtained absolute value of the difference is the weight difference information of the hidden layer node (namely in usual definition
Residual error)
The principle of traditional BP convolutional neural networks is: a convolutional neural networks is given, by a batch training image
After data input convolutional neural networks, what is carried out first is that " forward conduction " calculates, and is calculated according to forward conduction calculation method
Each hidden layer node is from the second layer (first layer is input layer) in convolutional neural networks to output layer in convolutional neural networks
Activation value calculate its residual error δ then for each hidden layer node ii (l), which shows each hidden layer section
How many contribution produced to network final output for point.The residual information of hidden layer node carries out back transfer from output layer at this time,
Then the weight amount of all hidden layer nodes is updated by " reverse conduction more new algorithm ".In the training of different batches, own
The weight amount of hidden layer node can be continuously updated, and the output result of network can more level off to required for us as a result, network
Stability and accuracy can also increase accordingly therewith, but at the same time, the related complexity and training duration of algorithm can also be shown
The promotion of work.
The experimental results show during being trained to traditional BP convolutional neural networks, convolutional neural networks
All hidden layer nodes in a part of hidden layer node it is in fact in fact and insensitive to a batch training image data, i.e., this
The residual values that part hidden layer node calculates in the forward conduction training of network are smaller, it is to network output accordingly
It influences also just smaller.Also mean that the increase with the training time, the variation of this part hidden layer node weight is little.Therefore such as
Fruit directly carries out the right value update of hidden layer node, this part after completing forward conduction algorithm and calculating using reverse conduction algorithm
What the lesser hidden layer node of residual values continued to be updated has little significance.Continue to be updated actually this part hidden layer node
It is to flog a dead horse, increases with when training with apparent so as to cause the related complexity of algorithm.
The present invention is the improvement made on this basis to traditional BP convolutional neural networks, passes through forward conduction in network
After training calculates the residual error of each hidden layer node, further, according to the theoretical thought of shannon entropy, we can lead to
The entropy for the residual error for calculating each hidden layer node is crossed to evaluate it, it is subsequent reversed to decide whether to carry out the node
Conduction updates weight process, and in evaluation criteria system index weights, entropy is a highly desirable scale.It is based on entropy weight method
It determines an evaluation criterion, meaningless hidden layer node is continued to update in this part with this evaluation criterion and screens and carries out
Freeze (i.e. this part of nodes no longer carry out weight amount update), so, not frozen node need to only be carried out subsequent
Reverse conduction calculate update its weight, greatly reduce the computational complexity of network, save a large amount of net training time.
Step 104, the weight difference is analyzed based on entropy weight method, to obtain the entropy weight of each hidden layer node, and
The evaluation of estimate that hidden layer node is determined with this screens the hidden layer node based on the evaluation of estimate of hidden layer node, freezes
Evaluation of estimate is tied in evaluation of estimate threshold value hidden layer node below to maintain its weight amount as current weight amount, and transmits and is not frozen
Hidden layer node；
Specifically, analyze the weight difference based on entropy weight method, to obtain the entropy weight of each hidden layer node, and with
This determines that this step of the evaluation of estimate of hidden layer node specifically includes:
An indicator evaluation system is initially set up, (n weight of hidden layer node is poor if wherein there is n evaluation index
Value), m are evaluated object (hidden layer node number), are evaluated the raw data matrix of the corresponding index of object, as follows:
Then we need that raw data matrix is normalized, and are denoted as matrix S=(sij)m*n。
S is normalized, is denoted as
All S values obtained in this way will be in [0,1] section.
The entropy of j-th of weight difference is at this time
Wherein,
The coefficient of variation of j-th of weight difference are as follows:
αj=1-Hj(j=1,2, n)
The entropy weight of j-th of weight difference are as follows:
Under the setting of such entropy weight, we are it can be concluded that a judgment criteria, when obtained hidden layer node
Entropy it is bigger when, then its entropy weight will be smaller, also can accordingly reduce to the contribution of the training process of this batch.When gained entropy
Maximum value be 1, when entropy weight is 0, it is believed that the hidden layer node does not provide for this training any effective
Contribution.
As shown in figure 4, specific entropy weight analytic approach can be summarized as follows,
Step 401, according to hidden layer node number and its weight difference, data matrix is established；
Step 402, normalized data matrix, obtains normalization matrix；
Step 403, according to normalization matrix, the entropy of hidden layer node weight difference is calculated；
Step 404, coefficient of variation is calculated according to the entropy of gained hidden layer node；
Step 405, the entropy weight of hidden layer hiding node is calculated according to gained coefficient of variation；
Step 406, the evaluation of estimate of hidden layer node is calculated with following formula according to the entropy weight acquired.
Thus the evaluation of estimate of all hidden layer nodes is found out, the value of X is smaller, then the contribution that the node updates weight is got over
It is small.Then the evaluation of estimate of each hidden layer node is analyzed, by hidden layer node evaluation of estimate the smallest 20 percent
Hidden layer node, which screens, to be freezed, and institute's evaluation values threshold value is the evaluation of estimate at 20 percent this separation,
In every batch of training simultaneously, evaluation of estimate threshold value can change, in generally 0-0.05 a number.Freeze entropy weight in entropy
Weight threshold hidden layer node below no longer updates this in subsequent algorithm to maintain its weight amount as initial weight amount
The weight amount of the frozen hidden layer node in part, the weight amount of this part hidden layer node will be maintained at its initial weight amount.
Simultaneously from the output layer of convolutional neural networks to the hidden layer of the front (be the system to convolutional layer and pond layer before output layer
Claim) hidden layer node that is not frozen of back transfer.
Step 105, by reverse conduction algorithm update transmitting come be not frozen hidden layer node weight amount, obtain
It is not frozen the updated weight amount of hidden layer node, to get the weight amount of current each hidden layer node；
Specifically, utilizing reverse conduction formula to not frozen hidden layer node, its weight amount is updated, has been frozen at this time
The hidden layer node of knot keeps its current weight amount, along with the not frozen updated weight amount of hidden layer node, is
The weight amount of current each hidden layer node.
Step 106, error judgment is carried out according to current each hidden layer node weight amount, if error is not default first
In range, then repeatedly step 103~105, until error stops the training of this batch in the first preset range；
Specifically, the network for working as previous training is calculated by following formula according to the weight amount of current each hidden layer node
Error E (w):
Wherein k is the quantity of mode, and C is the quantity of hidden layer node, tpiIt is the output target value of hidden layer node p.Spi
It is real output value of the hidden layer node p in convolutional neural networks.
Wherein, h is the node in hidden layer in network, xiIt is a n dimension input pattern, i=1,2 ..., k, vmIt is to connect
Connect the C dimensional vector of the weight of the arc of m-th of hidden layer node and output layer.The activation primitive of output layer is sigmoid function σ
(y)=1/ (1+e-y), the activation primitive of hidden layer is hyperbolic tangent function:
δ (y)=(ey-e-y)/(ey+e-y)
At this point, if the error amount acquired within the first preset range (generally 0~0.05), i.e. hidden layer node at this time
The output target value t of ppiWith hidden layer node p convolutional neural networks real output value SpiBetween almost without deviation, say
Bright batch training objective is smoothly completed, and can terminating the training of this batch, (step 103- step 105 is the training of this batch
Overall process).Conversely, this batch training objective is not reached if the error amount acquired is not within tolerance interval, then repeat
Step 103- step 105 again inputs this batch training image data into convolutional neural networks, to convolutional neural networks
In hidden layer node carry out forward calculation training, be based on entropy weight method frozen fraction hidden layer node, reversed update is not frozen
The weight amount for tying hidden layer node, until the obtained error amount of weight amount according to all hidden layer nodes is in the first default model
Enclose interior this batch of end training.
Step 107, it uses multiple batch training datas instead to be trained convolutional neural networks, so that the error criterion of network
In the second preset range.
Specifically, inputting the 2nd, the 3rd, the the 4th ... the 120th training image data after the training of a batch
Step 103- step 106 is repeated, the training of remaining batch training image data is completed, so that so that the error criterion of network is the
In two preset ranges.Each of this example image data is all one obtained from MNIST hand-written volume data identification library
The image of handwritten numeral, since the writing of different people is different, so how by convolutional neural networks to identify that writing habit is poor
Not biggish same numbers are that our problems to be solved that is, by training network pair want that network can be made to carry out it correctly
Identification.Therefore we are trained network using training image data (60000) after so that network performance is tended towards stability, and utilize
Test image data (10000) are tested for the property network, to show that (i.e. can network correctly identify writing to test result
It is accustomed to the biggish same numbers of difference), the fault rate of network is found out by test result.The default error range of second at this time is
It is a restriction to the fault rate of network overall operation result, generally requires network fault rate within 1 15.
Further, we by original BP convolutional neural networks and improve BP convolutional neural networks operation result (including instruction
Practice time and fault rate) it is compared, obtain comparison result as shown in Table 1.
Table 1
It can be seen that improved BP convolutional neural networks through the invention from above-mentioned chart, when averagely training every time
Between can reduce 25 seconds, be greatly shortened convolutional neural networks training duration.The improved BP convolutional Neural net of the present invention simultaneously
Although (fault rate improves 2% or so, within an acceptable range) the generally training time that increases on network error rate is reduced
13.9%.Therefore improved convolutional neural networks significantly improve net in the case where sacrificing partially acceptable accuracy
The arithmetic speed of network makes network training complexity reach 5% the above object of expected reduction.Secondly the improved volume of the present invention
Product neural network can be working properly, network does not fall into endless loop or error rate is excessively high, illustrate present invention effectively prevents
The problems such as step-length and local minimum for being easy to appear in traditional BP convolutional neural networks, has stronger robustness.
Embodiment 2
Fig. 5 show it is according to an embodiment of the present invention improve convolutional neural networks device, i.e., electronic equipment 310 (such as
Have program execute function computer server) comprising at least one processor 311, power supply 314, and with it is described extremely
The memory 312 and input/output interface 313 of a few processor 311 communication connection；The memory 312 is stored with can be by institute
State the instruction of at least one processor 311 execution, described instruction executed by least one described processor 311 so that it is described extremely
A few processor 311 is able to carry out method disclosed in aforementioned any embodiment；The input/output interface 313 may include
Display, keyboard, mouse and USB interface are used for inputoutput data；Power supply 314 is used to provide electricity for electronic equipment 310
Energy.
It will be appreciated by those skilled in the art that: realize that all or part of the steps of above method embodiment can pass through program
Relevant hardware is instructed to complete, program above-mentioned can store in computer-readable storage medium, which is executing
When, execute step including the steps of the foregoing method embodiments；And storage medium above-mentioned includes: movable storage device, read-only memory
The various media that can store program code such as (Read Only Memory, ROM), magnetic or disk.
When the above-mentioned integrated unit of the present invention be realized in the form of SFU software functional unit and as the sale of independent product or
In use, also can store in a computer readable storage medium.Based on this understanding, the skill of the embodiment of the present invention
Substantially the part that contributes to existing technology can be embodied in the form of software products art scheme in other words, the calculating
Machine software product is stored in a storage medium, including some instructions are used so that a computer equipment (can be individual
Computer, server or network equipment etc.) execute all or part of each embodiment the method for the present invention.And it is aforementioned
Storage medium include: the various media that can store program code such as movable storage device, ROM, magnetic or disk.
Claims (10)
1. a kind of method for improving convolutional neural networks based on frost weight, which comprises the following steps:
Step 101, pretreatment image obtains multiple batch training image data；
Step 102, convolutional neural networks are constructed, and initialization tax is carried out to the hidden layer node weight amount in convolutional neural networks
Value；
Step 103, the training image data for inputting a batch calculate convolutional neural networks by forward conduction algorithm,
The activation weight of each hidden layer node in convolutional neural networks is obtained, according to each hidden layer node of the activation weight computing
Weight difference；
Step 104, the weight difference is analyzed based on entropy weight method, to obtain the entropy weight of each hidden layer node, and with this
The evaluation of estimate for determining hidden layer node screens the hidden layer node based on the evaluation of estimate of hidden layer node, freezes to comment
Value in evaluation of estimate threshold value hidden layer node below to maintain its weight amount as current weight amount, and transmit be not frozen it is hidden
Containing node layer；
Step 105, by reverse conduction algorithm update transmitting come be not frozen hidden layer node weight amount, obtain not by
Freeze the updated weight amount of hidden layer node, to get the weight amount of current each hidden layer node；
Step 106, error judgment is carried out according to current each hidden layer node weight amount, if error is not in the first preset range
It is interior, then repeatedly step 103~105, until error stops the training of this batch in the first preset range；
Step 107, it uses multiple batch training image data instead to be trained convolutional neural networks, so that the error criterion of network
In the second preset range.
2. the method as described in claim 1, which is characterized in that the pretreatment image is normalized from hand-written volume data
The a certain number of image datas got in identification library, by all image data scaleds to unified scale.
3. the method as described in claim 1, which is characterized in that the convolutional neural networks include an input layer, at least two
A convolutional layer and at least two sample levels, an output layer, wherein output layer merges full articulamentum.
4. the method as described in claim 1, which is characterized in that the weight difference is the activation weight of gained hidden layer node
With the absolute value of the difference of hidden layer node current weight amount.
5. the method as described in claim 1, which is characterized in that analyze the weight based on entropy weight method described in step 104
Difference to obtain the entropy weight of each hidden layer node, and is specifically wrapped with this step of this evaluation of estimate for determining hidden layer node
It includes:
According to hidden layer node number and its weight difference, data matrix is established；The normalized data matrix, obtains normalizing
Change matrix；According to normalization matrix, the entropy of hidden layer node weight difference is calculated；It is poor to be calculated according to the entropy of gained hidden layer node
Different coefficient；The entropy weight of hidden layer node is calculated according to gained coefficient of variation；Hidden layer node is calculated according to the entropy weight acquired
Evaluation of estimate.
6. method as claimed in claim 4, which is characterized in that the data matrix are as follows:
Wherein, m is hidden layer node number, and n is hidden layer node weight difference.
7. the method as described in claim 1, which is characterized in that the weight amount of current each hidden layer node includes being frozen
The initial weight amount of the hidden layer node of knot and the not frozen updated weight amount of hidden layer node.
8. the method as described in claim 1, which is characterized in that the error judgment formula are as follows:
Wherein, E (w, v) is the current error amount of convolutional neural networks, and wherein k is the quantity of mode, and C is the number of hidden layer node
Amount, tpiIt is the output target value of hidden layer node p, spiIt is real output value of the hidden layer node p in convolutional neural networks.
9. the method as described in claim 1, which is characterized in that step 107 includes, by the training image data of multiple batches point
It is not input in convolutional neural networks, repeats step 103- step 106, execute multiple batches of training image data to the instruction of network
Practice.
10. a kind of device for improving convolutional neural networks based on frost weight, which is characterized in that including at least one processor,
And the memory being connect at least one described processor communication；The memory is stored with can be by least one described processing
The instruction that device executes, described instruction is executed by least one described processor, so that at least one described processor is able to carry out
Method described in any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811044605.5A CN109146000B (en) | 2018-09-07 | 2018-09-07 | Method and device for improving convolutional neural network based on freezing weight |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811044605.5A CN109146000B (en) | 2018-09-07 | 2018-09-07 | Method and device for improving convolutional neural network based on freezing weight |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109146000A true CN109146000A (en) | 2019-01-04 |
| CN109146000B CN109146000B (en) | 2022-03-08 |
Family
ID=64823890
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811044605.5A Active CN109146000B (en) | 2018-09-07 | 2018-09-07 | Method and device for improving convolutional neural network based on freezing weight |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109146000B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110263925A (en) * | 2019-06-04 | 2019-09-20 | 电子科技大学 | A kind of hardware-accelerated realization framework of the convolutional neural networks forward prediction based on FPGA |
| CN111222465A (en) * | 2019-11-07 | 2020-06-02 | 深圳云天励飞技术有限公司 | Image analysis method based on convolutional neural network and related equipment |
| CN111624964A (en) * | 2020-05-27 | 2020-09-04 | 甬矽电子(宁波)股份有限公司 | Dynamic authority management and control method, device, server and readable storage medium |
| TWI731511B (en) * | 2019-12-12 | 2021-06-21 | 國立中興大學 | Finger vein or palm vein identification processing and neural network training method |
| CN116989510A (en) * | 2023-09-28 | 2023-11-03 | 广州冰泉制冷设备有限责任公司 | Intelligent refrigeration method combining frosting detection and hot gas defrosting |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5845051A (en) * | 1995-09-15 | 1998-12-01 | Electronics And Telecommunications Research Institute | Learning method for multilayer perceptron neural network with N-bit data representation |
| US20130061114A1 (en) * | 2011-09-02 | 2013-03-07 | Samsung Electronics Co., Ltd. | Freezing-based ldpc decoder and method |
| CN105654729A (en) * | 2016-03-28 | 2016-06-08 | 南京邮电大学 | Short-term traffic flow prediction method based on convolutional neural network |
| US20170109243A1 (en) * | 2015-10-16 | 2017-04-20 | Business Objects Software, Ltd. | Model-Based System and Method for Undoing Actions in an Application |
| CN107194376A (en) * | 2017-06-21 | 2017-09-22 | 北京市威富安防科技有限公司 | Mask fraud convolutional neural networks training method and human face in-vivo detection method |
| CN107301396A (en) * | 2017-06-21 | 2017-10-27 | 北京市威富安防科技有限公司 | Video fraud convolutional neural networks training method and human face in-vivo detection method |
-
2018
- 2018-09-07 CN CN201811044605.5A patent/CN109146000B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5845051A (en) * | 1995-09-15 | 1998-12-01 | Electronics And Telecommunications Research Institute | Learning method for multilayer perceptron neural network with N-bit data representation |
| US20130061114A1 (en) * | 2011-09-02 | 2013-03-07 | Samsung Electronics Co., Ltd. | Freezing-based ldpc decoder and method |
| US20170109243A1 (en) * | 2015-10-16 | 2017-04-20 | Business Objects Software, Ltd. | Model-Based System and Method for Undoing Actions in an Application |
| CN105654729A (en) * | 2016-03-28 | 2016-06-08 | 南京邮电大学 | Short-term traffic flow prediction method based on convolutional neural network |
| CN107194376A (en) * | 2017-06-21 | 2017-09-22 | 北京市威富安防科技有限公司 | Mask fraud convolutional neural networks training method and human face in-vivo detection method |
| CN107301396A (en) * | 2017-06-21 | 2017-10-27 | 北京市威富安防科技有限公司 | Video fraud convolutional neural networks training method and human face in-vivo detection method |
Non-Patent Citations (3)
| Title |
|---|
| HINTON G E等: "Improving neural networks by preventing co-adaptation of feature detectors", 《COMPUTER SCIENCE》 * |
| RUI ZHANG等: "Extreme Learning Machine with Adaptive Growth of Hidden Nodes and Incremental Updating of Output Weights", 《AIS 2011: AUTONOMOUS AND INTELLIGENT SYSTEMS》 * |
| 高大文等: "人工神经网络中隐含层节点与训练次数的优化", 《哈尔滨工业大学学报》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110263925A (en) * | 2019-06-04 | 2019-09-20 | 电子科技大学 | A kind of hardware-accelerated realization framework of the convolutional neural networks forward prediction based on FPGA |
| CN110263925B (en) * | 2019-06-04 | 2022-03-15 | 电子科技大学 | Hardware acceleration implementation device for convolutional neural network forward prediction based on FPGA |
| CN111222465A (en) * | 2019-11-07 | 2020-06-02 | 深圳云天励飞技术有限公司 | Image analysis method based on convolutional neural network and related equipment |
| TWI731511B (en) * | 2019-12-12 | 2021-06-21 | 國立中興大學 | Finger vein or palm vein identification processing and neural network training method |
| CN111624964A (en) * | 2020-05-27 | 2020-09-04 | 甬矽电子(宁波)股份有限公司 | Dynamic authority management and control method, device, server and readable storage medium |
| CN116989510A (en) * | 2023-09-28 | 2023-11-03 | 广州冰泉制冷设备有限责任公司 | Intelligent refrigeration method combining frosting detection and hot gas defrosting |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109146000B (en) | 2022-03-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109146000A (en) | A kind of method and device for improving convolutional neural networks based on frost weight | |
| Cui et al. | Efficient human motion prediction using temporal convolutional generative adversarial network | |
| Chen et al. | Dynamical channel pruning by conditional accuracy change for deep neural networks | |
| Zhou et al. | Context-reinforced semantic segmentation | |
| CN112052948B (en) | Network model compression method and device, storage medium and electronic equipment | |
| CN112613581B (en) | Image recognition method, system, computer equipment and storage medium | |
| CN111294646A (en) | Video processing method, device, equipment and storage medium | |
| Liu et al. | CNN-FCM: System modeling promotes stability of deep learning in time series prediction | |
| CN109817276A (en) | A kind of secondary protein structure prediction method based on deep neural network | |
| CN110383299A (en) | The generation time model of memory-enhancing effect | |
| CN105550749A (en) | Method for constructing convolution neural network in novel network topological structure | |
| Maliar et al. | Will artificial intelligence replace computational economists any time soon? | |
| CN108596327A (en) | A kind of seismic velocity spectrum artificial intelligence pick-up method based on deep learning | |
| US11538178B2 (en) | Machine learning-based 2D structured image generation | |
| CN111199216B (en) | Motion prediction method and system for human skeleton | |
| CN106796533A (en) | It is adaptive selected the system and method for execution pattern | |
| CN113343901A (en) | Human behavior identification method based on multi-scale attention-driven graph convolutional network | |
| WO2020030052A1 (en) | Animal count identification method, device, medium, and electronic apparatus | |
| CN110033089A (en) | Deep neural network parameter optimization method and system based on Distributed fusion algorithm | |
| CN112529149A (en) | Data processing method and related device | |
| de Prado et al. | Automated design space exploration for optimized deployment of dnn on arm cortex-a cpus | |
| Jaiswal et al. | Deep neural network compression via knowledge distillation for embedded applications | |
| US11568621B2 (en) | Dynamic character model fitting of three-dimensional digital items | |
| CN113111308B (en) | Symbolic regression method and system based on data-driven genetic programming algorithm | |
| CN114863508A (en) | Expression recognition model generation method, medium and device of adaptive attention mechanism |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |