CN112258377A - Method and equipment for constructing robust binary neural network - Google Patents

Method and equipment for constructing robust binary neural network Download PDF

Info

Publication number
CN112258377A
CN112258377A CN202011088661.6A CN202011088661A CN112258377A CN 112258377 A CN112258377 A CN 112258377A CN 202011088661 A CN202011088661 A CN 202011088661A CN 112258377 A CN112258377 A CN 112258377A
Authority
CN
China
Prior art keywords
neural network
binary neural
training
neurons
layer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011088661.6A
Other languages
Chinese (zh)
Inventor
王祥
李鹏霄
杜翠兰
李扬曦
佟玲玲
任博雅
项菲
井雅琪
段运强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
National Computer Network and Information Security Management Center
Original Assignee
National Computer Network and Information Security Management Center
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by National Computer Network and Information Security Management Center filed Critical National Computer Network and Information Security Management Center
Priority to CN202011088661.6A priority Critical patent/CN112258377A/en
Publication of CN112258377A publication Critical patent/CN112258377A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a method and equipment for constructing a robust binary neural network. The construction method of the robust binary neural network comprises the following steps: carrying out binarization processing on the training data; training a preset binary neural network based on the training data after binarization processing; and carrying out error correction coding on the trained preset binary neural network. By adopting the invention, the memory occupation can be reduced to 1/32 of the original floating point type weight by carrying out binarization processing on the training data; meanwhile, the trained binary neural network is subjected to error correction coding, so that the coded binary neural network has the function of resisting the erasing or wrong noise interference, and the coded binary neural network has good robustness.

Description

Method and equipment for constructing robust binary neural network
Technical Field
The invention relates to the field of neural networks, in particular to a method and equipment for constructing a robust binary neural network.
Background
With the introduction of Deep Neural Networks (DNNs) and the improvement of digital computing devices, Deep Neural Network technology and theory have been rapidly developed, which have become the leading strength of artificial intelligence and revolutionized science and technology. The deep neural network is realized in hardware in the fields of mobile phones, sensors, medical equipment and the like, so that an artificial intelligence system becomes more and more energy-saving and ubiquitous.
At present, on the one hand, a neural network based on floating point operation has a large parameter amount and a large calculation amount, and needs to occupy a lot of calculation resources. On the other hand, training a neural network to obtain various parameters requires intensive calculation and a large amount of storage resources; however, the inference (or prediction) of the neural network does not require retraining the neural network, and the inference process of the neural network does not require significant computational and memory resources. Meanwhile, the deep neural network is extremely susceptible to interference of antagonistic noise. For example, recent studies have shown that in some deep neural networks, the output results will vary greatly if only one or two of the inputs are changed.
Disclosure of Invention
The embodiment of the invention provides a method and equipment for constructing a robust binary neural network, which are used for at least solving the problem of poor noise interference resistance of the neural network in the prior art.
The method for constructing the robust binary neural network comprises the following steps:
carrying out binarization processing on the training data;
training a preset binary neural network based on the training data after binarization processing;
and carrying out error correction coding on the trained preset binary neural network.
According to some embodiments of the invention, the training of the preset binary neural network comprises:
on a Graphic Processing Unit (GPU), training a preset binary neural network to determine the number of neuron layers of the preset binary neural network, the weight of each layer of neuron and the bias value of each layer of neuron.
According to some embodiments of the present invention, the error correction coding the trained preset binary neural network includes:
and performing error correction coding on the original input data by adding redundant neurons, and correspondingly adjusting the weight values of all layers of neurons of the preset binary neural network which completes training and the bias values of all layers of neurons.
According to some embodiments of the present invention, the performing error correction coding on the original input data by adding redundant neurons, and correspondingly adjusting the weight values of each layer of neurons of the trained preset binary neural network and the bias values of each layer of neurons includes:
using triplets (E)t,Vt b,μt) Inputs to each layer of neurons of a pre-defined binary neural network that performs training
Figure BDA0002721236550000021
Weight value
Figure BDA0002721236550000022
And offset value
Figure BDA0002721236550000023
Carrying out error correction coding;
wherein the content of the first and second substances,
Figure BDA0002721236550000024
F2＝{0,1}，
Figure BDA0002721236550000025
Figure BDA0002721236550000026
r is a real number set, t is more than or equal to 1 and less than or equal to L, L represents the number of neuron layers, i is more than or equal to 1 and less than or equal to kt，ktRepresenting the number of t-layer neurons;
the triplet (E)t,Vt b,μt) For any p-subset
Figure BDA0002721236550000027
And a q-subset
Figure BDA0002721236550000028
Satisfies formula 1:
Figure BDA0002721236550000029
according to some embodiments of the invention, the method further comprises:
and transferring the error correction coded preset binary neural network to the FPGA.
The construction equipment of the robust binary neural network comprises the following steps:
the data processing module is used for carrying out binarization processing on the training data;
the training module is used for training a preset binary neural network based on the training data after binarization processing;
and the coding module is used for carrying out error correction coding on the trained preset binary neural network.
According to some embodiments of the invention, the training module comprises:
and the graphic processor GPU is used for training a preset binary neural network to determine the number of neuron layers of the preset binary neural network, the weight value of each layer of neuron and the bias value of each layer of neuron.
According to some embodiments of the invention, the encoding module is to:
and performing error correction coding on the original input data by adding redundant neurons, and correspondingly adjusting the weight values of all layers of neurons of the preset binary neural network which completes training and the bias values of all layers of neurons.
According to some embodiments of the invention, the encoding module is to:
using triplets (E)t,Vt b,μt) Inputs to each layer of neurons of a pre-defined binary neural network that performs training
Figure BDA0002721236550000031
Weight value
Figure BDA0002721236550000032
And offset value
Figure BDA0002721236550000033
Carrying out error correction coding;
wherein the content of the first and second substances,
Figure BDA0002721236550000034
F2＝{0,1}，
Figure BDA0002721236550000035
Figure BDA0002721236550000036
r is a real number set, t is more than or equal to 1 and less than or equal to L, L represents the number of neuron layers, i is more than or equal to 1 and less than or equal to kt，ktRepresenting the number of t-layer neurons;
the triplet (E)t,Vt b,μt) For any p-subset
Figure BDA0002721236550000037
And a q-subset
Figure BDA0002721236550000038
Satisfies formula 1:
Figure BDA0002721236550000039
according to some embodiments of the invention, the apparatus further comprises:
and the FPGA is used for storing the error correction coded preset binary neural network.
By adopting the embodiment of the invention, the memory occupation can be reduced to 1/32 of the original floating point type weight by carrying out binarization processing on the training data; meanwhile, the trained binary neural network is subjected to error correction coding, so that the coded binary neural network has the function of resisting the erasing or wrong noise interference, and the coded binary neural network has good robustness.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. In the drawings:
FIG. 1 is a flow chart of a method for constructing a robust binary neural network according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for constructing a robust binary neural network according to an embodiment of the present invention;
FIG. 3 is a flow chart of a method for constructing a robust binary neural network according to an embodiment of the present invention;
FIG. 4 is a flow chart of a binary neural network trained on a GPU in an embodiment of the present invention;
FIG. 5 is a diagram of a trained binary neural network before encoding according to an embodiment of the present invention;
FIG. 6 is a diagram illustrating encoding of first-layer entries, weights, and offsets of a bi-level neural network according to an embodiment of the present invention;
FIG. 7 is a diagram illustrating encoding of i-th layer entries, weights and offsets of a bi-level neural network model according to an embodiment of the present invention;
FIG. 8 is a flow chart of the encoded binary neural network reasoning on FPGA in the embodiment of the present invention;
fig. 9 is a network structure diagram for encoding the trained binary neural network using the parity check code in the embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
As shown in fig. 1, the method for constructing a robust binary neural network according to an embodiment of the present invention includes:
s1, carrying out binarization processing on the training data;
s2, training a preset binary neural network based on the training data after the binarization processing;
and S3, carrying out error correction coding on the trained preset binary neural network.
In the aspect of a deep neural Network, a Binary Neural Network (BNN) is a neural Network obtained by simultaneously binarizing a weight value and each activation function value (eigenvalue) in a weight matrix of a floating-point neural Network, that is, the binary neural Network is: the weight value and the activation function value are binarized to 1 or-1. Through binarization processing, the parameters can occupy smaller storage space (the memory consumption is reduced to 1/32 times in theory and from float32 to 1bit, and bit operation is used for replacing multiply-add operation in the network, so that the operation time is greatly reduced.
By adopting the embodiment of the invention, the memory occupation can be reduced to 1/32 of the original floating point type weight by carrying out binarization processing on the training data; meanwhile, the trained binary neural network is subjected to error correction coding, so that the coded binary neural network has the function of resisting the erasing or wrong noise interference, and the coded binary neural network has good robustness.
On the basis of the above-described embodiment, various modified embodiments are further proposed, and it is to be noted herein that, in order to make the description brief, only the differences from the above-described embodiment are described in the various modified embodiments.
According to some embodiments of the invention, the training of the preset binary neural network comprises:
on a Graphic Processing Unit (GPU), training a preset binary neural network to determine the number of neuron layers of the preset binary neural network, the weight of each layer of neuron and the bias value of each layer of neuron.
According to some embodiments of the present invention, the error correction coding the trained preset binary neural network includes:
and performing error correction coding on the original input data by adding redundant neurons, and correspondingly adjusting the weight values of all layers of neurons of the preset binary neural network which completes training and the bias values of all layers of neurons.
According to some embodiments of the present invention, the performing error correction coding on the original input data by adding redundant neurons, and correspondingly adjusting the weight values of each layer of neurons of the trained preset binary neural network and the bias values of each layer of neurons includes:
using triplets (E)t,Vt b,μt) For preset binary neural networks for trainingInput terms for layer neurons
Figure BDA0002721236550000061
Weight value
Figure BDA0002721236550000062
And offset value
Figure BDA0002721236550000063
Carrying out error correction coding;
wherein the content of the first and second substances,
Figure BDA0002721236550000064
F2＝{0,1}，
Figure BDA00027212365500000610
Figure BDA0002721236550000066
r is a real number set, t is more than or equal to 1 and less than or equal to L, L represents the number of neuron layers, i is more than or equal to 1 and less than or equal to kt，ktRepresenting the number of t-layer neurons;
the triplet (E)t,Vt b,μt) For any p-subset
Figure BDA0002721236550000067
And a q-subset
Figure BDA0002721236550000068
Satisfies formula 1:
Figure BDA0002721236550000069
as shown in fig. 2, according to some embodiments of the invention, the method further comprises:
and S4, transferring the error correction coded preset binary neural network to the FPGA.
The graphic processor GPU has parallel computing characteristics and powerful computing resources, is good at training the neural network and the inference process of the neural network, but requires much power consumption. Therefore, in a practical environment with relatively low power consumption, it is not favorable to use the GPU for reasoning. The FPGA can program hardware according to specific application, needs lower power consumption, can be applied to various low-power consumption and low-environment environments, but has far lower computing capability than a GPU.
The FPGA comprises a trigger (FF), a digital processing unit (DSP), a storage unit RAM and a phase-locked loop PLL. And the FPGA adopts an AXI bus to carry out data transmission on the chip and off the chip.
The method for constructing the robust binary neural network according to the embodiment of the present invention is described in detail in a specific embodiment with reference to fig. 3 to 9. It is to be understood that the following description is illustrative only and is not intended to be in any way limiting. All similar structures and similar variations thereof adopted by the invention are intended to fall within the scope of the invention.
At present, in some application scenarios with low power consumption, because of the low power consumption of the FGPA, it is of great significance and application to transplant the neural network to the FPGA. In the aspect of application of the binary neural network on the FPGA, one is to directly binarize the neural network to be migrated and then transplant the neural network to the FPGA; and the other method is to train the binary neural network directly in the FPGA and apply the binary neural network to the FPGA. The first solution has the following disadvantages: firstly, directly binarizing the neural network to be shifted can make the binarized neural network not well predict a target value, and because a training set is not used for optimizing the weight value, the offset value and the neural network structure of the binary neural network. Secondly, the binary neural network is extremely susceptible to interference of reactive noise, and when there is erasure or error in input data or neural units, the output item is greatly changed, so that the robustness of the binary neural network is poor. The second solution has the following disadvantages: first, the computational resources of the FPGA are less suitable for training the binary neural network than the GPU, which can slow the training speed. In addition, the binary neural network is not considered to be interfered by antagonism, and the defect of poor robustness exists. Minrui et al propose in "a method of transplanting a deep learning network to a FPAG platform", adopt 1bit quantization scheme to convert original quantization into binary deep learning, not only reduce memory occupation to 1/32 of original floating-point type weight, but also because its weight parameter has only binary state, such binary operation can be realized rapidly by a logic gate device, can greatly alleviate the problem of insufficient DSP resources to a certain extent.
As shown in fig. 3, in the method for constructing a robust binary neural network according to the embodiment of the present invention, training data is processed to binarize the data (1 or-1), then a GPU with strong parallel computation is used to train the binary neural network, error correction coding is performed on an input item, a weight and an offset of each layer in the trained binary neural network, and finally the coded binary neural network is transplanted to an FPGA with low power consumption to perform inference (or prediction), so that the coded robust binary neural network is applied to a heterogeneous resource platform. The method specifically comprises the following steps:
step 1: training the set binary neural network on a GPU to obtain the structure of the optimal binary neural network, wherein the structure comprises the number of layers of the binary neural network, the weight value and the deviation value of each layer of neuron.
Training a structure chart of a binary neural network on a GPU (graphics processing unit), as shown in figure 4, quantizing a floating point type parameter into a binary parameter when the binary neural network is transmitted forwards, then performing convolution on the binary parameter and an input characteristic value, adding a bias term, and obtaining an output characteristic value through a Sign activation function, wherein a final specific expression is as follows;
Figure BDA0002721236550000081
wherein the content of the first and second substances,
Figure BDA0002721236550000082
is a binary input value of the convolutional layer,
Figure BDA0002721236550000083
is a binary weight, θ, of the convolutional layerkThe value of the offset is the value of the offset,
Figure BDA0002721236550000084
is a warpThe over-convolution and the feature value after the activation function. In the reverse propagation, the derivative function of the HTanh function is used to replace the derivative function of the Sign function, that is:
q＝Sign(r),gr＝g q1|r|≤1，
wherein, gqTo obtain the gradient of the loss function, then, based on the chain rule, the gradient of the real activation value is first obtained
Figure BDA0002721236550000085
And updating to obtain a real number type weight and a real number type parameter, wherein a specific expression is as follows:
Figure BDA0002721236550000086
Figure BDA0002721236550000087
wherein, the Update function and the Clip function are both defined in the related literature (I.Hubara, M.Courbariaux, D.Soudry, R.El-Yaniv and Y.Bengio: "binary Neural networks," Proc.neural Information Processing Systems (NIPS), pp.4107-4115,2016), and will not be described herein again.
Through training the binary neural network on the GPU, the weight of the binary neural network of the L layer can be obtained
Figure BDA0002721236550000088
And the offset parameter value theta is (theta)1,θ2,...,θL)。
Step 2: the method for coding the trained binary neural network with error correction, specifically coding the binary input items and each layer of neurons, comprises the following steps: firstly, adding some redundant input items to carry out error correction coding on original input data to obtain new input items, and adjusting the weight and the offset of a first layer through a coding scheme to obtain a new weight and offset of the first layer; then, redundant items are added to the input data of the hidden layer for coding, and corresponding weight and offset are adjusted.
Specifically, it is provided with
Figure BDA0002721236550000089
Is a binary k0The sum of the dimensional vectors Wi bIs ki-1×kiWhere 1 ≦ i ≦ L.
First, for an input x, a weight W of a first layer1 bVector of offset values theta1Error correction coding is performed, and, here,
Figure BDA00027212365500000810
middle ith column
Figure BDA00027212365500000811
Corresponding to the associated weight in the ith neuron of the first layer.
For convenience, a triple (E) is defined1,V1 b,μ1) Wherein
Figure BDA0002721236550000091
Figure BDA0002721236550000092
Where F 20,1 and R are real number sets. More generally, in a binary neural network, only 1 or-1 is considered as an input value, then, at F2Element 0 in {0,1} is considered to be a 1 and element 1 is considered to be-1, and "logical exclusive nor" is considered to be { +1, -1} multiplication.
When i is more than or equal to 1 and less than or equal to k1Error correction coding of the ith neuron of the first layer, the triplet
Figure BDA0002721236550000093
Is the ith neural coding parameter of the first layer, for any one p-subset
Figure BDA0002721236550000094
And a q-subset
Figure BDA0002721236550000095
So that
Figure BDA0002721236550000096
Here, the first and second liquid crystal display panels are,
Figure BDA0002721236550000098
for the sign of the convolution,
Figure BDA0002721236550000099
representing the multiplication of the jth component of the two vectors.
If i is less than or equal to k for all 1 or less1Triple of
Figure BDA00027212365500000910
All satisfy equation (1), then pass through (E)1,V1 b,μ1) After encoding, first layer neurons (total k)1P) are able to combat p erasure errors (erasures) and q errors (errors), and the first layer network is said to be (p, q) -robust. In the first layer of neural network, some redundant neurons are added (in total)
Figure BDA00027212365500000911
One) encode x into E1(x) And the weight of the first layer is changed from the original W1 bCoded as V1 bThen, the offset value of the first layer is changed from the original value of theta1Is encoded as mu1。
Similarly, for a t-th layer network, where 2. ltoreq. t.ltoreq.L, a triplet (E) is constructed in the same wayt,Vt b,μt) To encode a t-th layer network, wherein
Figure BDA00027212365500000912
Figure BDA00027212365500000913
For 1 ≦ i ≦ ktError correction coding is performed on the ith neuron of the t layer, and the triplet
Figure BDA00027212365500000914
Is the ith neural coding parameter of the t-th layer, for any one p-subset
Figure BDA00027212365500000915
And a q-subset
Figure BDA00027212365500000916
Such that:
Figure BDA0002721236550000097
in the t neural network hidden layer, some redundant neurons are added (in total)
Figure BDA00027212365500000917
Ones) input item x of the t-th layer neural networkt-1Is coded as Et(xt-1) And the weight of the first layer is changed from the original Wt bCoded as Vt bThen, the offset value of the first layer is changed from the original value of thetatIs encoded as mut. If L triples (E) satisfying the formula (1) and the formula (2) are constructedt,Vt b,μt) Where 1 ≦ t ≦ L, the binary neural network is said to be able to combat p erasure errors and q errors, and is a (p, q) -robust binary neural network.
For the definition of the function for encoding the error correcting code, reference may be made to the relevant literature (N.Raviv, S.Jain, P.Upadhyaya, J.Bruckanda.Jiang.: "CodN-Robust Neurralnetworkfrom Coded classification." arXivproprintarXiv: 2004.10700v2, (2020)), which is not described again in detail
In the following, a block diagram is used to describe the encoding process of the binary neural network coding. The figure 5 shows the binary neural network after being trained by the GPU, in the figure,
Figure BDA0002721236550000101
as an input item, Wi bIs a weight matrix (1 ≦ i ≦ L), θ ≦ L1,θ2,...,θL) As an offset parameter, τi,jIs the ith layer jth neuron of the binary neural network and is activated by the jth neuron to generate an output value xi,j(here, 1. ltoreq. i.ltoreq.L and 1. ltoreq. j.ltoreq.ki). Encoding the first layer neural network to obtain the encoded first layer neural network is shown in FIG. 6, in which the first layer parameters (x, W) of the neural network1 b,θ1) By a triplet (E)1,V1 b,μ1) After encoding, another set of parameters (E) is obtained1(x),V1 b,μ1) Where the first tier entry x becomes E1(x)(E1(x) Is composed of
Figure BDA0002721236550000102
Dimensional vector), i.e., adding some redundant neurons will k0The dimension entry is coded as
Figure BDA0002721236550000103
An entry to a dimension. Coding the ith layer neural network to obtain the coded ith layer neural network shown in figure 7, wherein i is more than or equal to 2 and less than or equal to L, and the ith layer parameter (x) of the neural networki-1,Wi b,θi) By a triplet (E)i,Vi b,μi) After encoding, another set of parameters (E) is obtainedi(xi-1),Vi b,μi) Here, the input item x of the i-th layeri-1Become one
Figure BDA0002721236550000104
Dimension vector Ei(xi-1) That is, adding some redundant neurons in the hidden layer will ki-1The dimension entry is coded as
Figure BDA0002721236550000105
An entry to a dimension.
And step 3: and moving the coded binary neural network to the FPGA.
As shown in fig. 8, a specific quantization method is to perform binary preprocessing on a prediction input value to be preprocessed, then encode the preprocessed binary input value, and input the encoded binary neural network to obtain a prediction output value.
The embodiment of the invention encodes the binary neural network by adopting an error correction coding scheme and is realized on heterogeneous resources GPU and FPGA. Because the binary neural network needs to preprocess the data into binary type data, the memory occupation is reduced to 1/32 of the original floating point type weight; meanwhile, the characteristics of the GPU and the FPGA are fully utilized, the GPU is utilized to train the binary neural network and the FPGA to realize the inference process of the binary neural network, and therefore the binary neural network is more efficient; and finally, carrying out error correction coding on the trained binary neural network, so that the coded binary neural network has the function of resisting the erasing or wrong noise interference, and the coded binary neural network has good robustness.
Below with F2The above parity check code is used as a coding function of all layers of the binary neural network to construct a coded binary neural network for explanation, and the steps are as follows:
step one, training a binary neural network by using a GPU to obtain a binary neural network structure (as shown in figure 3) and the binary neural network is an L layer, wherein the parameter of the ith layer of the binary neural network is (x)i-1,Wi b,θi) Here, xi-1As an input to the i-th layer, Wi bIs a binary weight matrix of the i-th layer, thetaiIs the offset parameter of the ith layer, and i is more than or equal to 1 and less than or equal to L.
Step two, utilizing F2The parity-check code above encodes each layer of the binary neural network of fig. 3, allowing for the use of { +1, -1} instead of F 20,1, then used to
Figure BDA0002721236550000111
Substitute F2In (1)
Figure BDA0002721236550000112
More, a popcount algorithm is used instead of an accumulation operation, thereby realizing convolution operation. By F2Binary neural network with parity check code codingParameters (x, W) of the first layer1 b,θ1) Is coded as (E)1(x),V1 b,μ1) Here, the
Figure BDA0002721236550000113
μ1,t＝θ1,tWherein t is more than or equal to 1 and less than or equal to k1. Similarly, when i is more than or equal to 2 and less than or equal to L, the parameter (x) of the ith layer is measuredi-1,Wi b,θi) By a triplet (E)i,Vi b,μi) After encoding, another set of parameters (E) is obtainedi(xi-1),Vi b,μi) Here, the
Figure BDA0002721236550000114
Figure BDA0002721236550000115
μi,t＝θi,tWherein t is more than or equal to 1 and less than or equal to ki. Each layer of parameters of the binary neural network are encoded through the parity check code, and an encoded binary neural network structure diagram is obtained, as shown in fig. 9.
And step three, moving the coded binary neural network structure diagram to the FPGA. The prediction output value is obtained by performing binary preprocessing on the prediction input value to be preprocessed, then performing parity check coding on the preprocessed binary input value, and inputting the coded binary input value into the coded binary neural network, as shown in fig. 9.
As can be seen from fig. 9, only one redundant neuron needs to be added in each layer, and each layer of encoded neuron entries only have one more entry than the original entries, so that the encoded binary neural network only needs to add a small amount of computing and storage resources. Meanwhile, after the binary neural network is coded by the parity check code, the coded binary neural network can resist the noise interference of one erasure, and the robustness of the binary neural network is improved.
It should be noted that the above-mentioned embodiments are only preferred embodiments of the present invention, and are not intended to limit the present invention, and those skilled in the art can make various modifications and changes. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
The construction equipment of the robust binary neural network comprises the following steps:
the data processing module is used for carrying out binarization processing on the training data;
the training module is used for training a preset binary neural network based on the training data after binarization processing;
and the coding module is used for carrying out error correction coding on the trained preset binary neural network.
By adopting the embodiment of the invention, the memory occupation can be reduced to 1/32 of the original floating point type weight by carrying out binarization processing on the training data; meanwhile, the trained binary neural network is subjected to error correction coding, so that the coded binary neural network has the function of resisting the erasing or wrong noise interference, and the coded binary neural network has good robustness.
On the basis of the above-described embodiment, various modified embodiments are further proposed, and it is to be noted herein that, in order to make the description brief, only the differences from the above-described embodiment are described in the various modified embodiments.
According to some embodiments of the invention, the training module comprises:
and the graphic processor GPU is used for training a preset binary neural network to determine the number of neuron layers of the preset binary neural network, the weight value of each layer of neuron and the bias value of each layer of neuron.
According to some embodiments of the invention, the encoding module is to:
and performing error correction coding on the original input data by adding redundant neurons, and correspondingly adjusting the weight values of all layers of neurons of the preset binary neural network which completes training and the bias values of all layers of neurons.
According to some embodiments of the invention, the encoding module is to:
using triplets (E)t,Vt b,μt) Inputs to each layer of neurons of a pre-defined binary neural network that performs training
Figure BDA0002721236550000132
Weight value
Figure BDA0002721236550000133
And offset value
Figure BDA0002721236550000134
Carrying out error correction coding;
wherein the content of the first and second substances,
Figure BDA0002721236550000135
F2＝{0,1}，
Figure BDA0002721236550000136
Figure BDA0002721236550000137
r is a real number set, t is more than or equal to 1 and less than or equal to L, L represents the number of neuron layers, i is more than or equal to 1 and less than or equal to kt，ktRepresenting the number of t-layer neurons;
the triplet (E)t,Vt b,μt) For any p-subset
Figure BDA0002721236550000138
And a q-subset
Figure BDA0002721236550000139
Satisfies formula 1:
Figure BDA0002721236550000131
according to some embodiments of the invention, the apparatus further comprises:
and the FPGA is used for storing the error correction coded preset binary neural network.
It should be noted that in the description of the present specification, reference to the description of the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples", etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Although some embodiments described herein include some features included in other embodiments instead of others, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. The particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. For example, in the claims, any of the claimed embodiments may be used in any combination.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims (10)

1. A method for constructing a robust binary neural network is characterized by comprising the following steps:
carrying out binarization processing on the training data;
training a preset binary neural network based on the training data after binarization processing;
and carrying out error correction coding on the trained preset binary neural network.
2. The method of claim 1, wherein training the preset binary neural network comprises:
on a Graphic Processing Unit (GPU), training a preset binary neural network to determine the number of neuron layers of the preset binary neural network, the weight of each layer of neuron and the bias value of each layer of neuron.
3. The method of claim 2, wherein the error correction coding of the trained pre-set binary neural network comprises:
and performing error correction coding on the original input data by adding redundant neurons, and correspondingly adjusting the weight values of all layers of neurons of the preset binary neural network which completes training and the bias values of all layers of neurons.
4. The method of claim 3, wherein the error correction coding of the original input data by adding redundant neurons and adjusting the weights of the neurons in each layer of the trained pre-defined binary neural network and the bias values of the neurons in each layer accordingly comprises:
using triplets (E)t,Vt b,μt) Inputs to each layer of neurons of a pre-defined binary neural network that performs training
Figure FDA0002721236540000011
Weight value
Figure FDA0002721236540000012
And offset value
Figure FDA0002721236540000013
Carrying out error correction coding;
wherein the content of the first and second substances,
Figure FDA0002721236540000014
F2＝{0,1}，
Figure FDA0002721236540000019
Figure FDA0002721236540000016
r is a real number set, t is more than or equal to 1 and less than or equal to L, L represents the number of neuron layers, i is more than or equal to 1 and less than or equal to kt，ktDenotes the t-th layerThe number of neurons;
the triplet (E)t,Vt b,μt) For any p-subset
Figure FDA0002721236540000017
And a q-subset
Figure FDA0002721236540000018
Satisfies formula 1:
Figure FDA0002721236540000021
5. the method of claim 1, wherein the method further comprises:
and transferring the error correction coded preset binary neural network to the FPGA.
6. A robust binary neural network construction device, comprising:
the data processing module is used for carrying out binarization processing on the training data;
the training module is used for training a preset binary neural network based on the training data after binarization processing;
and the coding module is used for carrying out error correction coding on the trained preset binary neural network.
7. The apparatus of claim 6, wherein the training module comprises:
and the graphic processor GPU is used for training a preset binary neural network to determine the number of neuron layers of the preset binary neural network, the weight value of each layer of neuron and the bias value of each layer of neuron.
8. The device of claim 7, wherein the encoding module is to:
and performing error correction coding on the original input data by adding redundant neurons, and correspondingly adjusting the weight values of all layers of neurons of the preset binary neural network which completes training and the bias values of all layers of neurons.
9. The device of claim 8, wherein the encoding module is to:
using triplets (E)t,Vt b,μt) Inputs to each layer of neurons of a pre-defined binary neural network that performs training
Figure FDA0002721236540000022
Weight value
Figure FDA0002721236540000023
And offset value
Figure FDA0002721236540000024
Carrying out error correction coding;
wherein the content of the first and second substances,
Figure FDA0002721236540000025
F2＝{0,1}，
Figure FDA00027212365400000211
Figure FDA0002721236540000027
r is a real number set, t is more than or equal to 1 and less than or equal to L, L represents the number of neuron layers, i is more than or equal to 1 and less than or equal to kt，ktRepresenting the number of t-layer neurons;
the triplet (E)t,Vt b,μt) For any p-subset
Figure FDA0002721236540000028
And a q-subset
Figure FDA0002721236540000029
Satisfies formula 1:
Figure FDA00027212365400000210
10. the apparatus of claim 6, wherein the apparatus further comprises:
and the FPGA is used for storing the error correction coded preset binary neural network.
CN202011088661.6A 2020-10-13 2020-10-13 Method and equipment for constructing robust binary neural network Pending CN112258377A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011088661.6A CN112258377A (en) 2020-10-13 2020-10-13 Method and equipment for constructing robust binary neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011088661.6A CN112258377A (en) 2020-10-13 2020-10-13 Method and equipment for constructing robust binary neural network

Publications (1)

Publication Number Publication Date
CN112258377A true CN112258377A (en) 2021-01-22

Family

ID=74242465

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011088661.6A Pending CN112258377A (en) 2020-10-13 2020-10-13 Method and equipment for constructing robust binary neural network

Country Status (1)

Country Link
CN (1) CN112258377A (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108231086A (en) * 2017-12-24 2018-06-29 航天恒星科技有限公司 A kind of deep learning voice enhancer and method based on FPGA
CN110097186A (en) * 2019-04-29 2019-08-06 济南浪潮高新科技投资发展有限公司 A kind of neural network isomery quantization training method
CN110751150A (en) * 2019-09-29 2020-02-04 上海工程技术大学 FPGA-based binary neural network license plate recognition method and system
CN110929852A (en) * 2019-11-29 2020-03-27 中国科学院自动化研究所 Deep binary neural network training method and system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108231086A (en) * 2017-12-24 2018-06-29 航天恒星科技有限公司 A kind of deep learning voice enhancer and method based on FPGA
CN110097186A (en) * 2019-04-29 2019-08-06 济南浪潮高新科技投资发展有限公司 A kind of neural network isomery quantization training method
CN110751150A (en) * 2019-09-29 2020-02-04 上海工程技术大学 FPGA-based binary neural network license plate recognition method and system
CN110929852A (en) * 2019-11-29 2020-03-27 中国科学院自动化研究所 Deep binary neural network training method and system

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
NETANEL RAVIV 等: "CodNN–Robust Neural Networks From Coded Classification", 《ARXIV:2004.10700V1》 *
周进登 等: "基于神经网络的纠错输出编码方法研究", 《电子学报》 *
平嘉蓉等: "基于轻量级神经网络的人群计数模型设计", 《无线电工程》 *

Similar Documents

Publication Publication Date Title
Ahmadizar et al. Artificial neural network development by means of a novel combination of grammatical evolution and genetic algorithm
Kouda et al. Qubit neural network and its learning efficiency
CN101183873B (en) BP neural network based embedded system data compression/decompression method
Hayashi et al. Reinforcement learning and graph embedding for binary truss topology optimization under stress and displacement constraints
Livne et al. Pops: Policy pruning and shrinking for deep reinforcement learning
Zhao et al. Tuning the structure and parameters of a neural network using cooperative binary-real particle swarm optimization
CN110932734B (en) Deep learning channel decoding method based on alternative direction multiplier method
CN109389208B (en) Data quantization device and quantization method
Chowdhury et al. Towards ultra low latency spiking neural networks for vision and sequential tasks using temporal pruning
Zhu et al. Learning to denoise and decode: A novel residual neural network decoder for polar codes
Kim et al. Exploiting retraining-based mixed-precision quantization for low-cost DNN accelerator design
Zhang et al. Summary of convolutional neural network compression technology
Chen et al. Universal perceptron and DNA-like learning algorithm for binary neural networks: non-LSBF implementation
EP4035273B1 (en) Design and training of binary neurons and binary neural networks with error correcting codes
Hussain et al. Exploiting deep neural networks for digital image compression
Dai et al. Fast training and model compression of gated RNNs via singular value decomposition
CN109389209B (en) Processing apparatus and processing method
CN109697507B (en) Processing method and device
Chen et al. Universal perceptron and DNA-like learning algorithm for binary neural networks: LSBF and PBF implementations
CN112258377A (en) Method and equipment for constructing robust binary neural network
EP1267304A1 (en) A method of performing the superposition operation of a Grover's or a Deutsch-Jozsa's quantum algorithm and a relative quantum gate
Goel et al. CompactNet: High accuracy deep neural network optimized for on-chip implementation
Müller et al. Randomized unregulated step descent for limited precision synaptic elements
Oh et al. Multi-FNN identification based on HCM clustering and evolutionary fuzzy granulation
CN114169528A (en) Method for prolonging quantum entanglement service life

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20210122

RJ01 Rejection of invention patent application after publication