CN107977707B - Method and computing equipment for resisting distillation neural network model - Google Patents
Method and computing equipment for resisting distillation neural network model Download PDFInfo
- Publication number
- CN107977707B CN107977707B CN201711179045.XA CN201711179045A CN107977707B CN 107977707 B CN107977707 B CN 107977707B CN 201711179045 A CN201711179045 A CN 201711179045A CN 107977707 B CN107977707 B CN 107977707B
- Authority
- CN
- China
- Prior art keywords
- neural network
- network model
- training
- loss function
- label
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method for resisting distillation neural network model, wherein the neural network model comprises a forward network with a characteristic layer structure and a softmax layer outputting probability vectors under multi-classification, the method is suitable for being executed in a computing device and comprises the following steps: adding a scaling layer between a forward network and a softmax layer of the original neural network model according to the distillation temperature to generate a first neural network model; training a first neural network model by using a first label of a training sample to obtain a second neural network model; inputting the training sample into a second neural network model, and outputting a second label representing the probability vector of the training sample under multi-classification through a softmax layer; simultaneously constraining and training the second neural network model by using the second label and the first label to obtain a third neural network model; and deleting the scaling layer in the third neural network model to obtain the neural network model after the resistance distillation. The invention also discloses corresponding computing equipment.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a method and computing equipment for an anti-distillation neural network model.
Background
The deep neural network can always obtain very accurate results on the current classification regression problem, and the trained deep neural network model has strong generalization capability under the support of mass data, so that the deep neural network is widely applied to the aspects of computer vision, speech recognition and the like in recent years. However, in practical applications, these deep neural network models may have some defects and vulnerabilities. For example, in the case that the structure and parameters of the network model are not clear, the inputs to the network are specially disturbed slightly, which does not affect the judgment in human subjectivity, but can make the network model output error results with high confidence, and these inputs after being disturbed slightly are called "confrontation samples". The problems directly affect the generalization ability and the safety of the neural network model.
A common approach for improving the generalization ability and security of neural network models is: countermeasure samples are added into training data of the neural network model, so that the error rate of the network model for recognizing the countermeasure samples is reduced, and meanwhile, the generalization capability of the model is further improved. However, the diversity against sample construction, etc., has resulted in such treatment approaches not achieving the intended results.
Therefore, a solution that can provide generalization capability and security of neural network models is needed.
Disclosure of Invention
To this end, the present invention provides a method and computing device of countering distillation neural network models in an attempt to solve or at least alleviate at least one of the problems presented above.
According to one aspect of the present invention, there is provided a method of countering distillation neural network models, wherein a neural network model comprises a forward network having a feature layer structure and a softmax layer that outputs multi-class lower probability vectors, the method adapted for execution in a computing device, comprising the steps of: adding a scaling layer between a forward network and a softmax layer of the original neural network model according to the distillation temperature to generate a first neural network model; training a first neural network model by using a first label of a training sample to obtain a second neural network model; inputting the training sample into a second neural network model, and outputting a second label representing the probability vector of the training sample under multi-classification through a softmax layer; simultaneously constraining and training the second neural network model by using the second label and the first label to obtain a third neural network model; and deleting the scaling layer in the third neural network model to obtain the neural network model after the resistance distillation.
Optionally, in the method according to the invention, the scaling layer is adapted to scale down the input of the softmax layer in accordance with the distillation temperature.
Optionally, in the method according to the present invention, the step of training the first neural network model with the first label of the training sample itself to obtain the second neural network model includes: and monitoring the training of the first neural network model by using the first label through a first loss function to obtain a second neural network model.
Optionally, in the method according to the present invention, the step of training the second neural network model using the second label and the first label while constraining to obtain the third neural network model includes: carrying out classification supervision training on the second neural network model by using the first label through a first loss function; performing regression supervision training on the second neural network model through a second loss function by using a second label; and training by combining the first loss function and the second loss function to obtain a third neural network model.
Optionally, in the method according to the present invention, the step of training the third neural network model by combining the first loss function and the second loss function includes: weighting the first loss function and the second loss function to obtain a final loss function for training a third neural network model; and training a third neural network model using the final loss function.
Optionally, in the method according to the invention, the first loss function is:
loss1＝-logf(zk)
wherein the content of the first and second substances,
in the formula, loss1Is the first loss function value, N is the batch size at training, zkFor full connectivity in a forward networkThe output of the kth neuron of the next layer.
Optionally, in the method according to the invention, the second loss function is:
in the formula, loss2For the second loss function value, M is the total number of classes of classification output by the softmax layer, x1iProbability vector, x, output for the ith class of the current network2iIs the probability vector corresponding to the ith class characterized by the second label.
Optionally, in the method according to the invention, the final loss function for training the third neural network model is defined as: loss ═ w1×loss1+w2×loss2，
Where loss is the final loss function value, w1And w2Weighting factors representing the first loss function value and the second loss function value, respectively.
According to another aspect of the present invention, there is provided a computing device comprising: one or more processors; and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described above.
According to a further aspect of the invention there is provided a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform any of the methods described above.
According to the method for resisting the distillation neural network model, the scaling layer is added in the original neural network model, the neural network model is distilled, the characteristic layer structure in the neural network model is not changed, and the error rate of the neural network model in the process of dealing with the resisting sample is effectively reduced; and the second label of the training sample and the first label of the training sample are used for simultaneously monitoring the training of the neural network model, so that the generalization capability of the neural network model is improved.
Drawings
To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.
FIG. 1 shows a schematic diagram of a configuration of a computing device 100 according to one embodiment of the invention; and
FIG. 2 shows a flow diagram of a method 200 of countering a distillation neural network model according to one embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Fig. 1 is a block diagram of an example computing device 100. In a basic configuration 102, computing device 100 typically includes system memory 106 and one or more processors 104. A memory bus 108 may be used for communication between the processor 104 and the system memory 106.
Depending on the desired configuration, the processor 104 may be any type of processor, including but not limited to: a microprocessor (μ P), a microcontroller (μ C), a Digital Signal Processor (DSP), or any combination thereof. The processor 104 may include one or more levels of cache, such as a level one cache 110 and a level two cache 112, a processor core 114, and registers 116. The example processor core 114 may include an Arithmetic Logic Unit (ALU), a Floating Point Unit (FPU), a digital signal processing core (DSP core), or any combination thereof. The example memory controller 118 may be used with the processor 104, or in some implementations the memory controller 118 may be an internal part of the processor 104.
Depending on the desired configuration, system memory 106 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 106 may include an operating system 120, one or more applications 122, and program data 124. In some embodiments, application 122 may be arranged to operate with program data 124 on an operating system. In some embodiments, computing device 100 is configured to perform a method of countering distillation neural network models, with instructions for performing the method being included in program data 124.
Computing device 100 may also include an interface bus 140 that facilitates communication from various interface devices (e.g., output devices 142, peripheral interfaces 144, and communication devices 146) to the basic configuration 102 via the bus/interface controller 130. The example output device 142 includes a graphics processing unit 148 and an audio processing unit 150. They may be configured to facilitate communication with various external devices, such as a display or speakers, via one or more a/V ports 152. Example peripheral interfaces 144 may include a serial interface controller 154 and a parallel interface controller 156, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, image input device) or other peripherals (e.g., printer, scanner, etc.) via one or more I/O ports 158. An example communication device 146 may include a network controller 160, which may be arranged to facilitate communications with one or more other computing devices 162 over a network communication link via one or more communication ports 164.
A network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, Radio Frequency (RF), microwave, Infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media. In some embodiments, one or more programs are stored in the computer readable medium, the one or more programs including instructions for performing certain methods, such as a method for countering a distillation neural network model performed by the computing device 100 according to embodiments of the present invention.
Computing device 100 may be implemented as part of a small-form factor portable (or mobile) electronic device such as a cellular telephone, a digital camera, a Personal Digital Assistant (PDA), a personal media player device, a wireless web-watch device, a personal headset device, an application specific device, or a hybrid device that include any of the above functions. Computing device 100 may also be implemented as a personal computer including both desktop and notebook computer configurations.
The flow of implementing the method 200 for countering the neural network model according to one embodiment of the present invention will be described in detail below with reference to fig. 2.
The general structure of the neural network model according to an embodiment of the present invention can be divided into two parts, i.e., a forward network having a feature layer structure and a softmax layer that outputs probability vectors under multi-classification. The forward network generally has at least one convolution layer, a pooling layer and a full link layer, and input data is output after being subjected to, for example, multiple convolution and pooling operations, and then being communicated and combined through the full link layer. The softmax layer is understood as the result of normalizing the output of the forward network, and assuming that the neural network model is used to classify pictures, there are hundreds of current picture classifications, and the output through the softmax layer is a one-hundred-dimensional vector, the first value of the vector is the probability value … … that the current picture belongs to the first classification, the second value of the vector is the probability value … … that the current picture belongs to the second classification, and the sum of the vector values of the one hundred dimensions is 1.
It should be noted that, a general application scenario of the method 200 is to perform classification processing by using a neural network model, and a specific structure of the neural network model is not limited. In practical applications, the forward network of the neural network model may be any one of existing or redefined network structures such as AlexNet, VGGNet, Google inclusion Net, and ResNet, which is not limited by the embodiment of the present invention.
The method 200 begins at step S210, where the distillation temperature is T, and a scaling layer is added between the forward network and the softmax layer of the original neural network model according to the distillation temperature to generate a first neural network model. According to one embodiment of the invention, the scaling layer performs a scaling process on the input of the softmax layer (i.e. the output of the forward network) according to the distillation temperature T. That is, a scaling layer is added between the forward network of the original neural network model and the softmax layer, the scaling layer reduces the output of the forward network (i.e., the output of the last fully-connected layer in the forward network) by 1/T, and then inputs the reduced data into the softmax layer. The embodiment of the invention does not limit the value of the distillation temperature T, and in practical application, the value of T is selected according to the size of the forward network and the practical situation.
Subsequently, in step S220, the first neural network model is trained by using the first label of the training sample itself, so as to obtain a second neural network model. According to one embodiment of the invention, a training sample is input into a first neural network model, the training of the first neural network model is supervised on the training sample labeled by a first label through a first loss function, and the neural network model of each parameter in the trained network is used as a second neural network model, wherein the first label is a label of the training sample, and is called hard label.
Subsequently, in step S230, the training samples in step S220 are input into the second neural network model trained in step S220, and a second label, called softtarget, representing the probability vector of the training samples under multi-classification is output through the softmax layer, where the second label is the predicted probability vector of the second neural network model to the training samples.
Subsequently, in step S240, the second neural network model (i.e., the distilled neural network model) is trained using the second label and the first label while constraining, resulting in a third neural network model. According to one embodiment of the invention, when training the second neural network model, the network model is trained simultaneously with the first label (hard label) and the second label (soft target) of the training sample, and two sets of loss functions are assigned. The specific steps are described as follows:
in one aspect, a second neural network model is class supervised trained with a first loss function for a first label (hard label). Here, the training process may be synchronized to step S220, and the second neural network model is trained using the first label of the training sample itself. Alternatively, the first loss function is, for example, Softmax with loss in Caffe, which is collectively referred to as the conditional Architecture for Fast Feature Embedding, which provides an open source toolkit for training, testing, tuning, and deploying deep learning models. In one embodiment according to the invention, Softmax with loss in Caffe is selected for supervised learning by classification.
On the other hand, for a second label (soft target), regression supervised training is performed on the second neural network model through a second loss function. The second loss function is, for example, Euclidean loss in Caffe, which is used to learn the output vector that fits the softtarget. It should be noted that when training the second neural network here, the output of the forward network is scaled down by 1/T because of the existence of the scaling layer.
And finally, training by combining the first loss function and the second loss function to obtain a third neural network model. According to one embodiment of the invention, different weights are respectively set for the first loss function and the second loss function, the first loss function and the second loss function are weighted to obtain a final loss function for training the third neural network model, and the final loss function is used for training the third neural network model.
Then, in step S250, the scaling layer in the third neural network model is deleted to obtain the neural network model after the anti-distillation. That is, the distillation temperature T of the scaling layer in the third neural network model trained in step S240 is set to 1 (i.e., the scaling process is cancelled), so as to obtain the neural network model after the antagonistic distillation.
According to the embodiment of the invention, the scaling layer is added in the original neural network model, the neural network model is distilled without changing the characteristic layer structure in the neural network model, and the error rate of the neural network model in response to the countermeasure sample is effectively reduced; and the distilled probability vector (namely the second label) and the first label of the probability vector are used for simultaneously supervising the training of the neural network model, so that the generalization capability of the neural network model is improved.
To further illustrate the method 200, the following describes a specific implementation of the method 200 by taking the chin classification in the facial feature classification as an example.
In the first step, a traditional VGG-Face network is selected as a forward network, and the distillation temperature is taken as T-20. Adding a scaling layer between the forward network and the softmax layer, performing 1/20 reduction on the output of the forward network (namely, the output of the last fully-connected layer in the forward network), and inputting the reduced output into the softmax layer, wherein the neural network model added with the scaling layer is used as a first neural network model.
VGGNet is a deep convolutional neural network developed by the computer vision Group (Visual Geometry Group) at oxford university and researchers at google deep mind corporation, and is often used to extract image features, and VGG-Face is a network used for Face recognition in one of the VGG groups. The method comprises the steps of constructing a convolutional neural network with 16-19 layers of depths by repeatedly stacking a small convolutional kernel with the depth of 3 multiplied by 3 and a maximum pooling layer with the depth of 2 multiplied by 2 according to the relation between the depth and the performance of the convolutional neural network, wherein the whole network is simple in structure, and the convolutional kernel size (3 multiplied by 3) and the maximum pooling size (2 multiplied by 2) with the same size are used. For more details on VGGNet, see the paper: the network structure of the VERY DEEP CONVOLUTIONAL NETWORKS FORLARGE-SCALE IMAGE RECOGTION will not be described in more detail herein.
And secondly, training on the training sample (namely the training image) labeled by the first label by using the first loss function, and taking the trained neural network model as a second neural network model. The first label is the label of the training image itself. Wherein the first loss function (softmax loss) is defined as:
loss1＝-logf(zk)
wherein the content of the first and second substances,
in the above formula, loss1For the first loss function value, N is the batch size (i.e. batch _ size) during training, and in colloquial terms, N can be understood as the number of input samples, z, of the first neural network model during a forward propagation processkIs the output of the kth neuron of the fully connected layer in the forward network.
And thirdly, inputting the original training images into a second neural network model to obtain a probability vector output by each training image in the second neural network model (namely, the distilled model) as a second label. For example, the second label of a training image is [0.93,0.02,0.05], and the three probability values respectively correspond to the probabilities that the chin in the image is a square chin, a sharp chin, and a round chin.
And fourthly, simultaneously supervising the training of the second neural network model by using the first label and the second label, and taking the trained neural network model as a third neural network model.
As described previously, according to an embodiment of the present invention, for the first label, the first loss function (softmax loss) in the second step is still adopted to perform classification supervision on the second neural network model; for the second label, the second loss function selects a euclidean loss function (eutlidean loss) to perform regression supervision on the second neural network model. Wherein the second loss function (i.e., eutlidean loss) is defined as:
in the above formula, loss2For the second loss function value, M is the total number of classes (i.e., dimensions of the feature) of the classification output by the softmax layer, x1iProbability vector, x, output for the ith class of the current network2iIs the probability vector corresponding to the ith class characterized by the second label.
Then, different weights are given to the two loss functions, and the final loss function for training the third neural network model is finally defined as:
loss＝w1×loss1+w2×loss2
where loss is the final loss function value, w1And w2Weighting factors representing the first loss function value and the second loss function value, respectively. w is a1And w2The value of (b) depends on the training situation, and the embodiment of the present invention does not limit this.
And fifthly, deleting a scaling layer in the trained third neural network model, namely setting the distillation temperature T to be 1, and using the obtained neural network model as the neural network model after the anti-distillation.
According to the method for resisting the distillation neural network model, the error rate of the neural network model in response to the resisting sample is effectively reduced in a mode of distilling the neural network model, and meanwhile, the obtained classification plane in the classification problem is more reasonable and the classification result is more accurate. Moreover, the scheme of the invention does not need to construct a countermeasure sample, thereby well improving the safety of the neural network model.
It should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in a device as described in this embodiment or alternatively may be located in one or more devices different from the devices in this example. The modules in the foregoing examples may be combined into one module or may be further divided into multiple sub-modules.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.
In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to perform the method of the present invention according to instructions in the program code stored in the memory.
By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer-readable media includes both computer storage media and communication media. Computer storage media store information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of computer readable media.
Furthermore, some of the described embodiments are described herein as a method or combination of method elements that can be performed by a processor of a computer system or by other means of performing the described functions. A processor having the necessary instructions for carrying out the method or method elements thus forms a means for carrying out the method or method elements. Further, the elements of the apparatus embodiments described herein are examples of the following apparatus: the apparatus is used to implement the functions performed by the elements for the purpose of carrying out the invention.
As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed in an illustrative rather than a restrictive sense, and the scope of the present invention is defined by the appended claims.
Claims (9)
1. A method of countering a distillation neural network model, the method being adapted for classifying a chin in the five sense organs of a human face, wherein the neural network model comprises a forward network having a feature layer structure, the forward network being for use in face recognition, and a softmax layer that outputs multi-class lower probability vectors, the method being adapted for execution in a computing device, the method comprising the steps of:
adding a scaling layer between a forward network and a softmax layer of the original neural network model according to the distillation temperature to generate a first neural network model;
training the first neural network model by utilizing a first label of a training image to obtain a second neural network model;
inputting a training image into the second neural network model, and outputting a second label representing a probability vector of the training image under multi-classification through a softmax layer, wherein the second label represents the probability that the chin in the training image is square chin, sharp chin and round chin;
utilizing the second label and the first label to simultaneously constrain and train the second neural network model to obtain a third neural network model, wherein the method comprises the following steps: carrying out classification supervision training on the second neural network model by using the first label through a first loss function, carrying out regression supervision training on the second neural network model by using the second label through a second loss function, and obtaining a third neural network model by combining the training of the first loss function and the training of the second loss function; and
and deleting the scaling layer in the third neural network model to obtain the neural network model after the resistance distillation.
2. The method of claim 1, wherein the scaling layer is adapted to scale down the input to the softmax layer according to the distillation temperature.
3. The method of claim 1 or 2, wherein the step of training the first neural network model with the first labels of the training images themselves to obtain the second neural network model comprises:
and monitoring the training of the first neural network model by using the first label through a first loss function to obtain a second neural network model.
4. The method of claim 3, wherein training the third neural network model in combination with the first and second loss functions comprises:
weighting the first loss function and the second loss function to obtain a final loss function for training a third neural network model; and
a third neural network model is trained using the final loss function.
5. The method of claim 4, wherein the first loss function is:
loss1＝-log f(zk)
wherein the content of the first and second substances,
in the formula, loss1Is the first loss function value, N is the batch size at training, zkIs the output of the kth neuron of the fully connected layer in the forward network.
6. The method of claim 5, wherein the second loss function is:
in the formula, loss2For the second loss function value, M is the total number of classes of classification output by the softmax layer, x1iProbability vector, x, output for the ith class of the current network2iIs the probability vector corresponding to the ith class characterized by the second label.
7. The method of claim 6, wherein the final loss function for training the third neural network model is defined as:
loss＝w1×loss1+w2×loss2
where loss is the final loss function value, w1And w2Weighting factors representing the first loss function value and the second loss function value, respectively.
8. A computing device, comprising:
one or more processors; and
a memory;
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the methods of claims 1-7.
9. A computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform any of the methods of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711179045.XA CN107977707B (en) | 2017-11-23 | 2017-11-23 | Method and computing equipment for resisting distillation neural network model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711179045.XA CN107977707B (en) | 2017-11-23 | 2017-11-23 | Method and computing equipment for resisting distillation neural network model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107977707A CN107977707A (en) | 2018-05-01 |
| CN107977707B true CN107977707B (en) | 2020-11-06 |
Family
ID=62011190
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711179045.XA Active CN107977707B (en) | 2017-11-23 | 2017-11-23 | Method and computing equipment for resisting distillation neural network model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107977707B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241988A (en) * | 2018-07-16 | 2019-01-18 | 北京市商汤科技开发有限公司 | Feature extracting method and device, electronic equipment, storage medium, program product |
| WO2020062262A1 (en) | 2018-09-30 | 2020-04-02 | Shanghai United Imaging Healthcare Co., Ltd. | Systems and methods for generating a neural network model for image processing |
| CN111105008A (en) * | 2018-10-29 | 2020-05-05 | 富士通株式会社 | Model training method, data recognition method and data recognition device |
| CN109886160B (en) * | 2019-01-30 | 2021-03-09 | 浙江工商大学 | Face recognition method under non-limited condition |
| WO2020161797A1 (en) * | 2019-02-05 | 2020-08-13 | 日本電気株式会社 | Learning device, learning method, and program |
| CN109961442B (en) * | 2019-03-25 | 2022-11-18 | 腾讯科技（深圳）有限公司 | Training method and device of neural network model and electronic equipment |
| CN110427466B (en) * | 2019-06-12 | 2023-05-26 | 创新先进技术有限公司 | Training method and device for neural network model for question-answer matching |
| CN110245662B (en) * | 2019-06-18 | 2021-08-10 | 腾讯科技（深圳）有限公司 | Detection model training method and device, computer equipment and storage medium |
| US11443069B2 (en) | 2019-09-03 | 2022-09-13 | International Business Machines Corporation | Root cause analysis of vulnerability of neural networks to adversarial examples |
| CN111079574B (en) * | 2019-11-29 | 2022-08-02 | 支付宝(杭州)信息技术有限公司 | Method and system for training neural network |
| CN111027060B (en) * | 2019-12-17 | 2022-04-29 | 电子科技大学 | Knowledge distillation-based neural network black box attack type defense method |
| CN111832701B (en) * | 2020-06-09 | 2023-09-22 | 北京百度网讯科技有限公司 | Model distillation method, model distillation device, electronic equipment and storage medium |
| CN112561076B (en) * | 2020-12-10 | 2022-09-20 | 支付宝(杭州)信息技术有限公司 | Model processing method and device |
| CN112820313B (en) * | 2020-12-31 | 2022-11-01 | 北京声智科技有限公司 | Model training method, voice separation method and device and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101847019A (en) * | 2009-03-23 | 2010-09-29 | 上海都峰智能科技有限公司 | Multichannel temperature controller |
| CN102626557A (en) * | 2012-04-13 | 2012-08-08 | 长春工业大学 | Molecular distillation process parameter optimizing method based on GA-BP (Genetic Algorithm-Back Propagation) algorithm |
| CN105069212A (en) * | 2015-07-30 | 2015-11-18 | 南通航运职业技术学院 | Ballast water microbe quantity prediction method based on artificial neural network |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120066163A1 (en) * | 2010-09-13 | 2012-03-15 | Nottingham Trent University | Time to event data analysis method and system |
-
2017
- 2017-11-23 CN CN201711179045.XA patent/CN107977707B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101847019A (en) * | 2009-03-23 | 2010-09-29 | 上海都峰智能科技有限公司 | Multichannel temperature controller |
| CN102626557A (en) * | 2012-04-13 | 2012-08-08 | 长春工业大学 | Molecular distillation process parameter optimizing method based on GA-BP (Genetic Algorithm-Back Propagation) algorithm |
| CN105069212A (en) * | 2015-07-30 | 2015-11-18 | 南通航运职业技术学院 | Ballast water microbe quantity prediction method based on artificial neural network |
Non-Patent Citations (3)
| Title |
|---|
| Distilling the Knowledge in a Neural Network;Geoffrey Hinton等;《arXiv:1503.02531v1 [stat.ML]》;20150309;第1节第4-5段、第2节第1-3段 * |
| Model Compression;Cristian Bucil˘a等;《In Proceedings of the 12th ACMSIGKDD International Conference on Knowledge Discovery and Data Mining,KDD》;20061231;第535-541页 * |
| 集总动力学-BP神经网络混合模型用于预测延迟;杨文剑等;《石油炼制与化工》;20150731;第46卷(第7期);第101-106页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107977707A (en) | 2018-05-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107977707B (en) | Method and computing equipment for resisting distillation neural network model | |
| Wu et al. | Object detection based on RGC mask R‐CNN | |
| WO2019100724A1 (en) | Method and device for training multi-label classification model | |
| WO2021114832A1 (en) | Sample image data enhancement method, apparatus, electronic device, and storage medium | |
| WO2019100723A1 (en) | Method and device for training multi-label classification model | |
| CN111191526B (en) | Pedestrian attribute recognition network training method, system, medium and terminal | |
| CN109902716B (en) | Training method for alignment classification model and image classification method | |
| CN107808147B (en) | Face confidence discrimination method based on real-time face point tracking | |
| WO2021143267A1 (en) | Image detection-based fine-grained classification model processing method, and related devices | |
| CN112016543A (en) | Text recognition network, neural network training method and related equipment | |
| CN113326930B (en) | Data processing method, neural network training method, related device and equipment | |
| CN110287775B (en) | Palm image clipping method, palm image clipping device, computer equipment and storage medium | |
| US11334773B2 (en) | Task-based image masking | |
| CN111414946B (en) | Artificial intelligence-based medical image noise data identification method and related device | |
| CN112686234B (en) | Face image quality evaluation method, electronic device and storage medium | |
| CN113705769A (en) | Neural network training method and device | |
| CN111582267B (en) | Text detection method, computing device and readable storage medium | |
| CN113255557B (en) | Deep learning-based video crowd emotion analysis method and system | |
| CN109583367A (en) | Image text row detection method and device, storage medium and electronic equipment | |
| CN112115937A (en) | Target identification method and device, computer equipment and storage medium | |
| EP4343616A1 (en) | Image classification method, model training method, device, storage medium, and computer program | |
| CN113011532A (en) | Classification model training method and device, computing equipment and storage medium | |
| WO2022063076A1 (en) | Adversarial example identification method and apparatus | |
| CN114139564A (en) | Two-dimensional code detection method and device, terminal equipment and training method for detection network | |
| CN113139618A (en) | Robustness-enhanced classification method and device based on integrated defense |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |