CN111159776A - Self-adaptive neural network model verification method and system - Google Patents
Self-adaptive neural network model verification method and system Download PDFInfo
- Publication number
- CN111159776A CN111159776A CN201911343421.3A CN201911343421A CN111159776A CN 111159776 A CN111159776 A CN 111159776A CN 201911343421 A CN201911343421 A CN 201911343421A CN 111159776 A CN111159776 A CN 111159776A
- Authority
- CN
- China
- Prior art keywords
- neural network
- network model
- gene
- seed
- template
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Abstract
The invention discloses a method and a system for verifying a self-adaptive neural network model, belonging to the safety verification technology of a neural network model, aiming at solving the technical problems of avoiding the time consumption of verification before the model runs, detecting whether the model in a running state is falsified or not and ensuring that the reasoning time delay of the model is not obviously increased in the verification process, and the technical scheme is as follows: the method comprises the following specific steps: s1, constructing a neural network model gene template; the method comprises the following specific steps: s101, selecting a candidate layer of a gene module; s102, selecting the position of a candidate layer template; s2, verifying the neural network model based on the model gene template; the method comprises the following specific steps: s201, generating and comparing a seed sample with a seed gene; s202, verifying the running tense model. The system comprises a gene template construction unit and a neural network model verification unit; the gene template construction unit comprises a candidate layer selection module and a candidate layer template position selection module; the neural network model verification unit comprises a generation and comparison module and a verification module.
Description
Technical Field
The invention relates to a safety verification technology of a neural network model, mainly relates to automatic verification of whether the model is tampered in the deployment and operation processes of the neural network model, and is a safety verification technology for the neural network in an artificial intelligence system, in particular to a self-adaptive neural network model verification method and a self-adaptive neural network model verification system.
Background
Once the neural network model is modified, the accuracy of reasoning cannot be guaranteed, and in the neural network model in the security field, such as a face verification system, if the model is maliciously tampered, the whole system is at risk of being invaded or paralyzed. The purpose of cracking the system without modifying the system code can be achieved by modifying the parameters of the model file, and users or administrators are difficult to perceive the cracking mode, so the model file is also often used for attacking the application comprising the neural network model.
Signing the model file and verifying the model signature before running the model can reduce the possibility of tampering with the model. However, the method has the defect of slow verification process, particularly, the neural network model files are large, and the method is not suitable for signature verification for the application with high real-time requirement. On the other hand, the signature verification method can only verify the model file in non-runtime, and cannot verify whether the model is tampered in runtime.
After the neural network model training is completed, files such as model parameters, structure description and the like are fixed, so that the model input corresponds to the output of the middle layer and the output of the final output layer one by one, and if the model parameters are modified, the output of the middle layer and the output of the output layer are changed. Therefore, whether the model is tampered or not can be detected by verifying whether the output of the middle layer and the output layer corresponds to the input. Neural network models have been applied to various systems, but these models will result in inferential bias if tampered with by an attacker. Therefore, how to avoid time consumption of verification before model operation and detect whether the model in the operation state is tampered or not simultaneously is an urgent problem to be solved in the prior art, so that the reasoning time delay of the model can hardly be obviously increased in the verification process.
Disclosure of Invention
The technical task of the invention is to provide a self-adaptive neural network model verification method and a self-adaptive neural network model verification system, so as to solve the problems that the verification time consumption before the model operation is avoided, whether the model in the operation state is falsified or not can be detected, and the reasoning time delay of the model can not be increased obviously in the verification process.
The technical task of the invention is realized in the following way, and the method for verifying the self-adaptive neural network model specifically comprises the following steps:
s1, constructing a neural network model gene template; the method comprises the following specific steps:
s101, selecting a candidate layer of a gene module;
s102, selecting the position of a candidate layer template;
s2, verifying the neural network model based on the model gene template; the method comprises the following specific steps:
s201, generating and comparing a seed sample with a seed gene;
s202, verifying the running tense model.
Preferably, the step S101 of selecting candidate layers of gene modules includes the following steps:
s10101, setting the number M of candidate layers, wherein M is more than or equal to 1;
s10102, classifying the output layer of the neural network model into a candidate layer;
s10103, randomly selecting M-1 candidate layers from the convolution layers and the full connection layers which are left in the neural network model.
Preferably, when the candidate layer is a convolutional layer, the specific step of selecting the position of the template of the candidate layer is as follows:
s10201-01 convolution layer output data is a four-dimensional array, and the dimensions are the number of samples, the number of channels, the height and the width respectively;
s10201-02, randomly selecting three channel indexes, randomly selecting starting positions of height and width, and setting the lengths of the width and the height dimensions as P, Q;
s10201-03, and the data blocks with width P and height Q on the three selected channels are the gene template positions of the convolutional layer.
Preferably, when the candidate layer is a full-connection layer, the specific step of selecting the position of the template of the candidate layer is as follows:
s10202-01, outputting data of a candidate layer to be a two-dimensional array, wherein the first dimension is the number of samples, and the second dimension is the number of neurons;
s10202-02, randomly selecting the positions of K neurons as the gene template positions of the candidate layer.
Preferably, the specific steps of generating and aligning the seed sample and the seed gene in step S201 are as follows:
s20101, randomly generating a seed sample according to the size requirement of the input layer of the neural network model, and inputting the seed sample into the neural network model;
s20102, outputting data corresponding to the positions on the gene template to the gene template in the neural network model reasoning process, and obtaining a seed gene after the gene template is filled with the data;
s20103, each sample has a corresponding seed gene, and seed gene comparison only needs to compare whether all data of corresponding positions are equal:
①, if all are equal, the seed genes are consistent, which indicates that the neural network model parameters and weights are not modified, because the same sample should output the same seed genes;
②, if not all are equal, the seed genes are not consistent, indicating that the neural network model is tampered.
Preferably, the data stored inside the seed gene is of the floating point number and/or integer type.
Preferably, the specific steps of verifying the runtime model in step S202 are as follows:
s20201, forming batch data by the sample to be inferred, the seed sample and the randomly generated shadow sample;
s20202, obtaining seed genes of the seed sample and the shadow sample during reasoning, and comparing whether the seed genes are consistent with the seed genes operated at the last time:
①, if the comparison of the seed genes is consistent, the neural network model passes the verification, and the step S20203 is executed;
②, if the contrast of the seed genes is not consistent, the neural network model is tampered, and the reasoning result is unreliable;
s20203, updating the seed sample into a shadow sample, and updating the seed gene into a shadow gene.
An adaptive neural network model verification system, the system comprising,
the neural network model gene template is used for recording which layers of the neural network model the gene comes from and the specific position of the output data of the layers, and determining the specific position of the output data, namely determining the gene template of the neural network model; the gene template construction unit comprises a gene template construction unit,
the candidate layer selecting module is used for selecting a candidate layer of the gene module;
the candidate layer template position selection module selects the candidate layer template position;
the neural network model verification unit is used for verifying the neural network model based on the model gene template; the neural network model verifying unit includes a neural network model verifying unit,
the generation and comparison module is used for generating and comparing the seed sample with the seed gene;
and the verification module is used for verifying the running temporal model.
The self-adaptive neural network model verification method and the self-adaptive neural network model verification system have the following advantages that:
the invention provides a model self-adaptive safety verification method according to the characteristics of a neural network model, and the consistency of a gene verification model generated in the model reasoning process by constructing a sample is verified; meanwhile, whether the model is modified or not can be dynamically verified under the condition that the running speed of the model is almost not influenced, and the verification process and the inference process are organically integrated, so that the consistency of the model during storage and release is ensured, and the consistency of the model during running can be ensured;
the invention provides a consistency verification technology of the artificial intelligent neural network model during release, deployment and operation, and prevents the model from being maliciously tampered; the method and the device not only solve the problem of time consumption of verification before the model runs, but also can detect whether the running-time model is tampered, and the reasoning time delay of the model can hardly be obviously increased in the verification process.
Drawings
The invention is further described below with reference to the accompanying drawings.
FIG. 1 is a schematic representation of a gene template for a construction model;
FIG. 2 is a flow chart of the operation of the adaptive neural network model verification system.
Detailed Description
An adaptive neural network model verification method and system according to the present invention will be described in detail below with reference to the drawings and specific embodiments.
Example 1:
the invention discloses a self-adaptive neural network model verification method, which comprises the following steps:
s1, constructing a neural network model gene template; the method comprises the following specific steps:
s101, selecting a candidate layer of a gene module; the method comprises the following specific steps:
s10101, setting the number M of candidate layers, wherein M is more than or equal to 1;
s10102, classifying the output layer of the neural network model into a candidate layer;
s10103, randomly selecting M-1 candidate layers from the convolution layers and the full connection layers which are left in the neural network model.
S102, selecting the position of a candidate layer template; when the candidate layer is a convolutional layer, the concrete steps of selecting the position of the template of the candidate layer are as follows:
s10201-01 convolution layer output data is a four-dimensional array, and the dimensions are the number of samples, the number of channels, the height and the width respectively;
s10201-02, randomly selecting three channel indexes, randomly selecting starting positions of height and width, and setting the lengths of the width and the height dimensions as P, Q;
s10201-03, and the data blocks with width P and height Q on the three selected channels are the gene template positions of the convolutional layer.
When the candidate layer is a full connection layer, the concrete steps of selecting the position of the template of the candidate layer are as follows:
s10202-01, outputting data of a candidate layer to be a two-dimensional array, wherein the first dimension is the number of samples, and the second dimension is the number of neurons;
s10202-02, randomly selecting the positions of K neurons as the gene template positions of the candidate layer.
S2, verifying the neural network model based on the model gene template; the method comprises the following specific steps:
s201, generating and comparing a seed sample with a seed gene; the method comprises the following specific steps:
s20101, randomly generating a seed sample according to the size requirement of the input layer of the neural network model, and inputting the seed sample into the neural network model;
s20102, outputting data corresponding to the positions on the gene template to the gene template in the neural network model reasoning process, and obtaining a seed gene after the gene template is filled with the data;
s20103, each sample has a corresponding seed gene, and seed gene comparison only needs to compare whether all data of corresponding positions are equal:
①, if all the seed genes are equal, the seed genes are consistent, which means that the neural network model parameters and weights are not modified because the same samples should output the same seed genes, wherein the data stored inside the seed genes are floating point numbers and/or integer types.
②, if not all are equal, the seed genes are not consistent, indicating that the neural network model is tampered.
S202, verifying the running tense model; the method comprises the following specific steps:
s20201, forming batch data by the sample to be inferred, the seed sample and the randomly generated shadow sample;
s20202, obtaining seed genes of the seed sample and the shadow sample during reasoning, and comparing whether the seed genes are consistent with the seed genes operated at the last time:
①, if the comparison of the seed genes is consistent, the neural network model passes the verification, and the step S20203 is executed;
②, if the contrast of the seed genes is not consistent, the neural network model is tampered, and the reasoning result is unreliable;
s20203, updating the seed sample into a shadow sample, and updating the seed gene into a shadow gene.
Example 2:
based on the specific implementation of example 1:
constructing a gene template of the model: as shown in figure 1, setting the number M of gene template candidate layers as 3, selecting a model output layer as a candidate layer, and randomly selecting 2 from the remaining convolutional layers and the full-link layers as candidate layers; according to the candidate layer template position selection method, for each layer in the candidate layers, randomly selecting the number of channels, the number of neurons, the initial position of data, the length of data and the like according to the type of the layer;
(II) randomly constructing a seed sample: feeding the model to obtain seed genes; packing the model file, the seed sample and the seed gene; sending the data to a platform needing to be deployed;
(III) verifying the seed sample and the seed gene by using a traditional signature verification mode: after signature verification passes, loading a model, feeding a seed sample to obtain deployed genes, and comparing the seed genes with the current genes:
①, if the gene data are consistent, it indicates that the parameters and structure of the model are not tampered, and the model enters an inference mode through verification;
②, if the verification is not passed, the operation is terminated;
(IV) randomly initializing a new sample called a shadow sample in the inference mode; the method comprises the following steps of (1) forming batch data by using a sample with inference, a seed sample and a shadow sample, feeding the batch data into a model together for inference to obtain gene data corresponding to the seed sample and the shadow sample, and comparing a seed gene obtained by the inference with a seed gene obtained in the last operation:
①, if the gene data are not consistent, the verification fails and the reasoning result is invalid;
②, if the gene data are consistent, the verification is successful, the inference result is valid, the seed sample is updated to the shadow sample of the inference, and the seed gene is updated to the shadow gene of the inference.
Example 3:
the adaptive neural network model verification system of the present invention, the system comprising,
the neural network model gene template is used for recording which layers of the neural network model the gene comes from and the specific position of the output data of the layers, and determining the specific position of the output data, namely determining the gene template of the neural network model; the gene template construction unit comprises a gene template construction unit,
the candidate layer selecting module is used for selecting a candidate layer of the gene module;
the candidate layer template position selection module selects the candidate layer template position;
the neural network model verification unit is used for verifying the neural network model based on the model gene template; the neural network model verifying unit includes a neural network model verifying unit,
the generation and comparison module is used for generating and comparing the seed sample with the seed gene;
and the verification module is used for verifying the running temporal model.
As shown in fig. 2, the system works as follows:
(1) determining a model gene template, constructing a seed sample, and feeding the seed sample into the model to obtain a seed gene;
(2) packing model files, seed samples and seed genes;
(3) the deployment model is used for carrying out signature verification on the seed sample and the seed gene;
(4) feeding a model seed sample to obtain a seed gene, and comparing whether the seed gene is consistent or not;
(5) constructing a shadow sample, and forming a batch data feeding model with the seed sample and the sample with inference;
(6) acquiring seed genes and shadow genes, and comparing whether the seed genes are consistent or not;
(7) and updating the seed sample and the seed gene, and outputting the reasoning result of the sample to be reasoned.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. A self-adaptive neural network model verification method is characterized by comprising the following steps:
s1, constructing a neural network model gene template; the method comprises the following specific steps:
s101, selecting a candidate layer of a gene module;
s102, selecting the position of a candidate layer template;
s2, verifying the neural network model based on the model gene template; the method comprises the following specific steps:
s201, generating and comparing a seed sample with a seed gene;
s202, verifying the running tense model.
2. The adaptive neural network model verification method of claim 1, wherein the specific steps of selecting the candidate layers of the gene modules in step S101 are as follows:
s10101, setting the number M of candidate layers, wherein M is more than or equal to 1;
s10102, classifying the output layer of the neural network model into a candidate layer;
s10103, randomly selecting M-1 candidate layers from the convolution layers and the full connection layers which are left in the neural network model.
3. The adaptive neural network model verification method of claim 1 or 2, wherein when the candidate layer is a convolutional layer, the specific steps of selecting the candidate layer template position are as follows:
s10201-01 convolution layer output data is a four-dimensional array, and the dimensions are the number of samples, the number of channels, the height and the width respectively;
s10201-02, randomly selecting three channel indexes, randomly selecting starting positions of height and width, and setting the lengths of the width and the height dimensions as P, Q;
s10201-03, and the data blocks with width P and height Q on the three selected channels are the gene template positions of the convolutional layer.
4. The adaptive neural network model verification method according to claim 1 or 2, wherein when the candidate layer is a fully connected layer, the specific step of selecting the position of the candidate layer template is as follows:
s10202-01, outputting data of a candidate layer to be a two-dimensional array, wherein the first dimension is the number of samples, and the second dimension is the number of neurons;
s10202-02, randomly selecting the positions of K neurons as the gene template positions of the candidate layer.
5. The adaptive neural network model verification method of claim 1, wherein the specific steps of generating and comparing the seed sample and the seed gene in step S201 are as follows:
s20101, randomly generating a seed sample according to the size requirement of the input layer of the neural network model, and inputting the seed sample into the neural network model;
s20102, outputting data corresponding to the positions on the gene template to the gene template in the neural network model reasoning process, and obtaining a seed gene after the gene template is filled with the data;
s20103, each sample has a corresponding seed gene, and seed gene comparison only needs to compare whether all data of corresponding positions are equal:
①, if all the seed genes are equal, the seed genes are consistent, which indicates that the neural network model parameters and the weights are not modified;
②, if not all are equal, the seed genes are not consistent, indicating that the neural network model is tampered.
6. The adaptive neural network model verification method of claim 5, wherein the data stored inside the seed gene is of floating point number and/or integer type.
7. The adaptive neural network model verification method of claim 1, wherein the specific steps of verifying the runtime model in step S202 are as follows:
s20201, forming batch data by the sample to be inferred, the seed sample and the randomly generated shadow sample;
s20202, obtaining seed genes of the seed sample and the shadow sample during reasoning, and comparing whether the seed genes are consistent with the seed genes operated at the last time:
①, if the comparison of the seed genes is consistent, the neural network model passes the verification, and the step S20203 is executed;
②, if the contrast of the seed genes is not consistent, the neural network model is tampered, and the reasoning result is unreliable;
s20203, updating the seed sample into a shadow sample, and updating the seed gene into a shadow gene.
8. An adaptive neural network model verification system, comprising,
the neural network model gene template is used for recording which layers of the neural network model the gene comes from and the specific position of the output data of the layers; the gene template construction unit comprises a gene template construction unit,
the candidate layer selecting module is used for selecting a candidate layer of the gene module;
the candidate layer template position selection module selects the candidate layer template position;
the neural network model verification unit is used for verifying the neural network model based on the model gene template; the neural network model verifying unit includes a neural network model verifying unit,
the generation and comparison module is used for generating and comparing the seed sample with the seed gene;
and the verification module is used for verifying the running temporal model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911343421.3A CN111159776A (en) | 2019-12-24 | 2019-12-24 | Self-adaptive neural network model verification method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911343421.3A CN111159776A (en) | 2019-12-24 | 2019-12-24 | Self-adaptive neural network model verification method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111159776A true CN111159776A (en) | 2020-05-15 |
Family
ID=70558251
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911343421.3A Pending CN111159776A (en) | 2019-12-24 | 2019-12-24 | Self-adaptive neural network model verification method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111159776A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023164868A1 (en) * | 2022-03-03 | 2023-09-07 | Huawei Technologies Co., Ltd. | Methods, devices and systems for trustworthiness certification of inference requests and inference responses |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107808098A (en) * | 2017-09-07 | 2018-03-16 | 阿里巴巴集团控股有限公司 | A kind of model safety detection method, device and electronic equipment |
| CN109684839A (en) * | 2018-12-19 | 2019-04-26 | Oppo广东移动通信有限公司 | Self-definition model tamper resistant method, device, terminal device and storage medium |
| CN110309911A (en) * | 2019-07-05 | 2019-10-08 | 北京中科寒武纪科技有限公司 | Neural network model verification method, device, computer equipment and storage medium |
-
2019
- 2019-12-24 CN CN201911343421.3A patent/CN111159776A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107808098A (en) * | 2017-09-07 | 2018-03-16 | 阿里巴巴集团控股有限公司 | A kind of model safety detection method, device and electronic equipment |
| CN109684839A (en) * | 2018-12-19 | 2019-04-26 | Oppo广东移动通信有限公司 | Self-definition model tamper resistant method, device, terminal device and storage medium |
| CN110309911A (en) * | 2019-07-05 | 2019-10-08 | 北京中科寒武纪科技有限公司 | Neural network model verification method, device, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| PU ZHAO等: "Fault Sneaking Attack: a Stealthy Framework for Misleading Deep Neural Networks", 《2019 56TH ACM/IEEE DESIGN AUTOMATION CONFERENCE(DAC)》 * |
| YANNAN LIU等: "Fault Injection Attack on Deep Neural Network", 《2017 IEEE/ACM INTERNATIONAL CONFERENCE ON COMPUTER-AIDED DESIGN (ICCAD)》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023164868A1 (en) * | 2022-03-03 | 2023-09-07 | Huawei Technologies Co., Ltd. | Methods, devices and systems for trustworthiness certification of inference requests and inference responses |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11341415B2 (en) | Method and apparatus for compressing neural network | |
| CN110956263A (en) | Construction method of binarization neural network, storage medium and terminal equipment | |
| CN111062036A (en) | Malicious software identification model construction method, malicious software identification medium and malicious software identification equipment | |
| CN111741002A (en) | Method and device for training network intrusion detection model | |
| US8581751B2 (en) | Multi-cell voltage secure data encoding in hybrid vehicles | |
| CN112163299B (en) | Method and system for predicting oil consumption of excavator and electronic equipment | |
| CN112364388A (en) | Sensor data authentication method and device based on block chain | |
| CN111159776A (en) | Self-adaptive neural network model verification method and system | |
| CN113761525A (en) | Intelligent intrusion detection method and system based on federal learning | |
| CN115423159A (en) | Photovoltaic power generation prediction method and device and terminal equipment | |
| CN115758379A (en) | Cross-language compiler vulnerability mining method and device based on transfer learning | |
| CN112764791B (en) | Incremental update malicious software detection method and system | |
| CN111737319B (en) | User cluster prediction method, device, computer equipment and storage medium | |
| CN111539029B (en) | Industrial internet-based big data storage rate optimization method and cloud computing center | |
| CN116416486A (en) | Image recognition method and system | |
| CN112148355B (en) | System differential upgrading method and device based on deep learning | |
| CN111033532B (en) | Training method and system for generating countermeasure network, electronic device and storage medium | |
| CN110889114B (en) | Software credibility measurement method and device based on concept lattice | |
| US20230316063A1 (en) | Neural network processing | |
| CN112463578B (en) | System test case generation method based on step-by-step solution of component contract | |
| CN117456286B (en) | Ginseng grading method, device and equipment | |
| CN113202647B (en) | Control method, device and terminal for output power of vehicle engine | |
| CN117081773A (en) | Intelligent networking terminal multi-attack intrusion detection method based on OTA signature update | |
| CN116258068A (en) | Transient stability evaluation method and device for power system and computer equipment | |
| CN117195284A (en) | Heterogeneous power terminal data protection method based on cross-validation blockchain |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200515 |