CN117932457B - Model fingerprint identification method and system based on error classification - Google Patents

Abstract

The present invention provides a model fingerprint recognition method and system based on error classification, and relates to the field of model copyright protection. The model fingerprint recognition based on error classification first searches for samples of models that are incorrectly classified by both the target model and the pirated model (modified model). Then, under the premise of not changing the parameters of the target model, a GAN network is used to enhance the classification features of these error samples to generate fingerprint samples and classify them correctly, while ensuring that the fingerprint samples are natural and have little difference from the original samples. Finally, the error samples and fingerprint samples are used as query sets, and the model ownership is verified by comparing the predicted labels of the error samples and the fingerprint samples. This method not only greatly enhances the concealment of the fingerprint samples, but also improves the robustness to attacks such as model fine-tuning, pruning and noise addition.

Description

A model fingerprint recognition method and system based on error classification

Technical Field

The present invention relates to the technical field of model copyright protection, and in particular to a model fingerprint recognition method and system based on error classification.

Background technique

With the rapid development of deep learning, deep neural networks have achieved great success in many artificial intelligence fields, such as image recognition, visual understanding, and natural language processing. Companies such as Microsoft, Google, and Baidu have deployed DL models in their commercial products to provide higher quality and intelligent services. Although deep neural networks are superior to traditional methods, designing and training a high-performance deep model is not a simple task. It usually requires large-scale labeled training data, a large amount of computing resources, and expertise to design an excellent framework and a suitable learning strategy. The development cost is not affordable for ordinary people. However, high-performance deep models are full of huge commercial value. Malicious users may use proxy attacks to steal models by accessing the API of the target model, or steal model structures and parameters and modify the model. Therefore, the intellectual property rights of model protection products need to be protected to prevent piracy.

Model watermarking is a common method for protecting model intellectual property rights. Watermark information is embedded into the model by modifying model parameters. However, existing research shows that watermark-based model protection methods inevitably affect model performance. However, in key fields such as medicine and finance, even a 1% loss in accuracy is intolerable, so researchers have proposed a model fingerprinting method. Model fingerprinting does not require modifying the model training process or fine-tuning model parameters, but protects model intellectual property rights by finding model-specific features. The model fingerprinting method first finds some samples at the classification boundary of the target model, and then generates these samples into fingerprint samples through methods such as adversarial samples. Finally, the fingerprint sample and its predicted label are used as the fingerprint of the target model. For a suspicious classifier, the model owner remotely accesses the API and inputs the fingerprint sample set to obtain its label. By comparing the predicted labels of the fingerprint sample by the suspicious classifier and the target classifier, the model owner verifies whether the suspicious classifier is pirated from the target classifier.

Although the existing model fingerprint recognition method based on classification boundaries has achieved intellectual property protection for the model, the samples on the decision boundary are not robust to model attacks. Moreover, the fingerprint samples generated by adversarial samples are unnatural and less concealed, making them easy to be detected.

Summary of the invention

1. Technical issues to be resolved

In view of the shortcomings of the prior art, the present invention provides a model fingerprint recognition method and system based on misclassification, which solves the problem that the model fingerprint recognition method based on classification boundaries can realize the intellectual property protection of the model, but the robustness of using samples on the decision boundary to model attacks is poor and not robust.

(II) Technical solution

To achieve the above objectives, the present invention is implemented through the following technical solutions:

In a first aspect, a model fingerprint recognition method based on error classification is provided, comprising:

Input a public dataset D _m , use the public dataset D _m to frequently access the target model to obtain the predicted label of the public dataset D _m , use the public dataset D _m with the label as the original training set D _train , and train the piracy model through the original training set D _train ;

Filter out samples Z that are misclassified by both the target model and the pirated model in the original training set D _train ;

Find the sample with the smallest cumulative distance from other samples of the same category in each category of the training set D _train as the centroid sample D _s ;

Select a batch of samples with the largest distance from the centroid sample from the misclassified samples Z as the error samples _Ze in the query set, and record the label set of the error samples _Ze ;

Input the error sample _Ze into the pre-built GAN network to guide the error sample _{Ze to} be correctly classified and generate a natural fingerprint sample _Zr ;

A batch of samples with the smallest distance to the centroid sample are selected from the fingerprint samples Z _r generated by the GAN network as the fingerprint samples Z _w in the query set, and the label set of the fingerprint samples is recorded;

The error samples _Ze and fingerprint samples _Zw are respectively input into the pre-built suspicious model to obtain the label set of the error samples and the label set of the fingerprint samples.

Preferably, the method of finding the sample with the smallest cumulative distance from other samples of the same category in each category of the training set D _train as the centroid sample D _s is as follows:

Where N represents the number of data in class k, Indicates the length of a vector.

Preferably, the step of inputting the error sample _Ze into a pre-built GAN network, guiding the error sample _{Ze to} be correctly classified, and generating a natural fingerprint sample _Zr specifically includes:

Input the error sample Z _e into the generator G in the GAN network to obtain the fingerprint sample Z _r , and input the fingerprint sample Z _r into the target model to guide its classification correct;

Using classification loss and identification loss/> A weighted combination of Train the GAN network, /> is a hyperparameter that balances the quality of error samples and fingerprint samples;

Input the fingerprint sample _Zr into the discriminator, and guide the generation of natural fingerprint samples by calculating the discrimination loss _Ld ;

Calculate total loss , back propagation is performed to minimize the total loss function L, and the parameters of the GAN network are iteratively updated to obtain natural fingerprint samples.

Preferably, the classification loss The formula is as follows:

in For the target model F, the fingerprint sample is SoftMax function, Y is the guide fingerprint sample/> Category tags, /> It's the Carlini-Wagner loss.

Preferably, the Carlini-Wagner loss , the formula is as follows:

Z is , the parameter k encourages the GAN network to generate high-confidence samples classified as class Y.

Preferably, the identification loss L _d is expressed as follows:

.

Preferably, the error sample Z _e and the fingerprint sample Z _w are respectively input into the pre-built suspicious model, and after obtaining the label set of the error sample and the label set of the fingerprint sample, it is determined whether E _i '=E _i and _Wi '=W _i are satisfied, wherein E _i ' represents the label set of the error sample, E _i represents the label set of the retained error sample, _Wi ' represents the label set of the fingerprint sample, and _{Wi represents} the label set of the retained fingerprint sample. The formula for calculating the matching rate S is as follows:

Match Rate If the value is greater than 95%, the suspicious model is considered a stolen model.

In a second aspect, a model fingerprint recognition system based on error classification is provided, comprising:

A preprocessing module is used to input a public data set D _m , frequently access a target model using the public data set D _m to obtain a predicted label of the public data set D _m , use the public data set D _m with the label as an original training set D _train , and train a piracy model through the original training set D _train ;

The first screening module is used to screen out samples Z that are misclassified by both the target model and the pirated model in the original training set D _train ;

An extraction module is used to find the sample with the smallest cumulative distance from other samples of the same category in each category of the training set D _train as the centroid sample D _s ;

A recording module is used to select a batch of samples with the largest distance from the centroid sample from the misclassified samples Z as the error samples Z _e in the query set, and record the label set of the error samples Z _e ;

The generation module is used to input the error sample _Ze into the pre-built GAN network, guide the error sample _Ze to be correctly classified, and generate a natural fingerprint sample _Zr ;

The second screening module is used to screen out a batch of samples with the smallest distance from the centroid sample from the fingerprint samples Z _r generated by the GAN network as the fingerprint samples Z _w in the query set, and record the label set of the fingerprint samples;

The processing and output module is used to input the error sample _Ze and the fingerprint sample _Zw into the pre-built suspicious model respectively to obtain the label set of the error sample and the label set of the fingerprint sample.

In a third aspect, a computer-readable storage medium storing one or more programs is provided, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform the method of the first aspect.

In a fourth aspect, a computing device is provided, including:

One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method of the first aspect.

(III) Beneficial effects

The present invention discloses a model fingerprint recognition method based on misclassification, which is different from other model fingerprint recognition methods based on classification boundaries. It first searches for error samples in the area where both the target model and the stolen model are misclassified, and enhances the classification features of the error samples through a GAN network to generate fingerprint samples with correct classification. The present invention uses a GAN network to generate fingerprint samples, which greatly enhances the concealment of the samples. At the same time, it simulates the modification that an attacker can make to the model, and uses the centroid sample to filter out the error samples and fingerprint samples, which greatly improves the robustness to various model attacks.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a model fingerprint recognition method based on error classification according to the present invention;

FIG2 is a schematic diagram of a process of training a piracy model in an embodiment of the present invention;

FIG3 is a schematic diagram of a process for screening erroneous samples in an embodiment of the present invention;

FIG4 is a schematic diagram of constructing a centroid sample in an embodiment of the present invention;

FIG5 is a structural diagram of a fingerprint sample generated by a GAN network according to an embodiment of the present invention;

FIG. 6 is a flow chart of verifying model ownership in an embodiment of the present invention.

Detailed ways

The following will be combined with the accompanying drawings of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Example

As shown in FIG1 , an embodiment of the present invention provides a model fingerprint recognition method based on error classification, comprising:

Input a public dataset D _m , use the public dataset D _m to frequently access the target model to obtain the predicted label of the public dataset D _m , use the public dataset D _m with the label as the original training set D _train , and train the piracy model through the original training set D _train ;

Filter out samples Z that are misclassified by both the target model and the pirated model in the original training set D _train ;

Find the sample with the smallest cumulative distance from other samples of the same category in each category of the training set D _train as the centroid sample D _s ;

Select a batch of samples with the largest distance from the centroid sample from the misclassified samples Z as the error samples _Ze in the query set, and record the label set of the error samples _Ze ;

Input the error sample _Ze into the pre-built GAN network to guide the error sample _{Ze to} be correctly classified and generate a natural fingerprint sample _Zr ;

A batch of samples with the smallest distance to the centroid sample are selected from the fingerprint samples Z _r generated by the GAN network as the fingerprint samples Z _w in the query set, and the label set of the fingerprint samples is recorded;

The error samples _Ze and fingerprint samples _Zw are respectively input into the pre-built suspicious model to obtain the label set of the error samples and the label set of the fingerprint samples.

Specifically, the process of training the pirated model is described in detail in conjunction with Figure 2. According to the different permissions of the attacker to access the model, there are two ways to train the pirated model. One is to access the target model to obtain the label of the data set, and use the labeled data set to train a proxy model with similar functions to the target model to steal the model. The other is to directly modify the target model by means of model fine-tuning, pruning, and noise addition. The structure of the target model is WideResNet, and the structure of the proxy model is PreacTresnet.

The process of selecting the wrong samples is described in detail below in conjunction with FIG3. After the training of the pirated model is completed, the training set D _train is used to screen out the samples Z that are incorrectly classified by both the target model and the pirated model. CIFAR-10 is selected as the training set D _train . The data set consists of 60,000 color images, including 50,000 training images (a total of 10 classes, 5,000 images for each class) and 10,000 test images (a total of 10 classes, 1,000 images for each class).

Furthermore, since the attack model has complex and diverse means, it is necessary to further screen out the sample _Ze with the largest distance from the centroid sample to enhance the robustness of the fingerprint. Figure 4 is a schematic diagram of constructing the centroid sample. The centroid sample is the sample closest to the center of the decision range of a class in the training set. Find the sample with the smallest cumulative distance to other samples of the same class in each class of the training set D _train as the centroid sample D _s , and the formula is as follows:

Where N represents the number of data in class k, Indicates the length of a vector.

Furthermore, the process of generating fingerprint samples by the GAN network is described in detail below in conjunction with FIG5. The error sample _Ze is input into the pre-built GAN network to guide the error sample _{Ze to} be correctly classified and generate a natural fingerprint sample _Zr , which specifically includes:

Input the error sample Z _e into the generator G in the GAN network to obtain the fingerprint sample Z _r , and input the fingerprint sample Z _r into the target model to guide its classification correct;

Using classification loss and identification loss/> A weighted combination of Train the GAN network, /> is a hyperparameter that balances the quality of error samples and fingerprint samples;

Input the fingerprint sample _Zr into the discriminator, and guide the generation of natural fingerprint samples by calculating the discrimination loss _Ld ;

Calculate total loss , back propagation is performed to minimize the total loss function L, and the parameters of the GAN network are iteratively updated to obtain natural fingerprint samples.

Furthermore, the classification loss The formula is as follows:

in For the target model F, the fingerprint sample is SoftMax function, Y is the guide fingerprint sample/> Category tags, /> It's the Carlini-Wagner loss.

Furthermore, the Carlini-Wagner loss , the formula is as follows:

Z is , the parameter k encourages the GAN network to generate high-confidence samples classified as class Y.

Furthermore, the identification loss L _d is given by the following formula:

.

Furthermore, the following is a detailed description of the implementation process of verifying model ownership in conjunction with FIG6: the error sample _Ze and the fingerprint sample _Zw are respectively input into the pre-built suspicious model, and after obtaining the label set of the error sample and the label set of the fingerprint sample, it is determined whether _Ei '= _Ei and _Wi '= _Wi are satisfied, where _Ei ' represents the label set of the error sample, _Ei represents the label set of the retained error sample, _Wi ' represents the label set of the fingerprint sample, and _{Wi represents} the label set of the retained fingerprint sample. The formula for calculating the matching rate S is as follows:

Match Rate If the value is greater than 95%, the suspicious model is considered a stolen model.

Yet another embodiment of the present invention provides a model fingerprint recognition system based on error classification, comprising:

A preprocessing module is used to input a public data set D _m , frequently access a target model using the public data set D _m to obtain a predicted label of the public data set D _m , use the public data set D _m with the label as an original training set D _train , and train a piracy model through the original training set D _train ;

The first screening module is used to screen out samples Z that are misclassified by both the target model and the pirated model in the original training set D _train ;

An extraction module is used to find the sample with the smallest cumulative distance from other samples of the same category in each category of the training set D _train as the centroid sample D _s ;

A recording module is used to select a batch of samples with the largest distance from the centroid sample from the misclassified samples Z as the error samples Z _e in the query set, and record the label set of the error samples Z _e ;

The generation module is used to input the error sample _Ze into the pre-built GAN network, guide the error sample _Ze to be correctly classified, and generate a natural fingerprint sample _Zr ;

The second screening module is used to screen out a batch of samples with the smallest distance from the centroid sample from the fingerprint samples Z _r generated by the GAN network as the fingerprint samples Z _w in the query set, and record the label set of the fingerprint samples;

The processing and output module is used to input the error sample _Ze and the fingerprint sample _Zw into the pre-built suspicious model respectively to obtain the label set of the error sample and the label set of the fingerprint sample.

The embodiments of the present application may be provided as methods or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present application may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

The present application is described with reference to the flowcharts and/or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the processes and/or boxes in the flowchart and/or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims (9)

1. The model fingerprint identification method based on the error classification is characterized by comprising the following steps of:

Inputting a public data set D _m, using the public data set D _m to access the target model frequently to obtain a predictive tag of the public data set D _m, taking the public data set D _m with the tag as a primary training set D _train, and training a piracy model through the primary training set D _train;

Screening samples Z with class errors equally classified by the target model and the pirate model from the original training set D _train;

Finding out a sample with the smallest accumulated distance with other samples of the same class in each class of the training set D _train as a centroid sample D _s;

Screening a batch of samples with the largest distance from the centroid sample from the samples Z with the classification errors as error samples Z _e in the query set, and recording a label set of the error samples Z _e;

Inputting the error sample Z _e into a pre-constructed GAN network, guiding the error sample Z _e to be correctly classified, and generating a natural fingerprint sample Z _r;

Screening a batch of samples with the smallest distance from a mass center sample from fingerprint samples Z _r generated by the GAN network as fingerprint samples Z _w in a query set, and recording a tag set of the fingerprint samples;

respectively inputting the error sample Z _e and the fingerprint sample Z _w into a pre-constructed suspicious model to obtain a label set of the error sample and a label set of the fingerprint sample;

After the error sample Z _e and the fingerprint sample Z _w are respectively input into a pre-constructed suspicious model to obtain a tag set of the error sample and a tag set of the fingerprint sample, judging whether E _i'=E_i and W _i'=W_i are satisfied, wherein E _i 'represents the tag set of the error sample, E _i represents the tag set of the reserved error sample, W _i' represents the tag set of the fingerprint sample, W _i retains the tag set of the fingerprint sample, and the calculation matching rate S is as follows:

Matching rate Greater than 95%, the suspicious model is considered a piracy model.

2. The error classification-based model fingerprint identification method as claimed in claim 1, wherein: the sample with the smallest accumulated distance with other samples of the same class in each class of the training set D _train is found out as a centroid sample D _s, and the formula is as follows:

where N represents the number of data in k classes, Representing the length of the vector.

3. The error classification-based model fingerprint identification method as claimed in claim 1, wherein: the inputting the error sample Z _e into the pre-constructed GAN network, guiding the error sample Z _e to be correctly classified, and generating a natural fingerprint sample Z _r, which specifically includes:

Inputting error sample Z _e into generator G in GAN network to obtain fingerprint sample Z _r, inputting fingerprint sample Z _r into target model, and guiding classification thereof Correct;

Using classification losses And identifying loss/>Weighted combination/>Training the GAN network,/>Is a super parameter that balances the quality of the erroneous sample and the fingerprint sample;

Inputting the fingerprint sample Z _r into a discriminator, and guiding to generate a natural fingerprint sample by calculating discrimination loss L _d;

Calculation of total loss The back propagation is performed to minimize the total loss function L, and the parameters of the GAN network are iteratively updated to obtain natural fingerprint samples.

4. A model fingerprint recognition method based on error classification as claimed in claim 3, wherein: said classification lossThe formula is as follows:

Wherein the method comprises the steps of Fingerprint sample/>, for object model FIs the guided fingerprint sample/>Classified tag,/>Is Carlini-Wagner loss.

5. The error classification based model fingerprint identification method of claim 4, wherein: the Carlini-Wagner lossThe formula is as follows:

Z is The parameter k encourages the GAN network to generate high confidence samples that are classified as class Y.

6. The error classification based model fingerprint identification method according to claim 5, wherein: the discrimination loss L _d is as follows:

。

7. A model fingerprint recognition system based on error classification, comprising:

The preprocessing module is used for inputting a public data set D _m, frequently accessing a target model by using the public data set D _m to obtain a predictive tag of the public data set D _m, taking the public data set D _m with the tag as a primary training set D _train, and training a piracy model by the primary training set D _train;

The first screening module is used for screening samples Z with class errors equally classified by the target model and the pirate model from the original training set D _train;

The extraction module is used for finding out a sample with the smallest accumulated distance with other samples of the same class in each class of the training set D _train to be used as a centroid sample D _s;

The recording module is used for screening a batch of samples with the largest distance from the mass center sample from the samples Z with the classification errors as error samples Z _e in the query set, and recording a label set of the error samples Z _e;

The generation module is used for inputting the error sample Z _e into a pre-constructed GAN network, guiding the error sample Z _e to be correctly classified, and generating a natural fingerprint sample Z _r;

The second screening module is used for screening a batch of samples with the smallest distance from the centroid sample from fingerprint samples Z _r generated by the GAN network as fingerprint samples Z _w in the query set, and recording a tag set of the fingerprint samples;

The processing and outputting module is used for respectively inputting the error sample Z _e and the fingerprint sample Z _w into a pre-constructed suspicious model to obtain a label set of the error sample and a label set of the fingerprint sample;

After the error sample Z _e and the fingerprint sample Z _w are respectively input into a pre-constructed suspicious model to obtain a tag set of the error sample and a tag set of the fingerprint sample, judging whether E _i'=E_i and W _i'=W_i are satisfied, wherein E _i 'represents the tag set of the error sample, E _i represents the tag set of the reserved error sample, W _i' represents the tag set of the fingerprint sample, W _i retains the tag set of the fingerprint sample, and the calculation matching rate S is as follows:

Matching rate Greater than 95%, the suspicious model is considered a piracy model.

8. A computer readable storage medium storing one or more programs, wherein the one or more programs comprise instructions, which when executed by a computing device, cause the computing device to perform any of the methods of claims 1-6.

9. A computing device, comprising:

One or more processors, memory, and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the methods of claims 1-6.