CN113297547B - Back door watermark adding method, verification method and system for data set - Google Patents
Back door watermark adding method, verification method and system for data set Download PDFInfo
- Publication number
- CN113297547B CN113297547B CN202110564333.7A CN202110564333A CN113297547B CN 113297547 B CN113297547 B CN 113297547B CN 202110564333 A CN202110564333 A CN 202110564333A CN 113297547 B CN113297547 B CN 113297547B
- Authority
- CN
- China
- Prior art keywords
- trained
- samples
- test set
- original
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000012795 verification Methods 0.000 title claims abstract description 24
- 238000012360 testing method Methods 0.000 claims description 132
- 238000012549 training Methods 0.000 claims description 91
- 230000000694 effects Effects 0.000 claims description 12
- 238000013473 artificial intelligence Methods 0.000 abstract description 5
- 238000002474 experimental method Methods 0.000 description 9
- 230000000007 visual effect Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 238000002347 injection Methods 0.000 description 6
- 239000007924 injection Substances 0.000 description 6
- 239000011159 matrix material Substances 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000009466 transformation Effects 0.000 description 4
- 230000004931 aggregating effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 239000000243 solution Substances 0.000 description 2
- 206010000117 Abnormal behaviour Diseases 0.000 description 1
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/10—Protecting distributed programs or content, e.g. vending or licensing of copyrighted material ; Digital rights management [DRM]
- G06F21/16—Program or content traceability, e.g. by watermarking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/0021—Image watermarking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2201/00—General purpose image data processing
- G06T2201/005—Image watermarking
Abstract
The invention relates to a method for adding a back door watermark of a data set, a verification method and a system, belonging to the field of artificial intelligence safety.
Description
Technical Field
The invention relates to the field of artificial intelligence security, in particular to a method and a system for adding and verifying a backdoor watermark of a data set.
Background
With the development of science and technology, the artificial intelligence technology becomes a key factor for promoting the progress of human society, and a large number of expert and scholars worldwide are transferred into the field of artificial intelligence, so that the technology is greatly developed and advanced. Deep learning is a core technology of artificial intelligence, and is widely applied in many fields, such as medical diagnosis, industrial control, financial analysis, computer vision and the like.
Deep neural networks are suitable for most tasks, but their effectiveness relies on a large number of training samples. Many open-source datasets on the network are open, but are only allowed for academic educational purposes and not for business purposes; there are also many enterprise or personal data sets that are only allowed for internal use due to privacy or confidentiality concerns. If a third party steals the data set to train the model, the owner of the data set cannot judge the model, and therefore a great privacy threat can be caused. There is a need to provide a method to verify that a data set is misappropriated to train third party models.
Disclosure of Invention
The invention aims to provide a backdoor watermark adding method, a verification method and a system of a data set.
In order to achieve the purpose, the invention provides the following scheme:
a method of back-door watermarking of a data set, the method comprising:
and performing hyper-resolution reconstruction on the sample of the data set, and modifying the label of the sample into a specified category to obtain the watermark data set.
The invention also provides a verification method for data set embezzlement, which comprises the following steps:
dividing a data set to be verified into an original training set and an original testing set;
performing hyper-resolution reconstruction on part of samples of the original training set, and modifying labels of the part of samples into specified categories to obtain watermark samples; forming a back door training set by the watermark sample and the residual part of sample, and forming the original training set by the part of sample and the residual part of sample;
performing hyper-resolution reconstruction on the original test set sample, and modifying the label of the original test set sample into a specified category to obtain a back door test set;
training the ResNet-34 classification network by adopting the back door training set to obtain the trained ResNet-34 classification network;
and inputting the back door test set into the trained ResNet-34 classification network to obtain the prediction category of the sample of the back door test set, and judging whether the data set to be verified is stolen according to the prediction category and the specified category.
The invention also provides a verification method for the influence of the data set, which is used for verifying the influence of the verification method for the stealing of the data set on the data set to be verified, and the method comprises the following steps:
dividing a data set to be verified into an original training set and an original testing set;
training a baseline network by adopting the original training set to obtain a trained baseline network;
inputting the original test set into the trained baseline network to obtain the prediction category of the trained baseline network to the samples of the original test set;
comparing the prediction category of the sample of the original test set by the trained base line network with the category to which the sample of the original test set belongs to obtain the accuracy rate of predicting the category of the sample of the original test set by the trained base line network;
performing hyper-resolution reconstruction on part of samples of the original training set, and modifying labels of the part of samples into specified categories to obtain watermark samples; forming a back door training set by the watermark sample and the residual part of sample, and forming the original training set by the part of sample and the residual part of sample;
training the ResNet-34 classification network by adopting the back door training set to obtain the trained ResNet-34 classification network;
inputting the original test set into the trained ResNet-34 classification network to obtain the prediction category of the trained ResNet-34 classification network on the original test set;
comparing the prediction category of the original test set by the trained ResNet-34 classification network with the category to which the sample of the original test set belongs to obtain the accuracy of the category prediction of the sample of the original test set by the trained ResNet-34 classification network;
and judging whether the back door watermark influences the normal use of the data set to be verified or not according to the class prediction accuracy of the trained base line network on the samples of the original test set and the class prediction accuracy of the trained ResNet-34 classification network on the original test set.
The present embodiment further provides a verification system for data set theft, where the system includes:
the dividing module is used for dividing the data set to be verified into an original training set and an original testing set;
the back door training set acquisition module is used for performing super-resolution reconstruction on part of samples of the original training set, modifying the labels of the part of samples into specified categories and acquiring watermark samples; forming a back door training set by the watermark sample and the residual part of sample, and forming the original training set by the part of sample and the residual part of sample;
the back door test set acquisition module is used for performing super-resolution reconstruction on the original test set sample, modifying the label of the original test set sample into a specified category and acquiring a back door test set;
the trained ResNet-34 classification network acquisition module is used for training the ResNet-34 classification network by adopting the back door training set to obtain the trained ResNet-34 classification network;
and the first verification module is used for inputting the back door test set into the trained ResNet-34 classification network, obtaining the prediction category of the sample of the back door test set, and judging whether the data set to be verified is stolen or not according to the prediction category and the specified category.
The invention also provides a verification system for data set influence, which is used for verifying the influence of a verification method for data set embezzlement on a data set to be verified, and comprises the following steps:
the dividing module is used for dividing the data set to be verified into an original training set and an original testing set;
the trained baseline network acquisition module is used for adopting the original training set to train a baseline network to obtain the trained baseline network;
a first prediction category module, configured to input the original test set into the trained baseline network, and obtain a prediction category of the trained baseline network for a sample of the original test set;
the first class prediction accuracy module is used for comparing the prediction class of the trained basic-line network on the samples of the original test set with the class of the samples of the original test set to obtain the class prediction accuracy of the trained basic-line network on the samples of the original test set;
the back door training set acquisition module is used for performing super-resolution reconstruction on part of samples of the original training set, modifying the labels of the part of samples into specified categories and acquiring watermark samples; forming a back door training set by the watermark sample and the residual part of sample, and forming the original training set by the part of sample and the residual part of sample;
the trained ResNet-34 classification network acquisition module is used for training the ResNet-34 classification network by adopting the back door training set to obtain the trained ResNet-34 classification network;
a second prediction category module, configured to input the original test set into the trained ResNet-34 classification network, and obtain a prediction category of the trained ResNet-34 classification network for the original test set;
the second class prediction accuracy module is used for comparing the prediction class of the original test set with the class of the sample of the original test set by the trained ResNet-34 classification network to obtain the class prediction accuracy of the sample of the original test set by the trained ResNet-34 classification network;
and the second judging module is used for judging whether the backdoor watermark influences the normal use of the data set to be verified according to the class prediction accuracy of the trained base line network on the samples of the original test set and the class prediction accuracy of the trained ResNet-34 classification network on the original test set.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
according to the method and the system for adding the back door watermark of the data set, provided by the invention, the super-resolution technology is adopted as a back door watermark mode to be added into part of training samples and is associated with a specific class, so that whether the data set is embezzled or not is verified.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a flowchart of a method for performing hyper-resolution reconstruction on a sample of a data set according to embodiment 1 of the present invention;
fig. 2 is a flowchart of a verification method provided in embodiment 2 of the present invention;
fig. 3 is a flowchart of a verification method provided in embodiment 3 of the present invention;
fig. 4 is a diagram of the watermark effect of the back door of sample 1 in embodiment 3 of the present invention;
fig. 5 is a diagram of the watermark effect of the back door of sample 2 in embodiment 3 of the present invention;
FIG. 6 is a visual activation chart of different models for a sample in embodiment 3 of the present invention;
FIG. 7 is a diagram of a verification system for data set theft according to embodiment 4 of the present invention;
fig. 8 is a diagram of a verification system for data set influence according to embodiment 5 of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention aims to provide a backdoor watermarking method, a verification method and a system for a data set, so as to effectively verify whether the data set is stolen or not and train a third-party model.
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Example 1
The embodiment provides a method for adding a watermark to a backdoor of a data set, which includes:
and performing hyper-resolution reconstruction on the sample of the data set, and modifying the label of the sample into a specified category to obtain the watermark data set.
The present embodiment may select 12 types of samples from the ImageNet dataset (1000 types, 1300 pictures per type) as the dataset, wherein the 12 types of samples are randomly selected, and the images are uniformly sized to 224 × 224. And performing hyper-resolution reconstruction on the data set, and modifying the label into a 0 th class. The network model of the hyper-resolution reconstruction uses SRCNN, and the parameters of the following convolution kernels are obtained by SRCNN training.
Referring to fig. 1, performing a hyper-resolution reconstruction on a sample of the data set specifically includes:
s11, carrying out bicubic interpolation on the samples of the data set to obtain an interpolation image;
in this embodiment, a sample image of a data set is defined as an original image a, and the original image a (i.e., a low-resolution image) is subjected to bicubic interpolation and enlarged to a target size to obtain an interpolated image B, where the enlargement factor is determined by a scale factor scale.
Optionally, the performing bicubic interpolation on the sample of the data set to obtain an interpolated image specifically includes:
s111, obtaining coordinates of 16 first pixel points closest to the coordinates of each pixel point of each sample in the samples;
BiCubic interpolation (BiCubic interpolation) is the most common interpolation method in a two-dimensional space, each pixel point of an original image A is known, an interpolation image B is unknown, if the value of each pixel point (X, Y) in B is required to be solved, a corresponding pixel point P (X, Y) in the original image A is firstly found out, 16 pixel points which are closest to the pixel point P (X, Y) in A are used as parameters for calculating the pixel value of the interpolation image B (X, Y),
s112, calculating the weight of the coordinates of the 16 pixel points by using a weight formula to obtain the weight of the coordinates of the 16 pixel points;
the weight formula is:
and a is a constant, x represents the distance from each first pixel point to a specified pixel point, and the specified pixel point is each pixel point of the sample in the training set.
Wherein, aijAnd W represents the pixel value of the pixel point, and W represents the weight.
S113, overlapping the weighted values of the coordinates of the 16 pixel points to obtain a pixel value of a pixel point corresponding to each pixel point in the sample in the interpolation image;
and S114, obtaining the interpolation image according to each pixel value in the interpolation image.
S12, carrying out convolution operation on the interpolation image to obtain an n1 dimensional feature matrix;
where the convolution kernel size is 9 × 9 and the number of convolution kernels is 64(n1), a 64-dimensional feature matrix is obtained.
S13, performing convolution operation on the n1 dimensional feature matrix again to obtain an n2 dimensional feature matrix;
the convolution kernel size is 1 x 1, the convolution kernel number is 32(n2), and another 32-dimensional feature matrix is obtained;
and S14, aggregating the n2 dimensional feature matrixes to obtain a hyper-resolution image.
Using 1 convolution kernel with size 5 x 5 and the above result to perform calculation, aggregating n2 dimension feature matrix to obtain final super-divided image (i.e. watermark).
In the embodiment, a Super-Resolution (Super-Resolution) technology is adopted as a backdoor watermark mode to be added to the sample of the data set and associated with a specific class, so that the watermark can be effectively added to the data set, and the data set with the backdoor watermark can be successfully acquired.
Example 2
The scheme of the embodiment comprises two stages: adding a watermark to the data set and verifying the validity of the data set. Adding a backdoor watermark pattern to a part of samples in the data set and associating the part of samples with a specific category so as to verify whether the data set is illegally used, for example, the data set is used for training a third-party model, and if the data set is used in the training of the third-party model, a backdoor is left; when verifying whether a third-party model steals the data set, adding the backdoor watermark pattern to a sample of the test set, and if the third-party model predicts the data set as a specified category with high precision, proving that the third-party model steals the data set. The specific scheme is as follows:
a method for verifying data set theft, please refer to fig. 2, the method includes:
s1, dividing the data set to be verified into an original training set and an original testing set;
the embodiment may select 12 types of samples from the ImageNet dataset (1000 types, 1300 pictures per type) as the dataset to be verified, where the 12 types of samples are randomly selected, and the images are uniformly sized to 224 × 224. 80% of the samples in each class are selected to form the original training set, and the rest 20% form the original testing set, and the two parts are not overlapped.
S2, performing hyper-resolution reconstruction on partial samples of the original training set, and modifying labels of the partial samples into designated categories to obtain watermark samples; forming a back door training set by the watermark samples and residual samples, wherein the residual samples are other samples except the partial samples in the original training set;
in this embodiment, 50 pictures are randomly selected from each type of sample in the original training set, and then the super-resolution reconstruction is performed, and the label is modified to be the 0 th type. The network model of the hyper-resolution reconstruction uses SRCNN, and the parameters of the following convolution kernels are obtained by SRCNN training.
It should be noted that, in this step, the method for adding the back-gate watermark to part of the samples of the original training set (i.e., performing the super-resolution reconstruction and modifying the sample labels) is the same as the method for performing the super-resolution reconstruction to the samples of the data set and modifying the sample labels in embodiment 1, and specific contents may be as in embodiment 1, and details are not described here again. Note that the samples for both are different.
S3, performing hyper-resolution reconstruction on the original test set sample, and modifying the label of the original test set sample into a designated category to obtain a back door test set;
here, performing the super-resolution reconstruction on the original test set sample refers to performing the super-resolution reconstruction on all the original test set samples, and the method of adding the back-gate watermark (i.e., performing the super-resolution reconstruction and modifying the sample label) to the original test set sample in step S3 is the same as the method of performing the super-resolution reconstruction and modifying the sample label on the partial sample of the original training set in step S2, i.e., the method of performing the super-resolution reconstruction and modifying the sample label in embodiment 1 is the same, and specific contents refer to embodiment 1, which is not described herein again.
S4, training the ResNet-34 classification network by adopting the back door training set to obtain the trained ResNet-34 classification network;
s5, inputting the back door test set into the trained ResNet-34 classification network, obtaining the prediction category of the sample of the back door test set, and judging whether the data set to be verified is stolen according to the prediction category and the specified category.
Specifically, whether the probability that the prediction category is the specified category is higher than or equal to a preset probability is judged;
if the probability is higher than or equal to the preset probability, the data set to be verified is stolen;
and if the probability is lower than the preset probability, the data set to be verified is not stolen.
The method for verifying the data set by using the backdoor watermark should have validity, that is, when the test set added with the watermark is used for testing the model trained by the watermark data set, the model should be predicted as the specified category with high precision, but the watermark success rate of the scheme in the prior art is not high enough. In this regard, the present embodiment adopts a Super-Resolution (Super-Resolution) technique as a backdoor watermark pattern to be added to a part of the training samples and associated with a specific class, so as to verify whether the data set is stolen or not and train a third-party model. Compared with the existing scheme, the back door watermark mode of the scheme (SRNet) is more concealed, and the high watermark success rate can be achieved under the condition of low injection rate.
Example 3
The verification method for the stealing of data set provided in example 2 should be harmless, i.e. the watermark added in example 2 should not prevent the normal use of the data set to be verified, and the model trained with the watermark data set (i.e. the trained ResNet-34 classification network) should perform on the benign test set (i.e. the original test set) as well as the model trained with the original data set (i.e. the trained baseline network). Therefore, in order to verify whether the method in embodiment 2 affects the normal use of the data set to be verified, this embodiment provides a method for verifying the influence of the data set, please refer to fig. 3, where the method includes:
s101, dividing a data set to be verified into an original training set and an original testing set;
the method of this step is the same as that of step S1 in embodiment 2, and specific contents thereof may be referred to step S1 in embodiment 2.
S102, training a baseline network by adopting the original training set to obtain a trained baseline network;
in this embodiment, an Adam optimizer can be adopted, cross entropy is used as a loss function, the initial learning rate is set to be 0.002, each 30 rounds are reduced by 10 times, and 200 rounds are trained in total.
S103, inputting the original test set into the trained baseline network to obtain the prediction type of the trained baseline network on the sample of the original test set;
here the baseline network is trained for comparison of the effect with the later ResNet-34 classification network.
S104, comparing the prediction categories of the samples of the original test set by the trained base line network with the categories to which the samples of the original test set belong to obtain the accuracy rate of the category prediction of the samples of the original test set by the trained base line network;
s105, performing hyper-resolution reconstruction on a part of samples of the original training set, and modifying the labels of the part of samples into specified categories to obtain watermark samples; forming a back door training set by the watermark samples and residual samples, wherein the residual samples are other samples except the partial samples in the original training set;
the method of this step is the same as that of step S2 in embodiment 2, and specific contents refer to step S2 in embodiment 2.
S106, training the ResNet-34 classification network by adopting the back door training set to obtain the trained ResNet-34 classification network;
the method of this step is the same as that of step S4 in embodiment 2, and specific contents refer to step S4 in embodiment 2.
S107, inputting the original test set into the trained ResNet-34 classification network to obtain the prediction type of the trained ResNet-34 classification network on the original test set;
s108, comparing the prediction class of the original test set by the trained ResNet-34 classification network with the class of the sample of the original test set to obtain the class prediction accuracy of the sample of the original test set by the trained ResNet-34 classification network;
s109, judging whether the backdoor watermark has influence on the normal use of the data set to be verified according to the class prediction accuracy of the trained base line network on the samples of the original test set and the class prediction accuracy of the trained ResNet-34 classification network on the original test set.
Specifically, whether the class prediction accuracy of the trained base line network on the samples of the original test set and the interpolation of the class prediction accuracy of the trained ResNet-34 classification network on the original test set are smaller than or equal to a preset difference value is judged;
if the back door watermark is smaller than or equal to the preset difference value, the back door watermark has no influence on the normal use of the data set to be verified;
if the difference value is larger than the preset difference value, the backdoor watermark has influence on the normal use of the data set to be verified.
In order to enable those skilled in the art to more accurately understand the effectiveness of the verification method for the theft of data sets provided in example 2 above and the harmlessness of the verification method effected by data sets provided in example 3, the following experiments are now provided for verification.
1. Experimental setup
In the experiment, ResNet-34 was chosen as the classifier, and the baseline network and the back-gate network (ResNet-34 classification network) were trained with the original training set and the back-gate training set, respectively. 50 pictures are selected from each type of sample to perform super-resolution reconstruction, and the target class is designated as class 0, and the detailed settings are shown in table 1.
TABLE 1 Experimental setup
The first column indicates the dataset used, the second the number of classes, the third the size of the input image, the fourth the number of training samples and the fifth the classification network used.
The effect of the back door watermarking scheme is verified with benign accuracy and watermarking success rate. The higher the benign accuracy rate is, the closer the benign accuracy rate is to the performance of a basic line network, and the smaller the influence of the backdoor watermark mode on a normal sample is proved to be; the higher the watermark success rate, the more reliably the scheme can verify that the back door model is trained by stealing a data set.
2. Effect of the experiment
The randomly selected images are subjected to over-resolution reconstruction, the scale factors scale are set to be 4 and 2 respectively, a backdoor training set is manufactured by adopting a backdoor injection rate (backdoor sample number/total sample number) of 4.8%, and an experiment is carried out on a classification task of the backdoor training set by taking a WANet method as a backdoor watermark scheme for comparison. The back door network is trained and tested by the same configuration, during testing, a test set divided from an original data set is used as an original test set, all samples of the original test set are added with a back door watermark mode to be used as a back door test set, and the two test sets are used for respectively testing the effect of the back door network. The results are shown in Table 2.
TABLE 2 Experimental results
In table 2, the first column indicates different back gate watermark methods, where scale indicates a scale factor used in super-resolution reconstruction, the second column indicates a used data set, the third column indicates accuracy of a baseline network to a benign test set, the fourth and fifth columns respectively indicate performances of a back gate model in an original test set and a back gate test set, and the last column indicates an injection rate of a back gate watermark sample.
As can be seen from the experimental results in Table 2, the method of the present invention achieves excellent results. When scale is 4, the Watermark Success Rate (WSR) reaches 98.87%, the test accuracy rate of the original test set reaches 87.86%, and is only reduced by 1.14% compared with the baseline network; experiments have found that when the scale factor is set to scale 2, a watermark success rate of 85.94% can be achieved, while the impact on the original test set remains small, only 1.77% lower than that of the baseline network. In the above experiment, the injection rate of the back gate of the invention is only 4.8%, which shows that the watermark mode of the back gate of the invention is strong enough.
The WANet method does not show ideal effects on the classification task of the present invention. The watermark success rate is only 10.38% when the injection rate of the back gate is set to 4.8%, and is only 15.63% when the injection rate is doubled and adjusted to 9.6.
3. Visual effects
Besides the success rate of the watermark, the concealment is also an element to be considered in the back door watermark mode, and the good back door watermark mode is not easy to be discovered and cracked by a third party. Several samples are randomly selected in the experiment, a backdoor watermark (image distortion and hyper-resolution reconstruction) is embedded according to a backdoor method of WANet and SRNet work, the visual effect of the samples is analyzed, all the calculations are carried out on the premise that the image pixels are 224 × 224, and the results are shown in fig. 4 and fig. 5. The first column is the original image, the second column is the image after image distortion, the third column and the fourth column are the super-resolution images, and the scale factors are 2 and 4 respectively. In order to evaluate the visual quality of the images before and after transformation, the PSNR (Peak Signal-to-Noise Ratio) and LPIPS (Learned Perceptual Image Patch Similarity) indexes were calculated for the images in this experiment.
In FIG. 4, the PSNR index of FIG. 2 is 37.62 and the LPIPS index is 0.009; the PSNR index for fig. 3 is 33.78, LPIPS index is 0.039; the PSNR index of FIG. 4 is 32.64, and the LPIPS index is 0.200.
In FIG. 5, the PSNR index of FIG. 2 is 35.50 and the LPIPS index is 0.016; the PSNR index for fig. 3 is 31.16, the LPIPS index is 0.138; the PSNR index of fig. 4 is 30.22 and the LPIPS index is 0.361.
As can be seen from the two groups of figures, in terms of visual effect, when the scale factor is 2, the difference between the two groups of figures before and after the super-resolution reconstruction can be hardly seen; at a scale factor of 4, the image appears slightly blurred. However, in various data sets, the data themselves have different definitions, and are normal in the physical world, so that the data are not questioned. On PSNR and LPIPS indices, the super-resolution reconstruction effect is slightly worse than image distortion, but no difference is visually perceived.
In order to obtain the general effect of the super-resolution reconstruction, 50 pictures are randomly selected in each class, and the 600 pictures are subjected to distortion transformation and super-resolution reconstruction and are calculated with the original data to obtain average PSNR and LPIPS values. The results are shown in Table 3:
TABLE 3 evaluation index of image quality for different transformations
| Image warping | Reconstruction (scale 4) | Reconstruction of super (scale 2) | |
| PSNR | 34.81 | 31.41 | 32.75 |
| LPIPS | 0.0263 | 0.2960 | 0.1107 |
As can be seen from the data in fig. 4, fig. 5 and table 3, the average PSNR and LPIPS indexes before and after the super-resolution reconstruction and before and after the warping transformation are not very different, and there is no influence on the visual effect, and the sample does not cause any doubt at all, but the watermark success rate of SRNet is much higher than that of WANet.
The visualization tool can detect the abnormal behavior of the network, if the traditional back door mode based on the image patch is used as the back door watermark, the abnormality can be easily detected because the local response is overlarge, and the method of the invention transforms the whole image, so the concealment is good and the detection is difficult. Visual analysis was performed using GradCam, which finds the key regions in the input image that best activate the output of the model, as shown in fig. 6.
In fig. 6, the 1 st graph is a base network, the 2 nd graph is a back gate network 1(scale 2), and the 3 rd graph is a back gate network 2(scale 4). And activating the original label and the appointed label respectively in the baseline network and the backdoor network. It can be seen from the figure that the backdoor network only slightly shifts the model attention out of the correct area, and the critical area after the shift is still in the object itself and not transferred elsewhere. In particular the back door network 1, whose visualized heatmap closely resembles the baseline model, illustrates the concealment of the inventive method from another perspective.
4. Summary of the invention
The scheme provided by the embodiment 2 can be used for verifying whether the data set is used for training a third-party model or not, normal use cannot be influenced, meanwhile, good concealment cannot be discovered and cracked by a third party, and experiments prove that the reliability of the scheme can be well achieved in all aspects.
Example 4
This embodiment provides a verification system for data set theft, please refer to fig. 7, the system includes:
the dividing module M1 is used for dividing the data set to be verified into an original training set and an original testing set;
a backdoor training set obtaining module M2, configured to perform hyper-resolution reconstruction on a part of samples of the original training set, and modify the labels of the part of samples into specified categories, so as to obtain watermark samples; forming a back door training set by the watermark samples and residual samples, wherein the residual samples are other samples except the partial samples in the original training set;
a back door test set obtaining module M3, configured to perform hyper-resolution reconstruction on the original test set sample, and modify the label of the original test set sample into a specified category, so as to obtain a back door test set;
the trained ResNet-34 classification network acquisition module M4 is used for training the ResNet-34 classification network by adopting the back door training set to obtain the trained ResNet-34 classification network;
and the first verification module M5 is configured to input the backdoor test set into the trained ResNet-34 classification network, obtain a prediction category of a sample of the backdoor test set, and determine whether the data set to be verified is stolen according to the prediction category and the specified category.
Example 5
This embodiment provides a verification system for data set influence, the system is used to verify the influence of the method of embodiment 2 on the data set to be verified, please refer to fig. 8, the system includes:
the dividing module M11 is used for dividing the data set to be verified into an original training set and an original testing set; this step is the same as M2 in example 4.
A trained baseline network obtaining module M21, configured to train a baseline network using the original training set to obtain a trained baseline network;
a first prediction category module M31, configured to input the original test set into the trained baseline network, and obtain a prediction category of the trained baseline network for a sample of the original test set;
a first class prediction accuracy module M41, configured to compare the prediction class of the sample of the original test set by the trained base line network with the class to which the sample of the original test set belongs, to obtain the class prediction accuracy of the sample of the original test set by the trained base line network;
a backdoor training set obtaining module M51, configured to perform hyper-resolution reconstruction on a part of samples of the original training set, and modify the labels of the part of samples into specified categories, so as to obtain watermark samples; forming a back door training set by the watermark samples and residual samples, wherein the residual samples are other samples except the partial samples in the original training set; this step is the same as M1 in example 4.
A trained ResNet-34 classification network obtaining module M61, configured to train the ResNet-34 classification network using the back door training set, so as to obtain a trained ResNet-34 classification network; this step is the same as M4 in example 4.
A second prediction category module M71, configured to input the original test set into the trained ResNet-34 classification network, and obtain a prediction category of the trained ResNet-34 classification network for the original test set;
a second class prediction accuracy module M81, configured to compare the prediction class of the original test set with the class to which the sample of the original test set belongs by the trained ResNet-34 classification network, and obtain the class prediction accuracy of the sample of the original test set by the trained ResNet-34 classification network;
a second judging module M91, configured to judge, according to the class prediction accuracy of the trained base line network on the samples of the original test set and the class prediction accuracy of the trained ResNet-34 classification network on the original test set, whether the back-gate watermark affects normal use of the data set to be verified.
In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system disclosed by the embodiment, the description is relatively simple because the system corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description.
The principles and embodiments of the present invention have been described herein using specific examples, which are provided only to help understand the method and the core concept of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed. In view of the above, the present disclosure should not be construed as limiting the invention.
Claims (3)
1. A method for validating the effects of a data set, the method comprising:
dividing a data set to be verified into an original training set and an original testing set;
training a baseline network by adopting the original training set to obtain a trained baseline network;
inputting the original test set into the trained baseline network to obtain the prediction category of the trained baseline network to the samples of the original test set;
comparing the prediction category of the sample of the original test set by the trained base line network with the category to which the sample of the original test set belongs to obtain the accuracy rate of predicting the category of the sample of the original test set by the trained base line network;
performing hyper-resolution reconstruction on part of samples of the original training set, and modifying labels of the part of samples into specified categories to obtain watermark samples; forming a back door training set by the watermark samples and residual samples, wherein the residual samples are other samples except the partial samples in the original training set;
training the ResNet-34 classification network by adopting the back door training set to obtain the trained ResNet-34 classification network;
inputting the original test set into the trained ResNet-34 classification network to obtain the prediction category of the trained ResNet-34 classification network on the original test set;
comparing the prediction category of the original test set by the trained ResNet-34 classification network with the category to which the sample of the original test set belongs to obtain the accuracy of the category prediction of the sample of the original test set by the trained ResNet-34 classification network;
and judging whether the backdoor watermark has influence on the normal use of the data set to be verified according to the class prediction accuracy of the trained base line network on the samples of the original test set and the class prediction accuracy of the trained ResNet-34 classification network on the original test set.
2. The method of claim 1, wherein the determining whether the back-gate watermark has an effect on the normal use of the data set to be verified according to the class prediction accuracy of the trained base-line network on the samples of the original test set and the class prediction accuracy of the trained ResNet-34 classification network on the original test set comprises:
judging whether the class prediction accuracy of the trained base line network on the samples of the original test set and the interpolation of the class prediction accuracy of the trained ResNet-34 classification network on the original test set are smaller than or equal to a preset difference value or not;
if the back door watermark is smaller than or equal to the preset difference value, the back door watermark has no influence on the normal use of the data set to be verified;
if the difference value is larger than the preset difference value, the backdoor watermark has influence on the normal use of the data set to be verified.
3. A verification system of data set influence, the system comprising:
the dividing module is used for dividing the data set to be verified into an original training set and an original testing set;
the trained baseline network acquisition module is used for adopting the original training set to train a baseline network to obtain the trained baseline network;
a first prediction category module, configured to input the original test set into the trained baseline network, and obtain a prediction category of the trained baseline network for a sample of the original test set;
the first class prediction accuracy module is used for comparing the prediction class of the trained basic-line network on the samples of the original test set with the class of the samples of the original test set to obtain the class prediction accuracy of the trained basic-line network on the samples of the original test set;
the back door training set acquisition module is used for performing super-resolution reconstruction on part of samples of the original training set, modifying the labels of the part of samples into specified categories and acquiring watermark samples; forming a back door training set by the watermark samples and residual samples, wherein the residual samples are other samples except the partial samples in the original training set;
the trained ResNet-34 classification network acquisition module is used for training the ResNet-34 classification network by adopting the back door training set to obtain the trained ResNet-34 classification network;
a second prediction category module, configured to input the original test set into the trained ResNet-34 classification network, and obtain a prediction category of the trained ResNet-34 classification network for the original test set;
the second class prediction accuracy module is used for comparing the prediction class of the original test set with the class to which the sample of the original test set belongs by the trained ResNet-34 classification network to obtain the class prediction accuracy of the sample of the original test set by the trained ResNet-34 classification network;
and the second judging module is used for judging whether the backdoor watermark influences the normal use of the data set to be verified according to the class prediction accuracy of the trained base line network on the samples of the original test set and the class prediction accuracy of the trained ResNet-34 classification network on the original test set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110564333.7A CN113297547B (en) | 2021-05-24 | 2021-05-24 | Back door watermark adding method, verification method and system for data set |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110564333.7A CN113297547B (en) | 2021-05-24 | 2021-05-24 | Back door watermark adding method, verification method and system for data set |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113297547A CN113297547A (en) | 2021-08-24 |
| CN113297547B true CN113297547B (en) | 2022-07-08 |
Family
ID=77324173
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110564333.7A Active CN113297547B (en) | 2021-05-24 | 2021-05-24 | Back door watermark adding method, verification method and system for data set |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113297547B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1320792A2 (en) * | 2000-06-16 | 2003-06-25 | Koninklijke Philips Electronics N.V. | Protecting audio data by proof of the existence of a complete data set using watermarking |
| CN109740316A (en) * | 2018-12-27 | 2019-05-10 | 北京三未信安科技发展有限公司 | A kind of insertion of dynamic watermark, verification method and system and dynamic watermark processing system |
| CN112699867A (en) * | 2020-09-27 | 2021-04-23 | 民生科技有限责任公司 | Fixed format target image element information extraction method and system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2313847A4 (en) * | 2008-08-19 | 2015-12-09 | Digimarc Corp | Methods and systems for content processing |
| CN105931179B (en) * | 2016-04-08 | 2018-10-26 | 武汉大学 | A kind of image super-resolution method and system of joint sparse expression and deep learning |
| CN108596882B (en) * | 2018-04-10 | 2019-04-02 | 中山大学肿瘤防治中心 | The recognition methods of pathological picture and device |
| CN111815523A (en) * | 2020-06-08 | 2020-10-23 | 天津中科智能识别产业技术研究院有限公司 | Image restoration method based on generation countermeasure network |
| CN112364310A (en) * | 2020-11-16 | 2021-02-12 | 山西三友和智慧信息技术股份有限公司 | Data set protection and verification method based on backdoor attack |
-
2021
- 2021-05-24 CN CN202110564333.7A patent/CN113297547B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1320792A2 (en) * | 2000-06-16 | 2003-06-25 | Koninklijke Philips Electronics N.V. | Protecting audio data by proof of the existence of a complete data set using watermarking |
| CN109740316A (en) * | 2018-12-27 | 2019-05-10 | 北京三未信安科技发展有限公司 | A kind of insertion of dynamic watermark, verification method and system and dynamic watermark processing system |
| CN112699867A (en) * | 2020-09-27 | 2021-04-23 | 民生科技有限责任公司 | Fixed format target image element information extraction method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113297547A (en) | 2021-08-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liao et al. | Robust detection of image operator chain with two-stream convolutional neural network | |
| Li et al. | How to prove your model belongs to you: A blind-watermark based framework to protect intellectual property of DNN | |
| Zhao et al. | Passive forensics for copy-move image forgery using a method based on DCT and SVD | |
| Liu et al. | No-reference image quality assessment based on spatial and spectral entropies | |
| Kirchner et al. | Hiding traces of resampling in digital images | |
| Zhang et al. | The application of visual saliency models in objective image quality assessment: A statistical evaluation | |
| Benrhouma et al. | Tamper detection and self-recovery scheme by DWT watermarking | |
| Ouyang et al. | Robust copy-move forgery detection method using pyramid model and Zernike moments | |
| Zhou et al. | Image quality assessment based on inter-patch and intra-patch similarity | |
| Mahmood et al. | A passive technique for detecting copy-move forgeries by image feature matching | |
| Zeng et al. | A multi-purpose countermeasure against image anti-forensics using autoregressive model | |
| Camacho et al. | Convolutional neural network initialization approaches for image manipulation detection | |
| Ouyang et al. | A semi-fragile watermarking tamper localization method based on QDFT and multi-view fusion | |
| Liu et al. | Your model trains on my data? Protecting intellectual property of training data via membership fingerprint authentication | |
| Warif et al. | A comprehensive evaluation procedure for copy-move forgery detection methods: results from a systematic review | |
| CN113297547B (en) | Back door watermark adding method, verification method and system for data set | |
| Jamali et al. | Robust watermarking using diffusion of logo into auto-encoder feature maps | |
| Al‐Bandawi et al. | Blind image quality assessment based on Benford's law | |
| CN113034332B (en) | Invisible watermark image and back door attack model construction and classification method and system | |
| Fang et al. | Attacking image splicing detection and localization algorithms using synthetic traces | |
| Chen et al. | The forensicability of operation detection in image operation chain | |
| Zhang et al. | A semi‐parametric model for microstructure analysis of advanced high‐strength dual‐phase steels considering sample variation | |
| Goel et al. | An approach for anti-forensic contrast enhancement detection using grey level co-occurrence matrix and Zernike moments | |
| Gong et al. | Kerbnet: A qoe-aware kernel-based backdoor attack framework | |
| Jiang et al. | A novel quality assessment for visual secret sharing schemes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |