CN114862650B - Neural network watermark embedding method and verification method - Google Patents

Abstract

The invention discloses a neural network watermark embedding method and a verification method, which are used for acquiring an original model and an image training set of a neural network to be embedded with a watermark; setting any layer of original model as watermark layerl _wm At the watermark layerl _wm Adding an original sub-layer to the watermark layerlAdditive sub-layers of identical structurelObtaining a neural network model added with a new structure; and training the neural network model added with the new structure by using the image training set and a preset forward propagation constraint condition to obtain the neural network model embedded with the watermark. The advantages are that: the invention does not use weight as a carrier, can effectively resist common attacks such as fine tuning, pruning and the like, and can control the relation between the characteristic vectors, so that the model precision can be greatly reduced due to the false watermark after the watermark is embedded, the model is invalid, and the model can be protected from being stolen while the ownership of the model is verified.

Description

Neural network watermark embedding method and verification method

Technical Field

The invention relates to a neural network watermark embedding method and a verification method, and belongs to the technical field of deep learning.

Background

With the development of deep learning, neural networks are widely used in the fields of automatic driving, image recognition, natural language processing, and the like due to their excellent learning capabilities. However, training an efficient neural network model requires engineers to have a lot of expertise, and consumes a lot of computing resources and time cost, and is highly susceptible to abuse and infringement once disclosed, so that the problem of copyright property of the neural network is widely concerned by scholars. The white-box digital watermarking method based on the internal information of the neural network generally utilizes the unique topological structure and fitting mode of the neural network to realize copyright protection by embedding the watermark into the weight.

At present, a white box protection method based on a neural network mainly includes: (1) the training process is constrained by adding regularization terms to adjust the weight distribution embedded watermark. For example, a regular term is added into a loss function to constrain a training process, so that the weight is continuously adjusted and then the watermark is embedded, and watermark information is extracted through a key projection matrix. (2) The distribution of different classes of activation values is adjusted during the training process to embed the watermark. For example, a watermark is embedded in the probability density distribution of the activation value of each layer, and different classes of pictures pass through the activation layer to obtain corresponding activation value distributions. The method for embedding the watermark has the advantages that the condition for triggering the watermark is dynamic and has good concealment, the picture set for triggering the watermark is complicated to construct, and the process is relatively complex. (3) The watermark is embedded by adding a structure in the model by associating the layer parameters with weights during training. For example, a passport layer structure is added between a convolutional layer and an activation layer of a convolutional neural network, so that the watermark, the weight and a scale factor and a deviation term of the layer are in strengthened association in a training process to embed the watermark.

In the white-box protection method based on the internal information of the neural network, the existence mode of the watermark is various, but the weight is used as a carrier. However, in common attacks such as fine tuning and pruning, the weights are easily modified, resulting in poor robustness of the current method. Even in extreme cases, when the weights are fully modified, the watermark will not be detectable and difficult to recover, resulting in a complete failure of the copyright protection process.

Disclosure of Invention

The technical problem to be solved by the invention is to overcome the defects of the prior art and provide a neural network watermark embedding method and a verification method, wherein the correlation between the watermark and the weight is reduced by restricting a forward propagation rule in the embedding process, so that the watermark can effectively resist common attacks such as fine adjustment, pruning and the like and does not depend on a data set during verification.

In order to solve the above technical problem, the present invention provides a neural network watermark embedding method, including:

acquiring an original model of a neural network to be embedded with a watermark and an image training set;

setting any layer of original model as watermark layerl _wm At the watermark layerl _wm Adding an original sub-layer to the watermark layerlStructurally identical added sublayerslObtaining a neural network model added with a new structure;

training the neural network model added with the new structure by using an image training set and a preset forward propagation constraint condition to obtain a neural network model embedded with the watermark;

the forward propagation constraint is:

in the forward propagation process of training model, the watermark to be embeddedb，b∈｛0,1｝ ^T Adding different constraint conditions to the feature vector groups at different positions as a basis; the set of feature vectors consists of original sub-layerslAnd adding a sublayerlThe output feature vector of the' same position is composed,Twhich represents the length of the watermark or watermarks,T≤Klength ofTIs watermarkbAnd a length ofKThe sets of feature vectors of (a) are in one-to-one correspondence in order, exceeding the length of the watermarkK-TThe group feature vector does not add constraints, where the value of K is the original sub-layerlAnd adding a sublayerlNumber of convolution kernels of.

Further, the constraint conditions are as follows:

when the watermark is 1, constraint is applied to make two output feature vectors in the group parallel in the same direction, and the expression is:

v _k ¹ = u _k ¹

v _k ² = u _k ²

wherein the content of the first and second substances,v _k ¹ indicating the position of the original sub-layer after the constraint is appliedkThe output feature vector of (a) is,u _k ¹ indicating the position of the original sub-layer before applying the constraintkThe output feature vector of (a) is,v _k ² showing the position of the added sublayer after the application of the constraintkThe output feature vector of (a) is,u _k ² indicating the position of the added sublayer before applying the constraintkThe output feature vector of (a) is,k∈｛1,2,…, K｝；

at a watermark of 0, a constraint is applied to make the two eigenvectors in the group orthogonal to each other, expressed as:

v _k ¹ = u _k ¹

。

further, the original sub-layerlAnd adding a sublayerlThe character vector of' input image is input into the watermark layerl _wm The original sub-layer, the original sub-layerlAnd adding sublayerslAfter concat concatenation is performed on all image feature vectors output, the output image feature vectors are passed backward.

A neural network watermark verification method, comprising:

acquiring n verification pictures, and sequentially inputting the verification pictures into an embedded picture obtained by using a neural network watermark embedding methodObtaining the watermark layer in the neural network model of the watermarkl _wm Including the original sub-layerlOutput feature vector ofv _k ¹ And adding sublayersl' s output feature vectorv _k ² ；

For each verification picture, selecting an original sub-layerlOutput feature vector ofv _k ¹ And adding a sublayerl' s output feature vectorv _k ² The feature vectors at the same position form a pair of feature vector groups, and a single picture can be obtainedKFor the feature vector group;

before selectionTFor the feature vector groups, calculating two output feature vectors in each pair of feature vector groupsv _t ¹ Andv _t ² angle of (2)

The length of a single picture isTAngle set ofangle，t∈｛1,2,…, T｝；

The length is set to beTAngle set of single pictureangleThe angle binarization is carried out, and the watermark to be verified is extracted and obtainedb＇，b＇∈｛0,1｝ ^T ；

Using watermarks to be authenticatedb' with pre-acquired embedded watermarkbCalculating to obtain the error rate of a single verification picture;

and acquiring the error rate of the n verification pictures, calculating the error rate mean value of the n verification pictures, and verifying the ownership of the model when the error rate mean value is within a preset error rate threshold range.

Further, the length is set to be the length by utilizing a preset angle threshold valueTAngle set of single pictureangleThe angle binarization is carried out, and the watermark to be verified is extracted and obtainedb＇，b＇∈｛0,1｝ ^T The method comprises the following steps:

acquired angle setangle∈[0°,180°]Setting an angleThe degree threshold value is 90 degrees, and the watermark to be verified is obtained through calculation according to the following formulab＇，b＇∈｛0,1｝ ^T ，

。

Further, the calculation formula of the bit error rate is as follows:

in the formula, BER represents a bit error rate,b _t "indicating the first place in the watermark to be verifiedtA watermark of a single bit of data,b _t indicating embedding in watermarktA single bit watermark.

The invention achieves the following beneficial effects:

the invention does not take the weight as a carrier, can effectively resist common attacks such as fine tuning, pruning and the like, and can control the relation between the characteristic vectors, so that the model precision can be greatly reduced due to the forged watermark after the watermark is embedded, the model is invalid, and the model can be protected from being stolen while the ownership of the model is verified.

Drawings

Fig. 1 is a schematic diagram of an adding structure taking Lenet-5 as an example in the implementation process of watermark embedding;

fig. 2 is an overall structural view of a watermark embedding method;

FIG. 3 is a graph of a training process with and without embedded watermarks in a VGG16 network;

fig. 4 is an angle distribution diagram obtained by randomly inputting 100 pictures after embedding a watermark in VGG 16.

Detailed Description

The invention is further described below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby.

As shown in fig. 1 and 2, a neural network watermark embedding method, VGG16 is used as a neural network model M to be embedded with a watermark, and a data set uses a cfar-10 data set, which is composed of 60000 color pictures of 32 × 32, and has 10 categories in total. 6000 pictures per category. Wherein 50000 pictures are used for training, 10000 pictures are used for testing, and the method comprises the following steps:

in the first step, the first layer of VGG16 is selected as the watermark layer to be embedded with the watermarkl _wm In the existing original sub-layerlOn the basis ofl _wm Adding an additional sub-layer having the same structure as the additional sub-layerl' obtaining a neural network model with a new structureMOf wherein the original sub-layerlAnd adding a sublayerl' has a convolution kernel number of 64 and a size of 3 × 3.

Second, obtain the watermark layerl _wm Including the original sub-layerlOutput feature vector ofu _k ¹ And adding a sublayerl' s output feature vectoru _k ² 。

Thirdly, selecting an original sub-layerlOutput feature vector ofu _k ¹ And adding sublayersl' s output feature vectoru _k ² Feature vectors at the same position and grouped into a pair of feature vector groups, resulting in 64 pairs of feature vector groups in total.

Fourth step, adding constraint condition in forward propagation rule to embed watermark with length of 64bWhere { 1,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,1,0,1,0,0,1,1,0,1,1,0,1,0,0,1,1,1,0,0,0,1,1,0,0,0,0,1,1,0,0,0,1, 0,0,0,0,0 } is used as a basis, and different constraint conditions are added to each pair of feature vector groups according to the difference in bit. Length 64 watermarkbCorresponding to the feature vector group with the length of 64 in sequence.

When the watermark is 1, a constraint is applied to make the two eigenvectors in the group parallel in the same direction, that is:

v _k ¹ = u _k ¹

v _k ² = u _k ²

when the watermark is 0, a constraint condition is applied to make two feature vectors in the group orthogonal to each other, that is:

v _k ¹ = u _k ¹

。

fifthly, adding the VGG16 with a new structure by using the 50000 picture training model, and before training, setting the learning rate to be 0.1, the batch size to be 256 and the training batch to be 100. The neural network watermark requires that the performance of an original model cannot be influenced in the training process of embedding the watermark, and under the same setting, a training process curve of embedding the watermark in VGG16 and a training process curve of not embedding the watermark in VGG16 are shown in figure 3. In the training process, no loss function needs to be additionally arranged, and the weight is updated to the adaptive watermark according to the forward propagation constraint condition arranged in the fourth stepbWhen the training batch is 1, the watermark can be embedded, but the iteration is still needed to make the accuracy of the original task of the model converge. After training is finished, obtaining the neural network model embedded with the watermarkM _wm 。

A neural network watermark verification method comprises the following steps:

the first step is that pictures in 100 data sets are randomly selected and output feature vectors of a VGG16 model embedded with a watermark at a first layer are obtained, wherein the output feature vectors include original sub-layerslOutput feature vector ofv _k ¹ And adding a sublayerl' s output feature vectorv _k ² 。

Secondly, selecting an original sub-layer for each picturelAnd adding a sublayerl' feature vectors in the same positionv _k ¹ Andv _k ² and make up ofAnd a pair of feature vector groups, and 64 pairs of feature vector groups are obtained by a single picture.

Thirdly, for each picture, selecting the first 64 pairs of feature vector groupsv _T ¹ Andv _T ² calculating two output feature vectors in each pairv _t ¹ Andv _t ² angle of (2)

The length of a single picture isTAngle set of (2)angle，tE { 1,2, …,64 }; in order to reduce the influence of statistical deviation, 100 pictures are randomly selected in the first step, so that an angle set with a length of 6400 is obtained in total, and the angle distribution of 100 pictures is shown in fig. 4.

Fourthly, setting a threshold value to be 90 degrees for each picture, and then collecting angles with the length of 64angleAnd (5) carrying out angle binarization, wherein the angle is converted into 1 when the angle is less than 90 degrees, and the angle is converted into 0 when the angle is greater than 90 degrees. Set of anglesangleThe extracted watermark is obtained after binarizationb＇=

｛1,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,1,0,1,0,0,1,1,0,1,1,0,1,0,0,1,1,1,0,0,0,1,1,0,0,0,0,1,1,0,0,0,1,1,0,0,1,0,1,1,1,0,0,0,0,0,0,0,0｝。

Fifthly, extracting the watermark from the model to be verified for each pictureb' AND embedded watermarkbAnd comparing, and calculating the bit error rate to obtain a result 0. In the first step, 100 pictures are input, each picture can calculate the error rate, and the error rate of the 100 pictures is calculated to be 0 on average, so that the ownership of the model can be verified within an acceptable range, such as 20%.

The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.

Claims (5)

1. A neural network watermark embedding method, comprising:

acquiring an original model of a neural network to be embedded with a watermark and an image training set;

setting any layer of original model as watermark layerl _wm At the watermark layerl _wm Adding an original sub-layer to the watermark layerlAdditive sub-layers of identical structurelObtaining a neural network model added with a new structure;

training the neural network model added with the new structure by using an image training set and a preset forward propagation constraint condition to obtain a neural network model embedded with the watermark;

the forward propagation constraint is:

in the forward propagation process of the training model, the watermark to be embeddedb，b∈｛0,1｝ ^T Adding different constraint conditions to the feature vector groups at different positions as a basis; the set of feature vectors consists of original sub-layerslAnd adding a sublayerlThe output feature vector of the' same position is composed,Twhich represents the length of the watermark or watermarks,T≤Klength ofTIs watermarkbAnd a length ofKThe sets of feature vectors of (a) are in one-to-one correspondence in order, exceeding the length of the watermarkK-TThe group feature vector does not add constraints, where,Kis the original sublayerlAnd adding a sublayerlNumber of convolution kernels of';

the constraint conditions are as follows:

at watermark 1, a constraint is applied to make the two output feature vectors in the group parallel with the same direction, which is expressed as:

v _k ¹ = u _k ¹

v _k ² = u _k ²

wherein the content of the first and second substances,v _k ¹ indicating the position of the original sub-layer after the constraint is appliedkThe output feature vector of (a) is,u _k ¹ indicating the original sub-layer before applying the constraint is in placekThe output feature vector of (a) is,v _k ² showing the position of the added sublayer after the application of the constraintkThe output feature vector of (a) is,u _k ² indicating the position of the added sublayer before applying the constraintkThe output feature vector of (a) is,k∈｛1,2,…, K｝；

at a watermark of 0, a constraint is applied to make two eigenvectors in the group orthogonal to each other, which is expressed as:

v _k ¹ = u _k ¹

。

2. the neural network watermark embedding method of claim 1, wherein the original sub-layerlAnd adding sublayerslThe input image feature vector being input to the watermark layerl _wm The original sub-layer, the original sub-layerlAnd adding a sublayerlAfter concat concatenation is performed on all image feature vectors output, the output image feature vectors are passed backward.

3. A neural network watermark verification method, comprising:

acquiring n verification pictures, and sequentially inputting the verification pictures into the neural network model with the embedded watermark obtained by the neural network watermark embedding method of claim 1 to obtain the verification pictures in the watermark layerl _wm Including the original sub-layerlOutput feature vector ofv _k ¹ And adding a sublayerl' s output feature vectorv _k ² ；

Selecting an original sub-layer for each verification picturelOutput feature vector ofv _k ¹ And adding sublayersl' s output feature vectorv _k ² Feature vectors at the same position form a pair of feature vector groups, and a single picture is obtainedKFor the feature vector group;

before selectionTFor the feature vector groups, calculating two output feature vectors in each pair of feature vector groupsv _t ¹ Andv _t ² angle of (2)

The length of a single picture isTAngle set of (2)angle，t∈｛1,2,…, T｝；

The length is set to beTAngle set of single pictureangleThe angle binarization is carried out, and the watermark to be verified is extracted and obtainedb＇，b＇∈｛0,1｝ ^T ；

Using watermarks to be authenticatedb' with pre-acquired embedded watermarkbCalculating to obtain the error rate of a single verification picture;

and acquiring the error rate of the n verification pictures, calculating the error rate mean value of the n verification pictures, and verifying the ownership of the model when the error rate mean value is within a preset error rate threshold range.

4. The neural network watermark verification method according to claim 3, wherein the length is determined by using a preset angle threshold valueTAngle set of single pictureangleThe angle binarization is carried out, and the watermark to be verified is extracted and obtainedb＇，b＇∈｛0,1｝ ^T The method comprises the following steps:

acquired angle setangle∈[0°,180°]Setting the angle threshold value to be 90 degrees, and calculating according to the following formula to obtain the watermark to be verifiedb＇，b＇∈｛0,1｝ ^T ，

。

5. The neural network watermark verification method of claim 3, wherein the bit error rate is calculated by the formula:

in the formula, BER represents a bit error rate,b _t "indicating the first place in the watermark to be verifiedtA watermark of a single bit of data,b _t indicating embedding in watermarktA single bit watermark.

Images