CN115311117A - Image watermarking system and method for style migration depth editing - Google Patents
Image watermarking system and method for style migration depth editing Download PDFInfo
- Publication number
- CN115311117A CN115311117A CN202211109211.XA CN202211109211A CN115311117A CN 115311117 A CN115311117 A CN 115311117A CN 202211109211 A CN202211109211 A CN 202211109211A CN 115311117 A CN115311117 A CN 115311117A
- Authority
- CN
- China
- Prior art keywords
- image
- watermark
- module
- style migration
- style
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/0021—Image watermarking
- G06T1/005—Robust watermarking, e.g. average attack or collusion attack resistant
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2201/00—General purpose image data processing
- G06T2201/005—Image watermarking
- G06T2201/0065—Extraction of an embedded watermark; Reliable detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2201/00—General purpose image data processing
- G06T2201/005—Image watermarking
- G06T2201/0202—Image watermarking whereby the quality of watermarked images is measured; Measuring quality or performance of watermarking methods; Balancing between quality and robustness
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Abstract
The invention discloses an image watermarking system and method facing to style migration depth editing, wherein the system comprises a style migration depth editing module, a watermarking coding sub-module and a watermarking coding sub-module, wherein the style migration depth editing module is used for extracting the characteristics of an original image and performing style migration depth editing; the discriminator module is used for judging the quality of the image output by the style transition depth editing module and improving the visual effect of the generated image through confrontation training; the watermark extraction module decouples host image features and watermark features by utilizing a pre-constructed convolutional neural network, and guides extraction and recovery of the watermark by using reconstruction loss. Aiming at the style migration task, the method completes the encoding and embedding of the watermark in the process of the style migration deep editing, so that a user cannot bypass the embedding process of the watermark, the method is favorable for assisting network supervision, and the use of a style migration deep editing model is standardized.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to an image watermarking system and method for style migration depth editing.
Background
Style migration may be defined as, for a particular scene, how it translates into one possible representation of the scene, given the other possible representation. In recent years, with the development of deep learning, a large number of false images generated by style migration, deep editing and counterfeiting are streamed on the network, and potential risks and challenges are brought to the monitoring of national and social security and network environments, wherein one representative method is pix2pix based on a U-Net network, which is a general framework of the first style migration problem, and after a pair of training sets of an input image and an output image are given for training, the generation network of pix2pix can directly convert the style of the input image into the style of the output image.
Currently, a series of counterfeit image detection methods exist for image counterfeit modes generated by deep editing. Most passive detection methods discriminate by distortions and artifacts in the forged image, but this approach may become increasingly difficult to cope with more complex and realistic generation algorithms, and the generalization tends to be poor, with poor migration capability among different data sets. However, the active detection method, such as the watermarking technology, usually encodes and embeds the watermark before the image is transmitted, and verifies and judges whether the image is tampered by extracting the watermark in the image after the transmission, however, the method still cannot prevent malicious tampering, and a tamperer can modify the original image and then embed the watermark to confuse the detector, which is difficult to effectively monitor.
Disclosure of Invention
In order to solve the defects of the prior art, the invention provides an image watermarking method facing style migration depth editing, which aims to finish the encoding and embedding of the watermark in the process of the image style migration depth editing, so that a user cannot bypass the embedding process of the watermark and assist network supervision. Firstly, designing an image watermark coding module based on a convolutional neural network, and extracting the coding characteristics of an image watermark; based on the modification of a U-Net generator in pix2pix work, splicing and subsequently reconstructing the watermark characteristics and the characteristics of an input image in an image reconstruction stage, and outputting an image which is embedded with the watermark and finishes the style migration; then, extracting the characteristics of an output image and reconstructing the watermark by using a watermark extractor based on a convolutional network, and guiding the extraction and recovery of the watermark by using reconstruction loss; and finally, performing countermeasure training by using a distortion simulation module, and further improving the robustness of the watermark. The specific technical scheme of the invention is as follows:
the invention provides an image watermarking system facing style migration depth editing, which comprises: a style migration depth editing module, a discriminator module, a watermark extraction module and a distortion simulation module, wherein,
the style migration depth editing module comprises a style migration generator module, a watermark coding module and a characteristic fusion module; the style migration generator module is based on a generating framework of U-Net, an image to be edited is input, the image is of a symmetrical structure of an encoder and a decoder, the image is subjected to feature extraction through the encoder, the image is converted into high-dimensional features and then is subjected to up-sampling by the decoder to be reconstructed, feature information sharing is performed by using jump connection at corresponding stages of the encoder and the decoder, and image feature information of a middle layer at an image reconstruction stage is reserved; the watermark encoding module is used for extracting the characteristics of the input watermark image to be embedded based on the convolutional neural network, converting the watermark image into high-dimensional characteristics and retaining the output watermark characteristic information; the characteristic fusion module is used for fusing the image characteristic information and the watermark characteristic information of the middle layer, converting the fused characteristic and inputting the converted characteristic to a subsequent reconstruction layer to finish the style migration and watermark embedding of the image;
the discriminator module is based on a PatchGAN network and is used for judging the quality of the image output by the style migration depth editing module;
the watermark extraction module takes the output image of the style migration depth editing module as input and extracts the features of the watermark image based on a convolutional neural network;
the distortion simulation module transforms an output image of the style migration depth editing module based on a convolutional neural network, inputs the output image into the watermark extraction module, and improves robustness of the watermark to unknown post-processing operation by adopting a mode of confrontation training.
Further, the style migration generator module is constructed based on U-Net, and the watermark encoding module applies 5 × 5, padding is 2, and the transposed convolution and the activation function ReLU are applied;
after the characteristic fusion module applies splicing operation, performing characteristic dimension reduction by using a convolution layer of 1 multiplied by 1 and a ReLU activation function;
the watermark extraction module is constructed based on a convolutional neural network, firstly, 4 layers of convolutional network modules with 16 channels are used for characteristic decoupling of a host image and a watermark image, each convolutional network module selects a ReLU as an activation function, and then a downsampling module based on the convolutional network, an average pooling layer and a ReLU activation layer is used for extracting the watermark image;
the distortion simulation module is composed of two layers of convolution networks, wherein the first layer of convolution network changes an input characteristic channel from 3 to 16, the second layer of convolution network restores the number of the channels from 16 to 3, and LeakyReLU is used as an activation function between the two layers of convolution networks.
Furthermore, the output dimensions of the style migration depth editing module, the watermark extraction module and the distortion simulation module are 3 channels.
Further, the image to be edited and the watermark image to be embedded are both 3-channel images, the image to be edited is 256 × 256 pixels, and the watermark image to be embedded is 64 × 64 pixels.
The invention also provides an image watermarking method facing style migration depth editing, which adopts the image watermarking system to edit images and comprises the following steps:
s1: constructing a training data set;
s1-1: constructing a style migration training data set, and taking a pair-form image pair { A, B } as training data, wherein A and B are different description forms of the same scene;
s1-2: constructing a watermark image training data set, collecting a watermark image to be embedded as training data, preprocessing the watermark image to be embedded into a black-and-white watermark image w with the size of 64 multiplied by 64, white foreground color and black background color;
s2: inputting the training data set into a style migration depth editing module, and outputting a depth editing image;
s2-1: performing style migration training on the images { A, B } by using the image pair obtained in the step S1-1, wherein the migration direction is A → B, inputting the image A to be converted into a style migration generator module, and outputting the size of the image feature as C A *H A *W A ，C A 、H A 、W A The number of output channels, length and width are respectively;
s2-2: inputting the black-and-white watermark image w obtained in the step S1-2 into a watermark encoding module to obtain the high-dimensional characteristics of the watermark image, wherein the size of the output watermark characteristics is C w *H A *W A ，C w 、H A 、W A The number of output channels, length and width are respectively;
s2-3: the image characteristics C obtained in the step S2-1 are A *H A *W A And watermark characteristic C obtained by S2-2 w *H A *W A The input feature fusion module carries out splicing operation to obtain (C) A +C w )*H A *W A After merging the features, using 1 × 1 convolution network to reduce the dimension to C A *H A *W A Inputting the image data into a subsequent reconstruction layer to finish style migration, and outputting an image B 'after the style migration is finished, wherein the image B' is a 3-channel 256 multiplied by 256 image which is a migration generated image after the image watermark is embedded;
s3: inputting the image B' obtained in the step S2-3 into a discriminator module, wherein the network output of the discriminator module is an NxN matrix, each element in the matrix only has two real or forged values, the two real or forged values correspond to a local area in the input image, and finally, the NxN output result is averaged to be used as the final judgment of the discriminator module on the input image;
s4: inputting the image B 'obtained in the step S2-3 into a watermark extraction module for watermark extraction, firstly decoupling watermark image characteristics and host image characteristics by using a convolutional neural network module, then reconstructing a watermark image, and outputting a 3-channel 64 multiplied by 64 watermark image w';
s5: repeating the step S2 to the step S4 until the loss function is converged and the quality of the image B' generated by style migration meets the given requirement;
s6: performing watermark robustness countermeasure training, namely adding a distortion simulation module, and simultaneously performing training and parameter updating with a grid migration depth editing module, a discriminator module and a watermark extraction module;
s6-1: finishing the image style migration and watermark embedding according to the step S2 to obtain a generated image B ', and inputting the B' into a discriminator module;
s6-2: inputting the image B' into a distortion simulation module, generating disturbance by the distortion simulation module through a convolutional neural network, simulating distortion or post-processing operation encountered by the image in the actual transmission process, and outputting a distortion simulation sample B ″ adv ，B″ adv Size of 3 channels 256 × 256 images;
s6-3: will distort the simulation sample B ″ adv Inputting the image B' into a watermark extraction module, and respectively extracting to obtain watermark images w ″) adv And w';
s7: repeating the step S6 until the loss function is converged, and generating an image B 'and extracting a watermark image w' with the quality meeting the given requirement;
s8: testing and using;
s8-1: inputting the image to be subjected to style migration depth editing and the watermark image to be embedded into the style migration depth editing module obtained in the step S7, and outputting the watermark embedded depth editing image;
s8-2: and (4) inputting the depth editing image obtained in the step (8-1) into the watermark extraction module obtained in the step (7) and outputting a watermark image.
Further, in the steps S2 and S3, the quality of the image generated by the style transition depth editing module is improved by selecting a mode of alternately training with a generator and a discriminator in GAN, a training target of the style transition depth editing module is that the discriminator module cannot judge whether the input image is a real image or an image generated by depth editing, a training target of the discriminator module is that whether the input image is real or not is effectively discriminated, and the training target and the image are finally subjected to nash equilibrium through zero-sum game.
Further, the loss function of the grid migration depth editing module in step S2 is:
wherein, the first and the second end of the pipe are connected with each other,
wherein x is the input image of the style migration depth editing module, y is the true value image corresponding to x, z is the noise input into the style migration generator module, G (x, z) represents the generated image, w is the true value of the watermark image, w' is the watermark image extracted by the watermark extraction module, λ and β are the hyperparameters,
generating a confrontation training function of the network for the condition, D (x, y) is the output result of the discriminator module,
in order to improve the optimization function of the image generation quality,
representing a loss of the watermark image.
Further, the loss function of the lattice migration depth editing module in step S6 is:
wherein the content of the first and second substances,
wherein λ, β, γ are hyper-parameters, G (x, z) is a generated image of the depth editing module, y' adv The distortion countercheck sample generated by the distortion simulation module is w is the true value of the watermark image, w 'is the watermark extracted from G (x, z) by the watermark extraction module, w' adv Is watermark extraction module from y' adv The watermark extracted in (1);
the watermark distortion brought by adding the distortion simulation module;
the distortion simulation module is added to generate image distortion.
Further, the loss function of the watermark extraction module in step S4 is:
in the formula, dec represents a watermark extraction module, w is a watermark image true value, and w' is a watermark image extracted by the watermark extraction module;
the loss function of the watermark extraction module in step S6 is:
in the formula, dec represents a watermark extraction module, w is a true value of the watermark image, w "is the watermark image extracted from G (x, z) by the watermark extraction module, w ″ adv Is watermark extraction module from y' adv The extracted watermark image.
Further, the loss function of the distortion simulation module in step S6 is:
in the formula, G adv For distortion simulation modules, α 1 ，α 2 Is hyperparametric, α 1 Controlling the intensity of the distortion, alpha, produced by the distortion simulation module 2 The strength of the information loss generated by the distortion simulation module is controlled.
The invention has the beneficial effects that:
1. the image watermarking method facing the style migration deep editing can complete the style migration task and simultaneously encode and embed the watermark, so that a user generating the model can not intentionally avoid the embedding of watermark information, and the use supervision of the deep editing model can be assisted;
2. the network architecture and the loss function designed by the invention can complete the embedding of the watermark without influencing the image generation quality obviously, and can extract the watermark information more completely;
3. the invention further improves the robustness of the watermark against the unknown distortion by applying the countermeasure training, and the watermark can be applied to various common post-processing operations.
Drawings
In order to illustrate embodiments of the invention or solutions in the prior art more clearly, the drawings that are needed in the embodiments will be briefly described below, so that the features and advantages of the invention will be more clearly understood by referring to the drawings that are schematic and should not be understood as limiting the invention in any way, and other drawings may be obtained by those skilled in the art without inventive effort. Wherein:
FIG. 1 is an architecture diagram of the style-migration-depth-editing-oriented image watermarking method of the present invention;
FIG. 2 is a schematic diagram of a training process of the style migration depth editing-oriented image watermarking method of the present invention;
FIG. 3 is a schematic diagram of the feature fusion of the style migration depth editing module of the present invention;
fig. 4 is a schematic diagram of a testing process of the style migration depth editing-oriented image watermarking method of the present invention.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention, taken in conjunction with the accompanying drawings and detailed description, is set forth below. It should be noted that the embodiments of the present invention and features of the embodiments may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.
The invention provides an image watermarking system facing style migration depth editing, as shown in figure 1, comprising: a style migration depth editing module, a discriminator module, a watermark extraction module and a distortion simulation module, wherein,
the style migration depth editing module comprises a style migration generator module, a watermark coding module and a feature fusion module, the input of the style migration depth editing module is an image to be edited and a watermark image to be embedded, the watermark coding module is used for extracting watermark features, the watermark features and the image features extracted by the style migration generator module are spliced in the image reconstruction stage, then subsequent image reconstruction is carried out, and meanwhile the style migration task and the watermark embedding task of the image are completed;
the style migration generator module is based on a U-Net generation framework, inputs an image to be edited and is a symmetrical structure of an encoder and a decoder, performs feature extraction on an original image through the encoder, performs up-sampling on the image by using the decoder after converting the image into high-dimensional features, performs feature information sharing by using jump connection at corresponding stages of the encoder and the decoder, and retains image feature information of a middle layer at an image reconstruction stage;
the watermark encoding module is used for extracting the characteristics of the input watermark image to be embedded based on the convolutional neural network, converting the characteristics into high-dimensional characteristics and retaining the output watermark characteristic information;
the characteristic fusion module is used for fusing the characteristic information of the image intermediate layer and the watermark characteristic information, converting the fused characteristic and inputting the converted characteristic to a subsequent reconstruction layer to continue to complete image reconstruction;
the discriminator module is a PatchGAN-based Network and is used for judging the quality of the image output by the style migration depth editing module, and according to the previous research on style migration tasks and generation of a countermeasure Network (GAN), the PatchGAN discriminator can effectively punish the structural difference between the output image and the target image;
the watermark extraction module takes the output image of the style migration depth editing module as input, utilizes a pre-constructed convolutional neural network to extract image characteristics, guides extraction and recovery of the watermark by using reconstruction loss, and can more completely extract watermark information from the migrated image;
the distortion simulation module transforms the output image of the style migration depth editing module through a convolutional neural network, inputs the output image into the watermark extraction module, and improves the robustness of the watermark to the unknown post-processing operation by adopting a mode of countertraining.
Furthermore, the style migration generator module is constructed based on U-Net, the watermark encoding module applies 5 × 5, padding is 2, and the transposition convolution and activation function ReLU are applied, and the feature fusion module performs feature dimension reduction by using 1 × 1 convolution layer and ReLU activation function after splicing operation is applied; the watermark extraction module is constructed based on a convolutional neural network, firstly, a convolutional network module with 16 channels in 4 layers is used for characteristic decoupling of a host image and a watermark image, each convolutional network module selects a ReLU as an activation function, and then a downsampling module based on the convolutional network, an average pooling layer and the ReLU activation layer is used for extracting the watermark image; the distortion simulation module is composed of two layers of convolution networks, wherein the first layer of convolution network changes an input characteristic channel from 3 to 16, the second layer of convolution network restores the number of the channels from 16 to 3, and LeakyReLU is used between the two layers of convolution networks as an activation function.
Preferably, the feature fusion of the feature fusion module is completed in the penultimate layer of the U-Net upsampling stage.
Furthermore, the output dimensions of the style migration depth editing module, the watermark extraction module and the distortion simulation module are 3 channels.
Furthermore, the image to be edited and the watermark image to be embedded are both 3-channel images, the image to be edited is 256 × 256 pixels, and the watermark image to be embedded is 64 × 64 pixels.
The invention also provides an image watermarking method facing style migration depth editing, as shown in fig. 2, comprising the following steps:
s1: constructing a training data set;
s1-1: constructing a style migration training data set, wherein a pair-form image pair { A, B } needs to be constructed as training data, wherein A and B are different description forms of the same scene, such as { tag map, real photo };
s1-2: constructing a watermark image training data set, collecting school badges of different universities as watermark images to be embedded, and preprocessing the original school badge images into images with the size of 64 multiplied by 64 in order to ensure the quality of style transition images, wherein the foreground color is white, and the background color is black, namely a black-and-white watermark image w;
s2: completing style migration depth editing, as shown in FIG. 3;
s2-1: performing image style migration training by using the image pair { A, B } obtained in the step S1-1, wherein the migration direction is A → B, and inputting the image A to be converted into a U-Net-based style migration generatorIn the module, an image B is used as a true value to participate in the calculation of a subsequent loss function, the generation training is guided, and the size of the output image characteristic is C A *H A *W A ，C A 、H A 、W A The number, length and width of output channels are respectively;
s2-2: inputting the black-and-white watermark image w obtained in the step S1-2 into a watermark encoding module to obtain the high-dimensional characteristics of the watermark image, wherein the size of the output watermark characteristics is C w *H A *W A ，C w 、H A 、W A The number, length and width of output channels are respectively;
s2-3: the image characteristics C obtained in the step S2-1 are A *H A *W A And watermark characteristic C obtained by S2-2 w *H A *W A Inputting the data into a feature fusion module to perform splicing (concatenate) operation to obtain (C) A +C w )*H A *W A After merging the features, using 1 × 1 convolution network to reduce the dimension to C A *H A *W A Inputting the image B 'into a subsequent reconstruction layer to finish style migration, and outputting an image B' after the style migration is finished, wherein the B 'is a migration generated image after the image watermark is embedded, and the sizes of the B' and the B are kept consistent and are still 256 images of 3 channels;
s3: inputting the image B' obtained in the step S2-3 into a discriminator module, wherein the network output of the discriminator module is an NxN matrix, each element in the matrix only has two real or forged values, and the output result of the NxN is averaged finally corresponding to a local area in the input image and is used as the final judgment of the discriminator module on the input image;
s4: inputting the image B 'obtained in the step S2-3 into a watermark extraction module for watermark extraction, firstly decoupling the watermark image characteristics and the characteristics of a host image by using a convolutional neural network module, then reconstructing the watermark image, and outputting a 3-channel 64 x 64 watermark image w';
s5: repeating the step S2 to the step S4 until the loss function is converged, and the quality of the image B 'generated by style migration is stable and the quality of the watermark w' is stable;
s6: performing watermark robustness confrontation training, namely adding a distortion simulation module, and performing training and parameter updating simultaneously with a grid migration depth editing module, a discriminator module and a watermark extraction module;
s6-1: finishing image style migration and watermark embedding according to the step S2 to obtain a generated image B ', inputting the B' into a discriminator module in order to prevent the quality of the generated image B 'obtained after the distortion simulation module is added for training from being reduced, and constraining the quality of the image B' through the confrontation training of the discriminator module and a style migration depth editing module;
s6-2: inputting the image B' of S6-1 into a distortion simulation module, generating disturbance by the distortion simulation module through a convolutional neural network, simulating unknown distortion or various post-processing operations possibly encountered by the image in the actual transmission process, and outputting a distortion simulation sample B ″ adv ，B″ adv Is still a 256 × 256 image of 3 channels;
s6-3: the distortion is simulated to a sample a ″ adb And the image B 'of S6-1 is input into a watermark extraction module, and watermark images w' are respectively extracted adv And w ";
s7: and repeating the step S6 until the loss function is converged, and generating the image B 'and the quality of the extracted watermark image w' is stable.
S8: testing and use, as shown in fig. 4;
s8-1: and inputting the image to be subjected to style migration depth editing and the watermark image to be embedded into the style migration depth editing module obtained in the step S7, and outputting the depth editing image embedded with the watermark.
S8-2: and inputting the deep editing image obtained in the step S8-1 into the watermark extraction module obtained in the step S7, and outputting a watermark image.
Further, in the steps S2 and S3, the quality of the image generated by the depth editing module is improved by selecting a mode of alternately training with the generator and the discriminator in the GAN, and the training target of the depth editing module is that the generated image is vivid enough, so that the discriminator module cannot judge whether the input image is a real image or a depth editing generated image, and the training target of the discriminator module is that whether the input image is real or not is effectively discriminated, and the training target and the discriminator finally achieve nash balance through a zero-sum game.
Further, in S5, network parameters of the discriminator module, the watermark extraction module, and the style migration depth editing module are sequentially updated until the loss function converges; and in the S7, network parameters of the discriminator module, the watermark extraction module, the distortion simulation module and the style migration depth editing module are updated in sequence until the loss function is converged.
Further, the loss function of the lattice migration depth editing module in step S2 is:
wherein the content of the first and second substances,
wherein x is an input image of the style migration depth editing module, y is a true value image corresponding to x, z is noise input into the style migration generator module for improving the diversity of the generated image, G (x, z) represents the generated image, w is a true value of the watermark image, w' is the watermark image extracted by the watermark extraction module, and λ and β are hyper-parameters;
generating a competing training function for the network for the conditions, D (x, y) being the output of the arbiter module whose optimization objective is to maximize
The style migration depth editing module should make the style migration depth editing module as possible
Becomes smaller, and the two reach Nash equilibrium through the countertraining;
to further improve the optimization function of the image generation quality,
selection of L 1 The distance reduces the fuzziness of the depth editing output image, so that the generated image is as close to the true value of the image as possible while the discriminator is confused;
representing the loss of the watermark image, the L2 distance is used for modeling the difference between the extracted watermark and the true value of the watermark, so that the watermark image embedded by the deep editing module can be extracted as completely as possible.
Further, in step S6, after the distortion simulation module is added for the joint training, the loss function of the style migration depth editing module is adjusted as follows:
wherein the content of the first and second substances,
wherein the content of the first and second substances,
and
in accordance with the above, λ, β, γ are hyper-parameters, G (x, z) is the generated image of the depth editing module, y' adv W is the true value of the watermark image, w is the watermark image extracted from G (x, z) by the watermark extraction module, w ″, for the distortion countermeasures sample generated by the distortion simulation module adv Is watermark extraction module from y' adv Extracting the watermark image;
for watermark distortion brought by adding a distortion simulation module, L2 distance is used for measuring watermark distortion, so that the watermark embedded in a generation module can effectively resist distortion;
in order to generate image distortion brought by adding the distortion simulation module, the L2 distance is used for measuring the image distortion intensity brought by the distortion simulation module, and the style migration network is constrained to generate a more robust image.
Further, the loss function of the watermark extraction module in step S4 is:
and the Dec represents a watermark extraction module, w is a true value of the watermark image, w' is the watermark image extracted by the watermark extraction module, and the watermark extraction module is constrained by the L2 distance to extract relatively complete watermark information, so that the visual quality of the extracted watermark image is ensured.
Further, in step S6, after the distortion simulation module is added for joint training, the loss function of the watermark extraction module is adjusted as follows:
where Dec represents a watermark extraction module, w is a true value of the watermark image, w "is the watermark image extracted from G (x, z) by the watermark extraction module, and w ″ adv Is watermark extraction Module from y' adv The watermark images extracted in (1), G (x, z) and y' adv The definition of (2) is consistent with the above, and the robustness of the watermark against the unknown distortion is further improved.
Further, in step S6, the loss function of the distortion simulation module is:
wherein G is adv For the distortion simulation module, G (x, z), y ′ adv ,w,w″ adv The meaning of (A) is in accordance with the foregoing, a 1 ，α 2 Is hyperparametric, α 1 Controlling the intensity of the distortion, alpha, produced by the distortion simulation module 2 The strength of the information loss generated by the distortion simulation module is controlled. Needs to pay attention to alpha in the training process 1 And alpha 2 Too large a strength of the distortion disturbance may result in a slow training process and may also reduce the adaptivity of the watermark to the distortion, and too small a strength of the distortion disturbance may result in an insufficient robustness of the watermark against the unknown distortion.
To verify the effectiveness and practicability of the present invention, facades is used as a training data set (more than 500 pairs), a model is trained according to steps S1-S7, adam is used as an optimizer of the model, the initial learning rate in the experiment is set to be 0.0002, and two ridge beam parameters are set to be beta 1 ＝0.5，β 2 =0.999. Training 200 iterations with training data 400 for training the model and 50 for testing the model, wherein the first 100 iterations only carry out the training of the deep editing module, the watermark extraction module and the discriminator module, and the last 100 iterations add the distortion simulationThe network performs joint training.
The method comprises the steps of using a test set to evaluate a model, evaluating a data set 50 to evaluate an image, using a trained model to test according to the step S8, using a non-reference image evaluation method FID, NIQE, PIQE and BRIQSUE to evaluate the quality of the generated image, wherein the scores of the non-reference image evaluation method FID, NIQE, PIQE and BRIQSUE are 148.879, 51.314,4.643 and 0.496, and the scores of the style depth editing model without embedding the watermark (the style migration depth editing module only comprises a style migration generator module) under four methods are 154.035, 61.405,5.495 and 0.506. According to the test in the step S8, the extracted watermark is evaluated, the average Mean Square Error (MSE) and the Structural Similarity (SSIM) are respectively 0.000722 and 0.943, and the visual quality of the extracted watermark image icon is high, which indicates that the watermark can be more completely embedded and extracted. And (5) testing according to the step (S8), carrying out post-processing operations such as brightness adjustment, hue, contrast, saturation, gaussian noise, gaussian blur and the like on the image obtained in the step (S8-1) to simulate image distortion in the practical application process, inputting the image subjected to the post-processing operations into a watermark extraction module, and ensuring higher visual quality and integrity of the output watermark, which shows that the embedded watermark has higher robustness. In conclusion, the invention is effective and feasible.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. An image watermarking system for style-migration-oriented deep editing, comprising: a style migration depth editing module, a discriminator module, a watermark extraction module and a distortion simulation module, wherein,
the style migration depth editing module comprises a style migration generator module, a watermark coding module and a characteristic fusion module; the style migration generator module is based on a U-Net generation framework, inputs an image to be edited and is a symmetrical structure of an encoder and a decoder, performs feature extraction on the image through the encoder, performs up-sampling on the image by using the decoder after converting the image into high-dimensional features, performs feature information sharing by using jump connection at corresponding stages of the encoder and the decoder, and retains image feature information of a middle layer at an image reconstruction stage; the watermark encoding module is used for extracting the characteristics of the input watermark image to be embedded based on the convolutional neural network, converting the watermark image into high-dimensional characteristics and keeping the output watermark characteristic information; the characteristic fusion module is used for fusing the image characteristic information and the watermark characteristic information of the middle layer, converting the fused characteristic and inputting the converted characteristic to a subsequent reconstruction layer to finish the style migration and watermark embedding of the image;
the discriminator module is based on a PatchGAN network and is used for judging the quality of the image output by the style migration depth editing module;
the watermark extraction module takes the output image of the style migration depth editing module as input and extracts the watermark image characteristics based on the convolutional neural network;
the distortion simulation module transforms an output image of the style migration depth editing module based on a convolutional neural network, inputs the output image into the watermark extraction module, and improves robustness of the watermark to unknown post-processing operation by adopting a mode of countertraining.
2. The image watermarking system for style transition depth editing according to claim 1,
the style migration generator module is constructed based on U-Net, the watermark coding module applies 5 multiplied by 5, padding is 2 of transposition convolution and an activation function ReLU;
after the characteristic fusion module applies splicing operation, performing characteristic dimension reduction by using a convolution layer of 1 multiplied by 1 and a ReLU activation function;
the watermark extraction module is constructed based on a convolutional neural network, firstly, 4 layers of convolutional network modules with 16 channels are used for characteristic decoupling of a host image and a watermark image, each convolutional network module selects a ReLU as an activation function, and then a downsampling module based on the convolutional network, an average pooling layer and a ReLU activation layer is used for extracting the watermark image;
the distortion simulation module is composed of two layers of convolution networks, wherein the first layer of convolution network changes an input characteristic channel from 3 to 16, the second layer of convolution network restores the number of the channels from 16 to 3, and LeakyReLU is used as an activation function between the two layers of convolution networks.
3. The image watermarking system for style transition depth editing according to claim 1 or 2, wherein the output dimensions of the style transition depth editing module, the watermark extracting module and the distortion simulating module are 3 channels.
4. The image watermarking system for style-oriented migration depth editing according to claim 1 or 2, wherein the image to be edited and the watermark image to be embedded are both 3-channel images, the image to be edited is 256 x 256 pixels, and the watermark image to be embedded is 64 x 64 pixels.
5. An image watermarking method facing style migration depth editing, characterized in that the image watermarking system of any one of claims 1-4 is adopted for image editing, and the method comprises the following steps:
s1: constructing a training data set;
s1-1: constructing a style migration training data set, and taking a pair-form image pair { A, B } as training data, wherein A and B are different description forms of the same scene;
s1-2: constructing a watermark image training data set, collecting a watermark image to be embedded as training data, preprocessing the watermark image to be embedded into a black-and-white watermark image w with the size of 64 multiplied by 64, white foreground color and black background color;
s2: inputting the training data set into a style migration depth editing module, and outputting a depth editing image;
s2-1: performing style migration training on the images { A, B } by using the image pair obtained in the step S1-1, wherein the migration direction is A → B, inputting the image A to be converted into a style migration generator module, and outputting the size of the image feature as C A *H A *W A ，C A 、H A 、W A The number, length and width of output channels are respectively;
s2-2: inputting the black-and-white watermark image w obtained in the step S1-2 into a watermark encoding module to obtain the high-dimensional characteristics of the watermark image, wherein the size of the output watermark characteristics is C w *H A *W A ，C w 、H A 、W A The number, length and width of output channels are respectively;
s2-3: the image characteristics C obtained in the step S2-1 are processed A *H A *W A And watermark characteristic C obtained by S2-2 w *H A *W A The input feature fusion module carries out splicing operation to obtain (C) A +C W )*H A *W A After fusing the features, using 1 × 1 convolution network to reduce the dimension to C A *H A *W A Inputting the image data into a subsequent reconstruction layer to finish style migration, and outputting an image B 'after the style migration is finished, wherein the image B' is a 3-channel 256 multiplied by 256 image which is a migration generated image after the image watermark is embedded;
s3: inputting the image B' obtained in the step S2-3 into a discriminator module, wherein the network output of the discriminator module is an NxN matrix, each element in the matrix only has two real or forged values, the matrix corresponds to a local area in the input image, and finally, the NxN output result is averaged to be used as the final judgment of the discriminator module on the input image;
s4: inputting the image B 'obtained in the step S2-3 into a watermark extraction module for watermark extraction, firstly decoupling watermark image characteristics and host image characteristics by using a convolutional neural network module, then reconstructing a watermark image, and outputting a 3-channel 64 x 64 watermark image w';
s5: repeating the step S2 to the step S4 until the loss function is converged and the quality of the image B' generated by style migration meets the given requirement;
s6: performing watermark robustness confrontation training, namely adding a distortion simulation module, and performing training and parameter updating simultaneously with a grid migration depth editing module, a discriminator module and a watermark extraction module;
s6-1: finishing the image style migration and watermark embedding according to the step S2 to obtain a generated image B ', and inputting the B' into a discriminator module;
s6-2: inputting the image B' into a distortion simulation module, generating disturbance by the distortion simulation module through a convolutional neural network, simulating distortion or post-processing operation encountered by the image in the actual transmission process, and outputting a distortion simulation sample B ″ adv ，B″ adv Size of 3 channels 256 × 256 images;
s6-3: the distortion is simulated as a sample B ″ adv Inputting the image B' into a watermark extraction module, and respectively extracting to obtain watermark images w ″ adv And w ";
s7: repeating the step S6 until the loss function is converged, and generating an image B 'and extracting a watermark image w' with the quality meeting the given requirement;
s8: testing and using;
s8-1: inputting the image to be subjected to style migration depth editing and the watermark image to be embedded into the style migration depth editing module obtained in the step S7, and outputting the watermark embedded depth editing image;
s8-2: and (4) inputting the depth editing image obtained in the step (8-1) into the watermark extraction module obtained in the step (7) and outputting a watermark image.
6. The image watermarking method facing style migration depth editing of claim 5, wherein in steps S2 and S3, a generator in GAN and a discriminator are selected for alternate training to improve the quality of the image generated by the style migration depth editing module, the training target of the style migration depth editing module is that the discriminator module cannot judge whether the input image is a real image or an image generated by depth editing, the training target of the discriminator module is that whether the input image is real or not is effectively distinguished, and the training targets and the discriminator module finally achieve nash balance through zero-sum game.
7. The image watermarking method facing style transition depth editing according to claim 6, wherein the loss function of the style transition depth editing module in the step S2 is:
wherein the content of the first and second substances,
wherein x is the input image of the style migration depth editing module, y is the true value image corresponding to x, z is the noise input into the style migration generator module, G (x, z) represents the generated image, w is the true value of the watermark image, w' is the watermark image extracted by the watermark extraction module, λ and β are the hyperparameters,
generating a confrontation training function of the network for the condition, D (x, y) is the output result of the discriminator module,
in order to improve the optimization function of the image generation quality,
representing a loss of the watermark image.
8. The image watermarking method facing style migration depth editing of claim 7, wherein the loss function of the style migration depth editing module in the step S6 is:
wherein the content of the first and second substances,
wherein λ, β, γ are hyper-parameters, G (x, z) is a generated image of the depth editing module, y' adv The distortion countercheck sample generated by the distortion simulation module is w is the true value of the watermark image, w 'is the watermark extracted from G (x, z) by the watermark extraction module, w' adv Is watermark extraction module from y' adv The watermark extracted in (1);
watermark distortion brought by adding a distortion simulation module;
brought after adding a distortion simulation moduleThe resulting image distortion.
9. The image watermarking method for style-migration-depth-editing-oriented editing according to claim 8,
the loss function of the watermark extraction module in step S4 is:
in the formula, dec represents a watermark extraction module, w is a watermark image true value, and w' is a watermark image extracted by the watermark extraction module;
the loss function of the watermark extraction module in step S6 is:
in the formula, dec represents a watermark extraction module, w is a true value of the watermark image, w "is the watermark image extracted from G (x, z) by the watermark extraction module, w ″ adv Is watermark extraction module from y' adv The extracted watermark image.
10. The method for watermarking an image for style-migration-depth editing according to claim 9, wherein the loss function of the distortion simulation module in step S6 is:
in the formula, G adv For distortion simulation modules, α 1 ，α 2 Is hyperparametric, α 1 Controlling the intensity of the distortion, alpha, produced by the distortion simulation module 2 The strength of the information loss generated by the distortion simulation module is controlled.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211109211.XA CN115311117A (en) | 2022-09-13 | 2022-09-13 | Image watermarking system and method for style migration depth editing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211109211.XA CN115311117A (en) | 2022-09-13 | 2022-09-13 | Image watermarking system and method for style migration depth editing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115311117A true CN115311117A (en) | 2022-11-08 |
Family
ID=83866994
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211109211.XA Pending CN115311117A (en) | 2022-09-13 | 2022-09-13 | Image watermarking system and method for style migration depth editing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115311117A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117314713A (en) * | 2023-07-12 | 2023-12-29 | 惠州市悠悠网络科技有限公司 | License anti-counterfeiting data processing method based on digital watermark |
-
2022
- 2022-09-13 CN CN202211109211.XA patent/CN115311117A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117314713A (en) * | 2023-07-12 | 2023-12-29 | 惠州市悠悠网络科技有限公司 | License anti-counterfeiting data processing method based on digital watermark |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109587372B (en) | Invisible image steganography based on generation of countermeasure network | |
| CN111292264B (en) | Image high dynamic range reconstruction method based on deep learning | |
| Chen et al. | The face image super-resolution algorithm based on combined representation learning | |
| CN111062872A (en) | Image super-resolution reconstruction method and system based on edge detection | |
| CN112288627B (en) | Recognition-oriented low-resolution face image super-resolution method | |
| CN111161158B (en) | Image restoration method based on generated network structure | |
| CN112884758B (en) | Defect insulator sample generation method and system based on style migration method | |
| CN112115951A (en) | RGB-D image semantic segmentation method based on spatial relationship | |
| CN115311117A (en) | Image watermarking system and method for style migration depth editing | |
| CN115809953A (en) | Attention mechanism-based multi-size image robust watermarking method and system | |
| CN113298689B (en) | Large-capacity image steganography method | |
| CN113689382B (en) | Tumor postoperative survival prediction method and system based on medical images and pathological images | |
| CN113887573A (en) | Human face forgery detection method based on visual converter | |
| CN113076806A (en) | Structure-enhanced semi-supervised online map generation method | |
| CN116721320A (en) | Universal image tampering evidence obtaining method and system based on multi-scale feature fusion | |
| CN116703885A (en) | Swin transducer-based surface defect detection method and system | |
| Hepburn et al. | Enforcing perceptual consistency on generative adversarial networks by using the normalised laplacian pyramid distance | |
| CN114764754B (en) | Occlusion face restoration method based on geometric perception priori guidance | |
| CN114529794B (en) | Infrared and visible light image fusion method, system and medium | |
| CN114782568A (en) | Multi-scale stage feature progressive fusion remote sensing image coloring method and system | |
| CN114549341A (en) | Sample guidance-based face image diversified restoration method | |
| Surabhi et al. | Advancing Faux Image Detection: A Hybrid Approach Combining Deep Learning and Data Mining Techniques | |
| CN113538615B (en) | Remote sensing image coloring method based on double-flow generator depth convolution countermeasure generation network | |
| CN118037641A (en) | Multi-scale image tampering detection and positioning method based on double-flow feature extraction | |
| CN116563147B (en) | Underwater image enhancement system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |