CN116703687A - Image generation model processing, image generation method, image generation device and computer equipment - Google Patents
Image generation model processing, image generation method, image generation device and computer equipment Download PDFInfo
- Publication number
- CN116703687A CN116703687A CN202310961674.7A CN202310961674A CN116703687A CN 116703687 A CN116703687 A CN 116703687A CN 202310961674 A CN202310961674 A CN 202310961674A CN 116703687 A CN116703687 A CN 116703687A
- Authority
- CN
- China
- Prior art keywords
- image
- watermark
- initial
- target
- original
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000012545 processing Methods 0.000 title claims abstract description 29
- 238000012549 training Methods 0.000 claims abstract description 101
- 238000004590 computer program Methods 0.000 claims abstract description 35
- 239000013598 vector Substances 0.000 claims description 107
- 238000004364 calculation method Methods 0.000 claims description 53
- 238000000605 extraction Methods 0.000 claims description 32
- 238000003672 processing method Methods 0.000 abstract description 16
- 238000010586 diagram Methods 0.000 description 13
- 238000013528 artificial neural network Methods 0.000 description 12
- 238000004422 calculation algorithm Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 12
- 238000006243 chemical reaction Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 10
- 241001465754 Metazoa Species 0.000 description 9
- 238000004891 communication Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 230000008859 change Effects 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000010428 oil painting Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/0021—Image watermarking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
- G06T9/002—Image coding using neural networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- Multimedia (AREA)
- Editing Of Facsimile Originals (AREA)
- Image Processing (AREA)
Abstract
The application relates to an image generation model processing method, an image generation model processing device, a computer device, a storage medium and a computer program product. The method comprises the following steps: inputting an original image into an original watermark image generation model to generate an original image matched initial generation image, coding the initial generation image and the original watermark to obtain a target watermark, fusing the target watermark and the initial generation image to obtain a target generation image, and extracting the watermark from the target generation image to obtain the initial watermark. Calculating initial image quality loss information of a target generated image and an initial generated image, calculating initial watermark loss information of an original watermark and the initial watermark, and performing iterative training by using the initial image quality loss information and the initial watermark loss information to obtain a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to an input image and the input watermark. The method can improve the image quality of the generated watermark image.
Description
Technical Field
The present application relates to the field of computer vision, and in particular, to an image generation model processing method, an image generation device, a computer apparatus, a storage medium, and a computer program product.
Background
With the development of computer vision, AIGC (AI generated content, artificial intelligence generation content) technology has emerged. The graph generation task can be realized through AIGC, namely, by inputting images into an AIGC model, the AIGC model can change the images into another style, but the contents are largely unchanged. Currently, images generated by the AIGC model require embedding watermarks to protect the intellectual property of the AIGC generated images. For example, a dark watermark can be added into an image generated by the AIGC through an image steganography technology, so that copyright attribution of the image is clarified. However, watermark embedding is performed on the generated image by a watermark adding module after the image is generated by the AIGC model, which tends to reduce the image quality of the resulting watermark embedded image.
Disclosure of Invention
In view of the foregoing, it is desirable to provide an image generation model process, an image generation method, an apparatus, a computer device, a computer-readable storage medium, and a computer program product that can improve the image quality of an embedded watermark image.
In a first aspect, the present application provides an image generation model processing method. The method comprises the following steps:
acquiring an original image and an original watermark;
inputting an original image into an initial watermark image generation model, performing matching image generation by the initial watermark image generation model based on the original image to obtain an initial generated image, encoding the initial generated image and the original watermark to obtain a target watermark, fusing the target watermark and the initial generated image to obtain a target generated image, and performing watermark extraction based on the target generated image to obtain the initial watermark;
performing image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and performing watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information;
updating an initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the iterative execution of the steps of acquiring the original image and the original watermark until the training completion condition is reached, obtaining a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark.
In a second aspect, the application further provides an image generation model processing device. The device comprises:
the image acquisition module is used for acquiring an original image and an original watermark;
the model processing module is used for inputting the original image into the initial watermark image generation model, performing matching image generation on the basis of the original image by the initial watermark image generation model to obtain an initial generated image, encoding the initial generated image and the original watermark to obtain a target watermark, fusing the target watermark with the initial generated image to obtain a target generated image, and performing watermark extraction on the basis of the target generated image to obtain the initial watermark;
the loss calculation module is used for carrying out image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and carrying out watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information;
the iteration module is used for updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, returning to the step of acquiring the original image and the original watermark, and carrying out iteration until the training completion condition is reached, obtaining a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:
acquiring an original image and an original watermark;
inputting an original image into an initial watermark image generation model, performing matching image generation by the initial watermark image generation model based on the original image to obtain an initial generated image, encoding the initial generated image and the original watermark to obtain a target watermark, fusing the target watermark and the initial generated image to obtain a target generated image, and performing watermark extraction based on the target generated image to obtain the initial watermark;
performing image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and performing watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information;
updating an initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the iterative execution of the steps of acquiring the original image and the original watermark until the training completion condition is reached, obtaining a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark.
In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
acquiring an original image and an original watermark;
inputting an original image into an initial watermark image generation model, performing matching image generation by the initial watermark image generation model based on the original image to obtain an initial generated image, encoding the initial generated image and the original watermark to obtain a target watermark, fusing the target watermark and the initial generated image to obtain a target generated image, and performing watermark extraction based on the target generated image to obtain the initial watermark;
performing image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and performing watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information;
updating an initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the iterative execution of the steps of acquiring the original image and the original watermark until the training completion condition is reached, obtaining a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark.
In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:
acquiring an original image and an original watermark;
inputting an original image into an initial watermark image generation model, performing matching image generation by the initial watermark image generation model based on the original image to obtain an initial generated image, encoding the initial generated image and the original watermark to obtain a target watermark, fusing the target watermark and the initial generated image to obtain a target generated image, and performing watermark extraction based on the target generated image to obtain the initial watermark;
performing image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and performing watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information;
updating an initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the iterative execution of the steps of acquiring the original image and the original watermark until the training completion condition is reached, obtaining a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark.
The image generation model processing method, the image generation model processing device, the computer equipment, the storage medium and the computer program product are used for acquiring an original image and an original watermark; inputting an original image into an initial watermark image generation model, performing matching image generation by the initial watermark image generation model based on the original image to obtain an initial generated image, encoding the initial generated image and the original watermark to obtain a target watermark, fusing the target watermark and the initial generated image to obtain a target generated image, and performing watermark extraction based on the target generated image to obtain the initial watermark; performing image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and performing watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information; updating an initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the iterative execution of the step of acquiring the original image and the original watermark until the training completion condition is reached, thereby obtaining a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark, namely, the initial watermark image generation model is trained by calculating the image quality loss information and the watermark loss information, so that the watermark can be better adapted to the image when the watermark is embedded into the trained target watermark image generation model, the image quality influence on the target watermark image after the watermark is embedded is reduced, and the image quality of the generated target watermark image is improved.
In a first aspect, the present application provides an image generation method. The method comprises the following steps:
obtaining a target image and a watermark to be embedded;
inputting the target image and the watermark to be embedded into a target watermark image generation model to obtain a target image and a target watermark image corresponding to the watermark to be embedded;
the target watermark image generation model is obtained by updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of obtaining the original image and the original watermark for iterative execution until reaching the training completion condition; the initial image quality loss information is obtained by performing image quality loss calculation based on the target generated image and the initial generated image; the initial watermark loss information is obtained by watermark loss calculation based on the original watermark and the initial watermark; the initial generated image is generated by matching an image based on the original image through an initial watermark image generation model; the target generated image is obtained by encoding the initial generated image and the original watermark through an initial watermark image generation model to obtain a target watermark and fusing the target watermark and the initial generated image; the initial watermark is obtained by watermark extraction based on the target generated image through an initial watermark image generation model.
In a second aspect, the application further provides an image generation device. The device comprises:
the acquisition module is used for acquiring the target image and the watermark to be embedded;
the image generation module is used for inputting the target image and the watermark to be embedded into the target watermark image generation model to obtain a target image and a target watermark image corresponding to the watermark to be embedded; the target watermark image generation model is obtained by updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of obtaining the original image and the original watermark for iterative execution until reaching the training completion condition; the initial image quality loss information is obtained by performing image quality loss calculation based on the target generated image and the initial generated image; the initial watermark loss information is obtained by watermark loss calculation based on the original watermark and the initial watermark; the initial generated image is generated by matching an image based on the original image through an initial watermark image generation model; the target generated image is obtained by encoding the initial generated image and the original watermark through an initial watermark image generation model to obtain a target watermark and fusing the target watermark and the initial generated image; the initial watermark is obtained by watermark extraction based on the target generated image through an initial watermark image generation model.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:
obtaining a target image and a watermark to be embedded;
inputting the target image and the watermark to be embedded into a target watermark image generation model to obtain a target image and a target watermark image corresponding to the watermark to be embedded;
the target watermark image generation model is obtained by updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of obtaining the original image and the original watermark for iterative execution until reaching the training completion condition; the initial image quality loss information is obtained by performing image quality loss calculation based on the target generated image and the initial generated image; the initial watermark loss information is obtained by watermark loss calculation based on the original watermark and the initial watermark; the initial generated image is generated by matching an image based on the original image through an initial watermark image generation model; the target generated image is obtained by encoding the initial generated image and the original watermark through an initial watermark image generation model to obtain a target watermark and fusing the target watermark and the initial generated image; the initial watermark is obtained by watermark extraction based on the target generated image through an initial watermark image generation model.
In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
obtaining a target image and a watermark to be embedded;
inputting the target image and the watermark to be embedded into a target watermark image generation model to obtain a target image and a target watermark image corresponding to the watermark to be embedded;
the target watermark image generation model is obtained by updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of obtaining the original image and the original watermark for iterative execution until reaching the training completion condition; the initial image quality loss information is obtained by performing image quality loss calculation based on the target generated image and the initial generated image; the initial watermark loss information is obtained by watermark loss calculation based on the original watermark and the initial watermark; the initial generated image is generated by matching an image based on the original image through an initial watermark image generation model; the target generated image is obtained by encoding the initial generated image and the original watermark through an initial watermark image generation model to obtain a target watermark and fusing the target watermark and the initial generated image; the initial watermark is obtained by watermark extraction based on the target generated image through an initial watermark image generation model.
In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:
obtaining a target image and a watermark to be embedded;
inputting the target image and the watermark to be embedded into a target watermark image generation model to obtain a target image and a target watermark image corresponding to the watermark to be embedded;
the target watermark image generation model is obtained by updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of obtaining the original image and the original watermark for iterative execution until reaching the training completion condition; the initial image quality loss information is obtained by performing image quality loss calculation based on the target generated image and the initial generated image; the initial watermark loss information is obtained by watermark loss calculation based on the original watermark and the initial watermark; the initial generated image is generated by matching an image based on the original image through an initial watermark image generation model; the target generated image is obtained by encoding the initial generated image and the original watermark through an initial watermark image generation model to obtain a target watermark and fusing the target watermark and the initial generated image; the initial watermark is obtained by watermark extraction based on the target generated image through an initial watermark image generation model.
The image generation method, the device, the computer equipment, the storage medium and the computer program product are used for obtaining the target image and the watermark to be embedded, and inputting the target image and the watermark to be embedded into a target watermark image generation model to obtain the target image and the target watermark image corresponding to the watermark to be embedded; the target watermark image generation model is obtained by updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of obtaining the original image and the original watermark for iterative execution until reaching the training completion condition; the initial image quality loss information is obtained by performing image quality loss calculation based on the target generated image and the initial generated image; the initial watermark loss information is obtained by watermark loss calculation based on the original watermark and the initial watermark; the initial generated image is generated by matching an image based on the original image through an initial watermark image generation model; the target generated image is obtained by encoding the initial generated image and the original watermark through an initial watermark image generation model to obtain a target watermark and fusing the target watermark and the initial generated image; the initial watermark is obtained by watermark extraction based on the target generated image through an initial watermark image generation model. The initial watermark image generation model is trained by calculating the image quality loss information and the watermark loss information, so that the watermark can be better adapted to the image when the watermark is embedded in the target watermark image generation model obtained by training, the influence on the image quality of the target watermark image after the watermark is embedded is reduced, and the image quality of the generated target watermark image is improved.
Drawings
FIG. 1 is a diagram of an application environment for an image generation model processing method in one embodiment;
FIG. 2 is a flow diagram of a method of image generation model processing in one embodiment;
FIG. 3 is a flow diagram of obtaining a target generated image in one embodiment;
FIG. 4 is a flow chart of obtaining initial image quality loss information according to an embodiment;
FIG. 5 is a flow diagram of an embodiment of obtaining an initial watermark;
FIG. 6 is a flow diagram of an updated watermark image generation model in one embodiment;
FIG. 7 is a flow diagram of an image generation method in one embodiment;
FIG. 8 is a flow diagram of an image generation model process in one embodiment;
FIG. 9 is a diagram of a network architecture for training a watermark image generation model in one embodiment;
FIG. 10 is a block diagram showing a structure of an image generation model processing apparatus in one embodiment;
FIG. 11 is a block diagram showing the structure of an image generating apparatus in one embodiment;
FIG. 12 is an internal block diagram of a computer device in one embodiment;
fig. 13 is an internal structural view of a computer device in another embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
Computer Vision (CV) is a science of studying how to "look" a machine, and more specifically, to replace human eyes with a camera and a Computer to perform machine Vision such as recognition, following and measurement on a target, and further perform graphic processing, so that the Computer is processed into an image more suitable for human eyes to observe or transmit to an instrument to detect. As a scientific discipline, computer vision research-related theory and technology has attempted to build artificial intelligence systems that can acquire information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning and mapping, autopilot, intelligent transportation, etc., as well as common biometric technologies such as face recognition, fingerprint recognition, etc.
The scheme provided by the embodiment of the application relates to technologies such as image processing of artificial intelligence, and is specifically described by the following embodiments:
the image generation model processing method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be provided separately, may be integrated on the server 104, or may be located on a cloud or other server. The server 104 may obtain the original image and the original watermark from the data storage system. Then, the server 104 inputs the original image into an initial watermark image generation model, the initial watermark image generation model performs matching image generation based on the original image to obtain an initial generated image, encodes the initial generated image and the original watermark to obtain a target watermark, fuses the target watermark and the initial generated image to obtain a target generated image, and performs watermark extraction based on the target generated image to obtain the initial watermark. The server 104 performs image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and performs watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information. The server 104 updates the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returns to the iterative execution of the steps of acquiring the original image and the original watermark until reaching the training completion condition, a target watermark image generation model is obtained, and the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark. The terminal 102 may be, but not limited to, various desktop computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, where the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server 104 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services. The terminal and the server may be directly or indirectly connected through wired or wireless communication, and the present application is not limited herein.
In one embodiment, as shown in fig. 2, an image generation model processing method is provided, and the method is taken as an example for describing that the method is applied to the server in fig. 1, it is understood that the method can also be applied to a terminal, and can also be applied to a system comprising the terminal and the server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:
s202, acquiring an original image and an original watermark.
The original image refers to an image used in training, and the original image may be an image in any image format, and the image format may be a JPEG (Joint Photographic Experts Group, a product of the JPEG standard), PNG (Portable NetworkGraphics ) format, GIF format (a compressed picture format), or the like. The original image may also be any type of image, for example, the original image may be an animal image, a plant image, a place image, a person image, an object image, or the like. The original watermark refers to a digital watermark used in training, and is usually in the form of pictures, character strings and the like, for example, the original watermark can be a picture marked by a platform, the original watermark can be a character string of company or department names, and the original watermark can be a character string of a video or image self identification number and the like. The original watermark may be either a watermark that is a superficial or a watermark that is a hidden watermark. The emerging watermark can be seen by naked eyes when embedded in an image, and the hidden watermark cannot be seen by naked eyes when embedded in an image, for example, the hidden watermark can be embedded in the image by an image steganography technology, which is a technology of hiding information in an image and recovering original information from the image by an additional means.
In particular, the server may obtain from the database the original image and the original watermark used in the training. The server may obtain the original image and the original watermark used in training from the server providing the data service. The server may also obtain the original image and the original watermark used in training from the internet. The server may also obtain the original image and the original watermark uploaded by the terminal. The server may also obtain the original image and the original watermark from a service party providing the business service.
S204, inputting the original image into an initial watermark image generation model, performing matching image generation by the initial watermark image generation model based on the original image to obtain an initial generated image, encoding the initial generated image and the original watermark to obtain a target watermark, fusing the target watermark and the initial generated image to obtain a target generated image, and performing watermark extraction based on the target generated image to obtain the initial watermark.
The initial watermark image generation model is a watermark image generation model initialized by model parameters, and the watermark image generation model is used for generating an image embedded with the digital watermark. The model parameter initialization may be zero initialization, gaussian distribution initialization, random initialization, initialization using pre-training parameters, etc. The initial watermark image generation model may be built using a deep neural network. The initial generated image refers to an image matched with the original image generated by using the parameters of the initial model, the generated image can be an image matched with the content of the original image and the style is different, and the style refers to an artistic style, for example, a film style, a cartoon style, an oil painting style, a watercolor style, a water ink style and the like. The style of the image to be generated may be preset, may be determined according to a selection of a user, or may be determined according to a style of a reference image input by the user. The target watermark is a digital watermark obtained by updating the original watermark according to the picture quality of the original watermark image, and the picture quality of the picture corresponding to the target watermark can be as close as possible to the picture quality of the generated image through training, so that the influence of the watermark on the picture quality of the generated watermark image can be reduced to the minimum. The target generated image refers to an image in which a target watermark is embedded in the initial generated image. The initial watermark refers to a digital watermark extracted from the target generated image using the initialized model parameters.
Specifically, the server uses the deep neural network to build a watermark image generation model, and initializes model parameters of the watermark image generation model to obtain an initial watermark image generation model. The server then inputs the original image into the initial watermark image generation model. The initial watermark image generation model uses trained matching image generation parameters to generate an initial generated image of the original image match. The trained matching image generation parameters may be pre-trained and used as parameters for matching image generation in the initial watermark image generation model. The original image can be converted into an image vector, and then the image vector is used for matching image generation, so that an initial generated image vector is obtained. And then converting the original watermark into an original watermark vector, and recoding the original watermark vector according to the image quality information in the initially generated image vector to obtain a target watermark vector, thus obtaining the target watermark. And finally, vector fusion can be carried out on the target watermark vector and the initial generated image vector, for example, the sum of the target watermark vector and the initial generated image vector can be calculated, the product of the target watermark vector and the initial generated image vector can be calculated, and the target generated image can be obtained. And extracting the watermark from the target generated image through initialized watermark extraction parameters to obtain an initial watermark.
S206, performing image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and performing watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information.
Wherein the initial image quality loss information is used to characterize an image quality error between the target generated image and the initial generated image. The smaller the initial image quality loss information, the smaller the degree of image quality change between the generated image and the watermark embedded image. The original watermark loss information refers to an error between the original watermark and the original watermark, and the information of the original watermark is ensured to be consistent with the information of the original watermark.
Specifically, the server calculates an error between the target generated image and the initial generated image using a loss function, for example, loss information between a vector corresponding to the target generated image and a vector corresponding to the initial generated image may be calculated, and then loss information between a vector corresponding to the original watermark and a vector corresponding to the initial watermark may be calculated, so as to obtain initial image quality loss information and initial watermark loss information, where the loss function may be a least squares error loss function, a mean square error loss function, an average data pair error loss function, or the like.
And S208, updating an initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of acquiring the original image and the original watermark for iterative execution until reaching the training completion condition, obtaining a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark.
The training completion condition refers to a condition that a model is generated by a target watermark image obtained through training, and the training completion condition comprises, but is not limited to, that the iteration number reaches a preset maximum iteration number, that the loss information of the model reaches a preset loss information threshold, that the parameters of the model are not changed any more, and the like. The target watermark image generation model refers to a trained deep neural network model for generating an embedded digital watermark image, i.e. the target watermark image generation model can generate a target watermark image according to an input image and an input watermark. The input image is an image that needs to be input into the target watermark image generation model for image generation. An input watermark refers to a digital watermark that needs to be embedded in an image generated from an input image. The target watermark image refers to the generated image embedded with the input watermark and matched with the input image.
The server calculates the sum of the original image quality loss information and the original watermark loss information, d starves to the trained total loss information, and then reversely updates initialization parameters in the original watermark image generation model through the trained total loss information by using a gradient descent algorithm to obtain an updated watermark image generation model. And then the server takes the updated watermark image generation model as an initial watermark image generation model, and carries out iterative training, namely, returns to the step of acquiring the original image and the original watermark for iterative execution, and takes the initial watermark image generation model when the training completion condition is reached as a final training to obtain a target watermark image generation model when the training completion condition is reached. The target watermark image generation model may then be deployed and used. When the watermark generation method is used, the server acquires the input image and the input watermark, and inputs the input image and the input watermark into the target watermark image generation model to generate the watermark image, so as to obtain the target watermark image.
The image generation model processing method, the image generation model processing device, the computer equipment, the storage medium and the computer program product are used for acquiring an original image and an original watermark; inputting an original image into an initial watermark image generation model, performing matching image generation by the initial watermark image generation model based on the original image to obtain an initial generated image, encoding the initial generated image and the original watermark to obtain a target watermark, fusing the target watermark and the initial generated image to obtain a target generated image, and performing watermark extraction based on the target generated image to obtain the initial watermark; performing image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and performing watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information; updating an initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the iterative execution of the step of acquiring the original image and the original watermark until the training completion condition is reached, thereby obtaining a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark, namely, the initial watermark image generation model is trained by calculating the image quality loss information and the watermark loss information, so that the watermark can be better adapted to the image when the watermark is embedded into the trained target watermark image generation model, the image quality influence on the target watermark image after the watermark is embedded is reduced, and the image quality of the generated target watermark image is improved.
In one embodiment, as shown in fig. 3, S204, encoding the initial generated image and the original watermark to obtain a target watermark, and fusing the target watermark with the initial generated image to obtain a target generated image, includes:
s302, vectorizing the initial generated image and the original watermark respectively to obtain an initial generated image vector and an original watermark vector.
The initial generated image vector refers to a vector corresponding to the initial generated image, and the original watermark vector is a vector for representing the original watermark.
Specifically, the server performs vector conversion on the initially generated image, for example, the pixel values in the initially generated image can be directly used as element values in the vector to obtain the initially generated image vector. And then vectorizing the original watermark to obtain a corresponding original watermark vector. In one embodiment, when the original watermark is a string text, the original watermark may be vectorized by a text vectorization algorithm to obtain an original watermark vector. The text vectorization algorithm may be thermal coding, word bag model, TF-IDF (Term Frequency-Inverse Document Frequency word Frequency-reverse file Frequency), neural network language model, etc. When the original watermark is an original watermark picture, the original watermark picture can be vectorized through an image vectorization algorithm to obtain an original watermark vector. For example, the pixel value of the original watermark picture may be taken as the original watermark vector by extracting the pixel value of the original watermark picture.
S304, extracting image quality characteristics corresponding to the initially generated image vector, and updating the original watermark vector based on the image quality characteristics to obtain a target watermark vector.
S306, superposing the target watermark vector and the initial generated image vector to obtain a target generated image.
The image quality feature refers to a feature vector used for representing the image quality of an initially generated image, and may include feature vectors of different scales.
Specifically, the server may extract image quality features corresponding to the initially generated image vectors using a deep neural network algorithm, where the deep neural network may be a convolutional neural network, a recurrent neural network, a feed forward neural network, or the like. And updating the original watermark vector by using the image quality characteristics, wherein the vector sum between the image quality characteristics and the original watermark vector can be calculated, then carrying out convolution operation to obtain a target watermark vector, and finally, superposing the target watermark vector and the initial generated image vector by the server to obtain a target generated image, namely, calculating the vector sum between the target watermark vector and the initial generated image vector by the server to obtain the target generated image.
In the above embodiment, the initially generated image and the original watermark are respectively vectorized to obtain an initially generated image vector and an original watermark vector, then image quality features corresponding to the initially generated image vector are extracted, and the original watermark vector is updated based on the image quality features to obtain the target watermark vector. And superposing the target watermark vector and the initial generated image vector to obtain a target generated image. The original watermark vector is updated through the image quality characteristics and then is overlapped with the original generated image vector, so that the influence on the image quality of the generated image after watermark embedding can be reduced, and the image quality of the obtained target generated image can be improved.
In one embodiment, as shown in fig. 4, S206, performing image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, includes:
s402, calculating peak signal-to-noise ratio between the target generated image and the initial generated image, and obtaining image-level image quality loss information.
S404, calculating the module length between the target generated image and the initial generated image, and obtaining pixel-level image quality loss information.
S406, calculating the sum of the image quality loss information at the image level and the image quality loss information at the pixel level to obtain initial image quality loss information.
The peak signal-to-noise ratio is an index for measuring the image quality. The image-level image quality loss information refers to loss information calculated using a peak signal-to-noise ratio for characterizing image-level image quality loss. The pixel-level image quality loss information refers to loss information calculated using a least squares error loss function, and is used to characterize the image quality loss at the pixel level.
Specifically, the server calculates a peak signal-to-noise ratio between the target generated image and the initial generated image, and uses the calculated peak signal-to-noise ratio value as image-level image quality loss information, wherein the server may calculate a mean square error between the target generated image and the initial generated image, and then calculate the peak signal-to-noise ratio by the mean square error. The server may also directly use the peak signal-to-noise ratio function to calculate the peak signal-to-noise ratio between the target generated image and the initial generated image. The server may also convert the target generated image and the initial generated image into images in YCbCr (one of color spaces) format, and then calculate the peak signal-to-noise ratio of the luminance component Y, resulting in image-level image quality loss information. And then the server calculates the modular length between the initial generated image and the target generated image by using the least square error loss function to obtain pixel-level image quality loss information. And finally, the server adds the image quality loss information at the image level and the image quality loss information at the pixel level to obtain initial image quality loss information.
In a specific embodiment, the initial image quality loss information may be calculated using equation (1) as shown below.
Formula (1)
Where loss1 refers to the original image quality loss information, I refers to the original generated image,refers to the object generating image. PSNR (Peak signal-to-noise ratio) refers to Peak signal-to-noise ratio, which is used to constrain image quality. />For characterizing image-level image quality loss information. />Refers to L2 norm, < >>The pixel-level image quality loss information is used for representing the image quality constraint of a pixel level with finer granularity. The image quality loss after embedding the watermark is constrained by calculating the initial image quality loss information.
In the above embodiment, by calculating the image-level image quality loss information and the pixel-level image quality loss information and then calculating the sum of the image-level image quality loss information and the pixel-level image quality loss information, the initial image quality loss information, that is, the image quality loss information is calculated from the image level and the pixel level, the accuracy of the obtained initial image quality loss information is improved.
In one embodiment, S204, inputting the original image into an initial watermark image generation model, and performing matching image generation by the initial watermark image generation model based on the original image to obtain an initial generated image, including the steps of:
Acquiring a reference image, inputting an original image and the reference image into an initial watermark image generation model, and respectively vectorizing the original image and the reference image by the initial watermark image generation model to obtain an original image vector and a reference image vector; the initial watermark image generation model generates an initial generated image matched with the original image vector according to the reference image vector.
The reference image refers to an image to be referred to when generating an image matched with the original image, that is, the style of the generated image needs to be consistent with the style of the reference image.
Specifically, the server may acquire the reference image from the database, may acquire the reference image uploaded by the terminal, may acquire the reference image sent by the server providing the data service, may acquire the reference image from the server providing the service, and may acquire the reference image from the internet. The server then inputs the original image and the reference image into an initial watermark image generation model. The initial watermark image generation model simultaneously carries out vectorization on an original image and a reference image through a vectorization network to obtain an original image vector and a reference image vector, wherein vectorization can be carried out through vectorization parameters in the vectorization network, and the vectorization parameters can be obtained through initialization or can be obtained from a pre-trained vectorization model. And finally, the server generates an initial generated image matched with the original image vector according to the reference image vector through a generated network, wherein the generated network can be a pre-trained network for generating the image, and the network structure and the network parameters of the generated network can be the same model structure and model parameters of the AIGC model.
In the above embodiment, the original image and the reference image are input into the initial watermark image generation model by acquiring the reference image, and then the initial watermark image generation model generates the initial generated image matched with the original image vector according to the reference image vector, so that the generated image accords with the reference image, and the accuracy of image generation is improved.
In one embodiment, the initial watermark image generation model includes a trained image generation network, an initial encoding network, and an initial decoding network;
as shown in fig. 5, S204, inputting an original image into an initial watermark image generation model, performing matching image generation based on the original image by the initial watermark image generation model to obtain an initial generated image, encoding the initial generated image and the original watermark to obtain a target watermark, fusing the target watermark with the initial generated image to obtain a target generated image, and performing watermark extraction based on the target generated image to obtain the initial watermark, including:
s502, inputting the original image into an initial watermark image generation model, and performing matching image generation on the original image through a trained image generation network in the initial watermark image generation model to obtain an initial generated image.
The trained image generation type network refers to a trained deep neural network for generating images by using input images. For example, the trained image-generating network may be an AIGC model, i.e., the network structure and network parameters of the trained image-generating network are the same as the model structure and model parameters of the AIGC model.
Specifically, the server inputs an original image into an initial watermark image generation model, and performs matching image generation on the original image by using image generation parameters through a trained image generation type network in the initial watermark image generation model to obtain an initial generated image. The server may obtain a model file of the AIGC model and then build an image-wise network in the initial watermark image-wise model from the model file.
In one embodiment, the network parameters of the image generation network may be obtained by initializing, and then training the image generation network during training, that is, updating the network parameters of the image generation network simultaneously, and when training is completed, obtaining a trained watermark image generation model, where the trained watermark image generation model includes the trained image generation network.
S504, coding the initial generated image and the original watermark through an initial coding network to obtain a target watermark, and fusing the target watermark and the initial generated image to obtain a target generated image.
The initial coding network refers to a coding network initialized by network parameters, the coding network is used for fusing the watermark and the generated image, and the coding network is established by using a deep neural network.
Specifically, the server encodes the initial generated image and the original watermark through initialized coding parameters in the initial coding network to obtain the target watermark. The initial coding network can extract the image quality adjustment corresponding to the initial generated image, and then update the original watermark by using the image quality characteristics to obtain the target watermark. That is, the image quality of the target watermark image can be kept consistent with that of the initially generated image. And finally, fusing the target watermark with the initial generated image to obtain the target generated image, wherein the fusing can be realized by superposing the target watermark with the initial generated image, calculating the vector product of the corresponding vector of the target watermark and the corresponding vector of the initial generated image, and the like.
In a specific embodiment, the target generated image may be obtained using the following formula (2) and formula (3).
Formula (2)
Formula (3)
Wherein I refers to an initially generated image,refers to the original watermark, E refers to the original encoding network,/->It is referred to as a target watermark,refers to the object generating image. The target watermark is obtained through an initial coding network, and then the target watermark is added with the initial generated image to obtain the target generated image.
S506, watermark extraction is carried out on the target generated image through the initial decoding network, and an initial watermark is obtained.
The initial decoding network refers to a decoding network initialized by network parameters, and the decoding network is used for extracting watermarks from an image with watermarks, wherein the watermarks in the image can be either emerging watermarks or hidden watermarks.
Specifically, the server extracts the watermark from the target generated image through initialized decoding parameters in the initial decoding network to obtain the initial watermark, wherein the decoding parameters can be obtained through initialization or obtained through pre-training. The initial decoding network is also established using a deep neural network.
In a specific embodiment, the watermark extraction may be performed using equation (4) shown in the image, resulting in an initial watermark.
Formula (4)
Wherein,,refers to the initial watermark. />Refers to the object generating image. D refers to the initial decoding network. And it is desirable to minimize the impact of embedded watermark on image quality without affecting watermark extraction, which can be expressed using equation (5)Showing the
Formula (5)
Wherein the optimization target is an initial watermark extracted by an initial decoding networkIs ++original watermark>As close as possible, i.e. minimized.
In the above embodiment, the original image is input into the initial watermark image generation model, and the target generated image and the initial watermark are obtained through the trained image generation type network, the initial encoding network and the initial decoding network in the initial watermark image generation model, that is, the trained image generation type network is used for training, so that the model training efficiency is improved.
In one embodiment, before S202, before the original image and the original watermark are acquired, the method further includes the steps of:
acquiring a trained image generation network, and establishing an initial coding network and an initial decoding network; and obtaining an initial watermark image generation model based on the trained image generation type network, the initial coding network and the initial decoding network.
In particular, the server may obtain a trained image-generating network from a database. The server can also acquire the model file of the image generation type network uploaded by the terminal, analyze the model file to obtain a network structure and network parameters, and obtain the trained image generation type network according to the network structure and the network parameters obtained by analysis. An initial encoding network and an initial decoding network are then established. In one embodiment, the network structures of the encoding network and the decoding network in the neural network translation model may be used as the network structures of the initial encoding network and the initial decoding network, and then the network parameters are initialized to obtain the initial encoding network and the initial decoding network. In one embodiment, the network structure of the U-Net (neural network of the encoder-decoder architecture) may be used as the network structure of the initial encoding network and the initial decoding network, and then network parameter initialization is performed to obtain the initial encoding network and the initial decoding network. And finally, the server connects the trained image generation network, the initial encoding network and the initial decoding network to obtain an initial watermark image generation model. The characteristic information of the image can be extracted through a convolutional layer operation through a coding network in the U-Net, and the extracted characteristic information of the image is overlapped with the original watermark to obtain the target watermark. And then positioning the watermark in the target generated image through a decoding network in the U-Net, namely extracting the watermark in the target generated image.
In one embodiment, the server adds the trained encoding network to the AIGC model in the form of a plug-in, thereby obtaining a watermark image generation model, i.e., an image embedded with a watermark can be generated directly by the encoding network plug-in after the image is generated by the AIGC model. So that the influence of the watermark on the quality of the generated image can be reduced.
In the above embodiment, the initial watermark image generation model is obtained by acquiring the trained image generation type network, and establishing the initial encoding network and the initial decoding network, and then using the trained image generation type network, the initial encoding network and the initial decoding network, so that the efficiency of obtaining the initial watermark image generation model is improved, and the training efficiency of the initial watermark image generation model can be improved.
In one embodiment, acquiring a trained image-generating network includes the steps of:
acquiring a training image and an image tag; inputting the training image into an initial image generation type network to obtain an initial generation type image; calculating the loss between the initial generation type image and the image label to obtain generation loss information; and reversely updating the initial image generation type network based on the generation loss information, and returning to the step of acquiring the training image and the image label for iterative execution until the generation type network training completion condition is reached, so as to obtain the trained image generation type network.
Wherein the training image refers to an image of a training image generation type network, and the image can be any type of image. The image label refers to a label corresponding to the training image, and the image label can be an image which is generated through an image generation network according to the requirement of the training image, namely, the image label can be an image which has the same content as the training image and has different styles. The initial image-generating network refers to an image-generating network that is initialized by network parameters, which may be random initialization, zero initialization, gaussian distribution initialization, and so forth. The condition of completing the generated network training refers to a condition of obtaining a trained image generated network, and the condition of completing the generated network training includes, but is not limited to, that the number of training iterations reaches the maximum number, that the training loss information reaches a preset loss threshold value, and that the network parameters of the trained image generated network are not changed any more. The initially generated image refers to an image generated using initialized network parameters at the time of training. The loss information is generated to characterize the error between the initially generated image and the image tag.
Specifically, the server may obtain training images and image tags from a database. The server may also obtain training images and image tags from a service party providing the data service. The server can also acquire training images and image tags uploaded by the terminal. And then the server inputs the training image into an initial image generation type network to generate an image, so as to obtain an initial generation type image. The server may then calculate the loss between the initially generated image and the image tag using a loss function, which may be a linear loss function, such as a mean square error loss function, a least squares error loss function, or the like, to yield generated loss information. And then the server reversely updates network parameters in the initial image generation type network by using the generation loss information through a gradient descent algorithm to obtain an updated image generation type network, takes the updated image generation type network as the initial image generation type network, returns to the step of acquiring the training image and the image label for iterative execution until the generation type network training completion condition is reached, and takes the initial image generation type network when the generation type network training completion condition is reached as the training obtained image generation type network.
In the above embodiment, the initial generation formula image is obtained by acquiring the training image and the image tag, and then inputting the training image into the initial image generation formula network. And calculating the loss between the initial generation type image and the image label to obtain generation loss information. And finally, reversely updating the initial image generation type network based on the generation loss information, returning to the iterative execution of the step of acquiring the training image and the image label, and obtaining the trained image generation type network when the training completion condition of the generation type network is reached, namely, obtaining the image generation type network through pre-training, thereby being convenient for subsequent use and further improving the training efficiency of the watermark image generation model.
In one embodiment, as shown in fig. 6, S208, updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, includes:
s602, calculating the sum of the original image quality loss information and the original watermark loss information to obtain target loss information.
S604, updating the initial coding network and the initial decoding network based on the target loss information to obtain an updated coding network and an updated decoding network.
S606, obtaining an updated watermark image generation model based on the trained image generation type network, the updated encoding network and the updated decoding network.
The target loss information is used for representing total loss information in the model training process. Updating the coding network refers to updating the network parameters. The updated decoding network refers to the decoding network after the network parameters are updated. The network parameters may include weight parameters and bias parameters. The updated watermark image generation model refers to a watermark image generation model obtained after updating model parameters in the initial watermark image generation model, wherein the model parameters comprise weight parameters and bias parameters.
Specifically, the server calculates the sum of the original image quality loss information and the original watermark loss information to obtain target loss information. The server then updates network parameters in the initial encoding network and the initial decoding network using the target loss information via a gradient descent algorithm, which may be a random gradient descent algorithm, a batch gradient descent algorithm, a small batch gradient descent algorithm, a momentum gradient descent algorithm, an adaptive gradient descent algorithm, and the like. And after the network parameters are updated, an updating coding network and an updating decoding network are obtained. And finally, the server obtains an updated watermark image generation model according to the trained image generation type network, the updated encoding network and the updated decoding network. The server does not need to train the trained image generation type network in the process of training the watermark image generation model.
In one embodiment, the server may also update the trained image generation network, the initial encoding network, and the initial decoding network using the target loss information to obtain an updated image generation network, an updated encoding network, and an updated decoding network, thereby obtaining an updated watermark image generation model, and improving accuracy of the watermark image generation model obtained by training in watermark image generation.
In a specific embodiment, the initial watermark loss information may be calculated using the formula shown below.
Where loss2 refers to the initial watermark loss information,refers to the original watermark->Refers to the initial watermark. The target loss information may then be calculated using the formula shown below.
L=loss1+loss2
Where L refers to target loss information. And obtaining target loss information by calculating the sum of the original image quality loss information and the original watermark loss information.
In the above embodiment, the target loss information is obtained by calculating the sum of the initial image quality loss information and the initial watermark loss information. And updating the initial coding network and the initial decoding network based on the target loss information to obtain an updated coding network and an updated decoding network. Based on the trained image generation type network, the updating coding network and the updating decoding network, an updated watermark image generation model is obtained, namely, when updating, only the initial coding network and the initial decoding network are updated to obtain the updated watermark image generation model, so that the updating efficiency of model parameters can be improved, and the training efficiency of the watermark image generation model is further improved.
In one embodiment, the modulo length of the target watermark vector corresponding to the target watermark is less than a preset modulo length threshold.
Specifically, the preset modulo-length threshold refers to a threshold of a preset modulo length, which is set as small as possible. The modulo length refers to the distance in vector space for the target watermark vector. The optimization target in the training process of the watermark image generation model comprises that the watermark extracted by the coding network is as close as possible to the original watermark, so that the accuracy of the watermark embedded in the generated image is ensured. The constraint of the optimization objective is that the modular length of the objective watermark vector is smaller than a preset modular length threshold, even if the modular length is as small as possible.
In one particular embodiment, this can be expressed using equation (6) as shown below,
formula (6)
Wherein,,refers to the target watermark. p refers to a real number, which is set according to the requirements. />Refers to a preset module length threshold. And obtaining the modular length through the calculated 1/p power of the sum of the absolute values p of the elements of the target watermark vector. />The constraint is represented by the fact that the modular length of the target watermark is less than a preset modular length threshold, which can be set as small as possible. The module length of the target watermark vector corresponding to the target watermark is smaller than the preset module length threshold in the training process, so that the extracted watermark is as close as possible to the original watermark, and the accuracy of the watermark embedded into the generated image is ensured.
In one embodiment, as shown in fig. 7, an image generation model processing method is provided, and the method is taken as an example for describing that the method is applied to the server in fig. 1, it is understood that the method can also be applied to a terminal, and can also be applied to a system comprising the terminal and the server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:
s702, acquiring a target image and a watermark to be embedded;
s704, inputting the target image and the watermark to be embedded into a target watermark image generation model to obtain a target image and a target watermark image corresponding to the watermark to be embedded; the target watermark image generation model is obtained by updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of obtaining the original image and the original watermark for iterative execution until reaching the training completion condition; the initial image quality loss information is obtained by performing image quality loss calculation based on the target generated image and the initial generated image; the initial watermark loss information is obtained by watermark loss calculation based on the original watermark and the initial watermark; the initial generated image is generated by matching an image based on the original image through an initial watermark image generation model; the target generated image is obtained by encoding the initial generated image and the original watermark through an initial watermark image generation model to obtain a target watermark and fusing the target watermark and the initial generated image; the initial watermark is obtained by watermark extraction based on the target generated image through an initial watermark image generation model.
The target image is an image to be input into the target watermark image generation model, and is an image required to be subjected to image style conversion. That is, the watermark image after the image style conversion can be generated by the target watermark image generation model. The watermark to be embedded refers to the watermark in the image after style conversion which needs to be embedded. The target watermark image refers to the watermark image after the generated image style is converted. The target watermark image generation model is obtained by training through the steps in any one embodiment of the image generation model processing method.
Specifically, the server may acquire the target watermark image generation model obtained by training through the steps in any one of the embodiments of the image generation model processing method described above. The target watermark image generation model is then deployed and used. When the target watermark image generation model is used, the server acquires the target image and the watermark to be embedded, wherein the watermark to be embedded can be the watermark corresponding to the authority of the target image and can represent the authority of the target image. And then the server inputs the target image and the watermark to be embedded into a target watermark image generation model to perform image style conversion, so as to obtain the target watermark image. The style of the target watermark image is an image obtained by converting the style of the input image. The watermark to be embedded is embedded in the target watermark image, and the authority for representing the target watermark image is the authority of the target image, so that the target watermark image is prevented from being abused, and the copyright of the target watermark image is defined.
In one embodiment, the server may also acquire the target image, the reference image and the watermark to be embedded, then input the target image, the reference image and the watermark to be embedded into the target watermark image generation model, and then the target watermark image generation model generates a generated image corresponding to the target image and embedded with the watermark according to the reference image, that is, the style of the generated image embedded with the watermark is the same as the style of the reference image.
The image generation method, the device, the computer equipment, the storage medium and the computer program product are used for obtaining the target image and the watermark to be embedded, and inputting the target image and the watermark to be embedded into a target watermark image generation model to obtain the target image and the target watermark image corresponding to the watermark to be embedded; the target watermark image generation model is obtained by updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of obtaining the original image and the original watermark for iterative execution until reaching the training completion condition; the initial image quality loss information is obtained by performing image quality loss calculation based on the target generated image and the initial generated image; the initial watermark loss information is obtained by watermark loss calculation based on the original watermark and the initial watermark; the initial generated image is generated by matching an image based on the original image through an initial watermark image generation model; the target generated image is obtained by encoding the initial generated image and the original watermark through an initial watermark image generation model to obtain a target watermark and fusing the target watermark and the initial generated image; the initial watermark is obtained by watermark extraction based on the target generated image through an initial watermark image generation model. The initial watermark image generation model is trained by calculating the image quality loss information and the watermark loss information, so that the watermark can be better adapted to the image when the watermark is embedded in the target watermark image generation model obtained by training, the influence on the image quality of the target watermark image after the watermark is embedded is reduced, and the image quality of the generated target watermark image is improved.
In one embodiment, the target watermark image generation model includes an image generation network and an encoding network;
s704, inputting the target image and the watermark to be embedded into a target watermark image generation model to obtain a target image and a target watermark image corresponding to the watermark to be embedded, wherein the method comprises the following steps:
inputting the target image and the watermark to be embedded into a target watermark image generation model, generating a matching image corresponding to the target image through an image generation network, and inputting the matching image and the watermark to be embedded into a coding network; and encoding the matched image and the watermark to be embedded through an encoding network to obtain an image watermark, and fusing the image watermark with the target image to obtain the target image and the target watermark image corresponding to the watermark to be embedded.
Specifically, the server inputs the target image and the watermark to be embedded into a target watermark image generation model, and the target watermark image generation model firstly uses an image generation network to convert the image style of the target image to obtain a matched image. And then inputting the matched image and the watermark to be embedded into a coding network, wherein the coding network codes the matched image and the watermark to be embedded to obtain an image watermark corresponding to the watermark to be embedded, and the image quality of the obtained image watermark is consistent with the image quality of the generated matched image. And finally, fusing the image watermark with the target image, for example, overlapping the image watermark with the target image to obtain the target image and the target watermark image corresponding to the watermark to be embedded.
In the above embodiment, the image generation type network and the encoding network in the target watermark image generation model are used to generate the target image and the target watermark image corresponding to the watermark to be embedded, so that the image quality of the generated target watermark image is improved, and the influence of the superimposed watermark on the image quality of the generated target watermark image is reduced.
In a specific embodiment, as shown in fig. 8, a flowchart of an image generation model processing method is provided, and the flowchart is executed by a computer device, and specifically includes the following steps:
s802, acquiring an original image and an original watermark, and inputting the original image into an original watermark image generation model.
S804, performing matching image generation on the original image through a trained image generation type network in the initial watermark image generation model to obtain an initial generated image.
S806, respectively vectorizing the initial generated image and the original watermark through an initial coding network in the initial watermark image generation model to obtain an initial generated image vector and an original watermark vector, extracting image quality characteristics corresponding to the initial generated image vector, updating the original watermark vector based on the image quality characteristics to obtain a target watermark vector, and superposing the target watermark vector and the initial generated image vector to obtain a target generated image.
S808, watermark extraction is carried out on the target generated image through the initial decoding network, and an initial watermark is obtained.
S810, calculating peak signal-to-noise ratio between the target generated image and the initial generated image to obtain image-level image quality loss information, calculating a modular length between the target generated image and the initial generated image to obtain pixel-level image quality loss information, calculating the sum of the image-level image quality loss information and the pixel-level image quality loss information to obtain initial image quality loss information, and performing watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information.
S812, calculating the sum of the original image quality loss information and the original watermark loss information to obtain target loss information; updating an initial coding network and an initial decoding network based on the target loss information to obtain an updated coding network and an updated decoding network; an updated watermark image generation model is obtained based on the trained image generation network, the updated encoding network and the updated decoding network.
S814, taking the updated watermark image generation model as an initial watermark image generation model, and returning to the step of acquiring the original image and the original watermark for iterative execution until reaching the training completion condition, thereby obtaining a target watermark image generation model.
In the above embodiment, the initial image quality loss information and the initial watermark loss information are calculated, and then the initial encoding network and the initial decoding network in the initial watermark image generation model are trained by using the initial image quality loss information and the initial watermark loss information, and when the training completion condition is reached, the target watermark image generation model is obtained, so that the image quality of the watermark image is improved when the target watermark image generation model generates the watermark image, and the influence on the image quality of the image after embedding the watermark in the image is reduced.
In a specific embodiment, as shown in fig. 9, a network structure diagram of watermark image generation model training is provided, specifically:
the server acquires an input image and an original watermark, then inputs the input image into an AIGC content generation network, generates an image through an AIGC content generation network, namely a style conversion model (style transfer module) shown in the figure, and obtains an initial generated image (stylized image), wherein the input image can comprise the original image and a corresponding reference style image, and the style conversion model converts the grids of the original image according to the style of the reference style image to obtain the initial generated image of the reference style. And then inputting the initial generated image and the original watermark into an initial coding network (encoder) for watermark embedding, namely updating the original watermark by using the initial generated image through the initial coding network to obtain a target watermark, namely, the target watermark can be understood as a watermark template (watermark template), and then overlapping the generated target watermark with the initial generated image to obtain a style conversion image (watermark styled image) for embedding the watermark. And then inputting the style-converted image embedded with the watermark into an initial decoding network (decoder) for watermark detection to obtain an initial watermark, namely, a watermark obtained by decoding and extracting (decoder watermark). The initial image quality loss information, loss1, is then calculated using the initial generated image and the watermark embedded image. The initial watermark loss information, loss2, is calculated using the initial watermark and the original watermark. And finally, calculating the sum of the loss1 and the loss2 to obtain total loss information, updating network parameters in the initial decoding network and the initial coding network by using the total loss information to obtain an updated decoding network and an updated coding network, and training the updated decoding network and the updated coding network again to obtain an iterative loop. The server returns to the step of obtaining the input image and the original watermark used in training to iterate the training, when the total loss information obtained through direct training is smaller than the preset loss threshold value, the AIGC content generating network and the coding network which reach the training completion are used as target watermark image generating models, and the AIGC content generating network and the coding network which reach the training completion are used as a whole to obtain target watermark image generating models. The target watermark image generation model is then used to generate a watermark embedded image for the input image. Wherein, the encoding network is added directly after the AIGC content generation network to generate the watermark embedded image, which can prevent AIGC data leakage caused by introducing extra links. And the image embedded with the watermark is generated through the coding network, so that the influence of the embedded watermark on the image quality of the generated image is reduced, and the image quality of the generated watermark image is improved.
In a specific embodiment, the target watermark image generation model is applied in a social platform, in particular: when a user performs social contact on the social contact platform, the original image is converted into images with different styles, and then the converted images are used for performing social contact. For example, the social platform server may obtain an image style conversion request, and analyze the image style conversion request to obtain a real animal image captured by the user terminal through the image capturing device, a watermark for identifying the user, and an image style to be generated, such as a cartoon style. And then the social platform server inputs the real animal image and the watermark for identifying the user into the cartoon style watermark image generation model to obtain the cartoon style animal image embedded with the watermark of the user, thereby improving the image quality of the obtained cartoon style animal image embedded with the watermark of the user. And then the social platform server returns the generated cartoon style animal image embedded with the user watermark to the user terminal for display, and then the user terminal can use the cartoon style animal image embedded with the user watermark for social contact, for example, a message embedded with the cartoon style animal image of the user can be sent in the instant messaging process, and the cartoon style animal image embedded with the user watermark can be shared with different social contact objects and the like.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides an image generation model processing device and an image generation device for realizing the image generation model processing method. The implementation of the solution provided by the apparatus is similar to the implementation described in the above method, so the specific limitations in the embodiments of the image generation model processing apparatus and the image generation apparatus provided below may refer to the limitations in the image generation model processing method and the image generation method hereinabove, and are not repeated herein.
In one embodiment, as shown in fig. 10, there is provided an image generation model processing apparatus 1000 including: an image acquisition module 1002, a model processing module 1004, a loss calculation module 1006, and an iteration module 1008, wherein:
an image acquisition module 1002, configured to acquire an original image and an original watermark;
the model processing module 1004 is configured to input an original image into an initial watermark image generation model, perform matching image generation based on the original image to obtain an initial generated image, encode the initial generated image and the original watermark to obtain a target watermark, fuse the target watermark with the initial generated image to obtain a target generated image, and perform watermark extraction based on the target generated image to obtain the initial watermark;
the loss calculation module 1006 is configured to perform image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and perform watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information;
and the iteration module 1008 is used for updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of acquiring the original image and the original watermark for iterative execution until reaching the training completion condition, obtaining a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark.
In one embodiment, the model processing module 1004 is further configured to vectorize the initially generated image and the original watermark respectively, so as to obtain an initially generated image vector and an original watermark vector; extracting image quality characteristics corresponding to the initially generated image vector, and updating the original watermark vector based on the image quality characteristics to obtain a target watermark vector; and superposing the target watermark vector and the initial generated image vector to obtain a target generated image.
In one embodiment, the loss calculation module 1006 is further configured to calculate a peak signal-to-noise ratio between the target generated image and the initial generated image, to obtain image-level image quality loss information; calculating the module length between the target generated image and the initial generated image to obtain pixel-level image quality loss information; and calculating the sum of the image quality loss information at the image level and the image quality loss information at the pixel level to obtain initial image quality loss information.
In one embodiment, the model processing module 1004 is further configured to obtain a reference image, input the original image and the reference image into an initial watermark image generation model, and the initial watermark image generation model vectorizes the original image and the reference image to obtain an original image vector and a reference image vector; the initial watermark image generation model generates an initial generated image matched with the original image vector according to the reference image vector.
In one embodiment, the initial watermark image generation model includes a trained image generation network, an initial encoding network, and an initial decoding network;
the model processing module 1004 is further configured to input an original image into an initial watermark image generation model, and perform matching image generation on the original image through a trained image generation network in the initial watermark image generation model to obtain an initial generated image; encoding the initial generated image and the original watermark through an initial encoding network to obtain a target watermark, and fusing the target watermark and the initial generated image to obtain a target generated image; and watermark extraction is carried out on the target generated image through an initial decoding network, so as to obtain an initial watermark.
In one embodiment, the image generation model processing apparatus 1000 further includes:
the model building module is used for obtaining the trained image generation type network and building an initial coding network and an initial decoding network; and obtaining an initial watermark image generation model based on the trained image generation type network, the initial coding network and the initial decoding network.
In one embodiment, the model building module is further configured to obtain a training image and an image tag; inputting the training image into an initial image generation type network to obtain an initial generation type image; calculating the loss between the initial generation type image and the image label to obtain generation loss information; and reversely updating the initial image generation type network based on the generation loss information, and returning to the step of acquiring the training image and the image label for iterative execution until the generation type network training completion condition is reached, so as to obtain the trained image generation type network.
In one embodiment, the iteration module 1008 is further configured to calculate a sum of the initial image quality loss information and the initial watermark loss information to obtain target loss information; updating an initial coding network and an initial decoding network based on the target loss information to obtain an updated coding network and an updated decoding network; an updated watermark image generation model is obtained based on the trained image generation network, the updated encoding network and the updated decoding network.
In one embodiment, the modulo length of the target watermark vector corresponding to the target watermark is less than a preset modulo length threshold.
In one embodiment, as shown in fig. 11, there is provided an image generating apparatus 1100 including: an acquisition module 1102 and an image generation module 1104, wherein
An obtaining module 1102, configured to obtain a target image and a watermark to be embedded;
the image generating module 1104 is configured to input the target image and the watermark to be embedded into a target watermark image generating model, so as to obtain a target image and a target watermark image corresponding to the watermark to be embedded; the target watermark image generation model is obtained by updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of obtaining the original image and the original watermark for iterative execution until reaching the training completion condition; the initial image quality loss information is obtained by performing image quality loss calculation based on the target generated image and the initial generated image; the initial watermark loss information is obtained by watermark loss calculation based on the original watermark and the initial watermark; the initial generated image is generated by matching an image based on the original image through an initial watermark image generation model; the target generated image is obtained by encoding the initial generated image and the original watermark through an initial watermark image generation model to obtain a target watermark and fusing the target watermark and the initial generated image; the initial watermark is obtained by watermark extraction based on the target generated image through an initial watermark image generation model.
In one embodiment, the target watermark image generation model includes an image generation network and an encoding network;
the image generation module 1104 is further configured to input the target image and the watermark to be embedded into a target watermark image generation model, generate a matching image corresponding to the target image through an image generation network, and input the matching image and the watermark to be embedded into the encoding network; and encoding the matched image and the watermark to be embedded through an encoding network to obtain an image watermark, and fusing the image watermark with the target image to obtain the target image and the target watermark image corresponding to the watermark to be embedded.
The image generation model processing apparatus and each module in the image generation apparatus described above may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 12. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer equipment is used for storing data such as an original image, an original watermark, a target image, a watermark to be embedded, a target watermark image generation model and the like. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement an image generation model processing method or an image generation method.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure thereof may be as shown in fig. 13. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement an image generation model processing method or an image generation method. The display unit of the computer equipment is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device, wherein the display screen can be a liquid crystal display screen or an electronic ink display screen, the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on a shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structures shown in fig. 12 or 13 are merely block diagrams of portions of structures associated with the present inventive arrangements and are not limiting of the computer device to which the present inventive arrangements may be implemented, and that a particular computer device may include more or fewer components than shown, or may be combined with certain components, or may have different arrangements of components.
In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (15)
1. A method of image generation model processing, the method comprising:
acquiring an original image and an original watermark;
inputting the original image into an initial watermark image generation model, performing matching image generation on the basis of the original image by the initial watermark image generation model to obtain an initial generated image, encoding the initial generated image and the original watermark to obtain a target watermark, fusing the target watermark and the initial generated image to obtain a target generated image, and performing watermark extraction on the basis of the target generated image to obtain the initial watermark;
Performing image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and performing watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information;
updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of acquiring the original image and the original watermark for iterative execution until reaching the training completion condition, obtaining a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark.
2. The method of claim 1, wherein encoding the initial generated image and the original watermark to obtain a target watermark, and fusing the target watermark with the initial generated image to obtain a target generated image, comprises:
vectorizing the initial generated image and the original watermark respectively to obtain an initial generated image vector and an original watermark vector;
extracting image quality characteristics corresponding to the initially generated image vector, and updating the original watermark vector based on the image quality characteristics to obtain a target watermark vector;
And superposing the target watermark vector and the initial generated image vector to obtain the target generated image.
3. The method according to claim 1, wherein the calculating the image quality loss based on the target generated image and the initial generated image to obtain initial image quality loss information includes:
calculating the peak signal-to-noise ratio between the target generated image and the initial generated image to obtain image-level image quality loss information;
calculating the module length between the target generated image and the initial generated image to obtain pixel-level image quality loss information;
and calculating the sum of the image-level image quality loss information and the pixel-level image quality loss information to obtain the initial image quality loss information.
4. The method of claim 1, wherein the inputting the original image into an initial watermark image generation model, the initial watermark image generation model performing matching image generation based on the original image, results in an initial generated image, comprises:
acquiring a reference image, inputting the original image and the reference image into the initial watermark image generation model, and respectively vectorizing the original image and the reference image by the initial watermark image generation model to obtain an original image vector and a reference image vector;
The initial watermark image generation model generates the initial generated image matched with the original image vector according to the reference image vector.
5. The method of claim 1, wherein the initial watermark image generation model comprises a trained image generation network, an initial encoding network, and an initial decoding network;
inputting the original image into an initial watermark image generation model, performing matching image generation by the initial watermark image generation model based on the original image to obtain an initial generated image, encoding the initial generated image and the original watermark to obtain a target watermark, fusing the target watermark and the initial generated image to obtain a target generated image, and performing watermark extraction based on the target generated image to obtain the initial watermark, wherein the method comprises the following steps:
inputting the original image into an initial watermark image generation model, and generating a matched image of the original image through the trained image generation network in the initial watermark image generation model to obtain the initial generated image;
encoding the initial generated image and the original watermark through the initial encoding network to obtain a target watermark, and fusing the target watermark and the initial generated image to obtain the target generated image;
And watermark extraction is carried out on the target generated image through the initial decoding network, so as to obtain an initial watermark.
6. The method of claim 5, further comprising, prior to said acquiring the original image and the original watermark:
acquiring the trained image generation network, and establishing the initial coding network and the initial decoding network;
and obtaining the initial watermark image generation model based on the trained image generation type network, the initial coding network and the initial decoding network.
7. The method of claim 6, wherein the acquiring the trained image-generating network comprises:
acquiring a training image and an image tag;
inputting the training image into an initial image generation type network to obtain an initial generation type image;
calculating the loss between the initial generation type image and the image label to obtain generation loss information;
and reversely updating the initial image generation type network based on the generation loss information, and returning to the step of acquiring the training image and the image label for iterative execution until the generation type network training completion condition is reached, so as to obtain the trained image generation type network.
8. The method of claim 5, wherein the updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information comprises:
calculating the sum of the original image quality loss information and the original watermark loss information to obtain target loss information;
updating the initial coding network and the initial decoding network based on the target loss information to obtain an updated coding network and an updated decoding network;
and obtaining an updated watermark image generation model based on the trained image generation type network, the updating coding network and the updating decoding network.
9. The method of claim 1, wherein a modulus of a target watermark vector corresponding to the target watermark is less than a preset modulus threshold.
10. An image generation method, the method comprising:
obtaining a target image and a watermark to be embedded;
inputting the target image and the watermark to be embedded into a target watermark image generation model to obtain target watermark images corresponding to the target image and the watermark to be embedded;
the target watermark image generation model is obtained by updating an initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of obtaining the original image and the original watermark for iterative execution until reaching the training completion condition; the initial image quality loss information is obtained by performing image quality loss calculation based on the target generated image and the initial generated image; the initial watermark loss information is obtained by watermark loss calculation based on the original watermark and the initial watermark; the initial generated image is generated by matching an image based on an original image through the initial watermark image generation model; the target generated image is obtained by encoding the initial generated image and the original watermark through the initial watermark image generation model to obtain a target watermark and fusing the target watermark and the initial generated image; the initial watermark is obtained by watermark extraction based on the target generated image through the initial watermark image generation model.
11. The method of claim 10, wherein the target watermark image generation model comprises an image generation network and an encoding network;
inputting the target image and the watermark to be embedded into the target watermark image generation model to obtain the target image and the target watermark image corresponding to the watermark to be embedded, wherein the method comprises the following steps:
inputting the target image and the watermark to be embedded into the target watermark image generation model, generating a matching image corresponding to the target image through the image generation network, and inputting the matching image and the watermark to be embedded into the coding network;
and encoding the matched image and the watermark to be embedded through the encoding network to obtain an image watermark, and fusing the image watermark with the target image to obtain the target image and the target watermark image corresponding to the watermark to be embedded.
12. An image generation model processing apparatus, characterized in that the apparatus comprises:
the image acquisition module is used for acquiring an original image and an original watermark;
the model processing module is used for inputting the original image into an initial watermark image generation model, the initial watermark image generation model is used for carrying out matching image generation based on the original image to obtain an initial generation image, encoding the initial generation image and the original watermark to obtain a target watermark, fusing the target watermark and the initial generation image to obtain a target generation image, and carrying out watermark extraction based on the target generation image to obtain the initial watermark;
The loss calculation module is used for carrying out image quality loss calculation based on the target generated image and the initial generated image to obtain initial image quality loss information, and carrying out watermark loss calculation based on the original watermark and the initial watermark to obtain initial watermark loss information;
the iteration module is used for updating the initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the iterative execution of the steps of acquiring the original image and the original watermark until the training completion condition is reached, obtaining a target watermark image generation model, wherein the target watermark image generation model is used for generating a target watermark image according to the input image and the input watermark.
13. An image generation apparatus, the apparatus comprising:
the acquisition module is used for acquiring the target image and the watermark to be embedded;
the image generation module is used for inputting the target image and the watermark to be embedded into the target watermark image generation model to obtain target watermark images corresponding to the target image and the watermark to be embedded; the target watermark image generation model is obtained by updating an initial watermark image generation model based on the initial image quality loss information and the initial watermark loss information, and returning to the step of obtaining the original image and the original watermark for iterative execution until reaching the training completion condition; the initial image quality loss information is obtained by performing image quality loss calculation based on the target generated image and the initial generated image; the initial watermark loss information is obtained by watermark loss calculation based on the original watermark and the initial watermark; the initial generated image is generated by matching an image based on an original image through the initial watermark image generation model; the target generated image is obtained by encoding the initial generated image and the original watermark through the initial watermark image generation model to obtain a target watermark and fusing the target watermark and the initial generated image; the initial watermark is obtained by watermark extraction based on the target generated image through the initial watermark image generation model.
14. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 11 when the computer program is executed.
15. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310961674.7A CN116703687B (en) | 2023-08-02 | 2023-08-02 | Image generation model processing, image generation method, image generation device and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310961674.7A CN116703687B (en) | 2023-08-02 | 2023-08-02 | Image generation model processing, image generation method, image generation device and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116703687A true CN116703687A (en) | 2023-09-05 |
| CN116703687B CN116703687B (en) | 2024-01-30 |
Family
ID=87839549
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310961674.7A Active CN116703687B (en) | 2023-08-02 | 2023-08-02 | Image generation model processing, image generation method, image generation device and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116703687B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111311472A (en) * | 2020-01-15 | 2020-06-19 | 中国科学技术大学 | Property right protection method for image processing model and image processing algorithm |
| CN111768327A (en) * | 2020-06-30 | 2020-10-13 | 苏州科达科技股份有限公司 | Watermark adding and extracting method and device based on deep learning and storage medium |
| CN115018688A (en) * | 2022-06-13 | 2022-09-06 | 杭州电子科技大学 | Network model watermark generation method based on DCT (discrete cosine transformation) coefficient and application |
| CN115660931A (en) * | 2022-11-01 | 2023-01-31 | 南京信息工程大学 | Robust watermarking method based on Transformer and denoising diffusion model |
-
2023
- 2023-08-02 CN CN202310961674.7A patent/CN116703687B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111311472A (en) * | 2020-01-15 | 2020-06-19 | 中国科学技术大学 | Property right protection method for image processing model and image processing algorithm |
| CN111768327A (en) * | 2020-06-30 | 2020-10-13 | 苏州科达科技股份有限公司 | Watermark adding and extracting method and device based on deep learning and storage medium |
| CN115018688A (en) * | 2022-06-13 | 2022-09-06 | 杭州电子科技大学 | Network model watermark generation method based on DCT (discrete cosine transformation) coefficient and application |
| CN115660931A (en) * | 2022-11-01 | 2023-01-31 | 南京信息工程大学 | Robust watermarking method based on Transformer and denoising diffusion model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116703687B (en) | 2024-01-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111291212B (en) | Zero sample sketch image retrieval method and system based on graph convolution neural network | |
| CN108460338B (en) | Human body posture estimation method and apparatus, electronic device, storage medium, and program | |
| CN112418292B (en) | Image quality evaluation method, device, computer equipment and storage medium | |
| CN110599395A (en) | Target image generation method, device, server and storage medium | |
| CN113762050B (en) | Image data processing method, device, equipment and medium | |
| CN111797834B (en) | Text recognition method and device, computer equipment and storage medium | |
| CN113781164B (en) | Virtual fitting model training method, virtual fitting method and related devices | |
| CN117078790B (en) | Image generation method, device, computer equipment and storage medium | |
| CN114419351A (en) | Image-text pre-training model training method and device and image-text prediction model training method and device | |
| CN115115552B (en) | Image correction model training method, image correction device and computer equipment | |
| CN116189265A (en) | Sketch face recognition method, device and equipment based on lightweight semantic transducer model | |
| CN113392791A (en) | Skin prediction processing method, device, equipment and storage medium | |
| CN117576248A (en) | Image generation method and device based on gesture guidance | |
| CN116030466B (en) | Image text information identification and processing method and device and computer equipment | |
| CN117237398A (en) | Matting method and device, electronic equipment and storage medium | |
| CN116703687B (en) | Image generation model processing, image generation method, image generation device and computer equipment | |
| CN116977714A (en) | Image classification method, apparatus, device, storage medium, and program product | |
| CN114493971B (en) | Media data conversion model training and digital watermark embedding method and device | |
| JP7479507B2 (en) | Image processing method and device, computer device, and computer program | |
| Ezekiel et al. | Investigating GAN and VAE to train DCNN | |
| Savitha et al. | Deep learning-based face hallucination: a survey | |
| CN116645700B (en) | Feature extraction model processing method and device and feature extraction method and device | |
| CN115984947B (en) | Image generation method, training device, electronic equipment and storage medium | |
| Parekh et al. | Image Super-Resolution using GAN-A study | |
| CN117934654A (en) | Image generation model training, image generation method and device and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40094476 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |