CN115272687A - Single-sample adaptive domain generator migration method - Google Patents

Single-sample adaptive domain generator migration method Download PDF

Info

Publication number
CN115272687A
CN115272687A CN202210811744.6A CN202210811744A CN115272687A CN 115272687 A CN115272687 A CN 115272687A CN 202210811744 A CN202210811744 A CN 202210811744A CN 115272687 A CN115272687 A CN 115272687A
Authority
CN
China
Prior art keywords
migration
domain
vector
generator
picture
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210811744.6A
Other languages
Chinese (zh)
Other versions
CN115272687B (en
Inventor
左旺孟
张亚博
姚明帅
魏于翔
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Harbin Institute of Technology
Original Assignee
Harbin Institute of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Harbin Institute of Technology filed Critical Harbin Institute of Technology
Priority to CN202210811744.6A priority Critical patent/CN115272687B/en
Publication of CN115272687A publication Critical patent/CN115272687A/en
Application granted granted Critical
Publication of CN115272687B publication Critical patent/CN115272687B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

A single-sample adaptive domain generator migration method relates to the field of image generation and migration learning in three-level vision. The method solves the problems that a target generator after migration by using the existing algorithm cannot accurately acquire the style of the guide map and the diversity of the images after migration is low. The method of the invention enables the picture synthesized by the target domain generator after migration to have the global characteristics of the guide graph through the global horizontal domain migration loss function; a local horizontal domain migration loss function is designed to solve the problem that the local representative characteristics of the guide map cannot be accurately acquired in the prior art; the method comprises the steps of utilizing a reverse mapper to map a synthesized picture to a hidden space, and providing the attribute of self-adaptive attribute maintenance loss function to select and maintain domain sharing in a self-adaptive mode, so that the transferred picture maintains partial attributes of the previous picture, the diversity of the synthesized picture is improved, and the transfer of the self-adaptive domain generator can be realized by giving a target domain guide graph. The method is mainly used for realizing the migration of the adaptive domain generator.

Description

Single-sample adaptive domain generator migration method
Technical Field
The present invention relates to the field of image generation and transfer learning in tertiary vision.
Background
The purpose of single sample generative model domain migration is: by using a given single guide picture, a generator which originally generates a real picture is converted into a generator which can generate a picture with a similar style as the guide picture. Training an available generator often requires a large amount of data, but it is very difficult to collect data of a sufficiently similar style in real life. Thus, for extreme data starvation situations, it is common practice to migrate a generator trained in a data-rich source domain into the target domain using given target domain data.
In recent years, as the deep learning technology is further developed, image generation and transfer learning thereof have also made breakthrough progress. The scholars synthesize a high-quality picture by using the generator, extract the characteristics of the picture by using a pre-trained model based on large-scale graphics and texts, and align the characteristics of the guide picture with the characteristics of the generated picture, so that the style of the generated picture is closer to that of the guide picture. However, the picture synthesized by the target generator after migration using the existing algorithm has two problems. On one hand, the synthesized picture and the guide map have larger difference in local characteristics and styles; on the other hand, the post-migration pictures have low diversity and do not match the pre-migration pictures.
Disclosure of Invention
The method aims to solve the problems that a target generator after migration by using the existing algorithm cannot accurately acquire the style of a guide map and the diversity of the migrated pictures is low. The invention provides a migration method of a single-sample self-adaptive domain generator.
A method for single sample adaptive domain generator migration, the method comprising:
s1, training a pre-trained generator by using a plurality of source domain pictures and a plurality of 512-dimensional vectors obtained by random sampling in multi-dimensional Gaussian distribution to obtain a source domain generator GB(ii) a Reuse source domain generator GBThe weight of (a) initializes the target domain generator to obtain an initialized target domain generator GAMeanwhile, a picture with a style different from that of the source domain is selected as an initialized target domain generator GBTarget of (2)A domain guidance diagram;
s2, randomly sampling M512-dimensional vectors from multi-dimensional Gaussian distribution, and inputting the M512-dimensional vectors into a source domain generator GBSynthesizing M source domain synthetic pictures, inputting the M source domain synthetic pictures and the target domain guide picture into a first CLIP picture encoder for feature extraction to obtain a source domain center feature vector vsrcAnd a global feature vector v of the target domain guide graphtarAnd a target domain guide map local feature vector sequence Ftar；
S3, randomly sampling a 512-dimensional vector from multi-dimensional Gaussian distribution, and simultaneously inputting the 512-dimensional vector to a source domain generator GAAnd an initialized target domain generator GBIn such a way that the source domain generator GATarget domain generator G for synthesizing a picture before migration and after initializationBSynthesizing a picture after migration;
s4, inputting the synthesized picture before the migration and the picture after the migration into a second CLIP picture encoder, and extracting the global feature vector v of the picture before the migrationAAnd the global feature vector v of the transferred pictureBAnd local feature vector sequence F of the migrated pictureBThen, combining the source domain center feature vector vsrcAnd a global feature vector v of the target domain guide graphtarAnd a target domain guide map local feature vector sequence FtarDetermining a global horizontal domain migration loss function LglobalLoss value of (D) and local horizontal domain migration loss function LlocalThe loss value of (d);
meanwhile, inputting the synthesized picture before the migration and the picture after the migration into an inverse mapper, and extracting the attribute vector omega of the picture before the migrationAAnd the attribute vector omega of the migrated pictureB(ii) a And according to the attribute vector omega of the picture before migrationAAnd the attribute vector omega of the migrated pictureBDetermining an adaptive attribute retention penalty function LsccThe loss value of (d);
s5, transferring a global horizontal domain to a loss function LglobalLoss value of (1), local horizontal domain migration loss function LlocalLoss value and adaptive property preserving loss function LsccTo obtain the total lossA value;
s6, judging whether the current total loss value is smaller than a preset threshold value or not, and if so, taking the current total loss value as an initialized target domain generator GBSo as to obtain the target domain generator after migration, and realize the migration of the adaptive domain generator; if not, the initialized target domain generator G is generated according to the current total loss valueBThe parameters are optimized and updated to realize the generation of the initialized target domain GBThen step S3 is performed again.
Preferably, in step S2, the central feature vector ν of the source domain is obtainedsrcAnd a global feature vector v of the target domain guide graphtarAnd a target domain guide map local feature vector sequence FtarThe implementation of the column is:
inputting the M source domain synthetic pictures into a first CLIP picture encoder for global feature extraction, obtaining global feature vectors of each source domain synthetic picture, calculating the average value of the global feature vectors of the M source domain synthetic pictures, and taking the average value as a source domain central feature vector vsrc；
Meanwhile, the target domain guide map is input into a first CLIP picture encoder for global feature extraction and local feature extraction, and a target domain guide map global feature vector v is obtainedtarAnd a target domain guide map local feature vector sequence Ftar。
Preferably, in step S4, a global horizontal domain migration loss function L is determinedglobalThe loss value of (c) is implemented as follows:
global feature vector v of picture before migrationAAnd the global feature vector v of the transferred pictureBAnd v characteristic vector of center of source regionsrcAnd a target domain guide map global feature vector vtarSubstituted into the global horizontal domain migration loss function LglobalThereby obtaining a global horizontal domain migration loss function LglobalThe loss value of (c).
Preferably, the global horizontal domain migration loss function LglobalThe expression of (a) is:
Figure BDA0003739510130000031
wherein ,
Δνsamp＝νB-νAformula 2;
Δνdom＝νtar-νsrcformula 3;
Δνsampglobal feature vector v for shifted picturesBGlobal feature vector v of picture before migrationADifference of (d), Δ νdomDirecting a graph global feature vector v for a target domaintarAnd source domain center feature vector vsrcThe difference of (c).
Preferably, in step S4, a local horizontal domain migration loss function L is determinedlocalThe loss value of (c) is implemented as follows:
global feature vector v of target domain guide graphtarAnd a target domain guide map local feature vector sequence FtarSubstituted into the local horizontal domain migration loss function LglobalThereby obtaining a loss value of the local horizontal domain migration loss function.
Preferably, the local horizontal domain migration loss function LlocalThe expression of (a) is:
Figure BDA0003739510130000032
Figure BDA0003739510130000033
wherein ,
Figure BDA0003739510130000034
for local feature vector sequence F of post-migration pictureBThe (ii) th vector of (a),
Figure BDA0003739510130000035
directing graphs for target domainsLocal feature vector sequence FtarThe j-th vector of (C)i,jIs composed of
Figure BDA0003739510130000036
And
Figure BDA0003739510130000037
cosine distance matrix between, n is the local characteristic vector sequence F of the picture after migrationBThe number of medium vectors, m is the local feature vector sequence F of the target domain guide maptarThe number of medium vectors, i and j are integers, and i =1,2 \ 8230, i 8230, n, j =1,2 \ 8230, m, n and m are integers.
Preferably, in step S6, the implementation manner of obtaining the target domain generator after migration is as follows:
s61, target domain generator G after initializationBIn the training process, the attribute vector omega of the P transferred pictures obtained in the training process is usedASumming and averaging to obtain average attribute vector of the images after migration, and obtaining attribute vector omega of P images before migration in the training processBAfter summing and averaging, obtaining an average attribute vector of the picture before migration;
s62, subtracting the average attribute vector of the image after the migration from the average attribute vector of the image before the migration to obtain an average attribute difference value vector delta omega; the average attribute difference vector delta omega is a multidimensional vector, the dimensionality is N, and N is an integer larger than 3;
s63, determining the position values of all elements in the condition vector mask (delta omega, alpha), thereby obtaining the condition vector mask (delta omega, alpha), and obtaining the attribute vector omega of the picture before migration through the condition vector mask (delta omega, alpha) and the last trainingAAnd finally training to obtain an attribute vector omega of the migrated pictureASubstituting adaptive attribute hold penalty function LsccIn (1), obtaining an adaptive attribute preserving loss function LsccAnd using the loss value as the initialized target domain generator GBThereby obtaining a migrated target domain generator;
the dimension of the preset vector mask (delta omega, alpha) is the same as that of the average attribute difference vector delta omega, and alpha is the proportion of the selected attribute.
Preferably, in S63, determining the values of the positions of all elements in the condition vector mask (Δ ω, α), so as to obtain the condition vector mask (Δ ω, α) is implemented as follows:
first, the value of the position of each element in the condition vector mask (Δ ω, α) is obtained in such a manner that,
Figure BDA0003739510130000041
wherein ,ΔωiIs the value of the ith element in the average attribute difference vector Δ ω, Δ ωsaNIs the value corresponding to the element with the small Nth alpha in the average attribute difference vector delta omega;
secondly, determining the values of all elements in the condition vector mask (delta omega, alpha) according to the value of the position of each element in the condition vector mask (delta omega, alpha), thereby obtaining the condition vector mask (delta omega, alpha); wherein,
mask(Δω，α)＝{mask(Δω，α)1，mask(Δω，α)2……mask(Δω，α)iequation 7;
the i =1,2,3 \8230, N.
Preferably, the adaptive property preserving loss function LsccThe expression of (a) is:
Lscc＝||mask(Δω，α)·(ωB-ωA)||1equation 7;
wherein α is the proportion of the selected attributes, mask (Δ ω, α) is the condition vector, Δ ω is the average attribute difference vector, | · | calc1Is a norm of 1.
Principle analysis: migration loss function L through global level domainglobalEnabling the picture synthesized by the migrated target domain generator to have the global characteristics of the guide picture; designing a local horizontal domain migration loss function LlocalThe problem that the local representative characteristics of the guide map cannot be accurately acquired in the prior art is solved; by usingThe inverse mapper maps the synthesized picture to a hidden space and proposes an adaptive attribute preserving loss function LsccAnd the shared attribute of the domain is selected and maintained adaptively, so that the transferred picture maintains partial attribute of the previous picture, and the diversity of the synthesized picture is improved. Therefore, given a guide map of the target domain, the picture synthesized by the trained generator can have rich content and a style similar to that of the target domain.
The invention has the following beneficial effects:
according to the single-sample adaptive domain generator migration method, the global style of the guide map is obtained by the synthetic picture by using the global horizontal domain migration loss function, and the local horizontal domain migration loss function and the adaptive attribute maintaining loss function are designed aiming at the problem that the guide map style cannot be accurately obtained and the diversity of the migrated pictures is low in the prior art, so that the local style of the guide map is obtained in a self-adaptive manner, the migrated pictures maintain partial attributes of the previous pictures, and the diversity of the synthetic pictures is improved, wherein the local horizontal domain migration loss function is used for realizing the alignment of local features.
In particular, the acquisition of representative features of global and local levels of a guide graph and the inheritance of source domain generator diversity are considered, and a global level and local level domain migration loss function and an adaptive attribute retention loss function are designed, so that the problems of insufficient acquisition of important attributes and styles and low diversity of a migrated picture in a generator migration process are solved, and the migrated picture retains a part of attributes. In addition, the method only gives a target domain guide graph, so that the picture synthesized by the target domain generator after migration (namely the target generator after migration) has rich content and a style similar to that of the target domain guide graph, the target domain generator is continuously trained and optimized by only one target domain guide graph, the number of used samples is small, the accuracy of attribute inheritance can be ensured, and the whole migration learning process is simple and convenient to implement.
Drawings
FIG. 1 is a flow chart of a single sample adaptive domain generator migration method of the present invention;
FIG. 2 is a schematic diagram of a single sample adaptive domain generator migration method.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
Example 1:
referring to fig. 1 to illustrate this embodiment 1, the method for migrating a single-sample adaptive domain generator described in this embodiment 1 includes:
s1, training a pre-trained generator by using a plurality of source domain pictures and a plurality of 512-dimensional vectors obtained by random sampling in multi-dimensional Gaussian distribution to obtain a source domain generator GB(ii) a Reuse source domain generator GBThe weight of (a) initializes the target domain generator to obtain an initialized target domain generator GAMeanwhile, selecting a picture with a style different from that of the source domain as an initialized target domain generator GBThe target domain guidance map of (1);
s2, randomly sampling M512-dimensional vectors from multi-dimensional Gaussian distribution, and inputting the M512-dimensional vectors to a source domain generator GBSynthesizing M source domain synthetic pictures, inputting the M source domain synthetic pictures and the target domain guide picture into a first CLIP picture encoder for feature extraction to obtain a source domain center feature vector vsrcAnd a global feature vector v of the target domain guide graphtarAnd a target domain guide map local feature vector sequence Ftar；
S3, randomly sampling a 512-dimensional vector from multi-dimensional Gaussian distribution, and simultaneously inputting the 512-dimensional vector to a source domain generator GAAnd an initialized target domain generator GBIn such a way that the source domain generator GATarget domain generator G for synthesizing picture before migration and after initializationBSynthesizing a transferred picture;
s4, inputting the synthesized picture before the migration and the picture after the migration into a second CLIP picture encoder, and extracting the global feature vector v of the picture before the migrationAAnd the global feature vector v of the transferred pictureBAnd local feature vector sequence F of the migrated pictureBThen, combining the source domain center feature vector vsrcAnd a global feature vector v of the target domain guide graphtarAnd a target domain guide map local feature vector sequence FtarDetermining a global horizontal domain migration loss function LglobalLoss value of (D) and local horizontal domain migration loss function LlocalThe loss value of (d);
meanwhile, inputting the synthesized picture before migration and the picture after migration into an inverse mapper, and extracting the attribute vector omega of the picture before migrationAAnd the attribute vector omega of the migrated pictureB(ii) a And according to the attribute vector omega of the picture before the migrationAAnd the attribute vector omega of the migrated pictureBDetermining an adaptive attribute retention penalty function LsccThe loss value of (d);
s5, transferring a global horizontal domain migration loss function LglobalLoss value of (1), local horizontal domain migration loss function LlocalLoss value and adaptive property preserving loss function LsccSumming the loss values to obtain a total loss value;
s6, judging whether the current total loss value is smaller than a preset threshold value or not, and if so, taking the current total loss value as an initialized target domain generator GBSo as to obtain the target domain generator after migration, and realize the migration of the adaptive domain generator; if not, the initialized target domain generator G is generated according to the current total loss valueBThe parameters are optimized and updated to realize the generation of the initialized target domain GBThen step S3 is performed again.
When in application, the current total loss value is larger than or equal toWhen the threshold is preset, the step S3 is executed again, and the initialized target domain generator G is generated according to the current total loss valueBThe parameters are continuously updated iteratively until the current total loss value is smaller than a preset threshold value, and the current total loss value is taken as an initialized target domain generator GBWhile the initialized target domain generator G at the moment is also generatedBAs a migrated target domain generator, this iteratively updates the target domain generator GBAnd the parameter mode ensures the generation precision of the migrated target domain generator.
In the migration learning process of the target domain generator, the single-sample self-adaptive domain generator migration method only continuously trains and optimizes the target domain generator through only one target domain guide graph, the number of used samples is small, the accuracy of attribute inheritance can be ensured, the acquisition of global and local level representative characteristics of the guide graph and the inheritance of source domain generator diversity are considered, and a global level domain migration loss function L is designedglobalLocal horizontal domain migration loss function LlocalAnd an adaptive attribute retention penalty function LsccTherefore, the problems of insufficient acquisition of important attributes and styles and low diversity of the transferred pictures in the generator transferring process are solved, the transferred pictures have rich contents and can have styles similar to the target domain guide picture, and the adaptive domain generator transferring method is simple in process.
The inverse mapper is implemented using a pixel2style2pixel frame, i.e., a pSp frame, which is a pixel-to-pixel coded image coding frame.
Further, in step S2, a feature vector v of the center of the source domain is obtainedsrcAnd a global feature vector v of the target domain guide graphtarAnd a target domain guide map local feature vector sequence FtarThe implementation of the column is:
inputting the M source domain synthetic pictures into a first CLIP picture encoder for global feature extraction, obtaining global feature vectors of each source domain synthetic picture, and calculating M source domain synthetic picturesThe average value of the global feature vectors of the source domain synthetic picture is used as the center feature vector v of the source domainsrc；
Meanwhile, the target domain guide map is input into a first CLIP picture encoder for global feature extraction and local feature extraction, and a target domain guide map global feature vector v is obtainedtarAnd a target domain guide map local feature vector sequence Ftar。
The implementation manner for obtaining the source domain central feature vector, the target domain guide map global feature vector and the target domain guide map local feature vector sequence provided by the preferred embodiment can enable the initialized target domain generator to accurately estimate the global feature vector in the source domain and fully obtain the guide map global feature vector and the target domain guide map local feature vector sequence.
Further, in step S4, a global level domain migration loss function L is determinedglobalThe loss value of (a) is implemented as:
global feature vector v of picture before migrationAAnd the global feature vector v of the transferred pictureBAnd v characteristic vector of center of source regionsrcAnd a target domain guide map global feature vector vtarSubstituted into the global horizontal domain migration loss function LglobalTo obtain a global horizontal domain migration loss function LglobalThe loss value of (c).
The preferred embodiment determines a global horizontal domain migration loss function LglobalThe goal of the penalty value of (a) is to have the target domain generator accurately acquire the global level features of the guidance map.
Further, a global horizontal domain migration loss function LglobalThe expression of (a) is:
Figure BDA0003739510130000081
wherein ,
Δνsamp＝νB-νAformula 2;
Δνdom＝νtar-νsrc formula 3;
Δνsampglobal feature vector v for post-shift picturesBGlobal feature vector v of picture before migrationADifference of (d), Δ νdomDirecting graph global feature vector v for target domaintarAnd source domain center feature vector vsrcThe difference of (c).
In the preferred embodiment, a structure of a global horizontal domain migration loss function is constructed, and the function of the loss function through the structure is to enable the target domain generator to accurately acquire the global horizontal features of the guide map.
Further, in step S4, a local horizontal domain migration loss function L is determinedlocalThe loss value of (a) is implemented as:
global feature vector v of target domain guide graphtarAnd a target domain guide map local feature vector sequence FtarSubstituted into the local horizontal domain migration loss function LglobalThereby obtaining a loss value of the local horizontal domain migration loss function.
The specific implementation mode for determining the loss value of the local horizontal domain migration loss function is given, the process for obtaining the loss value of the local horizontal domain migration loss function is simple, and the loss value of the local horizontal domain migration loss function can be obtained only by substituting the global feature vector of the target domain guide map and the local feature vector sequence of the target domain guide map into the local horizontal domain migration loss function.
Further, a local horizontal domain migration loss function LlocalThe expression of (a) is:
Figure BDA0003739510130000091
Figure BDA0003739510130000092
wherein ,
Figure BDA0003739510130000093
for local feature vector sequence F of post-migration pictureBThe (ii) th vector of (a),
Figure BDA0003739510130000094
local feature vector sequence F for target domain guide maptarThe jth vector of (C)i,jIs composed of
Figure BDA0003739510130000095
And with
Figure BDA0003739510130000096
Cosine distance matrix between, n is the local characteristic vector sequence F of the picture after migrationBThe number of medium vectors, m is the local feature vector sequence F of the target domain guide maptarThe number of the medium vectors, i and j are integers, and i =1,2 \ 8230, i 823030, n, j =1,2 \ 8230, m, n and m are integers.
The preferred embodiment enables the target domain generator to accurately acquire the local horizontal features of the target domain guide map.
Further, in step S6, referring to fig. 2, the implementation manner of obtaining the target domain generator after migration is:
s61, target domain generator G after initializationBIn the training process, the attribute vector omega of the P transferred pictures obtained in the training process is usedAAfter summing and averaging, obtaining the average attribute vector of the images after migration, and obtaining the attribute vectors omega of P images before migration obtained in the training processBAfter summing and averaging, obtaining an average attribute vector of the picture before migration;
s62, subtracting the average attribute vector of the image after the migration from the average attribute vector of the image before the migration to obtain an average attribute difference value vector delta omega; the average attribute difference vector delta omega is a multidimensional vector, the dimensionality is N, and N is an integer larger than 3;
s63, determining the position values of all elements in the condition vector mask (delta omega, alpha), thereby obtaining the condition vector mask (delta omega, alpha), and obtaining the attribute vector omega of the picture before migration through the condition vector mask (delta omega, alpha) and the last trainingAAnd obtaining the attribute vector omega of the image after the migration through the last trainingASubstituting adaptive attribute hold penalty function LsccIn (1), obtaining an adaptive attribute preserving loss function LsccAnd using the loss value as the initialized target domain generator GBTo obtain the migrated target domain generator;
the dimension of the preset vector mask (delta omega, alpha) is the same as the dimension of the average attribute difference vector delta omega, and alpha is the proportion of the selected attribute.
Further, in S63, determining the values of the positions of all elements in the condition vector mask (Δ ω, α), so as to obtain the condition vector mask (Δ ω, α) is implemented as follows:
first, the value of the position of each element in the condition vector mask (Δ ω, α) is obtained in such a manner that,
Figure BDA0003739510130000101
wherein ,ΔωiIs the value of the ith element in the average attribute difference vector Δ ω, Δ ωsaNIs the value corresponding to the element with the small Nth alpha in the average attribute difference vector delta omega;
secondly, determining the values of all elements in the condition vector mask (delta omega, alpha) according to the value of the position of each element in the condition vector mask (delta omega, alpha), thereby obtaining the condition vector mask (delta omega, alpha); wherein,
mask(Δω，α)＝{mask(Δω，α)1，mask(Δω，α)2……mask(Δω，α)iequation 7;
the i =1,2,3 \8230, N.
The condition vector mask (delta omega, alpha) of the preferred embodiment determines the attribute position required to be maintained in the domain migration process, and ensures that the content of the picture before the migration is similar to that of the picture after the migration.
Further, an adaptive attribute preserving loss function LsccThe expression of (a) is:
Lscc＝||mask(Δω，α)·(ωB-ωA)||1equation 7;
wherein α is the proportion of the selected attribute, mask (Δ ω, α) is the condition vector, Δ ω is the average attribute difference vector, | · | | luminance1Is a norm of 1.
The preferred embodiment preserves the loss function L by adaptive propertysccThe shared attribute of the source domain and the target domain in the domain migration process is kept, so that the generated pictures have more diversity.
Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. It is therefore to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that features described in different dependent claims and herein may be combined in ways different from those described in the original claims. It is also to be understood that features described in connection with individual embodiments may be used in other described embodiments.

Claims (9)

1. A method for migrating a single-sample adaptive domain generator, the method comprising:
s1, training a pre-trained generator by using a plurality of source domain pictures and a plurality of 512-dimensional vectors obtained by random sampling in multi-dimensional Gaussian distribution to obtain a source domain generator GB(ii) a Reuse source domain generator GBThe weight of (3) initializes the target domain generator to obtain an initialized target domain generator GAMeanwhile, a picture with a style different from that of the source domain is selected as an initialized target domain generator GBThe target domain guidance map of (1);
s2, randomly sampling M512-dimensional vectors from multi-dimensional Gaussian distribution, and inputting the M512-dimensional vectors to a source domain generator GBSynthesizing M source domain synthetic pictures, inputting the M source domain synthetic pictures and the target domain guide picture into a first CLIP picture encoder for feature extraction to obtain the central features of the source domainVector vsrcAnd global feature vector v of target domain guide graphtarAnd a target domain guide map local feature vector sequence Ftar；
S3, randomly sampling a 512-dimensional vector from multi-dimensional Gaussian distribution, and simultaneously inputting the 512-dimensional vector to a source domain generator GAAnd an initialized target domain generator GBIn such a way that the source domain generator GATarget domain generator G for synthesizing picture before migration and after initializationBSynthesizing a picture after migration;
s4, inputting the synthesized picture before the migration and the picture after the migration into a second CLIP picture encoder, and extracting the global feature vector v of the picture before the migrationAAnd the global feature vector v of the transferred pictureBAnd local feature vector sequence F of the migrated pictureBThen, combining the source domain center feature vector vsrcAnd global feature vector v of target domain guide graphtarAnd a target domain guide map local feature vector sequence FtarDetermining a global horizontal domain migration loss function LglobalLoss value of (D) and local horizontal domain migration loss function LlocalThe loss value of (d);
meanwhile, inputting the synthesized picture before migration and the picture after migration into an inverse mapper, and extracting the attribute vector omega of the picture before migrationAAnd the attribute vector omega of the migrated pictureB(ii) a And according to the attribute vector omega of the picture before the migrationAAnd the attribute vector omega of the migrated pictureBDetermining an adaptive attribute retention loss function LsccThe loss value of (d);
s5, transferring a global horizontal domain migration loss function LglobalLoss value of (1), local horizontal domain migration loss function LlocalAnd adaptive property preserving penalty function LsccSumming the loss values to obtain a total loss value;
s6, judging whether the current total loss value is smaller than a preset threshold value or not, and if so, taking the current total loss value as an initialized target domain generator GBSo as to obtain the target domain generator after migration, and realize the migration of the adaptive domain generator; a result is no, according toTarget domain generator G with initialized current total loss value pairBThe parameters are optimized and updated to realize the generation of the initialized target domain GBThen step S3 is performed again.
2. The method for migrating the single-sample adaptive domain generator of claim 1, wherein in step S2, the center feature vector v of the source domain is obtainedsrcAnd a global feature vector v of the target domain guide graphtarAnd a target domain guide map local feature vector sequence FtarThe implementation of the column is:
inputting the M source domain synthetic pictures into a first CLIP picture encoder for global feature extraction, obtaining global feature vectors of each source domain synthetic picture, calculating the average value of the global feature vectors of the M source domain synthetic pictures, and taking the average value as a source domain central feature vector vsrc；
Meanwhile, inputting the target domain guide graph into a first CLIP picture encoder for global feature extraction and local feature extraction to obtain a target domain guide graph global feature vector vtarAnd a target domain guide map local feature vector sequence Ftar。
3. The single-sample adaptive domain generator migration method according to claim 1, wherein in step S4, a global level domain migration loss function L is determinedglobalThe loss value of (a) is implemented as:
global feature vector v of picture before migrationAAnd the global feature vector v of the transferred pictureBSource domain center feature vector vsrcAnd a target domain guide map global feature vector vtarSubstituted into the global horizontal domain migration loss function LglobalTo obtain a global horizontal domain migration loss function LglobalThe loss value of (c).
4. The single-sample adaptive domain generator migration method of claim 1, wherein a global level domain migration loss function LglobalThe expression of (c) is:
Figure FDA0003739510120000021
wherein ,
Δνsamp＝νB-νAformula 2;
Δνdom＝νtar-νsrcformula 3;
Δνsampglobal feature vector v for shifted picturesBGlobal feature vector v of picture before migrationADifference of (d), Δ νdomDirecting a graph global feature vector v for a target domaintarV and source domain center feature vectorsrcThe difference of (a).
5. The single-sample adaptive domain generator migration method according to claim 1, wherein in step S4, a local horizontal domain migration loss function L is determinedlocalThe loss value of (a) is implemented as:
global feature vector v of target domain guide graphtarAnd a target domain guide map local feature vector sequence FtarSubstituted into the local horizontal domain migration loss function LglobalThereby obtaining a loss value of the local horizontal domain migration loss function.
6. The single-sample adaptive domain generator migration method of claim 1, wherein a local horizontal domain migration loss function LlocalThe expression of (a) is:
Figure FDA0003739510120000031
Figure FDA0003739510120000032
wherein ,
Figure FDA0003739510120000033
as a sequence of local feature vectors F for the post-migration pictureBThe (ii) th vector of (a),
Figure FDA0003739510120000034
local feature vector sequence F for target domain guide maptarThe jth vector of (C)i,jIs composed of
Figure FDA0003739510120000035
And
Figure FDA0003739510120000036
cosine distance matrix between the two images, n is the local characteristic vector sequence F of the image after migrationBThe number of medium vectors, m is the local feature vector sequence F of the target domain guide maptarThe number of the medium vectors, i and j are integers, and i =1,2 \ 8230, i 823030, n, j =1,2 \ 8230, m, n and m are integers.
7. The method for migrating a single-sample adaptive domain generator according to claim 1, wherein the implementation manner of obtaining the target domain generator after migration in step S6 is as follows:
s61, target domain generator G after initializationBIn the training process, the attribute vector omega of the P transferred pictures obtained in the training process is usedAAfter summing and averaging, obtaining the average attribute vector of the images after migration, and obtaining the attribute vectors omega of P images before migration obtained in the training processBAfter summing and averaging, obtaining an average attribute vector of the picture before migration;
s62, subtracting the average attribute vector of the image after the migration from the average attribute vector of the image before the migration to obtain an average attribute difference value vector delta omega; the average attribute difference vector delta omega is a multidimensional vector, the dimensionality is N, and N is an integer larger than 3;
s63, determining the position values of all elements in the condition vector mask (delta omega, alpha) to obtain the position valuesThe conditional vector mask (delta omega, alpha) is trained for the last time to obtain the attribute vector omega of the picture before migrationAAnd obtaining the attribute vector omega of the image after the migration through the last trainingASubstituting adaptive attribute preserving loss function LsccIn (1), obtaining an adaptive attribute preserving loss function LsccAnd using the loss value as the initialized target domain generator GBTo obtain the migrated target domain generator;
the dimension of the preset vector mask (delta omega, alpha) is the same as the dimension of the average attribute difference vector delta omega, and alpha is the proportion of the selected attribute.
8. The single-sample adaptive domain generator migration method according to claim 7, wherein in S63, determining the values of the positions of all elements in the condition vector mask (Δ ω, α) to obtain the condition vector mask (Δ ω, α) is implemented by:
first, the value of the position of each element in the condition vector mask (Δ ω, α) is obtained in such a manner that,
Figure FDA0003739510120000041
wherein ,ΔωiIs the value of the ith element in the average attribute difference vector Δ ω, Δ ωsaNIs the value corresponding to the element with the small Nth alpha in the average attribute difference vector delta omega;
secondly, determining the values of all elements in the condition vector mask (delta omega, alpha) according to the value of the position of each element in the condition vector mask (delta omega, alpha), and obtaining the condition vector mask (delta omega, alpha); wherein,
mask(Δω，α)＝{mask(Δω，α)1，mask(Δω，α)2……mask(Δω，α)iequation 7;
the i =1,2,3 \8230, 8230N.
9. The method of claim 1The single-sample adaptive domain generator migration method of (1), wherein the adaptive attribute preserving loss function LsccThe expression of (a) is:
Lscc＝||mask(Δω，α)·(ωB-ωA)||1equation 7;
wherein α is the proportion of the selected attribute, mask (Δ ω, α) is the condition vector, Δ ω is the average attribute difference vector, | · | | luminance1Is a norm of 1.
CN202210811744.6A 2022-07-11 2022-07-11 Single sample adaptive domain generator migration method Active CN115272687B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210811744.6A CN115272687B (en) 2022-07-11 2022-07-11 Single sample adaptive domain generator migration method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210811744.6A CN115272687B (en) 2022-07-11 2022-07-11 Single sample adaptive domain generator migration method

Publications (2)

Publication Number Publication Date
CN115272687A true CN115272687A (en) 2022-11-01
CN115272687B CN115272687B (en) 2023-05-05

Family

ID=83765505

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210811744.6A Active CN115272687B (en) 2022-07-11 2022-07-11 Single sample adaptive domain generator migration method

Country Status (1)

Country Link
CN (1) CN115272687B (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109299783A (en) * 2018-12-18 2019-02-01 哈尔滨工业大学 Public sentiment role based on isomery domain migration identifies migratory system
WO2020062120A1 (en) * 2018-09-29 2020-04-02 浙江大学 Method for generating facial animation from single image
CN111640057A (en) * 2020-05-25 2020-09-08 武汉理工大学 Hidden variable decoupling-based human face image local feature migration network and method
US20210133571A1 (en) * 2019-11-05 2021-05-06 California Institute Of Technology Systems and Methods for Training Neural Networks
CN113255813A (en) * 2021-06-02 2021-08-13 北京理工大学 Multi-style image generation method based on feature fusion
CN113361566A (en) * 2021-05-17 2021-09-07 长春工业大学 Method for migrating generative confrontation network by using confrontation learning and discriminant learning
CN113570500A (en) * 2021-08-04 2021-10-29 光华临港工程应用技术研发(上海)有限公司 Method for realizing real image style migration based on global information guide network
CN114493991A (en) * 2021-12-16 2022-05-13 之江实验室 Attention circulation countermeasure network-based style migration system, method and device

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020062120A1 (en) * 2018-09-29 2020-04-02 浙江大学 Method for generating facial animation from single image
CN109299783A (en) * 2018-12-18 2019-02-01 哈尔滨工业大学 Public sentiment role based on isomery domain migration identifies migratory system
US20210133571A1 (en) * 2019-11-05 2021-05-06 California Institute Of Technology Systems and Methods for Training Neural Networks
CN111640057A (en) * 2020-05-25 2020-09-08 武汉理工大学 Hidden variable decoupling-based human face image local feature migration network and method
CN113361566A (en) * 2021-05-17 2021-09-07 长春工业大学 Method for migrating generative confrontation network by using confrontation learning and discriminant learning
CN113255813A (en) * 2021-06-02 2021-08-13 北京理工大学 Multi-style image generation method based on feature fusion
CN113570500A (en) * 2021-08-04 2021-10-29 光华临港工程应用技术研发(上海)有限公司 Method for realizing real image style migration based on global information guide network
CN114493991A (en) * 2021-12-16 2022-05-13 之江实验室 Attention circulation countermeasure network-based style migration system, method and device

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
TERO KARRAS ETAL.: "A Style-Based Generator Architecture for Generative Adversarial Networks" *
王坤峰 ，左旺孟等: "生成式对抗网络: 从生成数据到创造智能" *

Also Published As

Publication number Publication date
CN115272687B (en) 2023-05-05

Similar Documents

Publication Publication Date Title
Yim et al. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning
CN111062951B (en) Knowledge distillation method based on semantic segmentation intra-class feature difference
Zhao et al. Uni-controlnet: All-in-one control to text-to-image diffusion models
Wang et al. Towards compact single image super-resolution via contrastive self-distillation
Wang et al. Land cover change detection at subpixel resolution with a Hopfield neural network
CN110427799B (en) Human hand depth image data enhancement method based on generation of countermeasure network
CN111783521B (en) Pedestrian re-identification method based on low-rank prior guidance and based on domain invariant information separation
CN109829959B (en) Facial analysis-based expression editing method and device
CN112070658A (en) Chinese character font style migration method based on deep learning
US11914841B2 (en) Automatic generation of stylized icons
CN106971189B (en) A kind of noisy method for recognising star map of low resolution
CN113436060A (en) Method and device for transferring styles of different-source remote sensing images
CN104488264B (en) The method and apparatus of image motion are estimated using the different information of multi-view image
Rexer Spectral solutions to the topographic potential in the context of high-resolution global gravity field modelling
Men et al. DCT-Net: domain-calibrated translation for portrait stylization
CN106407932A (en) Handwritten number recognition method based on fractional calculus and generalized inverse neural network
CN115272687B (en) Single sample adaptive domain generator migration method
CN106570928B (en) A kind of heavy illumination method based on image
CN111046958A (en) Image classification and recognition method based on data-dependent kernel learning and dictionary learning
CN113111906B (en) Method for generating confrontation network model based on condition of single pair image training
Wang et al. HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion Models
CN111680552B (en) Feature part intelligent recognition method
Csillik et al. Towards a global geomorphometric atlas using Google Earth Engine
Khan et al. Fast color transfer from multiple images
Hafner et al. Multi-Modal Deep Learning for Multi-Temporal Urban Mapping with a Partly Missing Optical Modality

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant