CN112529080A - Image generation method based on spectral feature discrimination - Google Patents

Image generation method based on spectral feature discrimination Download PDF

Info

Publication number
CN112529080A
CN112529080A CN202011457259.0A CN202011457259A CN112529080A CN 112529080 A CN112529080 A CN 112529080A CN 202011457259 A CN202011457259 A CN 202011457259A CN 112529080 A CN112529080 A CN 112529080A
Authority
CN
China
Prior art keywords
input image
truth
image
discriminator
frequency domain
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011457259.0A
Other languages
Chinese (zh)
Other versions
CN112529080B (en
Inventor
陈元祺
靳策策
刘杉
李宏
李革
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Instritute Of Intelligent Video Audio Technology Longgang Shenzhen
Original Assignee
Instritute Of Intelligent Video Audio Technology Longgang Shenzhen
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Instritute Of Intelligent Video Audio Technology Longgang Shenzhen filed Critical Instritute Of Intelligent Video Audio Technology Longgang Shenzhen
Priority to CN202011457259.0A priority Critical patent/CN112529080B/en
Publication of CN112529080A publication Critical patent/CN112529080A/en
Application granted granted Critical
Publication of CN112529080B publication Critical patent/CN112529080B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/14—Fourier, Walsh or analogous domain transformations, e.g. Laplace, Hilbert, Karhunen-Loeve, transforms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Optimization (AREA)
  • Health & Medical Sciences (AREA)
  • Pure & Applied Mathematics (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Analysis (AREA)
  • Computational Mathematics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Algebra (AREA)
  • Databases & Information Systems (AREA)
  • Editing Of Facsimile Originals (AREA)

Abstract

The invention discloses an image generation method based on spectral feature discrimination, which comprises the following steps: embedding the frequency spectrum classifier into the original discriminator to obtain an enhanced discriminator; inputting the input image into the enhanced discriminator, and simultaneously evaluating the truth of the input image from the two layers of the space domain and the frequency domain to obtain the frequency domain truth and the space domain truth of the input image; and weighting and integrating the frequency domain truth and the space domain truth to obtain the total truth of the input image. The image generation method based on the spectral feature discrimination uses the enhanced discriminator, can obtain effective excitation of learning high-frequency information, thereby obtaining better generation quality, and has certain improvement compared with a reference model on the image generation tasks of high resolution and low resolution.

Description

Image generation method based on spectral feature discrimination
Technical Field
The invention relates to the field of image generation and generation countermeasure networks, in particular to an image generation method based on spectral feature discrimination.
Background
The recent rapid development of creating an antagonistic network has greatly advanced the fields of computer vision and image processing, and models based on creating an antagonistic network have achieved good results on tasks such as image completion, image coloring, and image conversion. The generation of the countermeasure network comprises two parts of a generator and an arbiter, wherein the process of training the generation of the countermeasure network is a process of a infinitesimal game. Where the goal of the generator is to produce enough samples to confuse the arbiter with false positives; the discriminator then attempts to distinguish between samples from the true data distribution and the generated samples. With this kind of counter training, the final generator will be able to generate samples closer to the real data.
Under the framework of a standard generative confrontation network, for the discriminator, the confrontation loss function is:
Figure BDA0002829793070000011
wherein D (x) represents that sample x is from distribution PdataThe probability of (d), (x) measures the trueness of the sample x. If x is true, it will be considered a true sample from all perspectives. However, recent studies have shown that even though generating a countermeasure network can generate an image sufficiently realistic in the spatial domain, the generative model often does not achieve good performance when the degree of realism is evaluated from the frequency domain. This indicates that the trueness of the generated samples cannot be measured from the spatial domain alone.
The frequency domain of the image generated by the generation countermeasure network is not sufficiently realistic in that the distribution in the high frequency portion has a large difference compared with the real image. This difference can cause two problems: one is that due to the correspondence between the space domain and the frequency domain, the frequency domain difference means that the generated model does not well capture the characteristics of real data distribution in the space domain; secondly, the high frequency affects the detail accuracy of the image, and for the image generation task, the high frequency information is also important, and the abnormality of the high frequency part will cause the distortion of the detail in the generated image.
The reason why the countermeasure network is generated cannot learn the high frequency information of the data distribution is because the discriminators in the generation countermeasure network lack the discrimination ability for the high frequency part. When the high frequency part of the input image is adjusted differently, the discriminator network will output an approximate discrimination probability. This results in the generator lacking the excitation from the arbiter to learn the high frequency information of the true data distribution.
A discriminator discriminates the downsampled layer in a missing source domain discriminator network of capability for high frequency portions. The downsampling layers can be divided into two categories, an antialiasing-free downsampling layer and an antialiasing downsampling layer. For the anti-aliasing-free down-sampling layers, including the maximum pooling layer, the mean pooling layer, the convolution layer with step length and the like which are commonly used in the deep learning nowadays, due to neglecting the nyquist sampling theorem, aliasing is often generated in the high-frequency part after down-sampling, so that the high-frequency information is invalid. For the anti-aliasing down-sampling layer, the signal is low-pass filtered before the down-sampling operation, and part of high-frequency information is filtered. Although the aliasing effect is mitigated, the high frequency information is suppressed when low pass filtering is performed. In summary, both types of down-sampling layers result in the loss of discrimination capability of the discriminator for the high frequency part.
Disclosure of Invention
The invention provides an image generation method based on spectral feature discrimination, which can obtain effective excitation for learning high-frequency information so as to obtain better generation quality.
The technical scheme of the invention is as follows:
the invention discloses an image generation method based on spectral feature discrimination, which comprises the following steps: the method comprises the following steps: embedding the frequency spectrum classifier into the original discriminator to obtain an enhanced discriminator; step two: inputting the input image into the enhanced discriminator, and simultaneously evaluating the truth of the input image from the two layers of the space domain and the frequency domain to obtain the frequency domain truth and the space domain truth of the input image; and a third step: and weighting and integrating the frequency domain truth and the space domain truth to obtain the total truth of the input image.
Preferably, in the above image generation method, in the first step, a spectrum classifier is further configured to measure the degree of truth of the input image spectrum.
Preferably, in the image generation method, in the second step, for an input image, fourier transform is performed first, and then azimuth averaging is performed to obtain a spectral feature vector of the input image, and then the spectral feature vector is input to a spectral classifier to obtain the frequency domain truth of the input image; and meanwhile, the input image is brought into the original discriminator, and the space domain truth of the input image is obtained in the original discriminator.
Preferably, in the above image generation method, in step three, the total degree of realism of the input image is represented by Dss(x) λ d (x) + (1- λ) c (x), where x denotes the input image, λ controls the relative importance of spatial and frequency domain realism, d (x) denotes the spatial and c (x) denotes the frequency domain realism of the input image x.
According to the technical scheme of the invention, the beneficial effects are as follows:
the image generation method based on the spectral feature discrimination uses the enhanced discriminator, can obtain effective excitation of learning high-frequency information, thereby obtaining better generation quality, and has certain improvement compared with a reference model on the image generation tasks of high resolution and low resolution.
For a better understanding and appreciation of the concepts, principles of operation, and effects of the invention, reference will now be made in detail to the following examples, taken in conjunction with the accompanying drawings, in which:
drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the invention and, together with the description, serve to explain the invention and not to limit the invention.
FIG. 1 is a general framework diagram of the image generation method based on spectral feature discrimination according to the present invention; and
FIG. 2 is a schematic diagram of a frequency domain discrimination result according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without any creative effort, shall fall within the protection scope of the present invention.
The working principle of the invention is as follows: constructing a spectrum classifier C to measure the truth of the image spectrum, and embedding the classifier C into the original discriminator D to obtain an enhanced discriminator DssThereby making the enhanced discriminator DssThe image can be evaluated for the truth degree from the two levels of the space domain and the frequency domain at the same time. Using enhanced discriminators DssThe generator can obtain effective excitation for learning high-frequency information, and therefore better generation quality is obtained.
Fig. 1 is an overall frame diagram of an image generation method based on spectral feature discrimination according to the present invention. Referring to fig. 1, the image generation method based on spectral feature discrimination of the present invention mainly includes the following steps:
the method comprises the following steps: embedding the spectrum classifier C into the original discriminator D to obtain the enhanced discriminator Dss(ii) a The spectrum classifier C measures the truth of the spectrum of the input image x.
Step two: inputting the input image x to the enhanced discriminator DssAnd simultaneously evaluating the truth of the input image x from the spatial domain and the frequency domain to obtain the frequency domain truth and the spatial domain truth of the input image x. In particular, forCarrying out Fourier transform on an input image x, then carrying out azimuth angle averaging to obtain a frequency spectrum characteristic vector of the input image x, and then inputting the frequency spectrum characteristic vector into a frequency spectrum classifier C to obtain the frequency domain truth of the input image x; and meanwhile, bringing the input image x into the original discriminator D, and obtaining the space domain truth of the input image x in the original discriminator D.
Step three: and weighting and integrating the frequency domain truth and the space domain truth to obtain the total truth of the input image x for generating the countermeasure training of the countermeasure network. For an input image x, the overall degree of realism may be expressed as Dss(x) λ d (x) + (1- λ) c (x), where λ controls the relative importance of spatial and frequency domain realism, d (x) represents the spatial realism of the input image x, and c (x) represents the frequency domain realism of the input image x.
Fig. 2 shows a diagram of the classification result of the spectrum classifier in the present invention. For the real sample and the generated sample, according to the frequency spectrum classifier, images with high frequency spectrum quality have clear portraits and rich details; while images of low spectral quality either suffer from overexposure, lose detail, or have some distorted high frequency information.
Compared with the existing image generation method, the method provided by the invention has better generation quality and hidden space with better decoupling property.
The generation quality in image generation, the concept of implicit spatial decoupling and related evaluation indexes.
The quality of generation: and judging whether the converted image has higher image quality. On the evaluation index, the evaluation is divided into objective evaluation and subjective evaluation. Fraich perceptual distance (FID) is a commonly used objective method of generating an estimate of quality. To calculate the FID of an image conversion model, a batch of converted images is first generated using the model and sampled from the data set for comparison. Then, the characteristics of the two batches of images are extracted, the statistical characteristics of the two batches of images are calculated, and the difference of distribution between the generated image and the real image is measured based on the statistical characteristics to serve as the evaluation of the quality of the generated image.
Hidden spatial decoupling: and measuring whether the hidden space of the image generation model has more regular subspaces or not, wherein each subspace controls a type of variation factor. Perceptual path length is a commonly used evaluation index that measures how smooth an inter-image is generated when interpolating in a hidden space. For a hidden space with good decoupling, smooth change can be obtained during interpolation.
On high resolution (e.g., 1024 × 1024 high resolution) image generation tasks, the results of the present invention are plotted against the results of other algorithms as shown in table 1. The reference model is StyleGAN, and the enhanced model based on the invention is SSD-StyleGAN. As can be seen from table 1, the enhanced model (SSD-StyleGAN) of the present invention has a certain improvement in the index FID for measuring the generation quality and the index (perceptual path length) for measuring the implicit spatial decoupling compared to the reference model (StyleGAN).
Table 1 comparison of the results of the present invention with those of other algorithms on the task of high resolution image generation
Figure BDA0002829793070000041
The effectiveness of the invention was also verified experimentally on other low resolution image generation tasks (e.g., resolution 32 x 32,48 x 48 and 128 x 128). Table 2 shows a comparison between the present invention result and the fraiche perceived distance (FID) result of other algorithms on other image generation tasks, wherein the reference model is selected to be SNGAN, and the enhanced model based on the present invention is SSD-SNGAN. As can be seen from Table 2, the enhanced model according to the present invention achieves a certain improvement over three data sets CIFAR100-32, STL10-48 and LSUN-128 (wherein the data sets CIFAR100-32, STL10-48 and LSUN-128 have respective resolutions of 32X 32, 48X 48 and 128X 128).
TABLE 2 comparison of the results of the present invention with the Frey cut perceptual distance (FID) results of other algorithms on other low resolution image generation tasks
Model (model) CIFAR100-32 STL10-48 LSUN-128
Reference model 22.61 39.56 25.87
The invention 19.28 36.41 15.17
The present invention is not limited to the above preferred embodiments, and any modifications, equivalent replacements, improvements, etc. within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (4)

1. An image generation method based on spectral feature discrimination is characterized by comprising the following steps:
the method comprises the following steps: embedding the frequency spectrum classifier into the original discriminator to obtain an enhanced discriminator;
step two: inputting an input image into the enhanced discriminator, and simultaneously evaluating the truth of the input image from two layers of a space domain and a frequency domain to obtain the frequency domain truth and the space domain truth of the input image; and
step three: and weighting and integrating the frequency domain truth and the space domain truth to obtain the total truth of the input image.
2. The image generation method of claim 1, wherein in step one, the method further comprises constructing the spectrum classifier to measure the trueness of the input image spectrum.
3. The image generation method according to claim 1, wherein in step two, for one input image, fourier transform is performed first, and then azimuth averaging is performed to obtain a spectral feature vector of the input image, and then the spectral feature vector is input to the spectral classifier to obtain the frequency domain truth of the input image; and meanwhile, bringing the input image into the original discriminator to obtain the space domain truth of the input image in the original discriminator.
4. The image generation method according to claim 1, wherein in step three, the overall degree of realism of the input image is represented by Dss(x) λ d (x) + (1- λ) c (x), where x denotes the input image, λ controls the relative importance of spatial and frequency domain realism, d (x) denotes the spatial and c (x) denotes the frequency domain realism of the input image x.
CN202011457259.0A 2020-12-11 2020-12-11 Image generation method based on spectrum feature discrimination Active CN112529080B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011457259.0A CN112529080B (en) 2020-12-11 2020-12-11 Image generation method based on spectrum feature discrimination

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011457259.0A CN112529080B (en) 2020-12-11 2020-12-11 Image generation method based on spectrum feature discrimination

Publications (2)

Publication Number Publication Date
CN112529080A true CN112529080A (en) 2021-03-19
CN112529080B CN112529080B (en) 2023-07-25

Family

ID=74999073

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011457259.0A Active CN112529080B (en) 2020-12-11 2020-12-11 Image generation method based on spectrum feature discrimination

Country Status (1)

Country Link
CN (1) CN112529080B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117149359A (en) * 2023-10-27 2023-12-01 大尧信息科技(湖南)有限公司 Software definition simulation development platform and method for data chain equipment

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9159132B1 (en) * 2012-12-07 2015-10-13 Ricky Carl Crowsey Process and system for identifying damage
US20180239952A1 (en) * 2017-02-17 2018-08-23 Cogisen S.R.L. Method for image processing for content detection
CN111179189A (en) * 2019-12-15 2020-05-19 深圳先进技术研究院 Image processing method and device based on generation countermeasure network GAN, electronic equipment and storage medium
CN111275640A (en) * 2020-01-17 2020-06-12 天津大学 Image enhancement method for fusing two-dimensional discrete wavelet transform and generating countermeasure network
CN111696168A (en) * 2020-06-13 2020-09-22 中北大学 High-speed MRI reconstruction method based on residual self-attention image enhancement

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9159132B1 (en) * 2012-12-07 2015-10-13 Ricky Carl Crowsey Process and system for identifying damage
US20180239952A1 (en) * 2017-02-17 2018-08-23 Cogisen S.R.L. Method for image processing for content detection
CN111179189A (en) * 2019-12-15 2020-05-19 深圳先进技术研究院 Image processing method and device based on generation countermeasure network GAN, electronic equipment and storage medium
CN111275640A (en) * 2020-01-17 2020-06-12 天津大学 Image enhancement method for fusing two-dimensional discrete wavelet transform and generating countermeasure network
CN111696168A (en) * 2020-06-13 2020-09-22 中北大学 High-speed MRI reconstruction method based on residual self-attention image enhancement

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117149359A (en) * 2023-10-27 2023-12-01 大尧信息科技(湖南)有限公司 Software definition simulation development platform and method for data chain equipment
CN117149359B (en) * 2023-10-27 2024-01-26 大尧信息科技(湖南)有限公司 Software definition simulation development platform and method for data chain equipment

Also Published As

Publication number Publication date
CN112529080B (en) 2023-07-25

Similar Documents

Publication Publication Date Title
CN109522857B (en) People number estimation method based on generation type confrontation network model
CN110163813B (en) Image rain removing method and device, readable storage medium and terminal equipment
CN111583109A (en) Image super-resolution method based on generation countermeasure network
CN108414240B (en) Method and device for detecting abnormal vibration of machine
Ertürk et al. Hyperspectral image classification using empirical mode decomposition with spectral gradient enhancement
CN110909615B (en) Target detection method based on multi-scale input mixed perception neural network
CN111161224A (en) Casting internal defect grading evaluation system and method based on deep learning
CN108830829B (en) Non-reference quality evaluation algorithm combining multiple edge detection operators
CN107944497A (en) Image block method for measuring similarity based on principal component analysis
CN112529080A (en) Image generation method based on spectral feature discrimination
CN115908142A (en) Contact net tiny part damage testing method based on visual recognition
CN116205886A (en) Point cloud quality assessment method based on relative entropy
CN112767331B (en) Image anomaly detection method based on zero sample learning
CN111079893A (en) Method and device for obtaining generator network for interference fringe pattern filtering
CN109978855A (en) A kind of method for detecting change of remote sensing image and device
CN113378472A (en) Mixed boundary electromagnetic backscattering imaging method based on generation countermeasure network
CN108764112A (en) A kind of Remote Sensing Target object detecting method and equipment
Sahasrabudhe et al. Structured spatial domain image and data comparison metrics
CN107944474A (en) Multiple dimensioned cooperation table based on local auto-adaptive dictionary reaches hyperspectral classification method
CN114066786A (en) Infrared and visible light image fusion method based on sparsity and filter
Krishnan et al. A novel underwater image enhancement technique using ResNet
CN107977967B (en) No-reference image quality evaluation method for view angle synthesis
CN115880158A (en) Blind image super-resolution reconstruction method and system based on variational self-coding
CN115761606A (en) Box electric energy meter identification method and device based on image processing
CN104063864A (en) Image fuzziness assessment method based on quaternary phase congruency model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant