CN114758021A - Earth surface image generation method and system based on generation countermeasure network - Google Patents

Earth surface image generation method and system based on generation countermeasure network Download PDF

Info

Publication number
CN114758021A
CN114758021A CN202210249374.1A CN202210249374A CN114758021A CN 114758021 A CN114758021 A CN 114758021A CN 202210249374 A CN202210249374 A CN 202210249374A CN 114758021 A CN114758021 A CN 114758021A
Authority
CN
China
Prior art keywords
network
surface image
generation
texture
data set
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210249374.1A
Other languages
Chinese (zh)
Inventor
代磊
李华伟
王颖
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Institute of Computing Technology of CAS
Original Assignee
Institute of Computing Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institute of Computing Technology of CAS filed Critical Institute of Computing Technology of CAS
Priority to CN202210249374.1A priority Critical patent/CN114758021A/en
Publication of CN114758021A publication Critical patent/CN114758021A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/001—Texturing; Colouring; Generation of texture or colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a surface image generation method based on a generation countermeasure network, which comprises the following steps: acquiring an original earth surface image, and generating an original data set, a texture data set and a frame data set; constructing a texture generation network, and training the texture generation network by using the texture data set; constructing a frame generation network, and training the frame generation network by using the frame data set; embedding a first generator of the texture generation network into a second generator of the framework generation network to obtain a surface image generation network, and training the surface image generation network by using the original data set; the earth surface image generation is performed for the earth surface image generation model by using the generator of the earth surface image generation network. A ground surface image generation system based on the generation countermeasure network and a data processing device are also provided.

Description

Earth surface image generation method and system based on generation countermeasure network
Technical Field
The invention belongs to the technical field of computer vision and deep learning, and particularly relates to a method for generating a confrontation network (GAN) by training textures, and embedding the GAN into a complete image to generate the GAN so as to complete planet landform generation.
Background
PGGAN is stably trained in a staged training mode by gradually increasing the output resolution, and good effects are obtained in tasks such as face generation and the like. PGGAN training starts with low resolution images and then steps up the resolution by adding layers into the network. The training process of the network growth can firstly find a small number of features on a larger scale, then gradually enter the learning of more refined features, and gradually learn the features of all scales. In addition, the process of gradual training may also avoid the front layers from being difficult to train due to gradient disappearance problems if all layers are trained together. In the actual operation process, the incremental confrontation generation network introduces technologies including smooth transition in resolution, balanced learning rate, small batch standard deviation and the like for improving the training effect. The training process is shown in fig. 1.
The StyleGAN is inspired by style transfer of style migration, and a new generator network structure is designed on the basis of PGGAN. The new network structure can perform certain decoupling and separation on the high-level semantic attributes of the images through unsupervised automatic learning, such as the postures and identities of the face images and random changes of the generated images, such as freckles, hairs and the like, and can also perform control synthesis to a certain degree. The structure of the generator network is shown in fig. 2, and compared with the traditional structure, the structure is changed into two parts of a mapping network and a generating network.
The work of Multi-Scale Terrain computerized adapting Using A novel automatic generation process of detail maps is proposed, which can reduce tiling artifacts in real-time Terrain rendering. This is accomplished by training a generator of a competing network (GAN) using a single input texture and then using it to synthesize a huge texture across the entire terrain. During rendering, the low frequency components of the GAN output are extracted, scaled down, and combined with the high frequency components of the input texture. This results in highly detailed and non-repeating terrain texture, eliminating tiling artifacts without reducing overall image quality. Rendering is efficient in terms of both memory consumption and computational cost.
However, PGGAN is difficult to synthesize and contains texture that clearly fits real world features, and the types of terrain samples that can be generated are limited; StyleGAN and StyleGANV2 can generate images with sufficiently clear textures, but still have difficulty generating samples that are less dominant in the training data when faced with data skew; while Multi-Scale Terrain Texturing using genetic adaptive Networks uses a texture stitching method, generating a textured picture by stitching may be difficult to avoid tiling artifact problems during the stitching process.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method for generating a surface image based on a generation countermeasure network, including: acquiring an original earth surface image, and generating an original data set, a texture data set and a frame data set; constructing a texture generation network, and training the texture generation network by using the texture data set; constructing a frame generation network, and training the frame generation network by using the frame data set; embedding the generator of the texture generation network into the generator of the framework generation network to obtain a surface image generation network, and training the surface image generation network by using the original data set; the earth surface image generation is performed for the earth surface image generation model by using the generator of the earth surface image generation network.
The earth surface image generation method of the invention, wherein the step of obtaining the earth surface image generation network specifically comprises the following steps: the generator G1 ' of the texture generation countermeasure network GAN1 is embedded into the generator G2 ' of the framework generation countermeasure network GAN2, and a plurality of layers of fusion and resolution enhancement layers G3 are connected behind G2 ', so that the generator G2 of the ground surface image generation countermeasure network GAN2 is obtained; adding a resolution reduction network D3 before a discriminator D2' of the framework generation countermeasure network GAN2 to obtain a discriminator D2 of GAN 2; GAN2 is trained with this original data set R for a specified number of iterations.
The invention relates to a method for generating a surface image, wherein the surface image generates a loss function of a discriminator D2 of a network
Figure BDA0003546328300000021
Wherein, a2Is history LD2Loss of power
Figure BDA0003546328300000022
The proportion of the active ingredients is that,
Figure BDA0003546328300000023
for L of the last iterationD2，LwgangpIs Wasserstein-gp loss function.
The method for generating a surface image according to the present invention, wherein the texture generation network has a loss function L of a discriminator D1D1Comprises the following steps:
Figure BDA0003546328300000024
Figure BDA0003546328300000025
is LD1Current loss and LD1Historical value
Figure BDA0003546328300000026
Weighted new value, w is the weight occupied by the tag loss, L2As a function of scale label loss, a1Is history LD1Loss of power
Figure BDA0003546328300000027
The proportion of the active ingredients is that,
Figure BDA0003546328300000028
for the last iteration
Figure BDA0003546328300000029
LwgangpIs Wasserstein-gp loss function.
The invention also provides a ground surface image generation system based on the generation countermeasure network, which comprises the following components: the system comprises an original data acquisition module, a texture data acquisition module and a frame data acquisition module, wherein the original data acquisition module is used for acquiring an original earth surface image and generating an original data set, a texture data set and a frame data set; the first model training module is used for constructing a texture generation network and training the texture generation network by the texture data set; the second model training module is used for constructing a frame generation network and training the frame generation network by using the frame data set; the third model training module is used for embedding the generator of the texture generation network into the generator of the framework generation network to obtain a surface image generation network, and training the surface image generation network by using the original data set; and the earth surface image generation module is used for generating an earth surface image for the earth surface image generation model by using the generator of the earth surface image generation network.
The earth surface image generation system of the invention, wherein the third model training module specifically comprises: the network generation module is used for generating the earth surface image generation network; the generator G1 ' of the texture generation confrontation network GAN1 is embedded into the generator G2 ' of the framework generation confrontation network GAN2, and a plurality of layers of fusion and resolution enhancement layers G3 are connected behind G2 ', so that the generator G2 of the ground surface image generation confrontation network GAN2 is obtained; adding a resolution reduction network D3 before a discriminator D2' of the framework generation countermeasure network GAN2 to obtain a discriminator D2 of GAN 2; and the network training module is used for training the GAN2 for a specified iteration number by using the original data set R.
The earth surface image generation system according to the present invention, wherein the earth surface image generation network has a loss function of a discriminator D2
Figure BDA0003546328300000031
Wherein, a2Is history LD2Loss of power
Figure BDA0003546328300000032
The proportion of the active ingredients is that,
Figure BDA0003546328300000033
for L of the last iterationD2，LwgangpIs Wasserstein-gp loss function.
The earth surface image generation system of the present invention, wherein the texture generation network has a loss function L of a discriminator D1D1Comprises the following steps:
Figure BDA0003546328300000034
Figure BDA0003546328300000035
is LD1Current loss and LD1Historical value
Figure BDA0003546328300000036
Weighted new value, w is the weight occupied by the tag loss, L2As a function of scale label loss, a 1Is history LD1Loss of power
Figure BDA0003546328300000037
The proportion of the active ingredients is that,
Figure BDA0003546328300000038
for the last iteration
Figure BDA0003546328300000039
LwgangpIs Wasserstein-gp loss function.
The present invention also proposes a computer-readable storage medium storing computer-executable instructions that, when executed, implement the method for generating a surface image based on a generative confrontation network as described above.
The invention also proposes a data processing apparatus comprising a computer-readable storage medium as described above, the processor of the data processing apparatus generating a surface image when retrieving and executing computer-executable instructions in the computer-readable storage medium.
Drawings
Fig. 1 is a schematic diagram of a PGGAN countermeasure network training process.
Fig. 2 is a schematic diagram of a StyleGAN generator network structure.
Fig. 3 is a flowchart of a surface image generation method of the present invention.
Fig. 4A, 4B, and 4C are schematic diagrams of the earth surface image generation network structure according to the present invention.
FIG. 5 is a diagram of an embodiment of the texture generation network embedding method and the GAN2 complete network structure of the present invention.
FIG. 6 is a schematic diagram of the R middle terrain sample and R1 construction process of the present invention
FIG. 7 is a schematic diagram of a data processing apparatus of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention aims to solve the problem that the existing GAN network is difficult to generate sufficiently clear textures or generates few types of textures in the face of deflection data in an extraterrestrial planet sample generation task, and provides an extraterrestrial planet surface sample generation method based on GAN.
In an attempt to complete the generation of the mars surface samples by using the mars surface data set, the inventor finds that it is difficult to synthesize a network structure (PGGAN) containing a clear texture according to real world features and that the types of the terrain samples capable of being generated are limited; StyleGAN and StyleGANV2 are able to generate images with sufficiently clear texture, but still have difficulty generating samples that are smaller in the training data when faced with data skew. The inventors believe that the problem arises because the non-uniformity of the sample makes it difficult for a particular texture to be adequately learned in the process of directly generating a complete surface image.
The inventor researches the characteristics of the Mars surface image sample to find that the surface image mainly comprises several specific surface characteristics, namely several specific textures, the texture of each texture can be cut out from a complete data set, and the cut-out characteristic blocks have smaller resolution, so that the surface image is easy to learn by a GAN network. The difficulty of generating a complete surface sample image can be reduced by learning textures firstly and then learning an integral image.
Based on the analysis, the inventor proposes that a texture generation network is obtained through firstly intercepting a terrain texture block and countertraining; secondly, using the samples after the low-resolution samples are blurred to initially resist and train to completely generate a low-resolution part layer of the network; and finally, embedding the texture generation network into a complete frame network, and performing countermeasure training by using the real image to obtain the generation network capable of generating clear texture features and enough kinds of terrains.
The earth surface image generation method adopts a combined GAN structure, firstly trains texture generation networks to learn texture characteristics, and then embeds the trained texture generation networks into complete samples to generate GAN networks for training. Therefore, under the condition that the surface samples have certain deflection, the network is prevented from being difficultly trained to obtain a good generation effect due to the fact that all sample characteristics are directly learned, and the difficulty of generating better samples is reduced through stage training.
The texture generation network adopts the input noise, the input noise is converted by the full-connection network and then input into the rear resolution ratio promotion network, and the full-connection layer outputs a channel dimension protocol during independent training, so that only a single sample is generated, the training is facilitated, and the full-connection network can be trained; after the complete generation network is embedded, the output of the full connection layer adopts multi-channel dispersion and then is input into the rear network, so that the complete network can generate a plurality of samples in a single period. In this way, a single output of the texture generation network only contains less kinds of texture information, and multiple embedding of multiple textures can make the generated sample generate more kinds of texture features.
In addition, the scaling scale is used for constraint during the training of the texture learning network, so that the output texture of the texture generation network can be controlled by the constraint of the input scale; accordingly, when embedding the complete sample generation network, the multiple output feature maps are embedded in different resolution enhancement layers. The scale constraints and multi-resolution embedding enable the sample generation network to make use of different scales of texture information differently at different stages.
The general flow of the earth surface image generation method based on the attention texture generation of the generation confrontation network is shown in fig. 3, and comprises 5 general steps, wherein the network structures introduced in fig. 3 are shown in fig. 4A, 4B and 4C, and the specific steps are as follows:
step S1: collecting an original planet surface data set R, constructing a data set R1 containing texture scales by selecting, preprocessing, cutting and scaling the data set R, and constructing a low-resolution frame data set R2 by fuzzy downsampling the data set R.
The data set R can be constructed by the following steps:
in step S11, the data set R is a surface image data set that is shot with a relatively fixed viewing angle using a near-surface camera and cut into square blocks of a fixed size.
The data set R1 can be constructed by the following steps:
Step S12, uniformly selecting each terrain sample from R and adding each terrain sample into the set R'.
And step S13, for each sample in the R', performing rough segmentation on the foreground and the background by using a background segmentation algorithm, such as GrabCT and the like, and obtaining a foreground partial mask.
And step S14, performing expansion and contraction on the foreground mask to connect the adjacent masks, filtering by using the area of the mask area, and deleting the mask part with the smaller area.
Step S15, for each continuous mask region convex hull on one mask, according to the area size, the maximum square is cut from the inside or the minimum square is obtained from the outside to obtain a square mask region set, and each single square mask in the set is used for cutting an image block from an original image to be used as a texture sample block.
Step S16, for each cut sample block, calculating the proportion compared with the target size as a scale coefficient, scaling the sample block to the target size by using the scale coefficient, and adding the obtained sample block and the corresponding scale coefficient into a data set R1.
The data set R2 can be constructed by:
in step S17, for each sample in R, fine-grained noise is first filtered out by fuzzy filtering, for example, using gaussian fuzzy algorithm. The blurred sample samples are scaled to a lower target resolution and the resulting image is added to the sample set R2.
It should be understood that the collection of the data set R and the construction of the data sets R1 and R2 are not only completed by the above steps, for example, the data set R may be collected by unmanned aerial vehicle view shooting or/and satellite image capturing method, the data set R1 may be constructed by manual capturing or/and deep learning foreground segmentation method, and the data set R2 may also be constructed by direct downsampling method, which is not limited by the invention.
Step S2: a texture generation confrontation network GAN1 is constructed and GAN1 is trained using the data set R1 until preset requirements are met. The method specifically comprises the following steps:
at step S21, the texture generation countermeasure network GAN1 is constructed based on the GAN network, as shown in fig. 4A, the details of the GAN1 network are as follows:
at step S211, the GAN1 network generator G1 includes: multi-level fully-connected network portion fc1 ', multi-level convolution + resolution enhancement network portion up 1', primary RGB conversion layer LrgbWherein the sub-network connecting fc1 ' with up1 ' is used as the sub-generator g1 ' to be provided by the GAN1 network.
Step S212, set the input vector G1, G1 ', fc 1' to G1inA vector formed by connecting a noise vector Z' and a scale coefficient S is set as g1inLength Len (g 1)in) (ii) a Setting the output vector of fc 1' to fc1 outLength Len (fc 1)out) (ii) a Setting up 1' network input profile as vector up1inLength of Len (up 1)in)。Len(fc1out) Is Len (up 1)in) M is an integer greater than 1.
Step S213, fc1outFolded as M channels with length per channel of Len (up 1)in) The tensor is averaged over the M channels of the tensor to obtain a single-channel vector input g 1'; the output of g 1' is a multi-channel feature map f1′，f1' Jing LrgbThe converted RGB color image is output with the same size as the target texture block in the R1 set.
In step S214, the GAN1 network discriminator is D1, and D1 outputs the binary true/false discrimination ToRF and the predicted scale factor S'.
Loss function L of step S215, D1D1Comprises two parts, the first part is a history average Wasserstein-gp loss function LwgangpA second partAs a scale label loss function L2. The loss function is specifically defined as follows:
Figure BDA0003546328300000071
Figure BDA0003546328300000072
Figure BDA0003546328300000073
in the above formula, w is the weight occupied by the label loss, a1Is history LD1Loss of power
Figure BDA0003546328300000074
The proportion of the active ingredients is that,
Figure BDA0003546328300000075
is the current LD1And a weighted new value of the historical value, LwgangpIs Wasserstein-gp loss function, PgFor the generator to generate a sample distribution, PrIn order to be a true distribution of the sample,
Figure BDA0003546328300000076
for the distribution of all samples, D is the discriminator calculation,
Figure BDA0003546328300000077
for the last iteration of the saving
Figure BDA0003546328300000078
L2In order to scale the tag loss function,
Figure BDA0003546328300000079
For the sample true tag value(s),
Figure BDA00035463283000000710
and (4) predicting the sample label.
Step S22, training the texture generation confrontation network GAN1, inputting control noise Z' and scale coefficient S for the network generator G1 to generate a false sample P1; inputting P1 and the real sample in R1 into a network discriminator D1 to predict ToRF and S', calculating a loss function LD1And propagates backward, iterating the training generator G1 and the arbiter D1, until the end. The training end condition is that the preset iteration number is reached or the score of the generated sample stably exceeds a threshold value.
Step S3: constructing a low resolution generation countermeasure network gan2, training gan2 using data set R2 until a preset requirement is reached. The method specifically comprises the following steps:
in step S31, a low resolution part GAN2 of the surface sample generation countermeasure network GAN2 is constructed, as shown in fig. 4B, the details of the GAN2 network are as follows:
step S311, connecting RGB conversion layer L after the initial frame generator g 2' of gan2 networkrgb2. g2 'is formed as a single layer full link layer connecting convolution + resolution enhancement layers, the input of g 2' is control noise Z ', the output is a multi-channel feature map f 2', the feature map is processed through Lrgb2The converted RGB color image P2 is output with the same target texture block size as the data set R2.
In step S312, the network discriminator of gan2 is d2 ', and d 2' outputs the binary authenticity discrimination TorF.
Loss function L of step S313, d2d2′Loss function L of network arbiter D2 with GAN2D2And (5) the consistency is achieved.
Step S32, training the network: the gan2 network generator input controls the noise Z' to generate a dummy sample P2; inputting P2 and real samples in R2 into a discriminator to predict ToRF, calculating a loss function and propagating reversely, and iteratively training a generator and the discriminator until the end. The training end condition is that the preset iteration number is reached or the score of the generated sample stably exceeds a threshold value.
Step S4: a complete sample is constructed to generate the countermeasure network GAN2, and the GAN2 is trained using the data set R until a preset requirement is reached. The method specifically comprises the following steps:
step S41, constructing a complete surface sample generation countermeasure network GAN2, the abstract network structure is shown in fig. 4C, the example network structure is shown in fig. 5, and the construction details are as follows:
in step S411, in the generator G2 of GAN2, the sub-generator G1 'portion of the trained texture generation countermeasure network GAN1 is embedded as the sub-generator G1 of G2 (corresponding to output f1), and the G2' portion of the generator of the trained GAN2 network is embedded as the sub-generator G2 of G2 (corresponding to output f 2).
Step S412, the generator G2 adds a plurality of layers of amalgamation behind the G2 on the basis of the existing network layer &Raising resolution layer g3, where resolution raising network is denoted upg3In each layer of
Figure BDA0003546328300000081
G3 is followed by an RGB conversion layer to convert the feature map into an RGB image.
Step S413, embedding G1 into resolution enhancement layer up of G2g3Modifications of G1 and the manner in which G1 is embedded in G2 are described in detail below.
In step S414, in the discriminator D2 of GAN2, the network discriminator D2' of GAN2 is introduced as the sub-discriminator D2 of D2, and the reduced resolution network D3 is added on the basis of D2, and each layer of D3 is realized by convolutional layer. The per-layer network size of the deresolution layer in D2 is set according to the total number of parameters of the generator to balance the number of parameters of the generator G2 and the arbiter D2.
Loss function L of step S415, D2D2Is a historical average Wasserstein-gp loss function.
Figure BDA0003546328300000082
In the formula (4), the first and second groups,
Figure BDA0003546328300000083
for the saved L of the last iterationD2，LwgangpIs defined by the same formula (2), a2And a in the formula (2)1Define similarly, as History LD2Loss of power
Figure BDA0003546328300000084
The ratio of the active ingredients to the total amount of the active ingredients.
Step S42, the way to embed the G1 network in the back of G2 is:
in step S421, g2 network inputs the noise vector Z' and sets the length to Len (g 2)in) (ii) a G1 has connections in M layers in G2 resolution enhancement layer; g2 network input noise vector Z with length Len (G2) in)，Len(G2in)＝Len(g1in)+Len(g2in)。
Step S422, g1 network part modification: g1 part up1 input original fc 1' layer output fc1outThe vector resulting from the channel dimension reduction performed is of the shape [ Len (up 1)in)]The input is changed to fc1outM channel tensor fc1 obtained by dividing broadcast by M and foldingout′，fc1out' shape is [ M Len (up 1)in)]. This process may cause g1 to generate M sets of signatures of size f 1' at a time.
Step S423, splitting the input vector Z into correspondences g1inPart and corresponding g2inThe two partial sections were fed with g1 and g2, respectively. The g1 part outputs an M-channel characteristic diagram F1, and the g2 part outputs a characteristic diagram F2.
Step S424, the feature map F1 of one channel in the feature map F1iAdding a convolution splicing network layer Connect to the rear part of the characteristic diagramiPerforming tiling and convolution operations, and outputting the result as the output characteristic diagram fout of the layer before the target insertion position in G2iCharacteristic diagram f1 of same length and widthi′。
Step S425, put foutiAnd f 1'iThe connection becomes a new feature map fini+1Fin is toi+1Inputting the next resolution enhancement layer
Figure BDA0003546328300000091
Output fouti+1。
In step S426, for the insertion position i corresponding to each channel of the feature map F1 from 1 to M, the operations from step S422 to step S425 are performed to complete the insertion.
Step S43, GAN2 network training mode:
step S431, inputting control noise Z by a generator to generate a false sample P; inputting real samples in P and R into a discriminator to predict ToRF, and calculating a loss function L D2And performing back propagation and iterative training of the generator and the discriminator until the end.
And step S432, the training process is carried out in two stages of initial training and integral training.
a) In the initial training, the parameters of the layers such as g1, g2, d2 and the like of the original network are fixed, only the newly added layer is trained, and a gradual transition strategy is adopted at the connection part between the original network and the newly added network;
b) when initial network training reaches a specified number of iterations, an overall training phase is entered, in which all parameters of the network are made trainable, but the g1 network portion is set to a smaller learning rate.
Step S5: the extracted samples generate the generator portion G2 of the countermeasure network GAN2, and the input control vectors generate the surface samples.
The specific implementation example is as follows:
example parameter setting: the sample generation network target resolution, i.e., the resolution of samples in data set R is [512 x 512], the resolution of target samples in data set R1 is [64 x 64], the resolution of low resolution frame image samples in data set R2 is [64 x 64], the length of Z "is 255, the length of Z' is 256, the length of Z is 512, and M is 3.
Step 1: a data set R, R1, R2 was constructed.
The data set R is 10000 Mars sample images of [512 x 512] size, 500 samples of each terrain of 20 terrains are cut from R and are scaled to [64 x 64] and the contrast scaling is recorded as a sample set R1, and the fuzzy resolution reduction of each image of R types is obtained as a sample set R2.
An example of the terrain sample and R1 construction process in example R is shown in FIG. 6:
and 2, step: a texture generation confrontation network GAN1 network is constructed and trained using data set R1.
The generator G1 in the GAN1 is designed as: fc1 'layer is 3 layers, input Z' length is 255, S is 1, output length is768, change size to [3, 256]Size tensor, 3-channel convention [256 ]]Length vector up1inAnd then input up 1'. In the up 1' network, the data is first converted into [128 x 4] through a fully connected layer]Is represented by [128 x 4] 4]The feature map is subjected to resolution enhancement and convolution step by step to finally obtain [8 × 64 ]]The characteristic diagram of (1), namely the output characteristic diagram of each layer of network in up 1', from front to back is [128 × 4, 64 × 8, 32 × 16, 16 × 32, 8 × 64%]. The output f1' of up1 is converted over LrgbConversion of the layer to [3 x 64 ]]P1 ∈ P1.
The discriminator D1 in GAN1 is designed as: the input is a 3 x 64 image and the output is two scalars, TorF and S'. D1 consists of 5 convolutional deresolved layers and the last 3 fully-connected layers, with the output of each layer of the convolutional deresolved layers being [8 × 64, 16 × 32, 32 × 16, 64 × 8, 128 × 4] in sequence from front to back, the last fully-connected layer being the front layer output expanded into 2048 length vectors, and the output length of each fully-connected layer being 256, 256, 2.
The training process is as follows: loss function of LD1(ii) a The optimizer is an Adam optimizer; the training time ratio of the discriminator to the generator is 1: 1; the end condition is network convergence and the number of iterations reaches 5000.
And step 3: constructing surface samples generates a confrontation network low resolution partial gan2 network and is trained using data set R2.
generator g 2', L in gan2rgb2The design is as follows: the input noise Z' has a length of 256 and is converted into [512 x 4] through a layer of fully connected layers]The subsequent resolution enhancement layer constructs a generator similar to GAN1, the number of output channels with the same resolution of each layer of feature map is increased, and finally [64 × 64] is output]Characteristic diagram, meridian Lrgb2Conversion to [ 3X 64]]The sample image of (1).
The discriminator d 2' in gan2 is designed as: the shape of the input feature graph is [3 × 64], the input feature graph is converted into a [64 × 64] feature graph through one convolution layer, the structure of the subsequent resolution reduction layer is similar to a discriminator D1 of GAN1, the difference is that the number of output channels of each layer of the network is increased, and the final output is only true and false discrimination ToRF, namely the output length of the last layer of the full-connection layer is 1.
The training process is as follows: loss of powerFunction is LD2(ii) a The other configuration is the same as training GAN 1.
And 4, step 4: the constructed surface sample generation countermeasure network GAN2 is trained using the data set R.
The structure of the complete GAN2 is shown in fig. 3, which includes partial layers of GAN1 and GAN2, i.e., g1 corresponding to GAN1 network (including fc1, up1) and g2, d2 corresponding to GAN2 network.
Modification of g 1: output fc1 of fc1out' divide each channel by 3 and fold to shape [3 x 256]The multi-channel tensor input up1, no longer being as specified by the multi-channel rule in step two as [256 ]]A single channel vector of length, as shown in part g1 in fig. 3.
GAN2 adds four new sets of networks, including 3 sets of networks in the multi-layer fusion and resolution enhancement layer g3 of the generator part:
1)3 level resolution enhancement layer
Figure BDA0003546328300000111
Each output foutiThe shape is [ 32X 128 ]]，[16*256*256]，[16*512*512]。
2) 3-level convolution splicing layer Connect1，Connect2，Connect3. Each input is a characteristic diagram of one channel in F1, with the shape of [8 x 64]]Each Connect layer is tiled, repeated and convolved to output f 1'iThe shape is [ 32X 128 ]]，[16*256*256]，[16*512*512]。
3) And (3) converting the final characteristic diagram into an RGB conversion layer of an RGB image, wherein the output shape is [3 x 512 ].
And adding 1 group of networks in the discriminator part:
4) multi-level deresolved convolutional layer d 3. The structure of d3 is input as [3 x 512] signature graph and output as [8 x 64] signature graph (consistent with the rear d2 input).
And 5: the extracted samples generate network generator input control vector output samples.
And extracting a generator G2 in the GAN2, and inputting a control noise vector Z with the length of 512 to generate samples.
FIG. 7 is a schematic diagram of a data processing apparatus of the present invention. As shown in fig. 7, the embodiment of the present invention further provides a computer-readable storage medium and a data processing apparatus. The computer-readable storage medium of the present invention stores computer-executable instructions, which when executed by a processor of a data processing apparatus, implement the above-described earth surface image generation method based on generation of a countermeasure network. It will be understood by those skilled in the art that all or part of the steps of the above method may be implemented by instructing relevant hardware (e.g., processor, FPGA, ASIC, etc.) through a program, and the program may be stored in a readable storage medium, such as a read-only memory, a magnetic or optical disk, etc. All or some of the steps of the above embodiments may also be implemented using one or more integrated circuits. Accordingly, the modules in the above embodiments may be implemented in hardware, for example, by an integrated circuit, or in software, for example, by a processor executing programs/instructions stored in a memory. Embodiments of the invention are not limited to any specific form of hardware or software combination.
The method can stably generate the characteristics of the surface image samples under the condition that the surface samples have certain deflection, and increase the types of the surface samples which can be generated by the antagonistic generation network.
The above embodiments are only for illustrating the present invention and are not to be construed as limiting the present invention, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, therefore, all equivalent technical solutions also belong to the scope of the present invention, and the scope of the present invention should be defined by the claims.

Claims (10)

1. A surface image generation method based on generation of a countermeasure network is characterized by comprising the following steps:
step 1, acquiring an original earth surface image, and generating an original data set, a texture data set and a frame data set;
step 2, constructing a texture generation network, and training the texture generation network by using the texture data set;
step 3, constructing a frame generation network, and training the frame generation network by using the frame data set;
step 4, adding a generator of a frame generation network and a network layer of a discriminator, embedding the generator of the texture generation network into the generator of the frame generation network to obtain a surface image generation network, and training the surface image generation network by using the original data set;
And 5, generating the earth surface image by using the generator of the earth surface image generation network as an earth surface image generation model.
2. The earth surface image generation method according to claim 1, wherein the step 4 specifically includes:
step 41, embedding the generator G1' of the texture generation confrontation network GAN1 into the generator G2' of the framework generation confrontation network GAN2, and connecting a multilayer fusion and resolution raising layer G3 behind G2' to obtain the generator G2 of the surface image generation confrontation network GAN 2; adding a resolution reduction network D3 before a discriminator D2' of the framework generation countermeasure network GAN2 to obtain a discriminator D2 of GAN 2;
at step 42, GAN2 is trained with this original data set R for a specified number of iterations.
3. The earth's surface image generation method of claim 2, characterized in that the loss function of D2
Figure FDA0003546328290000011
Wherein, a2Is history LD2Loss of power
Figure FDA0003546328290000012
The proportion of the active ingredients is that,
Figure FDA0003546328290000013
for L of the last iterationD2，LwgangpIs Wasserstein-gp loss function.
4. The method of claim 1, wherein the texture generation network has a loss function L of a discriminator D1D1Comprises the following steps:
Figure FDA0003546328290000014
Figure FDA0003546328290000015
is LD1Current loss and LD1Historical value
Figure FDA0003546328290000016
Weighted new value, w is the weight occupied by the tag loss, L 2As a function of scale label loss, a1Is history LD1Loss of
Figure FDA0003546328290000017
The proportion of the active carbon is that,
Figure FDA0003546328290000018
for the last iteration
Figure FDA0003546328290000019
LwgangpIs Wasserstein-gp loss function.
5. A surface image generation system based on a generation countermeasure network, comprising:
the system comprises an original data acquisition module, a texture data acquisition module and a frame data acquisition module, wherein the original data acquisition module is used for acquiring an original earth surface image and generating an original data set, a texture data set and a frame data set;
the first model training module is used for constructing a texture generation network and training the texture generation network by the texture data set;
the second model training module is used for constructing a frame generation network and training the frame generation network by using the frame data set;
the third model training module is used for embedding the generator of the texture generation network into the generator of the frame generation network to obtain a ground surface image generation network, and training the ground surface image generation network by using the original data set;
and the earth surface image generation module is used for generating an earth surface image for the earth surface image generation model by using the generator of the earth surface image generation network.
6. The earth-surface image generation system of claim 5, wherein the third model training module specifically comprises:
The network generation module is used for generating the earth surface image generation network; the generator G1' of the texture generation confrontation network GAN1 is embedded into the generator G2' of the framework generation confrontation network GAN2, and a plurality of layers of fusion and resolution enhancement layers G3 are connected behind G2', so that the generator G2 of the ground surface image generation confrontation network GAN2 is obtained; adding a resolution reduction network D3 before a discriminator D2' of the framework generation countermeasure network GAN2 to obtain a discriminator D2 of GAN 2;
and the network training module is used for training the GAN2 for a specified iteration number by using the original data set R.
7. The earth-surface image generation system of claim 6, wherein the loss function of D2
Figure FDA0003546328290000021
Wherein, a2Is history LD2Loss of power
Figure FDA0003546328290000022
The proportion of the active ingredients is that,
Figure FDA0003546328290000023
for L of the last iterationD2，LwgangpIs Wasserstein-gp loss function.
8. The earth-surface image generation system of claim 5, wherein the loss function L of the discriminator D1 of the texture generation networkD1Comprises the following steps:
Figure FDA0003546328290000024
Figure FDA0003546328290000025
is LD1Current loss and LD1Historical value
Figure FDA0003546328290000026
Weighted new value, w is the weight occupied by the tag loss, L2As a function of scale label loss, a1Is history LD1Loss of power
Figure FDA0003546328290000027
The proportion of the active ingredients is that,
Figure FDA0003546328290000028
for the last iteration
Figure FDA0003546328290000029
LwgangpIs Wasserstein-gp loss function.
9. A computer-readable storage medium storing computer-executable instructions which, when executed, implement the earth surface image generation method based on generation of a countermeasure network according to any one of claims 1 to 4.
10. A data processing apparatus comprising the computer-readable storage medium of claim 9, wherein the processor of the data processing apparatus generates the earth's surface image when it retrieves and executes the computer-executable instructions in the computer-readable storage medium.
CN202210249374.1A 2022-03-14 2022-03-14 Earth surface image generation method and system based on generation countermeasure network Pending CN114758021A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210249374.1A CN114758021A (en) 2022-03-14 2022-03-14 Earth surface image generation method and system based on generation countermeasure network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210249374.1A CN114758021A (en) 2022-03-14 2022-03-14 Earth surface image generation method and system based on generation countermeasure network

Publications (1)

Publication Number Publication Date
CN114758021A true CN114758021A (en) 2022-07-15

Family

ID=82327984

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210249374.1A Pending CN114758021A (en) 2022-03-14 2022-03-14 Earth surface image generation method and system based on generation countermeasure network

Country Status (1)

Country Link
CN (1) CN114758021A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117574161A (en) * 2024-01-17 2024-02-20 航天宏图信息技术股份有限公司 Surface parameter estimation method, device and equipment based on generation of countermeasure network

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117574161A (en) * 2024-01-17 2024-02-20 航天宏图信息技术股份有限公司 Surface parameter estimation method, device and equipment based on generation of countermeasure network
CN117574161B (en) * 2024-01-17 2024-04-16 航天宏图信息技术股份有限公司 Surface parameter estimation method, device and equipment based on generation of countermeasure network

Similar Documents

Publication Publication Date Title
CN109859147B (en) Real image denoising method based on generation of antagonistic network noise modeling
CN111062872B (en) Image super-resolution reconstruction method and system based on edge detection
CN112614077B (en) Unsupervised low-illumination image enhancement method based on generation countermeasure network
CN112348743B (en) Image super-resolution method fusing discriminant network and generation network
CN110136062B (en) Super-resolution reconstruction method combining semantic segmentation
CN112288632B (en) Single image super-resolution method and system based on simplified ESRGAN
CN111986075B (en) Style migration method for target edge clarification
CN111696033B (en) Real image super-resolution model and method based on angular point guided cascade hourglass network structure learning
CN112614070B (en) defogNet-based single image defogging method
CN112598587B (en) Image processing system and method combining face mask removal and super-resolution
CN111160138A (en) Fast face exchange method based on convolutional neural network
CN114066747A (en) Low-illumination image enhancement method based on illumination and reflection complementarity
CN115546505A (en) Unsupervised monocular image depth estimation method based on deep learning
CN115511708A (en) Depth map super-resolution method and system based on uncertainty perception feature transmission
Zheng et al. T-net: Deep stacked scale-iteration network for image dehazing
CN114758021A (en) Earth surface image generation method and system based on generation countermeasure network
CN114067018B (en) Infrared image colorization method for generating countermeasure network based on expansion residual error
CN115187474A (en) Inference-based two-stage dense fog image defogging method
Liu et al. Facial image inpainting using multi-level generative network
CN113379606A (en) Face super-resolution method based on pre-training generation model
CN117151990A (en) Image defogging method based on self-attention coding and decoding
CN116597146A (en) Semantic segmentation method for laser radar sparse point cloud data
CN115100044A (en) Endoscope super-resolution method and system based on three-generator generation countermeasure network
CN109087247A (en) The method that a kind of pair of stereo-picture carries out oversubscription
CN117315735A (en) Face super-resolution reconstruction method based on priori information and attention mechanism

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination