CN112529774A - Remote sensing simulation image generation method based on cycleGAN - Google Patents
Remote sensing simulation image generation method based on cycleGAN Download PDFInfo
- Publication number
- CN112529774A CN112529774A CN202011572862.3A CN202011572862A CN112529774A CN 112529774 A CN112529774 A CN 112529774A CN 202011572862 A CN202011572862 A CN 202011572862A CN 112529774 A CN112529774 A CN 112529774A
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- image
- training
- game
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000004088 simulation Methods 0.000 title claims abstract description 25
- 238000012549 training Methods 0.000 claims abstract description 46
- 238000012360 testing method Methods 0.000 claims abstract description 22
- 230000011218 segmentation Effects 0.000 claims abstract description 11
- 230000000694 effects Effects 0.000 claims description 12
- 238000013508 migration Methods 0.000 claims description 8
- 230000005012 migration Effects 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 6
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims description 2
- 238000013135 deep learning Methods 0.000 abstract description 5
- 238000002372 labelling Methods 0.000 abstract description 2
- 238000013528 artificial neural network Methods 0.000 abstract 1
- 230000006978 adaptation Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The method for generating the remote sensing simulation image comprises the steps of firstly obtaining a game satellite image and a real remote sensing image, dividing the game satellite image and the real remote sensing image into a training set and a testing set after the game satellite image and the real remote sensing image are subjected to the same segmentation, then training a neural network for converting the game satellite image into the remote sensing image through a classical generation model CycleGAN algorithm in deep learning, and fixing generator parameters in the network after the training is finished, so that the game satellite image which is input randomly can be converted into the remote sensing simulation image. The method can be used for generating the simulation remote sensing image. Because a large number of labels are difficult to obtain from the real remote sensing data, the problem of insufficient data labeling in the remote sensing field can be effectively solved by using the method, and the generated simulation image can be further applied to other tasks, so that the method has wide market prospect and application value.
Description
Technical Field
A method for generating a remote sensing simulation image of CycleGAN belongs to the field of remote sensing image processing, and particularly relates to a remote sensing image simulation and image generation technology.
Background
The remote sensing image has the characteristics of high spectral resolution, multiple wave bands, strong spatial correlation and the like, and is widely applied to the fields of military affairs, geography, agriculture and the like. However, the labeling of remote sensing images is costly, time consuming and requires expert knowledge in the relevant field. To date, while there is already a large amount of telemetry data available for acquisition, the number of datasets with full pixel-level tags is still limited.
Aiming at the problem that pixel-level labels are lacked in the fields of remote sensing and computer vision, scholars at home and abroad carry out extensive research and provide a series of effective methods, such as a semi-supervised generation countermeasure network algorithm, a depth domain adaptation algorithm, a scene migration semantic segmentation algorithm and the like.
The semi-supervised variational generation is used for resisting a network algorithm, and an encoder-decoder structure is added on the basis to generate pseudo samples, so that a more robust classification effect can be achieved under the condition of less labels.
Depth Domain Adaptation (DAN) is an unsupervised domain adaptation method of depth domain adaptation proposed based on the Loss of maximum mean difference (MMD Loss) common in domain adaptation. Firstly, the parameters of the front three-layer network are fixed by using a pre-trained network, the parameters of the middle two-layer network are finely adjusted, and finally the three layers are trained by using MMD Loss. The method fully exerts the strong fitting effect of the deep convolutional neural network on the basis of keeping the distribution alignment of the source domain and the target domain, and improves the classification accuracy of the domain adaptation.
The scene migration semantic segmentation algorithm tries to perform a semantic segmentation training task by using the game scene image, so that a good effect is achieved, and the problem of insufficient tags in semantic segmentation is relieved to a great extent.
Disclosure of Invention
Technical problem to be solved
The invention provides a remote sensing simulation image generation method based on cycleGAN. The method is an improvement and an expansion of the traditional remote sensing simulation image generation method, and the game-style satellite image is converted into the display-style remote sensing image by using a cyclic generation countermeasure network (cycleGAN), so that the problem of difficulty in supervised learning tasks caused by insufficient pixel-level labels of the remote sensing image is solved.
(II) technical scheme
A remote sensing simulation image generation method based on CycleGAN is characterized by comprising the following specific steps:
step one, segmenting satellite images in a game, and dividing the satellite images into a training set and a testing set.
And step two, carrying out the same segmentation processing on the remote sensing data set in reality, and synchronously dividing the training set and the test set in the same proportion of the step one.
And step three, setting training network parameters including the number of rounds, the learning rate and the like.
And step four, inputting the training set data into a cycleGAN network for training, and testing the training effect by using the test set.
And step five, extracting the network parameters of the trained generator after the training is finished, and inputting the real remote sensing image into the network to generate the simulated remote sensing image with the converted style.
(III) advantageous effects
The remote sensing image simulation method adopts the loop generation countermeasure network in the deep learning to simulate the remote sensing image, fully utilizes data information contained in the image, can effectively generate a high-quality remote sensing simulation image, and achieves full mining and efficient utilization of data; the simulation image is generated by performing simulation processing on high-quality satellite images in the game, and a large amount of labeled game image data can be used as a data source of the method, so that the method has relatively high universality; the simulation image generated by the method can effectively solve the problem of insufficient data annotation in the remote sensing field, and the generated remote sensing simulation image can be further used for various common tasks of deep learning, such as semantic segmentation, target detection and the like; the method has the advantages that the mature CycleGAN algorithm in the field of computer vision is adopted to generate the remote sensing simulation image, reliable theoretical support and good actual effect are achieved, the convergence speed of the algorithm is high, and considerable operation efficiency is achieved.
Drawings
FIG. 1: a flow chart of a remote sensing simulation image generation method based on cycleGAN;
FIG. 2: the invention is an experimental diagram, wherein fig. 2a is a game image for training, and fig. 2b is a game image (left side) and an effect diagram (right side) after the game image is converted into a real remote sensing image.
[ Special description ] of: since fig. 2, which was submitted as a black and white picture, does not show the better contrast effect of the present invention, a color picture 2 was submitted in other documentations.
Detailed Description
For a better understanding of the technical aspects of the present invention, reference will now be made in detail to the embodiments of the present invention as illustrated in the accompanying drawings.
The invention relates to a remote sensing simulation image generation method based on cycleGAN, which mainly comprises the following steps:
1. segmenting satellite images in the game, and dividing the satellite images into a training set and a test set;
2. carrying out the same segmentation processing on the real remote sensing data set, and synchronously dividing the training set and the test set in the same proportion of one step;
3. setting training network parameters including the number of rounds, the learning rate and the like;
4. inputting training set data into a cycleGAN network for training, and testing a training effect by using a test set;
5. and after training is finished, extracting the network parameters of the trained generator, and inputting the real remote sensing image into the network, so that the simulated remote sensing image with the converted style can be generated.
The specific implementation flow of the invention is shown in fig. 1, and the specific implementation details of each part are as follows:
1. segmenting satellite images within a game into a training set and a test set
Taking the game GTA5 as an example, the satellite image in the game is a panoramic satellite remote sensing image of a virtual city "los saint city", which involves many scenes and ground features, and the training of deep learning requires to input a plurality of data at one time, so the image must be segmented. The method divides the image into 500 × 500 sub-images. The size of the segmented image can be adjusted according to the size of the original image and the computing power of the computer. And dividing the training set and the test set according to the general principle of deep learning training.
2. The same segmentation processing is carried out on the remote sensing data set in reality, and the training set and the test set are divided in the same proportion of one step synchronously
3. And setting the network parameters of training, including the number of rounds, the learning rate and the like. For the first 100 rounds of training, the learning rate was set to 2 x 10-4For the last 100 rounds of training, the learning rate was reduced to 2 x 10-6。
4. Inputting the training set data into a cycleGAN network for training, and testing the training effect by using the test set.
The loss function of this method is as follows:
Ltransfer(G，F，DX，DY)＝Lgan(G，DY，X，Y)
+Lgan(F，DX，Y，X)
+λLcycle(G，F).
the four losses are respectively: the method comprises the following steps of (1) resisting loss of migration of a game image domain X to a remote sensing image domain Y, resisting loss of migration of the remote sensing image domain Y to the game image domain X, cycle consistency loss and total loss of the game image domain and the remote sensing image domain, wherein lambda is a balance parameter between the resisting loss and the cycle consistency loss and has a value of 10.
5. After training is completed, the parameters of the generator G in the network are fixed. And inputting the game image into a network to generate a remote sensing simulation image after the migration.
The method can be applied to the generation of remote sensing simulation images, effectively solves the problem of insufficient labels of the remote sensing images, and the generated images can be further applied to various supervised computer vision methods, thereby having wide market prospect and application value.
Claims (5)
1. A remote sensing simulation image generation method based on CycleGAN is characterized by comprising the following steps:
(1) segmenting satellite images in the game, and dividing the satellite images into a training set and a test set;
(2) carrying out the same segmentation processing on the real remote sensing data set, and synchronously dividing the training set and the test set in the same proportion of one step;
(3) setting training network parameters including the number of rounds, the learning rate and the like;
(4) inputting training set data into a cycleGAN network for training, and testing a training effect by using a test set;
(5) and after training is finished, extracting the network parameters of the trained generator, and inputting the real remote sensing image into the network to generate the simulated remote sensing image with the converted style.
2. The method for generating remote sensing simulation images based on CycleGAN as claimed in claim 1, wherein the method comprises the following steps: and (1) and (2) respectively carrying out the same segmentation on the satellite image in the game and the remote sensing image in reality, and dividing the satellite image into a training set and a testing set. The method comprises the following steps: the images were each divided into 500 x 500 sub-images, allowing partial overlap between the images. The training set and the test set are divided according to the ratio of 8: 2 or 9: 1.
3. The method of claim 1, wherein the method comprises: and (3) setting the trained network parameters including the number of rounds, the learning rate and the like. The method comprises the following steps: set the number of rounds to 200 and for the first 100 rounds of training, the learning rate to 2 x 10-4For the last 100 rounds of training, the learning rate was set to 2 x 10-6。
4. The method of claim 1, wherein the method comprises: and (4) inputting the training set data into a cycleGAN network for training, and testing the training effect by using the test set. The method adopts a formula
Ltransfer(G，F，DX，DY)＝Lgan(G，DY，X，Y)
+Lgan(F，DX，Y，X)
+λLcycle(G，F)。
The four losses are respectively: the method comprises the following steps of (1) resisting loss of migration of a game image domain X to a remote sensing image domain Y, resisting loss of migration of the remote sensing image domain Y to the game image domain X, cyclic consistency loss and total loss of the game image domain and the remote sensing image domain, wherein lambda is a balance parameter between the resisting loss and the cyclic consistency loss, and the value of lambda is 10.
5. The method of claim 1, wherein the method comprises: and (5) after the training in the step (4) is completed, fixing the parameters of the generator G in the network. And inputting the game image into a network to generate a remote sensing simulation image after the migration.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011572862.3A CN112529774A (en) | 2020-12-28 | 2020-12-28 | Remote sensing simulation image generation method based on cycleGAN |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011572862.3A CN112529774A (en) | 2020-12-28 | 2020-12-28 | Remote sensing simulation image generation method based on cycleGAN |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112529774A true CN112529774A (en) | 2021-03-19 |
Family
ID=74976584
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011572862.3A Pending CN112529774A (en) | 2020-12-28 | 2020-12-28 | Remote sensing simulation image generation method based on cycleGAN |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112529774A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113723535A (en) * | 2021-09-02 | 2021-11-30 | 北京大学 | CycleGAN deep learning-based cell micronucleus image processing method and storage medium |
| CN114707402A (en) * | 2022-03-09 | 2022-07-05 | 中国石油大学(华东) | Method for converting curling simulation image into real image by reinforcement learning perception |
| CN116778335A (en) * | 2023-07-04 | 2023-09-19 | 中国科学院空天信息创新研究院 | Method and system for detecting collapsed building based on cross-domain teacher-student training |
-
2020
- 2020-12-28 CN CN202011572862.3A patent/CN112529774A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113723535A (en) * | 2021-09-02 | 2021-11-30 | 北京大学 | CycleGAN deep learning-based cell micronucleus image processing method and storage medium |
| CN114707402A (en) * | 2022-03-09 | 2022-07-05 | 中国石油大学(华东) | Method for converting curling simulation image into real image by reinforcement learning perception |
| CN116778335A (en) * | 2023-07-04 | 2023-09-19 | 中国科学院空天信息创新研究院 | Method and system for detecting collapsed building based on cross-domain teacher-student training |
| CN116778335B (en) * | 2023-07-04 | 2024-04-26 | 中国科学院空天信息创新研究院 | Method and system for detecting collapsed building based on cross-domain teacher-student training |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112465111B (en) | Three-dimensional voxel image segmentation method based on knowledge distillation and countermeasure training | |
| CN112529774A (en) | Remote sensing simulation image generation method based on cycleGAN | |
| Sheng et al. | Improving video temporal consistency via broad learning system | |
| CN109544442B (en) | Image local style migration method of double-countermeasure-based generation type countermeasure network | |
| CN111914617B (en) | Face attribute editing method based on balanced stack type generation type countermeasure network | |
| CN111833261A (en) | Image super-resolution restoration method for generating countermeasure network based on attention | |
| CN116682120A (en) | Multilingual mosaic image text recognition method based on deep learning | |
| CN111489405A (en) | Face sketch synthesis system for generating confrontation network based on condition enhancement | |
| CN115063655A (en) | Class activation mapping graph generation method fusing supercolumns | |
| CN112561782B (en) | Method for improving reality degree of simulation picture of offshore scene | |
| CN117173022A (en) | Remote sensing image super-resolution reconstruction method based on multipath fusion and attention | |
| Sun et al. | ESinGAN: Enhanced single-image GAN using pixel attention mechanism for image super-resolution | |
| CN116823647A (en) | Image complement method based on fast Fourier transform and selective attention mechanism | |
| Fan et al. | Image inpainting based on structural constraint and multi-scale feature fusion | |
| CN113537292B (en) | Multi-source domain adaptation method based on tensor high-order mutual attention mechanism | |
| Zhang et al. | New image processing: VGG image style transfer with gram matrix style features | |
| Pu et al. | Application of Image Style Transfer Based on Normalized Residual Network in Art Design | |
| Li et al. | FaceRefiner: High-Fidelity Facial Texture Refinement with Differentiable Rendering-based Style Transfer | |
| Ni et al. | Research on Visual Communication Page Design Theory of Multi-Source Information Fusion Using Deep Web Crawlers | |
| CN117932413B (en) | High generalization mechanical intelligent fault diagnosis method based on style migration under variable working conditions | |
| CN114881843B (en) | Fluid artistic control method based on deep learning | |
| CN114679605B (en) | Video transition method, device, computer equipment and storage medium | |
| Ji et al. | Research on Acceleration Methods of Semi-training Color Stripping DehazeNet | |
| Li et al. | Research on Remote Sensing Image Enhancement for Complex Environment Design Field | |
| Jin et al. | Unsupersived Image Texture Transfer Based On Generative Adversarial Network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |