CN110766797B - Three-dimensional map repairing method based on GAN - Google Patents
Three-dimensional map repairing method based on GAN Download PDFInfo
- Publication number
- CN110766797B CN110766797B CN201911045035.6A CN201911045035A CN110766797B CN 110766797 B CN110766797 B CN 110766797B CN 201911045035 A CN201911045035 A CN 201911045035A CN 110766797 B CN110766797 B CN 110766797B
- Authority
- CN
- China
- Prior art keywords
- color
- color image
- image
- repaired
- pseudo
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000008569 process Effects 0.000 claims abstract description 12
- 238000012549 training Methods 0.000 claims description 17
- 238000013528 artificial neural network Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 abstract description 7
- 238000013527 convolutional neural network Methods 0.000 abstract description 5
- 238000010606 normalization Methods 0.000 abstract description 4
- 238000013135 deep learning Methods 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 6
- 230000008439 repair process Effects 0.000 description 6
- 238000011160 research Methods 0.000 description 3
- 230000004888 barrier function Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Geometry (AREA)
- General Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Remote Sensing (AREA)
- Computer Graphics (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention relates to the technical field of deep learning, in particular to a three-dimensional map repairing method based on GAN. Firstly, carrying out scaling and normalization processing on an input original color image to be restored and a corresponding pseudo color disparity map to be restored to enable the size of the original color image to be restored to be H x W; secondly, inputting the color image to be repaired and the pseudo color parallax image to be repaired into a generation model; and thirdly, generating a model to finish the restoration of the color image to be restored and the pseudo color parallax image to be restored to obtain the restored color image and the restored pseudo color parallax image. According to the three-dimensional map repairing method based on the GAN, the generation model and the judgment model are both composed of the convolutional neural network, the convolutional neural network can well extract image characteristics, and the problem that the traditional method cannot process images without textures or with weak textures is solved.
Description
Technical Field
The invention relates to the technical field of deep learning, in particular to a three-dimensional map repairing method based on GAN.
Background
The high-precision map is one of the most core technologies of the unmanned technology, and directly determines the intelligence degree of the unmanned vehicle. The repair problem can be classified into: color image restoration, depth image restoration, and three-dimensional restoration. At present, color image restoration and depth image restoration are a hot research direction in the field of computer vision, but the restoration of a single color image or the restoration of a depth image cannot accurately carry out high-precision map three-dimensional construction. If a map is to be constructed with high accuracy, it needs to be repaired three-dimensionally. The three-dimensional restoration can be divided into direct three-dimensional point cloud restoration and collaborative restoration of a color image and a depth image (a parallax image or a pseudo-color parallax image) according to different restoration contents. The three-dimensional point cloud can directly express the space, but the three-dimensional point cloud is defined on an irregular non-Euclidean domain and has the characteristics of disorder, rotation and the like, so that the high-precision repair of the three-dimensional point cloud is difficult to realize. Another method of three-dimensional restoration is the cooperative restoration of color images and depth images. The repairing method considers that the dynamic barrier exists in both the color image and the depth image, and the color image and the depth image need to be repaired simultaneously when three-dimensional repairing is carried out. At present, researches on the aspects of the cooperative repair of the color image and the depth image are less, and the color image and the depth image are repaired by adopting a traditional method due to the data difference of the color image and the depth image. However, the conventional method has difficulty in processing images without texture or weak texture when processing color images and depth images. The convolutional neural network can well extract image features, but at present, the research of cooperative repair of a color image and a depth image (a disparity map or a pseudo-color disparity map) based on deep learning is not available.
Disclosure of Invention
In order to overcome the defects in the prior art, the invention provides the three-dimensional map repairing method based on the GAN, and the implementation process is simple and convenient.
In order to solve the technical problems, the invention adopts the technical scheme that: a three-dimensional map repairing method based on GAN, at first, carry on zoom and normalization process to the primitive color picture to be repaired and its correspondent pseudo-color disparity map to be repaired, make its size H x W, the output value of the picture is (-1, 1); secondly, inputting the color image to be repaired and the pseudo color parallax image to be repaired into a generation model; and thirdly, generating a model to finish the restoration of the color image to be restored and the pseudo color parallax image to be restored to obtain the restored color image and the restored pseudo color parallax image.
Further, the color image to be repaired and the pseudo color parallax image to be repaired are simultaneously transmitted into the generation model, and the generation model repairs the color image to be repaired and the pseudo color parallax image to be repaired.
The training device further comprises a discrimination model, wherein the discrimination model is used for discriminating the quality of the generated model, and if the discrimination result shows that the output result of the generated model is not good, the generated model continues to be trained.
Further, the generation model comprises a feature sharing module, a color image feature module and a pseudo-color parallax image feature module; the feature sharing module comprises 4 neural network layers, wherein the first layer is an input layer, and the other layers are convolutional layers; the data dimension of the first layer is H W6, the data dimensions of the second layer and the third layer are (H/2) W/2 256, and the data dimension of the fourth layer is (H/4) W/4 512; the color image feature module and the pseudo-color parallax map feature module respectively comprise 4 convolution layers, the data dimension of the first layer is (H/4) × (W/4) × 512, the data dimension of the second layer and the third layer is (H/2) × (W/2) × 256, and the data dimension of the fourth layer is H × W3.
Furthermore, the discrimination model comprises a feature sharing module, a color image feature module and a pseudo-color parallax image feature module; the characteristic sharing module comprises 3 neural network layers, wherein the first layer is an input layer, and the other layers are convolutional layers; the data dimension of the first layer is H W6, the data dimension of the second layer is (H/2) W/2 256, and the data dimension of the third layer is (H/4) W/4 512; the color image feature module and the pseudo-color parallax map feature module respectively comprise 3 convolution layers, the data dimension of the first layer and the second layer is (H/4) × (W/4) × 512, and the data dimension of the third layer is 1 × 1.
In the invention, the activation function of the generation model and the discrimination model is ReLU, the data processing batch is 64, the size of a convolution kernel is 3 x 3, the step size is 2, and the optimization function is Adam.
Further, the training process of the discriminant model includes the following steps:
s1, respectively inputting a true color image and a true and false color parallax image corresponding to the true color image, a color image generated by a generation model and a false color parallax image corresponding to the color image into a discrimination model D, wherein: discriminator loss function Ldis：
In the formula: x is a true color image or a true and false color image;
d (x) is a true color image or a true-false color image;
s2, training a discrimination model;
and S3, judging the authenticity of the output result of the judging model, namely the color image output result and the pseudo-color parallax image output result, through a sigmoid function.
Further, the training process of the generative model comprises the following steps:
s1, inputting a color image to be repaired and a corresponding pseudo color disparity map to be repaired into a generation model, wherein a generator loss function L of the color image or the pseudo color disparity map isgen(LgenThe distance between the true color image and the color image to be restored is expected as the judgment result L1, or the distance between the true color image and the false color parallax image to be restored is expected as the judgment result L1):
wherein, x is a true color image or a true color image;
d (x) is a true color image or a true-false color image;
s2, generating a model to finish the restoration of the color image to be restored and the pseudo color parallax image to be restored to obtain a restored color image and a restored pseudo color parallax image;
further, in the training process, a model and a discriminant model are generated for iterative training.
Compared with the prior art, the beneficial effects are:
1. in the three-dimensional map repairing method based on the GAN, a generation model and a discrimination model are both formed by a convolutional neural network, and the convolutional neural network can well extract image characteristics, so that the problem that the traditional method cannot process images without textures or weak textures is solved;
2. the three-dimensional map repairing method based on the GAN is an end-to-end learning method, and the implementation process is simple and convenient.
Drawings
FIG. 1 is a schematic flow diagram of the process of the present invention.
FIG. 2 is a schematic flow chart of the training method of the present invention.
Detailed Description
The drawings are for illustration purposes only and are not to be construed as limiting the invention; for the purpose of better illustrating the embodiments, certain features of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product; it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted. The positional relationships depicted in the drawings are for illustrative purposes only and are not to be construed as limiting the invention.
Example 1:
as shown in fig. 1 and 2, when implementing the GAN-based three-dimensional map repairing method, the following contents are included in part 3:
1) collecting and processing training data and test data;
step 1, calibrating a binocular camera;
step 3, calculating the parallax of the color image by using the data collected without the obstacle scene as a true value and the data collected with the obstacle scene as a false value through an optical flow algorithm, and further obtaining a pseudo-color parallax image;
step 5, carrying out panoramic segmentation on the color image through a Panoptic FPN neural network architecture, and extracting an obstacle region to be used as a region to be repaired;
and 6, setting 70% of data (color images and corresponding pseudo-color disparity maps) as training data, 20% of data as test data and 10% of data as verification data.
2) Generating model G and discriminating model D for training GAN
(1) The training steps of the discriminant model D are as follows:
step 1, respectively inputting a true color image and a true and false color disparity map corresponding to the true color image, a color image generated by a generation model G and a false color disparity map corresponding to the color image into a discrimination model D;
and 3, judging the authenticity of the output result of the judgment model D through the sigmoid function.
(2) The training steps for generating the model are as follows:
step 1, inputting a color image to be repaired and a corresponding pseudo color disparity map to be repaired into a generation model G;
3) generating model G based on GAN to realize three-dimensional map repair
Step 1, carrying out scaling and normalization processing on an input original color image to be restored and a corresponding pseudo color disparity map to be restored to enable the size of the original color image to be H x W and the image output value to be (-1, 1);
and 3, generating a model G to finish the restoration of the color image to be restored and the pseudo color parallax image to be restored, and obtaining the restored color image and the restored pseudo color parallax image.
It should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.
Claims (6)
1. A three-dimensional map repairing method based on GAN is characterized in that firstly, scaling and normalizing an original color image to be repaired and a corresponding pseudo color parallax image to be repaired in an input three-dimensional map to make the original color image to be repaired and the pseudo color parallax image to be repaired equal to H x W; secondly, simultaneously transmitting the color image to be repaired and the pseudo color parallax image to be repaired into a generation model, and simultaneously repairing the generated model; thirdly, generating a model to finish the restoration of the color image to be restored and the pseudo color parallax image to be restored to obtain a restored color image and a restored pseudo color parallax image;
the training process of the generative model comprises the following steps:
s21, inputting the color image to be repaired and the corresponding pseudo color disparity map to be repaired into a generation model, wherein a generator loss function L of the color image or the pseudo color disparity map is used for generating a loss function Lgen，LgenThe expectation of the distance between the true color image and the color image discrimination result L1 to be repaired or the expectation of the distance between the true-false color image and the false color parallax image discrimination result L1 to be repaired is as follows:
wherein, x is a true color image or a true color image;
d (x) is the true value of the true color image or the true and false color image;
and S22, generating a model to finish the restoration of the color image to be restored and the pseudo color parallax image to be restored to obtain the restored color image and the restored pseudo color parallax image.
2. The method of repairing a three-dimensional map based on GAN as claimed in claim 1, further comprising a discriminant model for determining whether the generated model is good or bad, wherein if the result of the determination shows that the output result of the generated model is not good, the generated model continues to be trained.
3. The GAN-based three-dimensional map repairing method according to claim 2, wherein the generated model comprises a feature sharing module, a color image feature module and a pseudo color parallax image feature module; the feature sharing module comprises 4 neural network layers, wherein the first layer is an input layer, and the other layers are convolutional layers; the data dimension of the first layer is H W6, the data dimensions of the second layer and the third layer are (H/2) W/2 256, and the data dimension of the fourth layer is (H/4) W/4 512; the color image feature module and the pseudo-color parallax map feature module respectively comprise 4 convolution layers, the data dimension of the first layer is (H/4) × (W/4) × 512, the data dimension of the second layer and the third layer is (H/2) × (W/2) × 256, and the data dimension of the fourth layer is H × W3.
4. The GAN-based three-dimensional map repairing method according to claim 2, wherein the discriminant model comprises a feature sharing module, a color image feature module and a pseudo-color disparity map feature module; the characteristic sharing module comprises 3 neural network layers, wherein the first layer is an input layer, and the other layers are convolutional layers; the data dimension of the first layer is H W6, the data dimension of the second layer is (H/2) W/2 256, and the data dimension of the third layer is (H/4) W/4 512; the color image feature module and the pseudo-color parallax map feature module respectively comprise 3 convolution layers, the data dimension of the first layer and the second layer is (H/4) × (W/4) × 512, and the data dimension of the third layer is 1 × 1.
5. The GAN-based three-dimensional map inpainting method according to any one of claims 2 to 4, wherein the discriminant model training process comprises the steps of:
s11, inputting the true color image and the true and false color parallax map corresponding to the true color image into a discrimination model D, wherein: discriminator loss function LdisThe expectation of the distance between the true color image and the color image discrimination result L1 to be repaired or the expectation of the distance between the true-false color image and the false color parallax image discrimination result L1 to be repaired is as follows:
in the formula: x is a true color image or a true and false color image;
d (x) is the true value of the true color image or the true and false color image;
s12, training a discrimination model;
and S13, judging the authenticity of the output result of the judging model, namely the color image output result and the pseudo-color parallax image output result, through a sigmoid function.
6. The GAN-based three-dimensional map inpainting method according to claim 5, wherein the generation model and the discriminant model are iteratively trained during the training process.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911045035.6A CN110766797B (en) | 2019-10-30 | 2019-10-30 | Three-dimensional map repairing method based on GAN |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911045035.6A CN110766797B (en) | 2019-10-30 | 2019-10-30 | Three-dimensional map repairing method based on GAN |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110766797A CN110766797A (en) | 2020-02-07 |
| CN110766797B true CN110766797B (en) | 2021-08-13 |
Family
ID=69333046
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911045035.6A Active CN110766797B (en) | 2019-10-30 | 2019-10-30 | Three-dimensional map repairing method based on GAN |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110766797B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106600583A (en) * | 2016-12-07 | 2017-04-26 | 西安电子科技大学 | Disparity map acquiring method based on end-to-end neural network |
| CN109191402A (en) * | 2018-09-03 | 2019-01-11 | 武汉大学 | The image repair method and system of neural network are generated based on confrontation |
| CN109801230A (en) * | 2018-12-21 | 2019-05-24 | 河海大学 | A kind of image repair method based on new encoder structure |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107818554B (en) * | 2016-09-12 | 2023-04-21 | 索尼公司 | Information processing apparatus and information processing method |
-
2019
- 2019-10-30 CN CN201911045035.6A patent/CN110766797B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106600583A (en) * | 2016-12-07 | 2017-04-26 | 西安电子科技大学 | Disparity map acquiring method based on end-to-end neural network |
| CN109191402A (en) * | 2018-09-03 | 2019-01-11 | 武汉大学 | The image repair method and system of neural network are generated based on confrontation |
| CN109801230A (en) * | 2018-12-21 | 2019-05-24 | 河海大学 | A kind of image repair method based on new encoder structure |
Non-Patent Citations (1)
| Title |
|---|
| 基于深度学习的视觉三维重建研究;刘立强;《中国优秀硕士学位论文数据库》;20190730(第7期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110766797A (en) | 2020-02-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Uittenbogaard et al. | Privacy protection in street-view panoramas using depth and multi-view imagery | |
| Vaudrey et al. | Differences between stereo and motion behaviour on synthetic and real-world stereo sequences | |
| CN112085728B (en) | Submarine pipeline and leakage point detection method | |
| CN110321877B (en) | Three-eye rearview mirror and three-eye vision safe driving method and system | |
| CN113159043B (en) | Feature point matching method and system based on semantic information | |
| CN109509156B (en) | Image defogging processing method based on generation countermeasure model | |
| CN113052066B (en) | Multi-mode fusion method based on multi-view and image segmentation in three-dimensional target detection | |
| CN112734915A (en) | Multi-view stereoscopic vision three-dimensional scene reconstruction method based on deep learning | |
| CN117036641A (en) | Road scene three-dimensional reconstruction and defect detection method based on binocular vision | |
| Zhang et al. | RangeLVDet: Boosting 3D object detection in LiDAR with range image and RGB image | |
| CN113538569A (en) | Weak texture object pose estimation method and system | |
| CN110688976A (en) | Store comparison method based on image identification | |
| CN111861935B (en) | Rain removing method based on image restoration technology | |
| CN110766797B (en) | Three-dimensional map repairing method based on GAN | |
| CN116229426B (en) | Unmanned parking space detection method based on panoramic all-around image | |
| Nouduri et al. | Deep realistic novel view generation for city-scale aerial images | |
| CN112001954A (en) | Polar curve constraint-based underwater PCA-SIFT image matching method | |
| Huang et al. | FFNet: A simple image dedusting network with feature fusion | |
| CN115294371B (en) | Complementary feature reliable description and matching method based on deep learning | |
| TWI805282B (en) | Methods and apparatuses of depth estimation from focus information | |
| CN111583131B (en) | Defogging method based on binocular image | |
| CN115330935A (en) | Three-dimensional reconstruction method and system based on deep learning | |
| CN113870162A (en) | Low-light image enhancement method integrating illumination and reflection | |
| CN111951319A (en) | Image stereo matching method | |
| Babu et al. | Development and performance evaluation of enhanced image dehazing method using deep learning networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |