CN111797836A - Extraterrestrial celestial body patrolling device obstacle segmentation method based on deep learning - Google Patents
Extraterrestrial celestial body patrolling device obstacle segmentation method based on deep learning Download PDFInfo
- Publication number
- CN111797836A CN111797836A CN202010562735.9A CN202010562735A CN111797836A CN 111797836 A CN111797836 A CN 111797836A CN 202010562735 A CN202010562735 A CN 202010562735A CN 111797836 A CN111797836 A CN 111797836A
- Authority
- CN
- China
- Prior art keywords
- convolutional
- conv2d
- convolutional layer
- cond2d
- extraterrestrial celestial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 230000011218 segmentation Effects 0.000 title claims abstract description 49
- 238000013135 deep learning Methods 0.000 title claims abstract description 24
- 238000012549 training Methods 0.000 claims abstract description 49
- 238000013528 artificial neural network Methods 0.000 claims abstract description 34
- 238000012360 testing method Methods 0.000 claims abstract description 31
- 238000012795 verification Methods 0.000 claims abstract description 14
- 238000002372 labelling Methods 0.000 claims abstract description 11
- 238000011176 pooling Methods 0.000 claims description 19
- 230000006870 function Effects 0.000 claims description 17
- 241001061260 Emmelichthys struhsakeri Species 0.000 claims description 13
- 238000010586 diagram Methods 0.000 claims description 13
- 238000004422 calculation algorithm Methods 0.000 claims description 8
- 238000010200 validation analysis Methods 0.000 claims description 4
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000007689 inspection Methods 0.000 abstract description 13
- 238000001514 detection method Methods 0.000 description 26
- 239000011435 rock Substances 0.000 description 23
- 238000004590 computer program Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000003709 image segmentation Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 241000651994 Curio Species 0.000 description 1
- 238000012952 Resampling Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000000084 colloidal system Substances 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012502 risk assessment Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004576 sand Substances 0.000 description 1
- 239000002689 soil Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention provides a method for segmenting obstacles of an extraterrestrial celestial body inspection device based on deep learning, which comprises the following steps: forming a sample set by using an image set obtained by an extraterrestrial celestial body patrolling device and a corresponding artificial labeling image set; dividing samples in the sample set into training samples, verification samples and test samples; constructing a neural network based on the VGGNet convolutional network and the U-Net network; inputting the training sample and the verification sample into the neural network, and training the neural network to obtain a training model; and testing the test sample by using the training model to obtain an obstacle segmentation result of the extraterrestrial celestial body. The method locally migrates the VGG network with high precision to the network, improves the precision of the obstacle segmentation, improves the segmentation speed, and can meet the real-time requirement of the extraterrestrial celestial body inspection device.
Description
Technical Field
The invention relates to a method for dividing obstacles of an extraterrestrial celestial body inspection device based on deep learning, which is suitable for detecting and identifying obstacles of the inspection device for performing detection tasks on the surface of the extraterrestrial celestial body and can also be used for detecting and identifying obstacles of field robots.
Background
Extraterrestrial celestial body patroller exploration has become an important element of deep space exploration. Since extraterrestrial celestial bodies have the characteristics of unknown working environment, unstructured working environment, long distance and the like, a series of problems of autonomous environment perception, path planning and the like of the patrol device need to be mainly solved. Whether the patrol device can correctly judge the barrier is the key for the patrol device to smoothly run on the surface of the extraterrestrial celestial body and is the basic guarantee of scientific detection tasks.
In the current inspection device obstacle detection, the traditional method mainly establishes a three-dimensional topographic map through technologies such as stereoscopic vision, laser radar or structured light and the like, and provides powerful support for judging obstacles and navigating the inspection device. For example, Yu et al (Yu, H., Zhu, J., Wang, Y., Obstacle classification and 3D measurement in unstructured settings based on ToF cameras, 14 (6): 10753 and 10782, 2014) propose to classify obstacles using a multi-correlation vector machine (RVM) classifier on the basis that 2D and 3D images can be obtained simultaneously by a ToF depth camera, however, ToF camera measurement distances are short. Kostavelis et al (Kostavelis, I., Nalpentidis, L., Gasteratos, A., colloid risk assessment for autonomus robots by of Flinetraversability searching, Robotics and Autonomus Systems, 60 (11): 1367-. Bellone et al (Bellone, m., Reina, g., Giannoccaro, n.i., et al., unknown point location for terrestrial analysis in mobile robot applications, international journal of Advanced Robotics Systems, 10: 1-10, 2013) propose to describe obstacles in the detection environment using uneven points according to the accurate dense three-dimensional point cloud obtained by the image sensor, and the method also has the problem of low accuracy of medium-and long-distance terrain three-dimensional reconstruction by using an RGB-D depth camera. Dawn et al (a method for detecting obstacles based on a monocular camera and active structured light, patent of the invention, 201410829101.X) propose a method for detecting obstacles of a patrol instrument based on active structured light, in which a backup camera can still detect obstacles even if a reference camera fails, but the method needs to be equipped with a structured light sensor.
Because the obstacle on the surface of the extraterrestrial celestial body is mainly rock, even if a flat area also has a large amount of rocks, the rock has higher research value, and therefore, rock detection not only can realize obstacle avoidance of the inspection tour device, but also can lock a scientific target. Many scholars have proposed rock detection algorithms based on a single image. For example, the edge profile-based rock detection algorithm (Burl, M.C., Thompson, D.R., et al, Rockster: on board rock segmentation method for rock clustering. journal of Aero space Information Systems, 13: 329-342, 2016) can detect rock edges with obvious brightness difference in background, but due to the influence of sand dust, sun highlight and the like, the contrast of the planet surface rock and soil is weak, so that the edge-based rock detection algorithm can detect only a part of the rock in most cases, and often has a large amount of missing detection and false detection and poor accuracy of rock detection. Shadow-based rock detection methods (Guick, V.C., Morris, R.L., et al., Automous image analysis during the1999Marsokhod river field test. journal of geographic Research-plants, 106: 7745-. The region-based rock detection algorithm combines pixel points with the same or similar characteristics into one region (Dunlop, h., Thompson, d.r., Wettergreen, d., Multi-scale features for detection and segmentation of rocks in Mars images. ieee Conference CVPR, 2007) through region splitting and growing, and belongs to an image segmentation technology, however, such an algorithm is poor in robustness and limited in segmentation accuracy.
With the development and rapid application of neural networks, researchers have developed obstacle detection based on machine learning as well as deep learning. Ono et al (Ono, M., fuels, T.J., Steffy, A., et al., Risk-aware planar operation: Autonomous terrainin classification and path planning, Proceeding of IEEE Aerospace Conference, 1-10, 2015) propose a classification method based on machine learning and visual features for the Haoqixin Mars train, with the disadvantage of requiring a large amount of labeled data. Hadsell et al (Hadsell, r., serman, p., Ben, j., et al, Learning-range vision for autonomous off-road driving. journal of Field Robotics, 26 (2): 120-144, 2009) propose convolution encoders for mobile robot vision on off-road, adaptively extracting terrain features through off-line training, which requires collecting a large number of Field scenes as samples and spending a long training time.
The inventor of the present application finds, in the process of implementing the present invention, that the above-mentioned solutions of the prior art have the above-mentioned series of drawbacks 1) on the one hand, the obstacle detection method based on stereoscopic three-dimensional reconstruction has the problem of low accuracy of medium-and long-distance terrain three-dimensional reconstruction; on the other hand, for three-dimensional reconstruction techniques such as structured light and laser radar, it is necessary to mount external devices and apparatuses. 2) The rock detection methods based on the edge contour, shadow, region segmentation and the like of the single image often have the problems of missing detection, false detection, poor robustness, poor segmentation precision and the like. 3) At present, robot obstacle detection based on machine learning and deep learning needs a large amount of collected data as samples, and the collection of a large amount of data as samples for a deep space exploration robot is difficult.
Disclosure of Invention
The embodiment of the invention aims to provide a method for segmenting an obstacle of an extraterrestrial celestial body patrolling device based on deep learning, which is used for solving the problems of missing detection and false detection, poor robustness, low segmentation precision and the like in the prior art.
The invention provides a method for segmenting obstacles of a celestial body inspection tour device outside the ground based on deep learning, which comprises the following steps: s1, forming a sample set by using an image set obtained by the extraterrestrial celestial body patrolling device and a corresponding manual labeling image set; s2, dividing the samples in the sample set into training samples, verification samples and test samples; s3, constructing a neural network based on the VGGNet convolution network and the U-Net network; s4, inputting the training sample and the verification sample into the neural network, and training the neural network to obtain a training model; and S5, testing the test sample by using the training model to obtain the obstacle segmentation result of the extraterrestrial celestial body.
Preferably, the step S1 includes: and manually labeling the image set obtained by the extraterrestrial celestial body patrolling device to obtain a manually labeled image set.
Preferably, the step S2 includes: samples in the sample set were randomly divided into training samples, validation samples, and test samples in proportions of 80%, 10%, and 10%.
Preferably, in step S3, the neural network is constructed by: s301, in an encoder of the neural network, all layers of block1, block2, block3 and block4 in a VGG16 network structure and convolutional layers of block5 are utilized, and all the layers are set to be trainable; s302, in a decoder of the neural network, restoring the characteristic diagram output by each layer in the encoder.
Preferably, in step S301, the encoder is constructed by: the block1 comprises 2 convolutional layer blocks 1_ conv1, block1_ conv2 and 1 pooling layer block1_ pool, the number of channels of each convolutional layer is 64, and the size of a convolutional core is 3 multiplied by 3; the block2 comprises 2 convolutional layer blocks 2_ conv1, block2_ conv2 and 1 pooling layer block2_ pool, the number of channels of each convolutional layer is 128, and the size of a convolutional core is 3 multiplied by 3; the block3 comprises 3 convolutional layer blocks 3_ conv1, block3_ conv2, block3_ conv3 and 1 pooling layer block3_ pool, the number of each convolutional layer channel is 256, and the size of a convolutional core is 3 multiplied by 3; the block4 comprises 3 convolutional layer blocks 4_ conv1, block4_ conv2, block4_ conv3 and 1 pooling layer block4_ pool, the number of each convolutional layer channel is 512, and the size of a convolutional core is 3 multiplied by 3; the block5 comprises 3 convolutional layer blocks 5_ conv1, 5_ conv2 and 5_ conv3, the number of channels of each convolutional layer is 512, and the size of a convolutional core is 3 multiplied by 3; all pooling layers employ maximum pooling.
Preferably, in step S302, the decoder is constructed by: 2 convolution layers conv2d-1, conv2d-2 and 1 dropout layer dropout-1 are closely arranged behind block5_ conv3, the number of channels of each convolution layer is 1024, and the size of a convolution kernel is 3 multiplied by 3; amplifying the output of dropout-1 by 1 time through an UpSampling function UpSampling2D, then connecting 1 convolutional layer conv2d-3 with the channel number of 512 and the convolution kernel size of 3 × 3, then connecting a feature map output by an encoder block4_ conv3 with a feature map output by conv2d-3 by using a Concatenate mode, and then immediately connecting 3 convolutional layers cond2d-4, cond2d-5 and cond2d-6, wherein the channel number of each convolutional layer is 512 and the convolution kernel size is 3 × 3; the output of cond2d-6 is amplified by 1 time through an UpSampling function UpSampling2D, then 1 convolutional layer conv2d-7 with the channel number of 256 and the convolution kernel size of 3 × 3 is carried out, then the feature map output by the encoder block3_ conv3 is connected with the feature map output by conv2d-7 by using a Concatenate mode, and then 3 convolutional layers cond2d-8, cond2d-9 and cond2d-10 are carried out, the channel number of each convolutional layer is 256, and the convolution kernel size is 3 × 3; the output of cond2d-10 is amplified by 1 time through an UpSampling function UpSampling2D, then 1 convolutional layer conv2d-11 with the channel number of 128 and the convolution kernel size of 3 × 3 is carried out, then the feature map output by the encoder block2_ conv2 is connected with the feature map output by conv2d-11 by using a Concatenate mode, and then 3 convolutional layers cond2d-12, cond2d-13 and cond2d-14 are carried out, the channel number of each convolutional layer is 128, and the convolution kernel size is 3 × 3; the output of cond2d-14 is amplified by 1 time through an UpSampling function UpSampling2D, then 1 convolutional layer conv2d-15 with the channel number of 64 and the convolution kernel size of 3 × 3 is carried out, then the feature map output by the encoder block1_ conv2 is connected with the feature map output by conv2d-15 by using a Concatenate mode, and then 3 convolutional layers conv2d-16, conv2d-17 and conv2d-18 are carried out after connection, the channel number of each convolutional layer is 64, and the convolution kernel size is 3 × 3; the decoder section finally outputs the feature map output by conv2d-18, followed by convolutional layer conv2d-19 with channel number 1 and convolutional kernel size 1 × 1, and the segmented image.
Preferably, in the step S4: and training the neural network by adopting an Adam optimization algorithm.
In another aspect, the present invention further provides a machine-readable storage medium, where instructions are stored on the machine-readable storage medium, and the instructions are used for causing a machine to execute the above method for segmenting the obstacle of the extraterrestrial celestial body patrolling device based on deep learning.
The invention also provides a processor for running a program, wherein the program is run to execute the extraterrestrial celestial body patrolling device obstacle segmentation method based on deep learning.
Compared with the prior art, the invention has the beneficial effects that:
(1) according to the invention, the VGG network with high precision is locally migrated to the network, so that the obstacle segmentation precision is improved, the segmentation speed is also improved, and the real-time requirement of the extraterrestrial celestial body inspection device can be met.
(2) The invention provides a method based on a U-shaped segmentation network, which can realize high-precision segmentation on a very small sample, has high training speed and high precision, is also suitable for middle-distance and long-distance obstacle segmentation detection, and is suitable for obstacle detection and application of an extraterrestrial astronomical inspection tour device based on a monocular camera.
(3) The obstacle segmentation method based on deep learning is adopted, so that the shadow of the rock on the surface of the extraterrestrial celestial body can be removed simultaneously, and the problem that the shadow is difficult to remove automatically by a tour device is solved.
Additional features and advantages of embodiments of the invention will be set forth in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the embodiments of the invention without limiting the embodiments of the invention. In the drawings:
FIG. 1 is a flow chart of a method of extraterrestrial celestial rover obstacle segmentation in accordance with the present invention;
FIG. 2 is a diagram of a extraterrestrial celestial rover obstacle segmentation network in accordance with a preferred embodiment of the present invention;
FIG. 3 is 96 training samples in accordance with a preferred embodiment;
FIGS. 4 a-4 e are five indicators corresponding to 12 test samples, respectively, in accordance with a preferred embodiment;
fig. 5 a-51 are segmentation results of a method for segmenting extraterrestrial celestial rover obstacles according to a preferred embodiment.
Detailed Description
The following detailed description of embodiments of the invention refers to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating embodiments of the invention, are given by way of illustration and explanation only, not limitation.
The invention provides a method for dividing obstacles of an extraterrestrial celestial body inspection tour based on deep learning, which comprises the following steps of:
s1, forming a sample set by using an image set obtained by the extraterrestrial celestial body patrolling device and a corresponding manual labeling image set;
s2, dividing the samples in the sample set into training samples, verification samples and test samples;
s3, constructing a neural network based on the VGGNet convolution network and the U-Net network;
s4, inputting the training sample and the verification sample into the neural network, and training the neural network to obtain a training model;
and S5, testing the test sample by using the training model to obtain the obstacle segmentation result of the extraterrestrial celestial body.
In step S1, according to a preferred embodiment, the manual labeling atlas may be obtained by manually labeling the image set obtained by the extraterrestrial celestial rover. For example, 120 images shot by a curio train may be used, an original image with a size of 560 × 500 pixels is resampled into an image with a size of 512 × 512 pixels, the 120 resampled images may be manually labeled by using label (an image labeling software), and then a sample set is formed by using the image set and a corresponding manual labeling set.
In step S2, according to a preferred embodiment, the samples in the sample set can be randomly divided into training samples, verification samples and test samples in the proportions of 80%, 10% and 10%. For the embodiment of the invention, 96 training samples, 12 verification samples and 12 test samples can be obtained
In step S3, the U-Net network is a full Convolutional network, which has a structure similar to U-type and is called U-Net (see: Ronneberger, O.A., Fischer, P.A., Brox, T.A., U-Net: volumetric network for biological image segmentation. Pro.of IEEE connection. on CVPR, 3431-3440, 2015). Compared with other neural networks, the U-Net neural network needs less training sets and high segmentation precision, and is widely applied to medical image segmentation. And the VGGNet convolution network studied by the Oxford university visual geometry team obtains the army of the ILSVRC 2014 game and the champion of the positioning project, and the error rate on top5 is 7.5%. The VGGNet convolutional network successfully constructs a convolutional neural network with 16-19 layers of depth by repeatedly stacking a 3X 3 small convolutional kernel and a 2X 2 maximum pooling layer.
The neural network constructed by the invention inherits the architecture of the semantic segmentation network U-Net and is divided into an encoder and a decoder.
According to a preferred embodiment, the neural network is constructed in said step S3 by:
s301, in an encoder of the neural network, all layers of block1, block2, block3 and block4 in a VGG16 network structure and convolution layers of block5 are utilized, and all the layers are set to be trainable, namely, a variable trackable attribute is set to true;
and S302, in a decoder of the neural network, restoring the characteristic diagram output by each layer in the encoder through an up-sampling function.
Fig. 2 is a diagram of an extraterrestrial celestial rover obstacle segmentation network according to a preferred embodiment of the present invention. According to the preferred embodiment shown in fig. 2, in step S301, the encoder may be constructed by:
the block1 comprises 2 convolutional layer blocks 1_ conv1, block1_ conv2 and 1 pooling layer block1_ pool, the number of channels of each convolutional layer is 64, and the size of a convolutional core is 3 multiplied by 3; the block2 comprises 2 convolutional layer blocks 2_ conv1, block2_ conv2 and 1 pooling layer block2_ pool, the number of channels of each convolutional layer is 128, and the size of a convolutional core is 3 multiplied by 3; the block3 comprises 3 convolutional layer blocks 3_ conv1, block3_ conv2, block3_ conv3 and 1 pooling layer block3_ pool, the number of each convolutional layer channel is 256, and the size of a convolutional core is 3 multiplied by 3; the block4 comprises 3 convolutional layer blocks 4_ conv1, block4_ conv2, block4_ conv3 and 1 pooling layer block4_ pool, the number of each convolutional layer channel is 512, and the size of a convolutional core is 3 multiplied by 3; the block5 comprises 3 convolutional layer blocks 5_ conv1, 5_ conv2 and 5_ conv3, the number of channels of each convolutional layer is 512, and the size of a convolutional core is 3 multiplied by 3; all pooling layers employ maximum pooling.
In step S302, a decoder may be constructed by:
2 convolution layers conv2d-1, conv2d-2 and 1 dropout layer dropout-1 are closely arranged behind block5_ conv3, the number of channels of each convolution layer is 1024, and the size of a convolution kernel is 3 multiplied by 3;
amplifying the output of dropout-1 by 1 time through an UpSampling function UpSampling2D, then connecting 1 convolutional layer conv2d-3 with the channel number of 512 and the convolution kernel size of 3 × 3, then connecting a feature map output by an encoder block4_ conv3 with a feature map output by conv2d-3 by using a Concatenate mode, and then immediately connecting 3 convolutional layers cond2d-4, cond2d-5 and cond2d-6, wherein the channel number of each convolutional layer is 512 and the convolution kernel size is 3 × 3;
the output of cond2d-6 is amplified by 1 time through an UpSampling function UpSampling2D, then 1 convolutional layer conv2d-7 with the channel number of 256 and the convolution kernel size of 3 × 3 is carried out, then the feature map output by the encoder block3_ conv3 is connected with the feature map output by conv2d-7 by using a Concatenate mode, and then 3 convolutional layers cond2d-8, cond2d-9 and cond2d-10 are carried out, the channel number of each convolutional layer is 256, and the convolution kernel size is 3 × 3;
the output of cond2d-10 is amplified by 1 time through an UpSampling function UpSampling2D, then 1 convolutional layer conv2d-11 with the channel number of 128 and the convolution kernel size of 3 × 3 is carried out, then the feature map output by the encoder block2_ conv2 is connected with the feature map output by conv2d-11 by using a Concatenate mode, and then 3 convolutional layers cond2d-12, cond2d-13 and cond2d-14 are carried out, the channel number of each convolutional layer is 128, and the convolution kernel size is 3 × 3;
the output of cond2d-14 is amplified by 1 time through an UpSampling function UpSampling2D, then 1 convolutional layer conv2d-15 with the channel number of 64 and the convolution kernel size of 3 × 3 is carried out, then the characteristic diagram output by the encoder block1_ conv2 and the characteristic diagram output by conv2d-15 are connected in a Concatenate (series connection) mode, 3 convolutional layers conv2d-16, conv2d-17 and conv2d-18 are carried out after connection, the channel number of each convolutional layer is 64, and the convolution kernel size is 3 × 3; the decoder section finally outputs the feature map output by conv2d-18, followed by convolutional layer conv2d-19 with channel number 1 and convolutional kernel size 1 × 1, and the segmented image.
According to the preferred embodiment shown in fig. 2, the image obtained by mars train is resampled to 512 × 512 pixels, input to the U-shaped neural network of the present invention, and after passing through the encoder and decoder, a segmented mars image can be output, where the white areas are the detected rocks.
In step S4, in order to obtain a training model with the minimum loss function, a neural network may be trained using the training samples and the verification samples. Preferably, in order to realize efficient and convenient calculation, the method adopts an Adam optimization algorithm to train the neural network. According to a preferred embodiment, the number of training iterations can be set to 500, the iteration precision can be set to 0.0001, and the loss function adopts the cross-entropy binary _ cross _ entry of the two classes, so that the input image can be divided into two classes of rock and non-rock.
The implementation and effects of a preferred embodiment of the present invention are described in detail below.
In this embodiment, 120 images of the surfaces of the sparks shot by the curio-number train are used as samples.
(1) Resampling an original image with the size of 560 × 500 pixels into an image with the size of 512 × 512 pixels, manually labeling the 120 resampled images by using Labelme image labeling software, and then forming a sample set by using the image set and a corresponding manual labeling set.
(2) In this embodiment, a sample set is randomly divided into training samples, validation samples, and test samples according to the ratio of 80%, 10%, and 10%, to obtain 96 training samples, 12 validation samples, and 12 test samples. Fig. 3 shows a set of 96 training samples randomly selected from 120 samples in the present embodiment.
(3) And constructing the neural network according to the neural network model provided by the invention.
(4) Inputting 96 training samples and 12 verification samples into the constructed neural network, and training the neural network to obtain a training model with the minimum loss function. The training iteration number of the embodiment of the invention is set to be 500, and the iteration precision is set to be 0.0001.
(5) And inputting the 12 test samples into the trained segmentation network, and testing the 12 test samples by using the trained training model to obtain the obstacle segmentation result of the extraterrestrial celestial body.
In order to better validate the method of the present invention, in this example, we compared the method of the present invention with the U-Net method and the Att-Unet method [ see Ozan Oktay, Jo Schlemper, Loic Le Folgoc, et al. Attention U-Net: learning where to look for the trade.2018CVPR.Main test criteria include Pixel accuracy PA (Pixel accuracy), class average Pixel accuracy MPA (mean Pixel accuracy), merge ratio IoU (Intersection over Unit), average merge ratio MIoU (mean Intersection over Unit), frequency weight merge ratio FWloU (free Weighted Intersection Unit), and the like.
The three methods respectively train 96 training samples (the training times are all 500) to obtain training models, and in the training process, 12 verification samples are simultaneously used for verification, and then the obtained training models are used for testing 12 test samples. The following table shows the average index of the three methods for 12 test samples, and fig. 4 a-4 e show five index values corresponding to 12 images in the test samples.
As can be seen from comparison of five indexes in the embodiment of the invention, the segmentation precision of the method is superior to that of U-Net and Att-UNet neural networks. As shown in FIG. 5 a-FIG. 51, the rock segmentation results of 12 test images according to the embodiment of the present invention are shown, and each row represents the test sample, the corresponding labeled graph, the segmentation result of the method of the present invention, the segmentation result of U-Net, and the segmentation result of Attention U-Net in turn from left to right. After the test of the training model obtained by the invention, the shadow in the graph is obviously removed.
Compared with the prior art, the invention has the beneficial effects that:
(1) according to the invention, the VGG network with high precision is locally migrated to the network, so that the obstacle segmentation precision is improved, the segmentation speed is also improved, and the real-time requirement of the extraterrestrial celestial body inspection device can be met.
(2) The invention provides a method based on a U-shaped segmentation network, which can realize high-precision segmentation on a very small sample, has high training speed and high precision, is also suitable for middle-distance and long-distance obstacle segmentation detection, and is suitable for obstacle detection and application of an extraterrestrial astronomical inspection tour device based on a monocular camera.
(3) The obstacle segmentation method based on deep learning is adopted, so that the shadow of the rock on the surface of the extraterrestrial celestial body can be removed simultaneously, and the problem that the shadow is difficult to remove automatically by a tour device is solved.
Additional features and advantages of embodiments of the invention will be set forth in the detailed description which follows.
The embodiment of the invention also provides a machine-readable storage medium, wherein the machine-readable storage medium is stored with instructions, and the instructions are used for enabling a machine to execute the method for segmenting the obstacle of the extraterrestrial celestial body patrolling device based on deep learning.
The embodiment of the invention also provides a processor which is characterized by being used for running a program, wherein the program is used for executing the extraterrestrial celestial body patrolling device obstacle segmentation method based on deep learning when being run.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. The use of the phrase "including an" as used herein does not exclude the presence of other, identical elements, components, methods, articles, or apparatus that may include the same, unless expressly stated otherwise.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (9)
1. A method for segmenting obstacles of an extraterrestrial celestial body patrolling device based on deep learning comprises the following steps:
s1, forming a sample set by using an image set obtained by the extraterrestrial celestial body patrolling device and a corresponding manual labeling image set;
s2, dividing the samples in the sample set into training samples, verification samples and test samples;
s3, constructing a neural network based on the VGGNet convolution network and the U-Net network;
s4, inputting the training sample and the verification sample into the neural network, and training the neural network to obtain a training model;
and S5, testing the test sample by using the training model to obtain the obstacle segmentation result of the extraterrestrial celestial body.
2. The deep learning-based extraterrestrial celestial rover obstacle segmentation method according to claim 1, wherein the step S1 includes:
and manually labeling the image set obtained by the extraterrestrial celestial body patrolling device to obtain a manually labeled image set.
3. The deep learning-based extraterrestrial celestial rover obstacle segmentation method according to claim 1, wherein the step S2 includes:
samples in the sample set were randomly divided into training samples, validation samples, and test samples in proportions of 80%, 10%, and 10%.
4. The deep learning-based extraterrestrial celestial rover obstacle segmentation method according to claim 1, wherein a neural network is constructed in the step S3 by:
s301, in an encoder of the neural network, all layers of block1, block2, block3 and block4 in a VGG16 network structure and convolutional layers of block5 are utilized, and all the layers are set to be trainable;
s302, in a decoder of the neural network, restoring the characteristic diagram output by each layer in the encoder.
5. The deep learning-based extraterrestrial celestial rover obstacle segmentation method according to claim 4, wherein in step S301, an encoder is constructed by:
the block1 comprises 2 convolutional layer blocks 1_ conv1, block1_ conv2 and 1 pooling layer block1_ pool, the number of channels of each convolutional layer is 64, and the size of a convolutional core is 3 multiplied by 3; the block2 comprises 2 convolutional layer blocks 2_ conv1, block2_ conv2 and 1 pooling layer block2_ pool, the number of channels of each convolutional layer is 128, and the size of a convolutional core is 3 multiplied by 3; the block3 comprises 3 convolutional layer blocks 3_ conv1, block3_ conv2, block3_ conv3 and 1 pooling layer block3_ pool, the number of each convolutional layer channel is 256, and the size of a convolutional core is 3 multiplied by 3; the block4 comprises 3 convolutional layer blocks 4_ conv1, block4_ conv2, block4_ conv3 and 1 pooling layer block4_ pool, the number of each convolutional layer channel is 512, and the size of a convolutional core is 3 multiplied by 3; the block5 comprises 3 convolutional layer blocks 5_ conv1, 5_ conv2 and 5_ conv3, the number of channels of each convolutional layer is 512, and the size of a convolutional core is 3 multiplied by 3; all pooling layers employ maximum pooling.
6. The deep learning-based extraterrestrial celestial rover obstacle segmentation method according to claim 5, wherein in step S302, a decoder is constructed by:
2 convolution layers conv2d-1, conv2d-2 and 1 dropout layer dropout-1 are closely arranged behind block5_ conv3, the number of channels of each convolution layer is 1024, and the size of a convolution kernel is 3 multiplied by 3;
amplifying the output of dropout-1 by 1 time through an UpSampling function UpSampling2D, then connecting 1 convolutional layer conv2d-3 with the channel number of 512 and the convolution kernel size of 3 × 3, then connecting a feature map output by an encoder block4_ conv3 with a feature map output by conv2d-3 by using a Concatenate mode, and then immediately connecting 3 convolutional layers cond2d-4, cond2d-5 and cond2d-6, wherein the channel number of each convolutional layer is 512 and the convolution kernel size is 3 × 3;
the output of cond2d-6 is amplified by 1 time through an UpSampling function UpSampling2D, then 1 convolutional layer conv2d-7 with the channel number of 256 and the convolution kernel size of 3 × 3 is carried out, then the feature map output by the encoder block3_ conv3 is connected with the feature map output by conv2d-7 by using a Concatenate mode, and then 3 convolutional layers cond2d-8, cond2d-9 and cond2d-10 are carried out, the channel number of each convolutional layer is 256, and the convolution kernel size is 3 × 3;
the output of cond2d-10 is amplified by 1 time through an UpSampling function UpSampling2D, then 1 convolutional layer conv2d-11 with the channel number of 128 and the convolution kernel size of 3 × 3 is carried out, then the feature map output by the encoder block2_ conv2 is connected with the feature map output by conv2d-11 by using a Concatenate mode, and then 3 convolutional layers cond2d-12, cond2d-13 and cond2d-14 are carried out, the channel number of each convolutional layer is 128, and the convolution kernel size is 3 × 3;
the output of cond2d-14 is amplified by 1 time through an UpSampling function UpSampling2D, then 1 convolutional layer conv2d-15 with the channel number of 64 and the convolution kernel size of 3 × 3 is carried out, then the feature map output by the encoder block1_ conv2 is connected with the feature map output by conv2d-15 by using a Concatenate mode, and then 3 convolutional layers conv2d-16, conv2d-17 and conv2d-18 are carried out after connection, the channel number of each convolutional layer is 64, and the convolution kernel size is 3 × 3; the decoder section finally outputs the feature map output by conv2d-18, followed by convolutional layer conv2d-19 with channel number 1 and convolutional kernel size 1 × 1, and the segmented image.
7. The deep learning-based extraterrestrial celestial rover obstacle segmentation method according to claim 1, wherein in the step S4:
and training the neural network by adopting an Adam optimization algorithm.
8. A machine-readable storage medium having stored thereon instructions for causing a machine to perform the method for deep learning based segmentation of extraterrestrial celestial rover obstacles of any one of claims 1-7.
9. A processor configured to execute a program, wherein the program is configured to perform: the deep learning based extraterrestrial celestial rover obstacle segmentation method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010562735.9A CN111797836B (en) | 2020-06-18 | 2020-06-18 | Depth learning-based obstacle segmentation method for extraterrestrial celestial body inspection device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010562735.9A CN111797836B (en) | 2020-06-18 | 2020-06-18 | Depth learning-based obstacle segmentation method for extraterrestrial celestial body inspection device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111797836A true CN111797836A (en) | 2020-10-20 |
| CN111797836B CN111797836B (en) | 2024-04-26 |
Family
ID=72803580
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010562735.9A Active CN111797836B (en) | 2020-06-18 | 2020-06-18 | Depth learning-based obstacle segmentation method for extraterrestrial celestial body inspection device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111797836B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112668696A (en) * | 2020-12-25 | 2021-04-16 | 杭州中科先进技术研究院有限公司 | Unmanned aerial vehicle power grid inspection method and system based on embedded deep learning |
| CN113128347A (en) * | 2021-03-24 | 2021-07-16 | 北京中科慧眼科技有限公司 | RGB-D fusion information based obstacle target classification method and system and intelligent terminal |
| CN113378390A (en) * | 2021-06-15 | 2021-09-10 | 浙江大学 | Extraterrestrial star traffic analysis method and extraterrestrial star traffic analysis system based on deep learning |
| CN113747058A (en) * | 2021-07-27 | 2021-12-03 | 荣耀终端有限公司 | Image content shielding method and device based on multiple cameras |
| CN114155372A (en) * | 2021-12-03 | 2022-03-08 | 长春工业大学 | Deep learning-based structured light weld curve identification and fitting method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019144575A1 (en) * | 2018-01-24 | 2019-08-01 | 中山大学 | Fast pedestrian detection method and device |
| CN110570432A (en) * | 2019-08-23 | 2019-12-13 | 北京工业大学 | CT image liver tumor segmentation method based on deep learning |
| CN110675411A (en) * | 2019-09-26 | 2020-01-10 | 重庆大学 | Cervical squamous intraepithelial lesion recognition algorithm based on deep learning |
| CN111028217A (en) * | 2019-12-10 | 2020-04-17 | 南京航空航天大学 | Image crack segmentation method based on full convolution neural network |
-
2020
- 2020-06-18 CN CN202010562735.9A patent/CN111797836B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019144575A1 (en) * | 2018-01-24 | 2019-08-01 | 中山大学 | Fast pedestrian detection method and device |
| CN110570432A (en) * | 2019-08-23 | 2019-12-13 | 北京工业大学 | CT image liver tumor segmentation method based on deep learning |
| CN110675411A (en) * | 2019-09-26 | 2020-01-10 | 重庆大学 | Cervical squamous intraepithelial lesion recognition algorithm based on deep learning |
| CN111028217A (en) * | 2019-12-10 | 2020-04-17 | 南京航空航天大学 | Image crack segmentation method based on full convolution neural network |
Non-Patent Citations (2)
| Title |
|---|
| JONATHAN LONG等: "Fully convolutional networks for semantic segmentation", 2015 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), pages 3431 - 3440 * |
| 许玥等: "基于深度学习模型的遥感图像分割方法", 计算机应用, pages 2905 - 2914 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112668696A (en) * | 2020-12-25 | 2021-04-16 | 杭州中科先进技术研究院有限公司 | Unmanned aerial vehicle power grid inspection method and system based on embedded deep learning |
| CN113128347A (en) * | 2021-03-24 | 2021-07-16 | 北京中科慧眼科技有限公司 | RGB-D fusion information based obstacle target classification method and system and intelligent terminal |
| CN113128347B (en) * | 2021-03-24 | 2024-01-16 | 北京中科慧眼科技有限公司 | Obstacle target classification method and system based on RGB-D fusion information and intelligent terminal |
| CN113378390A (en) * | 2021-06-15 | 2021-09-10 | 浙江大学 | Extraterrestrial star traffic analysis method and extraterrestrial star traffic analysis system based on deep learning |
| CN113378390B (en) * | 2021-06-15 | 2022-06-24 | 浙江大学 | Method and system for analyzing trafficability of extraterrestrial ephemeris based on deep learning |
| CN113747058A (en) * | 2021-07-27 | 2021-12-03 | 荣耀终端有限公司 | Image content shielding method and device based on multiple cameras |
| CN113747058B (en) * | 2021-07-27 | 2023-06-23 | 荣耀终端有限公司 | Image content shielding method and device based on multiple cameras |
| CN114155372A (en) * | 2021-12-03 | 2022-03-08 | 长春工业大学 | Deep learning-based structured light weld curve identification and fitting method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111797836B (en) | 2024-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zermas et al. | Fast segmentation of 3d point clouds: A paradigm on lidar data for autonomous vehicle applications | |
| CN111797836B (en) | Depth learning-based obstacle segmentation method for extraterrestrial celestial body inspection device | |
| Zhao et al. | Lidar mapping optimization based on lightweight semantic segmentation | |
| Broggi et al. | Terrain mapping for off-road autonomous ground vehicles using rational b-spline surfaces and stereo vision | |
| CN113706480B (en) | Point cloud 3D target detection method based on key point multi-scale feature fusion | |
| Zhang et al. | Visual SLAM for underwater vehicles: A survey | |
| Ding et al. | Vehicle pose and shape estimation through multiple monocular vision | |
| CN111311663B (en) | Real-time large-scene three-dimensional semantic modeling method | |
| Zhang et al. | Three-dimensional cooperative mapping for connected and automated vehicles | |
| Ku et al. | SHREC 2020: 3D point cloud semantic segmentation for street scenes | |
| Lowphansirikul et al. | 3D Semantic segmentation of large-scale point-clouds in urban areas using deep learning | |
| Arora et al. | Static map generation from 3D LiDAR point clouds exploiting ground segmentation | |
| Kampker et al. | Concept study for vehicle self-localization using neural networks for detection of pole-like landmarks | |
| JP2022080303A (en) | Lidar localization using optical flow | |
| CN112907625B (en) | Target following method and system applied to quadruped bionic robot | |
| Fehr et al. | Reshaping our model of the world over time | |
| CN114913519B (en) | 3D target detection method and device, electronic equipment and storage medium | |
| CN116503602A (en) | Unstructured environment three-dimensional point cloud semantic segmentation method based on multi-level edge enhancement | |
| Guo et al. | A feasible region detection method for vehicles in unstructured environments based on PSMNet and improved RANSAC | |
| Diaz et al. | Real-time ground filtering algorithm of cloud points acquired using Terrestrial Laser Scanner (TLS) | |
| Madake et al. | Visualization of 3D Point Clouds for Vehicle Detection Based on LiDAR and Camera Fusion | |
| Wang et al. | LiDAR-SLAM loop closure detection based on multi-scale point cloud feature transformer | |
| Kim et al. | LiDAR Based 3D object detection using CCD information | |
| JP2023508276A (en) | map containing covariances at multiresolution voxels | |
| Hu et al. | Accurate fiducial mapping for pose estimation using manifold optimization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |