CN113011354A

CN113011354A - Unmanned aerial vehicle hyperspectral image pine wood nematode disease identification method based on deep learning

Info

Publication number: CN113011354A
Application number: CN202110320557.3A
Authority: CN
Inventors: 徐琪
Original assignee: Zhejiang A&F University ZAFU
Current assignee: Zhejiang A&F University ZAFU
Priority date: 2021-03-25
Filing date: 2021-03-25
Publication date: 2021-06-22

Abstract

The invention provides an unmanned aerial vehicle hyperspectral image pine wilt disease identification method based on deep learning, which comprises the following steps of: s1, extracting a suspected diseased wood area from a hyperspectral image acquired by an unmanned aerial vehicle through an image segmentation network; s2, extracting spectral characteristic vectors and spatial characteristic vectors of suspected diseased wood areas through a deep learning network; and S3, identifying the type of the suspected diseased wood area according to the spectral feature vector and the spatial feature vector. The method can effectively distinguish the color-changing deciduous trees in autumn, bare lands in forests and the diseased trees with the pine wilt disease, can identify the diseased trees with high accuracy even aiming at low-canopy-density forest lands and middle and late autumn, and can realize timely and accurate monitoring effect on the diseased trees.

Description

Unmanned aerial vehicle hyperspectral image pine wood nematode disease identification method based on deep learning

Technical Field

The invention belongs to the technical field of forestry disease and pest monitoring, and particularly relates to an unmanned aerial vehicle hyperspectral image pine wood nematode disease identification method based on deep learning.

Background

The pine wood nematode disease is a destructive pine disease caused by the pine wood nematodes, and is the most serious disease and insect pest causing forest resource loss in China at present. The unmanned aerial vehicle remote sensing can rapidly acquire high-resolution images of large-area key forest regions, single-tree position information of trees suffering from the pine wilt disease can be acquired in time, and the monitoring effect is good and the potential is huge. The outbreak period of the pine wilt disease is in autumn every year, but at present, the pine wilt disease monitored by an unmanned aerial vehicle through a visible light image or a multispectral remote sensing image cannot effectively distinguish autumn color-changing deciduous trees, woodland bare land and damaged pine wilt disease trees, and can only be used for monitoring early-autumn high-canopy-closure forest regions, but the monitoring effect of low-canopy-closure forest regions and late-autumn middle-stage forest regions is poor.

Disclosure of Invention

The invention aims to solve the problems and provides an unmanned aerial vehicle hyperspectral image pine wilt disease identification method based on deep learning.

In order to achieve the purpose, the invention adopts the following technical scheme:

an unmanned aerial vehicle hyperspectral image autumn pine wood nematode disease identification method based on deep learning comprises the following steps:

s1, extracting a suspected diseased wood area from a hyperspectral image acquired by an unmanned aerial vehicle through an image segmentation network;

s2, extracting spectral characteristic vectors and spatial characteristic vectors of suspected diseased wood areas through a deep learning network;

and S3, identifying the type of the suspected diseased wood area according to the spectral feature vector and the spatial feature vector.

In the method for identifying autumn pine wood nematode disease based on deep learning of hyperspectral images of unmanned aerial vehicles, step S1 specifically includes:

s11, extracting 3 wave bands in a spectrum range of red, green and blue light of a hyperspectral image to synthesize a true color image;

s12, cutting the true color image into a plurality of block images with N pixels multiplied by N pixels:

s13, the image segmentation network judges the image pixel category of each block image;

and S14, determining and extracting a suspected diseased wood area according to the pixel type judgment result of the block image.

In the method for identifying the autumn pine wood nematode disease based on the hyperspectral image of the unmanned aerial vehicle by the deep learning, in step S12, the true color image is cut into a plurality of block images with the overlapping degree of 10% -30%;

in step S14, the classification results of the block images are combined, the pixels of the consecutive adjacent suspected diseased wood are combined into a suspected diseased wood area, and the area where the suspected diseased wood area is larger than the preset area and the area circularity is larger than the preset circularity is extracted as the suspected diseased wood area for the subsequent steps.

In the method for identifying the autumn pine wood nematode disease based on the hyperspectral image of the unmanned aerial vehicle by deep learning, in step S2, spectral feature vectors are extracted from partial or all pixel spectral data of a suspected diseased wood area;

and extracting a spatial feature vector from the hyperspectral image of the suspected diseased wood area.

In the method for identifying autumn pine wood nematode disease based on hyperspectral image of unmanned aerial vehicle by deep learning, the deep learning network comprises a spectral feature vector extraction network and a spatial feature vector extraction network which are trained independently, and in step S2, the spectral feature vector is extracted from the spectral feature vector extraction network, and the spatial feature vector is extracted from the spatial feature vector extraction network. The spectral feature vector extraction network and the spatial feature vector extraction network are trained independently, so that the extraction accuracy of the spectral feature vector and the spatial feature vector by the two networks can be ensured.

In the method for identifying autumn pine wood nematode disease based on hyperspectral image of unmanned aerial vehicle by deep learning, in step S3, the deep learning network combines the spectral feature vector and the spatial feature vector, and classifies the spectral feature vector and the spatial feature vector by two layers of fully-connected neural networks to judge the type of the suspected diseased wood area. By means of combination of spectral characteristics and spatial characteristics, distinguishing effects of deciduous trees, bare places in forests and diseased trees can be effectively improved.

In the method for identifying autumn pine wood nematode disease based on hyperspectral image of unmanned aerial vehicle by deep learning, in step S2, the spectral feature vector is extracted by the following method:

s211, randomly selecting M pixels from a suspected diseased wood area;

s212, extracting M pixels, and extracting the spectral reflectivity and the spectral characteristic index of the hyperspectral wave band of the pixels;

s213, the spectral feature vector extraction network extracts spectral feature vectors of M pixels according to the spectral reflectivity and the spectral feature index.

In the method for identifying pine wilt disease in autumn based on hyperspectral image of unmanned aerial vehicle by deep learning, after step S213, the method further includes:

s214, performing feature extraction on the M pixel features of each dimension of the spectral feature vector through two layers of fully-connected neural networks;

and S215, combining the extracted features of all dimensions to form a final spectral feature vector.

In the method for identifying autumn pine wood nematode disease based on hyperspectral image of unmanned aerial vehicle by deep learning, in step S2, the spatial feature vector is extracted by the following method:

s221, extracting a hyperspectral image of an external rectangular window area of a suspected diseased wood area;

s222, extracting three PCA main features through PCA conversion;

s223, synthesizing a false color image by using Principal Component Analysis (PCA) principal characteristics;

s224, extracting the spatial feature vector from the false color image by the spatial feature vector extraction network.

In the method for identifying autumn pine wood nematode disease based on the hyperspectral image of the unmanned aerial vehicle for deep learning, in step S1, the hyperspectral image is a hyperspectral orthographic image subjected to data processing;

step S3 is followed by:

and S4, acquiring a hyperspectral ortho-image, calculating the position of the diseased wood area according to the geographic coordinates of the ortho-image and the ground resolution, and outputting the position coordinates of the diseased wood.

The invention has the advantages that: the method can effectively distinguish the disease trees of color-changing deciduous trees in autumn, bare lands in forests and pine wood nematode diseases, can identify the disease trees with high accuracy even in low-canopy-density forest regions and middle and late autumn, and can realize timely and accurate disease tree monitoring effect.

Drawings

FIG. 1 is a flow chart of the identification of pine wood nematode disease wood according to the present invention combining spectral and spatial characteristics;

FIG. 2 is a diagram illustrating merging of block images according to the present invention;

fig. 3 is a diagram of a deep learning network structure for classifying suspected diseased wood areas according to the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.

As shown in fig. 1, the scheme provides an unmanned aerial vehicle hyperspectral image pine wood nematode disease identification method based on deep learning, the unmanned aerial vehicle hyperspectral image is used for monitoring diseases and trees caused by pine wood nematode disease in autumn, when the diseases and trees are identified, firstly, a hyperspectral image suspected disease and tree area is extracted through an image segmentation network, then, a deep learning network is used for respectively extracting a spectral feature vector and a spatial feature vector according to partial pixel spectral data and an area hyperspectral image of the suspected disease and tree area, and finally, the type of the suspected disease and tree area is identified and judged according to the spectral feature vector and the spatial feature vector so as to search the disease and tree area or the disease and tree area and determine the disease and tree area position, and the specific method:

1) collecting and acquiring a hyperspectral image of an unmanned aerial vehicle in a monitoring area, correcting the hyperspectral image through radiometric calibration and correction software, converting a pixel brightness value into a surface reflectivity, and performing data processing on the hyperspectral image by using unmanned aerial vehicle image processing software to generate a hyperspectral ortho-image;

2) extracting 3 wave bands in the spectrum range of the red, green and blue light of the hyperspectral ortho-image to synthesize a true color image, cutting the true color image into block images with 500 pixels by 500 pixels with 20 percent of overlapping degree, wherein in practical application, the overlapping degree and the size of the block images can be changed according to the situation, and then carrying out image pixel type judgment on each block image through an image segmentation network, wherein the image type has normal loosenessTree, suspected diseased wood (diseased wood, woodland bare land, discolored deciduous tree) and other ground areas. As shown in fig. 2, each small square in the diagram represents a unique pixel of each block image, adjacent block images have an overlap degree of 20%, shaded small squares represent block images classified as suspected diseased wood pixels, classification results of the block images are combined, continuous adjacent suspected diseased wood pixels are combined into suspected diseased wood regions, regions with areas larger than a preset area and areas with circularities larger than a preset circularity are extracted as the suspected diseased wood regions, and the next step is performed, wherein the preset area is generally set to be 1m²The preset circularity is generally set to 0.2. Of course, the size and the preset area of the block image can be adjusted according to the image ground resolution and the size of the crown of the monitoring area, and the circularity can be set to other sizes according to specific situations, such as 0.3, and the like, and is not limited here.

As shown in fig. 3, the following steps are entered for classifying the suspected diseased wood areas:

3) randomly selecting 100 pixels from a suspected diseased wood area, extracting spectral data of all hyperspectral bands of the pixels, namely spectral reflectivity and spectral feature indexes, extracting M pixels, such as spectral feature vectors of 100 pixels, by using a spectral feature extraction network, then extracting features of the M pixels of each dimension of the spectral feature vectors by using two layers of fully-connected neural networks FC1 and FC2, and combining the extracted features of each dimension to form a final spectral feature vector;

4) extracting a hyperspectral image of an external rectangular window area of a suspected diseased wood area, extracting three Principal Component Analysis (PCA) principal characteristics through PCA conversion, synthesizing a false color image by using the PCA principal characteristics, and extracting a spatial feature vector of the false color image by a spatial feature extraction network;

5) the deep learning network further combines the spectral feature vectors and the spatial feature vectors, the spectral feature vectors and the spatial feature vectors are combined, two layers of fully-connected neural networks are used for classifying each suspected diseased wood area object, a Softmax activation function is adopted to output a classification result, and the classification of the hyperspectral image suspected diseased wood area objects is determined, wherein the classification comprises three types: pine wood nematode disease trees, color-changing deciduous trees and woodland bare land;

6) and outputting the diseased wood objects in the classification result, calculating the pixel positions of the diseased wood according to the geographic coordinates of the orthoimage and the ground resolution, and outputting the position coordinates of the diseased wood.

Specifically, each network is constructed as follows:

high spectral image segmentation network construction

1. Establishing a sample data set for classification of hyperspectral images of unmanned aerial vehicles

Carry on high spectral imaging system through unmanned aerial vehicle, gather the high spectral image in pine wood nematode epidemic area, image ground resolution is for requiring about 10 cm. And correcting the hyperspectral image through radiometric calibration and correction software, and converting the pixel brightness value into the earth surface reflectivity. And performing unmanned aerial vehicle data processing on the hyperspectral image by using unmanned aerial vehicle image processing software to generate a hyperspectral ortho-image. Synthesizing a color image by using 3 wave bands in the spectral ranges of the red, green and blue light of the hyperspectral ortho-image, and cutting the image into 500 × 500 block images. Marking normal pine trees, suspected diseased trees (diseased trees, bare places in forests and discolored deciduous trees) and other ground areas on the segmented images by using sample marking software, and manufacturing an image segmentation sample data set;

2. construction and training of image segmentation networks

1) Network architecture

The image segmentation network adopts a deep learning image segmentation network Deeplab V3. The network input is a block image, the output is the category of image pixels, and the category is divided into 3 categories: the extraction network can also be replaced by other deep learning image segmentation networks for normal pine trees, suspected diseased trees (diseased trees, woodland bare land, discolored deciduous trees) and other ground areas.

2) Network construction and training

The method comprises the steps of constructing a network by utilizing deep learning tools such as tensorflow or pytorch, training the network by utilizing all image sample data sets, and expanding samples by data enhancement methods such as turning and rotating. 80% of the samples were used as training samples and 20% as validation samples. And setting a training optimizer, learning rate and training times, and training the network.

(II) deep learning network construction

The deep learning comprises the trained spectral feature extraction network and spatial feature extraction network, a network is constructed by utilizing tenorflow or pytorch, when the deep learning network is constructed and trained, the spectral feature extraction network and the spatial feature extraction network are firstly constructed and trained, then the deep learning network is integrally trained, when the deep learning network is integrally trained, the parameters of the networks are fixed, only the parameters of a full connection layer are trained, a suspected diseased wood area constructed in the step one is directly adopted for training, 80% of samples are used as training samples, and 20% of samples are used as verification samples. And setting a training optimizer, learning rate and training times, and training the network.

(1) Constructing and training a spectral feature extraction network:

1. establishing a ground reflection spectrum sample data set

Collecting normal pine needle leaves, pine wood nematode disease wood needle leaves, color-changing deciduous tree leaves and woodland bare soil, and measuring the reflection spectra of the normal pine needle leaves, the pine wood nematode disease wood needle leaves, the color-changing deciduous tree and the soil by using a spectral radiometer. The measured reflection spectrum range is 350-1000nm, standard white board correction is carried out before data acquisition every time, each sample is repeatedly measured for many times, the average value of the data measured for many times is taken as sample reflection spectrum data, and a sample spectrum data set containing deciduous trees, diseased pines and normal pines is established.

2. Reflectance spectrum data processing

And preprocessing the reflection spectrum data of the sample spectrum data set, wherein the preprocessing mainly comprises spectrum anomaly removal and noise processing. Removing abnormal data according to the standard deviation of the samples, calculating the average value and the standard deviation of each group of samples on each waveband, establishing a spectrum interval by taking the average value as a reference and taking 1.5 times of the standard deviation as upper and lower limits, and removing a spectrum curve exceeding the established spectrum interval. Noise is removed through data smoothing, a sliding window of 3 multiplied by 3 is selected by utilizing a Savitzky-Golay convolution smoothing method, a polynomial of degree k-1 is adopted to fit spectral data points in the window, and smoothing and denoising processing is carried out. And performing spectrum resampling on all sample spectrum data obtained by ground spectrum measurement according to the spectrum waveband range of the hyperspectral camera of the unmanned aerial vehicle, resampling the spectrum waveband into a waveband corresponding to the hyperspectral camera of the unmanned aerial vehicle, and extracting the spectrum reflectivity of the waveband.

3. Selection of characteristic bands

And analyzing the spectral reflectances of all the wave bands calculated in the previous step by adopting an ANOVA method, and determining the first 100 characteristic wave bands with significant differences on the spectra of deciduous trees, diseased pines and normal pines.

4. Spectral feature index calculation

Processing the sample reflection spectrum data, and calculating a spectrum first-order differential (derivative), wherein the calculating of the spectrum trilateral characteristic parameters comprises the following steps: three parameters of the amplitude, position and area of the red side (680nm-750nm), the yellow side (550nm-582nm) and the blue side (490-530nm), wherein the amplitude is the maximum first order differential value in the corresponding range, the position is the wavelength corresponding to the amplitude, and the area is the sum of the first order differential values in the corresponding range. Calculating an envelope curve of the spectrum curve, performing envelope elimination processing on the spectrum curve, and calculating the spectrum absorption characteristic: absorption position, absorption depth, absorption width and absorption symmetry. Calculating a hyperspectral soil adjusted vegetation index (OSAVI), a Structure Insensitive Pigment Index (SIPI), a vegetation decay index (PSRI), a carotenoid reflectance index 1(CRI1), a normalized demagnetisation index (NPQI), and a photochemical vegetation reflectance index (PRI).

5. Network construction and training

The spectral feature extraction network is constructed by adopting a BP neural network, the network inputs the spectral reflectivity of a sample reflection spectral data characteristic wave band and a spectral feature index calculated according to the spectral reflectivity, the output is the type of the sample, and the type of the sample is divided into three types: deciduous trees, diseased trees, and other trees. The method comprises the steps of training a network by utilizing sample spectrum data obtained by ground spectrum measurement, selecting 80% of samples as training samples and 20% of the samples as verification samples, setting a training optimizer, learning rate and training times, training the network, adjusting the number of neurons in the middle layer of the network, and determining the optimal network structure and parameters according to the accuracy of network training.

(2) Spatial feature extraction network

The method comprises the steps of carrying out data processing on a hyperspectral image to obtain a hyperspectral ortho-image, cutting normal pine trees, pine wood nematode disease trees, discolored deciduous trees and woodland bare lands on the hyperspectral ortho-image into single image blocks, labeling image types of the image blocks by using sample labeling software, and manufacturing an image sample data set.

2. Feature analysis and feature transformation

And performing Principal Component Analysis (PCA) on the spectral reflectivity of all sample pixels in the image sample data set, extracting the first 3 PCA principal characteristics, performing principal component transformation on all pictures in the image sample data set, and converting the pictures into RGB false color pictures synthesized by the first 3 PCA principal characteristic images.

3. Network construction and training

The spatial feature extraction network is constructed by adopting GoogleNet inclusion V4, the network input is a PCA principal component image, the output is the category of a sample image, and the sample category is divided into four categories: normal pine, wood with pine nematode disease, color-changing deciduous trees and woodland bare land. And training the network by using all image sample data sets. And expanding the sample by a data enhancement method such as turning and rotating. 80% of the samples were used as training samples and 20% as validation samples. After the network training is completed, global averaging potential is performed on the output of the inclusion-C module in the network (the average value of all pixels is calculated by each dimension feature map) and then the output is used as the extracted spatial feature vector. The spatial feature extraction network can also be replaced by other deep learning image classification networks.

Aiming at the hyperspectral image, firstly, the hyperspectral image is subjected to image segmentation, a suspected diseased wood area is screened out, then, a BP neural network is utilized to extract spectral characteristics, a convolutional neural network is utilized to extract spatial characteristics of a main component false color image, a full-connection neural network is utilized to combine the spectral characteristics and the spatial characteristics to distinguish the pine wood nematode diseased wood, the color-changing deciduous tree and the forest bare land, and the distinguishing effect of the deciduous tree, the forest bare land and the diseased wood can be effectively improved, so that the color-changing deciduous tree in autumn, the forest bare land and the damaged pine wood of the pine wood nematode disease can be effectively distinguished, and even the monitoring effect is higher for the low-canopy density forest area and the middle and late period in autumn.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or ambit of the invention as defined in the appended claims.

Although the terms image segmentation network, deep learning network, suspected diseased wood area, spectral feature vector, spatial feature vector, block image, etc. are used more often herein, the possibility of using other terms is not excluded. These terms are used merely to more conveniently describe and explain the nature of the present invention; they are to be construed as being without limitation to any additional limitations that may be imposed by the spirit of the present invention.

Claims

1. An unmanned aerial vehicle hyperspectral image autumn pine wood nematode disease identification method based on deep learning is characterized by comprising the following steps:

2. The unmanned aerial vehicle hyperspectral image autumn pine wood nematode disease identification method based on deep learning of claim 1, wherein the step S1 specifically comprises:

3. The unmanned aerial vehicle hyperspectral image autumn pine wood nematode disease identification method based on deep learning of claim 2 is characterized in that in step S12, the true color image is cut into a plurality of block images with the overlapping degree of 10% -30%;

4. The unmanned aerial vehicle hyperspectral image autumn pine wood nematode disease identification method based on deep learning of claim 1, wherein in step S2, spectral feature vector extraction is performed from partial or all pixel spectral data of a suspected diseased wood area;

5. The unmanned aerial vehicle hyperspectral image autumn pine wood nematode disease identification method based on deep learning of claim 4, wherein the deep learning network comprises a spectral feature vector extraction network and a spatial feature vector extraction network which are trained separately, and in step S2, the spectral feature vector is extracted from the spectral feature vector extraction network, and the spatial feature vector is extracted from the spatial feature vector extraction network.

6. The method for identifying autumn pine wood nematode disease based on hyperspectral image of unmanned aerial vehicle based on deep learning of claim 5, wherein in step S3, the deep learning network combines the spectral feature vector and the spatial feature vector, and classifies the spectral feature vector and the spatial feature vector by two layers of fully connected neural networks to determine the category of the suspected diseased wood area.

7. The method for identifying the unmanned aerial vehicle high-spectrum image autumn pine wood nematode disease based on deep learning of claim 6, wherein in step S2, the spectral feature vector is extracted by:

s211, randomly selecting M pixels from a suspected diseased wood area;

8. The unmanned aerial vehicle hyperspectral image autumn pine wood nematode disease identification method based on deep learning of claim 7 is characterized by further comprising, after step S213:

9. The method for identifying the autumn pine wood nematode disease based on the hyperspectral image of the unmanned aerial vehicle based on the deep learning of claim 8, wherein in the step S2, the spatial feature vector is extracted by:

s222, extracting three PCA main features through PCA conversion;

10. The method for identifying pine wood nematode disease in autumn based on hyperspectral image of unmanned aerial vehicle of any of claims 1-9, wherein in step S1, the hyperspectral image is a data-processed hyperspectral orthographic image;

step S3 is followed by: