CN114219870A - Auxiliary image generation method and system for super-resolution reconstruction of hyperspectral image of unmanned aerial vehicle - Google Patents
Auxiliary image generation method and system for super-resolution reconstruction of hyperspectral image of unmanned aerial vehicle Download PDFInfo
- Publication number
- CN114219870A CN114219870A CN202111493634.1A CN202111493634A CN114219870A CN 114219870 A CN114219870 A CN 114219870A CN 202111493634 A CN202111493634 A CN 202111493634A CN 114219870 A CN114219870 A CN 114219870A
- Authority
- CN
- China
- Prior art keywords
- unmanned aerial
- aerial vehicle
- frequency band
- hyperspectral
- channel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Abstract
The invention discloses an auxiliary image generation method and system for super-resolution reconstruction of hyperspectral images of an unmanned aerial vehicle, which comprises the steps of firstly, considering the characteristics of channels of various frequency bands of hyperspectral imaging equipment and establishing a spectral response output function; secondly, establishing a frequency band channel selection model of the convolutional neural network, embedding a frequency band selection process into the neural network, and adaptively selecting an optimal frequency band channel in training; and finally, converting the selected channel mark into a control signal, controlling the unmanned aerial vehicle to carry an auxiliary controlled adjustable spectral imager, and outputting a corresponding auxiliary image on the optimal frequency band channel. According to the method, the frequency band selection of the auxiliary image and the hyperspectral image with the scene target characteristic are integrated into the same model frame, the relationship between the auxiliary image and the hyperspectral image is fully considered, and the auxiliary image which is most beneficial to reconstruction is generated in a self-adaptive mode according to the target scene characteristic under the condition of no manual intervention.
Description
Technical Field
The invention relates to the technical field of hyperspectral image processing, in particular to an auxiliary image generation method and system for unmanned aerial vehicle hyperspectral image super-resolution reconstruction.
Background
An unmanned aerial vehicle remote sensing technology based on hyperspectral image analysis belongs to the cross technical field of combining the unmanned aerial vehicle technology with remote sensing science and computer science, and many researches at present mainly focus on the problems of information extraction, target detection and classification and the like of collected hyperspectral images, and are widely applied to application of homeland resource monitoring, military reconnaissance and the like. However, the above research results are often affected by the quality of the hyperspectral image acquired by the unmanned aerial vehicle, and the poor hyperspectral image cannot provide reliable, objective and accurate data support for research, which easily causes military decision errors. The hyperspectral image is three-dimensional spectral data, and on one hand, the hyperspectral image contains extremely high spectral resolution, and can accurately identify and distinguish the classes of objects in a target scene. On the other hand, because the solar photons are distributed on too many frequency bands, the spatial resolution of the hyperspectral imaging equipment must be reduced to ensure that each frequency band has a certain number of photons, so that the spatial resolution of the hyperspectral image of the unmanned aerial vehicle is often lower than that of a natural image. For reliable object detection and monitoring, extremely high spatial resolution is often required. Therefore, super-resolution reconstruction is carried out on the low-resolution hyperspectral image, the spatial resolution of the hyperspectral image is enhanced, the application requirement that the unmanned aerial vehicle acquires more accurate target information is met, the method is particularly important, and the method is one of the core problems of the unmanned aerial vehicle remote sensing technology.
The most basic problem faced by super-resolution reconstruction of hyperspectral images of unmanned aerial vehicles is information loss, namely, recovering a hyperspectral image with high spatial information content from a small amount of spatial information is a pathological problem, and serious information loss exists. In order to compensate for the information loss, a common mode is to mount a high spatial resolution camera on the unmanned aerial vehicle, so as to introduce information other than the hyperspectral image data to participate in the hyperspectral image super-resolution reconstruction. The existing super-resolution reconstruction algorithm for the hyperspectral image of the unmanned aerial vehicle usually adopts RGB, panchromatic cameras and multispectral cameras to generate auxiliary images participating in reconstruction. These auxiliary images are determined by using specific frequency bands, such as RGB images fixed on three frequency band channels of 0.630-0.680 μm (red), 0.525-0.600 μm (green), and 0.450-0.515 μm (blue). For the hyperspectral image reconstruction task of the unmanned aerial vehicle, a frequency band which is more favorable for reconstruction is designed and selected, and then a better reconstruction task can be achieved at the same image acquisition cost. Some work shows that the RGB image often does not achieve the optimal effect on the same reconstruction method, which makes the RGB image as an auxiliary reconstruction method have limited application prospects. When the full-color image is used as an auxiliary to participate in reconstruction, the spatial information of a single channel can generate obvious spectral distortion. Although the frequency bands of the multispectral image are rich (generally contain 3-10 frequency band information), the frequency bands are artificially predefined and difficult to ensure scene self-adaptation. In addition, unmanned aerial vehicle carries out the hyperspectral image of continuous collection in different target scenes, because material, element composition and structure that different scenes contain are different, and it has different spectral feature, and the different hyperspectral image that consequently probably cause the auxiliary image that the same frequency channel generated to guide rebuilds the result good time bad, and the generalization is relatively poor.
In conclusion, scene self-adaptation is difficult to be a great challenge faced by many existing reconstruction methods for super-resolution of hyperspectral images of unmanned aerial vehicles, and further research is needed for how to self-adaptively select an auxiliary image more beneficial to hyperspectral reconstruction according to a target scene.
Disclosure of Invention
The invention aims to solve the technical problem of providing an auxiliary image generation method and system for super-resolution reconstruction of hyperspectral images of an unmanned aerial vehicle, which can self-adaptively select the auxiliary image most beneficial to reconstruction according to the characteristics of a target scene under the condition of no manual intervention, thereby providing reliable data support for super-resolution reconstruction of the hyperspectral images.
In order to solve the technical problem, the invention provides an auxiliary image generation method for super-resolution reconstruction of hyperspectral images of an unmanned aerial vehicle, which comprises the following steps:
step one, for main hyperspectral image imaging equipment mounted by an unmanned aerial vehicle, the characteristics of each frequency band channel are described by establishing a spectral response output function;
step two, in order to select the frequency band channel which is beneficial to super-resolution reconstruction, a frequency band channel selection model of a convolutional neural network is established, a frequency band selection process is embedded into the neural network, and the optimal frequency band channel is selected in a self-adaptive manner in training;
and step three, converting the corresponding channel marks into control signals, controlling the unmanned aerial vehicle to carry an auxiliary controlled adjustable hyperspectral imager, and outputting corresponding auxiliary images at the optimal frequency band channel.
Preferably, in the first step, for the main hyperspectral image imaging device mounted by the unmanned aerial vehicle, the characteristics of each frequency band channel are described by establishing a spectral response output function, specifically:
Hk=S(k)L(k),k=1,2,...,N
wherein k represents a frequency band channel number;the method comprises the steps that image signals collected by imaging equipment in a k channel are obtained, k is less than or equal to N, and a x b represents the size of a channel image; all HkStacked into three-dimensional hyperspectral imagesS (k) represents the spectral response function of channel k, and the overall spectral response form can be expressed as S ═ S (1), S (2),.., S (k),.., S (n)](ii) a L (k) represents the spectral radiance at channel k.
Preferably, in the second step, in order to select a frequency band channel conducive to super-resolution reconstruction, a frequency band channel selection model of a convolutional neural network is established, a frequency band selection process is embedded into the neural network, and the adaptively selecting an optimal frequency band channel in training specifically comprises: in the spectral response output function of the first step, the process of generating the frequency band image by the spectral radiation l (k) through the corresponding spectral response function s (k) is equivalent to the convolution operation of a 1 × 1 convolution kernel in the convolution neural network; therefore, the response function S (k) of each frequency band channel is stacked into a multi-channel convolution kernel (the total number of channels is N) which is used as the first layer of the convolution neural network to simulate the frequency band selection process; meanwhile, a channel attention mechanism is utilized, and sparse constraint is applied, namely, the constraint network only uses a small number of channels to generate a frequency band image; in summary, the loss function of the convolutional neural network proposed for band channel selection is as follows:
Subject to,Sk∈S,wk∈W
wherein the first term is a reconstruction error,the method comprises the following steps of (1) training samples, wherein each pixel vector of a hyperspectral image is M ═ a × b; m is the number of training samples; n is the total number of channels; skRepresenting a 1 × 1 convolution kernel in the network;representing convolution operations for simulating the band channel selection process, for hmSampling is carried out; w is akRepresenting the parameters corresponding to each channel under the attention mechanism; f represents selecting a network; phi is a parameter of the selected network, the second term is a sparse regularization term, and alpha is a regularization parameter; w represents a parameter vector, under sparse constraint, most weight values are close to 0 value, and the rest channels are selected frequency band channels to obtain corresponding channel marks.
Correspondingly, an auxiliary image generation system for unmanned aerial vehicle hyperspectral image super-resolution reconstruction includes: the system comprises an unmanned aerial vehicle-mounted processing module, an unmanned aerial vehicle-mounted storage module, an unmanned aerial vehicle-mounted main hyperspectral image imaging device and an unmanned aerial vehicle-mounted auxiliary controlled adjustable hyperspectral imager; the unmanned aerial vehicle-mounted processing module is electrically connected with the unmanned aerial vehicle-mounted storage module, the unmanned aerial vehicle-mounted main hyperspectral image imaging equipment and the unmanned aerial vehicle-mounted auxiliary controlled adjustable hyperspectral imager respectively.
Preferably, the unmanned on-board memory module is programmable or configurable; the unmanned aerial vehicle-mounted main hyperspectral image imaging equipment is used for acquiring a hyperspectral image of a target scene; the unmanned aerial vehicle-mounted auxiliary controlled adjustable hyperspectral imager selects any number of frequency band channels with any number by inputting control signals so as to generate an auxiliary image.
The invention has the beneficial effects that: according to the method, the frequency band selection of the auxiliary image and the hyperspectral image with the scene target characteristic are integrated into the same model frame, the relationship between the auxiliary image and the hyperspectral image is fully considered, and the auxiliary image which is most beneficial to reconstruction is adaptively selected and generated according to the target scene characteristic under the condition of no manual intervention.
Drawings
FIG. 1 is a schematic flow chart of the method of the present invention.
FIG. 2 is a schematic diagram of the system framework of the present invention.
FIG. 3 is a schematic diagram of a process of generating an auxiliary image by the unmanned aerial vehicle-mounted auxiliary controlled adjustable hyperspectral imager.
Detailed Description
As shown in fig. 1, an auxiliary image generation method for unmanned aerial vehicle hyperspectral image super-resolution reconstruction includes the following steps:
step one, for main hyperspectral image imaging equipment mounted by an unmanned aerial vehicle, the characteristics of each frequency band channel are described by establishing a spectral response output function;
step two, in order to select the frequency band channel which is beneficial to super-resolution reconstruction, a frequency band channel selection model of a convolutional neural network is established, a frequency band selection process is embedded into the neural network, and the optimal frequency band channel is selected in a self-adaptive manner in training;
and step three, converting the corresponding channel marks into control signals, controlling the unmanned aerial vehicle to carry an auxiliary controlled adjustable hyperspectral imager, and outputting corresponding auxiliary images at the optimal frequency band channel.
In the first step, for a main hyperspectral image imaging device mounted by an unmanned aerial vehicle, the characteristics of each frequency band channel are described by establishing a spectral response output function, specifically:
Hk=S(k)L(k),k=1,2,...,N
wherein k represents a frequency band channel number;the method comprises the steps that image signals collected by imaging equipment in a k channel are obtained, k is less than or equal to N, and a x b represents the size of a channel image; all HkStacked into three-dimensional hyperspectral imagesS (k) represents the spectral response function of channel k, and the overall spectral response form can be expressed as S ═ S (1), S (2),.., S (k),.., S (n)](ii) a L (k) represents the spectral radiance at channel k.
In the second step, in order to select a frequency band channel which is beneficial to super-resolution reconstruction, a frequency band channel selection model of a convolutional neural network is established, a frequency band selection process is embedded into the neural network, and the self-adaptive selection of the optimal frequency band channel in the training specifically comprises the following steps: in the spectral response output function of the first step, the process of generating the frequency band image by the spectral radiation l (k) through the corresponding spectral response function s (k) is equivalent to the convolution operation of a 1 × 1 convolution kernel in the convolution neural network; therefore, the response function S (k) of each frequency band channel is stacked into a multi-channel convolution kernel (the total number of channels is N) which is used as the first layer of the convolution neural network to simulate the frequency band selection process; meanwhile, a channel attention mechanism is utilized, and sparse constraint is applied, namely, the constraint network only uses a small number of channels to generate a frequency band image; in summary, the loss function of the convolutional neural network proposed for band channel selection is as follows:
Subject to,Sk∈S,wk∈W
wherein the first term is a reconstruction error,the method comprises the following steps of (1) training samples, wherein each pixel vector of a hyperspectral image is M ═ a × b; m is the number of training samples; n is the total number of channels; skRepresenting a 1 × 1 convolution kernel in the network;representing convolution operations for simulating the band channel selection process, for hmSampling is carried out; w is akRepresenting the parameters corresponding to each channel under the attention mechanism; f represents selecting a network; phi is a parameter of the selected network, the second term is a sparse regularization term, and alpha is a regularization parameter; w represents a parameter vector, under sparse constraint, most weight values are close to 0 value, and the rest channels are selected frequency band channels to obtain corresponding channel marks.
Correspondingly, as shown in fig. 2, the auxiliary image generation system for super-resolution reconstruction of hyperspectral images of an unmanned aerial vehicle includes: the system comprises an unmanned aerial vehicle-mounted processing module, an unmanned aerial vehicle-mounted storage module, an unmanned aerial vehicle-mounted main hyperspectral image imaging device and an unmanned aerial vehicle-mounted auxiliary controlled adjustable hyperspectral imager; the unmanned aerial vehicle-mounted processing module is electrically connected with the unmanned aerial vehicle-mounted storage module, the unmanned aerial vehicle-mounted main hyperspectral image imaging equipment and the unmanned aerial vehicle-mounted auxiliary controlled adjustable hyperspectral imager respectively.
The unmanned airborne storage module can be programmed or configured; the unmanned aerial vehicle-mounted main hyperspectral image imaging equipment is used for acquiring a hyperspectral image of a target scene; the unmanned aerial vehicle-mounted auxiliary controlled adjustable hyperspectral imager selects any number and number of frequency band channels by inputting control signals so as to generate an auxiliary image, as shown in fig. 3.
According to the method, the frequency band selection of the auxiliary image and the hyperspectral image with the scene target characteristic are integrated into the same model frame, the relationship between the auxiliary image and the hyperspectral image is fully considered, and the auxiliary image which is most beneficial to reconstruction is adaptively selected and generated according to the target scene characteristic under the condition of no manual intervention.
Claims (5)
1. An auxiliary image generation method for super-resolution reconstruction of hyperspectral images of an unmanned aerial vehicle is characterized by comprising the following steps:
step one, for main hyperspectral image imaging equipment mounted by an unmanned aerial vehicle, the characteristics of each frequency band channel are described by establishing a spectral response output function;
step two, in order to select the frequency band channel which is beneficial to super-resolution reconstruction, a frequency band channel selection model of a convolutional neural network is established, a frequency band selection process is embedded into the neural network, and the optimal frequency band channel is selected in a self-adaptive manner in training;
and step three, converting the corresponding channel marks into control signals, controlling the unmanned aerial vehicle to carry an auxiliary controlled adjustable hyperspectral imager, and outputting corresponding auxiliary images at the optimal frequency band channel.
2. The auxiliary image generation method for unmanned aerial vehicle hyperspectral image super-resolution reconstruction as claimed in claim 1, wherein in the step one, for a main hyperspectral image imaging device mounted by an unmanned aerial vehicle, the characteristics of each frequency channel are described by establishing a spectral response output function, specifically:
Hk=S(k)L(k),k=1,2,…,N
wherein k represents a frequency band channel number;the method comprises the steps that image signals collected by imaging equipment in a k channel are obtained, k is less than or equal to N, and a x b represents the size of a channel image; all HkStacked into three-dimensional hyperspectral imagesS (k) represents general formulaThe spectral response function of trace k, the overall spectral response form, may be expressed as S ═ S (1), S (2), …, S (k), …, S (N)](ii) a L (k) represents the spectral radiance at channel k.
3. The auxiliary image generation method for unmanned aerial vehicle hyperspectral image super-resolution reconstruction as recited in claim 1, wherein in step two, in order to select a frequency band channel conducive to super-resolution reconstruction, a frequency band channel selection model of a convolutional neural network is established, a frequency band selection process is embedded into the neural network, and the adaptive selection of the optimal frequency band channel in training specifically comprises: in the spectral response output function of the first step, the process of generating the frequency band image by the spectral radiation l (k) through the corresponding spectral response function s (k) is equivalent to the convolution operation of a 1 × 1 convolution kernel in the convolution neural network; therefore, the response function S (k) of each frequency band channel is stacked into a multi-channel convolution kernel, the total number of the channels is N, the multi-channel convolution kernel is used as the first layer of the convolution neural network, and the frequency band selection process is simulated; meanwhile, a channel attention mechanism is utilized, and sparse constraint is applied, namely, the constraint network only uses a small number of channels to generate a frequency band image; in summary, the loss function of the convolutional neural network proposed for band channel selection is as follows:
Subject to,Sk∈S,wk∈W
wherein the first term is a reconstruction error,the method comprises the following steps of (1) training samples, wherein each pixel vector of a hyperspectral image is M ═ a × b; m is the number of training samples; n is the total number of channels; skRepresenting a 1 × 1 convolution kernel in the network;representing convolution operations for simulating the band channel selection process, for hmSampling is carried out; w is akRepresenting the parameters corresponding to each channel under the attention mechanism; f represents selecting a network; phi is a parameter of the selected network, the second term is a sparse regularization term, and alpha is a regularization parameter; w represents a parameter vector, under sparse constraint, most weight values are close to 0 value, and the rest channels are selected frequency band channels to obtain corresponding channel marks.
4. A supplementary image generation system for unmanned aerial vehicle hyperspectral image super-resolution is rebuild, its characterized in that includes: the system comprises an unmanned aerial vehicle-mounted processing module, an unmanned aerial vehicle-mounted storage module, an unmanned aerial vehicle-mounted main hyperspectral image imaging device and an unmanned aerial vehicle-mounted auxiliary controlled adjustable hyperspectral imager; the unmanned aerial vehicle-mounted processing module is electrically connected with the unmanned aerial vehicle-mounted storage module, the unmanned aerial vehicle-mounted main hyperspectral image imaging equipment and the unmanned aerial vehicle-mounted auxiliary controlled adjustable hyperspectral imager respectively.
5. The auxiliary image generation system for unmanned aerial vehicle hyperspectral image super-resolution reconstruction of claim 4, wherein the unmanned aerial vehicle-mounted storage module can be programmed or configured; the unmanned aerial vehicle-mounted main hyperspectral image imaging equipment is used for acquiring a hyperspectral image of a target scene; the unmanned aerial vehicle-mounted auxiliary controlled adjustable hyperspectral imager selects any number of frequency band channels with any number by inputting control signals so as to generate an auxiliary image.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111493634.1A CN114219870A (en) | 2021-12-08 | 2021-12-08 | Auxiliary image generation method and system for super-resolution reconstruction of hyperspectral image of unmanned aerial vehicle |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111493634.1A CN114219870A (en) | 2021-12-08 | 2021-12-08 | Auxiliary image generation method and system for super-resolution reconstruction of hyperspectral image of unmanned aerial vehicle |
Publications (1)
Publication Number | Publication Date |
---|---|
CN114219870A true CN114219870A (en) | 2022-03-22 |
Family
ID=80700338
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202111493634.1A Pending CN114219870A (en) | 2021-12-08 | 2021-12-08 | Auxiliary image generation method and system for super-resolution reconstruction of hyperspectral image of unmanned aerial vehicle |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114219870A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115542318A (en) * | 2022-10-12 | 2022-12-30 | 南京航空航天大学 | Air-ground combined multi-domain detection system and method for unmanned aerial vehicle group target |
-
2021
- 2021-12-08 CN CN202111493634.1A patent/CN114219870A/en active Pending
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115542318A (en) * | 2022-10-12 | 2022-12-30 | 南京航空航天大学 | Air-ground combined multi-domain detection system and method for unmanned aerial vehicle group target |
CN115542318B (en) * | 2022-10-12 | 2024-01-09 | 南京航空航天大学 | Unmanned aerial vehicle group target-oriented air-ground combined multi-domain detection system and method |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110533620B (en) | Hyperspectral and full-color image fusion method based on AAE extraction spatial features | |
Fu et al. | Joint camera spectral sensitivity selection and hyperspectral image recovery | |
Fu et al. | Joint camera spectral response selection and hyperspectral image recovery | |
CN112734646B (en) | Image super-resolution reconstruction method based on feature channel division | |
JP6945953B2 (en) | Imaging system and image processing method | |
CN110415199B (en) | Multispectral remote sensing image fusion method and device based on residual learning | |
WO2021135657A1 (en) | Image processing method and apparatus, and image processing system | |
CN108810418A (en) | Image processing method, device, mobile terminal and computer readable storage medium | |
CN109472752B (en) | Multi-exposure fusion system based on aerial images | |
CN113160053B (en) | Pose information-based underwater video image restoration and splicing method | |
CN110880163A (en) | Low-light color imaging method based on deep learning | |
CN111814545A (en) | Crop identification method and device, electronic equipment and storage medium | |
CN114219870A (en) | Auxiliary image generation method and system for super-resolution reconstruction of hyperspectral image of unmanned aerial vehicle | |
CN115410087A (en) | Transmission line foreign matter detection method based on improved YOLOv4 | |
CN110555877B (en) | Image processing method, device and equipment and readable medium | |
CN117409339A (en) | Unmanned aerial vehicle crop state visual identification method for air-ground coordination | |
CN113901928A (en) | Target detection method based on dynamic super-resolution, and power transmission line component detection method and system | |
CN111881924A (en) | Dim light vehicle illumination identification method combining illumination invariance and short-exposure illumination enhancement | |
CN116883799A (en) | Hyperspectral image depth space spectrum fusion method guided by component replacement model | |
CN111273376A (en) | Downscaling sea surface net radiation determination method, system, equipment and storage medium | |
CN116029954A (en) | Image fusion method and device | |
CN112991236B (en) | Image enhancement method and device based on template | |
CN116189160A (en) | Infrared dim target detection method based on local contrast mechanism | |
Kamal et al. | Resoluting multispectral image using image fusion and CNN model | |
CN110489584B (en) | Image classification method and system based on dense connection MobileNet model |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |