CN114972022A - Hyperspectral super-resolution method and system based on non-aligned RGB image fusion - Google Patents
Hyperspectral super-resolution method and system based on non-aligned RGB image fusion Download PDFInfo
- Publication number
- CN114972022A CN114972022A CN202210401740.0A CN202210401740A CN114972022A CN 114972022 A CN114972022 A CN 114972022A CN 202210401740 A CN202210401740 A CN 202210401740A CN 114972022 A CN114972022 A CN 114972022A
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- hyperspectral
- rgb
- subsystem
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 230000004927 fusion Effects 0.000 title claims description 16
- 238000013135 deep learning Methods 0.000 claims abstract description 9
- 230000003287 optical effect Effects 0.000 claims abstract description 6
- 238000012549 training Methods 0.000 claims description 46
- 238000012545 processing Methods 0.000 claims description 28
- 238000013528 artificial neural network Methods 0.000 claims description 18
- 230000003595 spectral effect Effects 0.000 claims description 14
- 238000005516 engineering process Methods 0.000 claims description 9
- 238000003062 neural network model Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 claims description 6
- 238000005316 response function Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 5
- 230000002194 synthesizing effect Effects 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 4
- 238000003384 imaging method Methods 0.000 abstract description 4
- 238000013507 mapping Methods 0.000 description 4
- 238000001228 spectrum Methods 0.000 description 4
- 238000000701 chemical imaging Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 239000002131 composite material Substances 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 239000003208 petroleum Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T3/4076—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution using the original low-resolution images to iteratively correct the high-resolution images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/269—Analysis of motion using gradient-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a method and a system for fusing hyperspectral super-resolution based on non-aligned RGB images, and belongs to the technical field of super-resolution imaging. Firstly, a depth RGB image feature extractor and a hyperspectral image feature extractor are respectively constructed for RGB images and hyperspectral images based on a deep learning theory. And respectively extracting multi-level features of the RGB reference image and the hyperspectral image by using a feature extractor. And aligning the RGB reference image with the multi-level features of the hyperspectral image by using a multi-level depth optical flow estimation network. And after the aligned RGB image features and the hyperspectral image features are obtained, a depth self-adaptive feature decoder is constructed, the aligned features are decoded, and a high-resolution hyperspectral image is reconstructed. According to the method, the spatial super-resolution is carried out on the low-resolution hyperspectral image by utilizing the captured misaligned high-resolution RGB image without explicit intermediate steps and manual intervention under the condition of only using the hyperspectral camera, the RGB camera and a necessary fixing device.
Description
Technical Field
The invention relates to a method and a system for fusing hyperspectral super-resolution based on non-aligned RGB images, and belongs to the technical field of super-resolution imaging in computational camera science.
Background
Different from the traditional black-and-white image and the traditional RGB image, the hyperspectral image is divided more finely in spectral dimension and may contain hundreds to thousands of wave bands, so that the hyperspectral image not only can acquire the image characteristics of an object, but also can acquire the spectral characteristics of the object. This property makes hyperspectral images extremely practical in a variety of detection fields, as different physical objects leave unique "spectral fingerprints" in the electromagnetic spectrum that can be used to identify the constituent components of the object. For example, the spectral characteristics of petroleum can help miners find oil fields.
The existing hyperspectral imaging equipment usually depends on a large number of high-sensitivity sensors, high-speed computers and mass storage equipment to shoot hyperspectral images. This results in a very complex and expensive hyperspectral imaging system. In order to reduce the cost, the existing commercial hyperspectral camera usually sacrifices partial spatial resolution under the condition of ensuring the spectral resolution.
The hyperspectral super-resolution technology aims at improving the spatial resolution of a hyperspectral image by using a software method. The existing high spectral resolution technology can be divided into two types according to input, one type is to input only a single low-resolution high spectral image and reconstruct missing high frequency details through an algorithm to improve the spatial resolution, and the method is generally called hyper-spectral single image hyper-resolution. The other method is to input a low-resolution hyperspectral image and a matched high-resolution RGB image at the same time and use the high-resolution spatial information of RGB to assist the super-resolution of hyperspectral, and the method is called hyperspectral fusion.
Most of the existing hyperspectral single-image hyper-resolution algorithms rely on deep learning methods. In the hyper-spectral simple graph hyper-resolution algorithm based on deep learning, a well-designed deep nonlinear neural network is generally used for modeling the mapping from a low-resolution hyper-spectral image to a high-resolution hyper-spectral image, and then the parameters of the network are optimized by using related data and a proper loss function for training so as to enable the network to approach a real mapping relation. Such methods tend to achieve relatively good results at a certain magnification (less than four times). However, for larger magnifications (greater than four times), such single-map hyper-segmentation algorithms do not achieve satisfactory results.
The hyperspectral fusion method is used for performing hyperspectral image hyper-resolution by using the matched RGB image with high resolution as an aid. Some of the methods are based on optimization methods to design various prior constraints, and some are based on deep learning. Due to the matched high-resolution RGB images, the method can obtain better effect of a higher-spectrum single-image super-resolution method on high magnification. The main disadvantage is that most of the existing methods rely on that the RGB image and the hyperspectral image are accurately aligned, and if the RGB reference image and the hyperspectral image are not aligned, the super-resolution effect of the methods is greatly reduced.
The accurate alignment high-resolution RGB reference image is not easy to obtain in practical applications, and an input optical path is often divided into two paths by means of a spectroscope, and then the two paths are imaged by a hyperspectral camera and an RGB camera at the same time. At the same time, the entire imaging system also needs to be accurately calibrated in order to achieve accurate alignment. The series of requirements can greatly improve the complexity of the system, greatly increase the cost, and further reduce the brightness of an input light path due to the use of the spectroscope, so that the hyperspectral imaging is very unfavorable.
Therefore, around how to improve the imaging quality of the hyperspectral image, reduce the cost of the whole system and expand the application scene of the hyperspectral image, a hyperspectral fusion super-resolution method and a hyperspectral fusion super-resolution system which can still keep better performance under the condition that a high-resolution RGB reference image and a low-resolution hyperspectral image are not completely aligned are urgently needed.
Disclosure of Invention
The invention aims to overcome the defects in the prior art, and creatively provides a method and a system for fusing hyperspectral super-resolution based on non-aligned RGB images in order to reduce the dependence of the existing hyperspectral fused super-resolution technology on the accurate alignment of RGB reference images.
The invention is realized by adopting the following technical scheme.
A fusion hyperspectral super-resolution method based on non-aligned RGB images takes a hyperspectral image with low resolution and a corresponding high-resolution RGB reference image as input, and does not require the RGB reference image to be completely aligned with the hyperspectral image.
The hyperspectral super-resolution method is used for improving the spatial resolution of a hyperspectral image. The image to be processed is called a low-resolution hyperspectral image, and the image after super-resolution is called a relative high-resolution hyperspectral image. The high-resolution RGB image has the same resolution as the high-resolution hyperspectral image. The resolution gap between high resolution and low resolution depends on the scaling factor specified at the runtime of the model.
Step 1: constructing a neural network;
firstly, respectively constructing a depth RGB image feature extractor and a hyperspectral image feature extractor aiming at an RGB image and a hyperspectral image based on a deep learning theory; respectively extracting multi-level features of the RGB reference image and the hyperspectral image by using the two feature extractors;
then, aligning the RGB reference image with the multi-level features of the hyperspectral image by using a multi-level depth optical flow estimation network;
after the aligned RGB image features and the hyperspectral image features are obtained, a depth self-adaptive feature decoder is constructed, the aligned features are decoded, and a high-resolution hyperspectral image is reconstructed;
step 2: a training stage;
using the processed non-aligned hyperspectral fusion data set, iteratively training trainable parameters of the neural network constructed in the step 1 and storing the trainable parameters;
then, aligning the image pair based on an alignment algorithm of SIFT and RANSAC, and synthesizing a synthesized RGB image corresponding to the hyperspectral image by using a spectral response function; performing color matching on the synthetic RGB image and the non-aligned reference RGB image by using a histogram-based color matching algorithm; down-sampling the acquired high-resolution hyperspectral image to obtain a synthesized low-resolution hyperspectral image; taking the processed data as training data;
and step 3: a using stage, predicting a corresponding high-resolution hyperspectral image according to the input hyperspectral image and the RGB reference image by using the model parameters obtained in the training stage in the step 1;
firstly, inputting a synthesized low-resolution hyperspectral image and a matched RGB image into a deep neural network to obtain a predicted high-resolution hyperspectral image; then, calculating a mean square error loss function of the predicted image and a real high-resolution image of the training scene; then, calculating the gradient of each node of the deep neural network by using a back propagation technology, and updating network parameters by using a parameter optimizer; the updating is performed repeatedly using each sample in the data set until the loss falls below a set threshold.
Further, in order to effectively implement the method, the invention provides a fusion hyperspectral super-resolution system based on non-aligned RGB images, which comprises a data acquisition subsystem, a data processing subsystem, a training subsystem and an inference subsystem.
The data acquisition subsystem is used for acquiring the paired non-aligned high-resolution RGB image and the low-resolution hyperspectral image. These data will be used for training.
Optionally, the data acquisition subsystem comprises a hyperspectral camera, an RGB camera and a camera fixture, wherein the hyperspectral camera and the RGB camera are fixed on the fixture in parallel. By adjusting the angle and the focal length of the two cameras, clear images containing the same scene can be shot.
The data processing subsystem is used for processing the misaligned image acquired by the data acquisition subsystem. Specifically, the processing content may include: and aligning the image pair by using an alignment algorithm based on SIFT (Scale-invariant feature transform) and RANSAC (RANdom SAmple Consensus) and synthesizing a synthesized RGB image corresponding to the hyperspectral image by using a spectral response function. Color matching is performed on the composite RGB image and the non-aligned reference RGB image using a histogram-based color matching algorithm. And performing down-sampling on the acquired high-resolution hyperspectral image to obtain a synthesized low-resolution hyperspectral image. And taking the processed data as training data.
The training subsystem trains the deep neural network model by using the training data processed by the data processing subsystem. Specifically, firstly, a low-resolution hyperspectral image and a paired RGB image are synthesized and input into a deep neural network, and a predicted high-resolution hyperspectral image is obtained. Then, the predicted image and the real high-resolution image of the training scene are used for calculating a mean square error loss function. After the calculation is finished, calculating the gradient of each node of the deep neural network by using a back propagation technology, and then updating the network parameters by using a parameter optimizer. The updating is performed repeatedly using each sample in the data set until the loss falls below a set threshold.
The reasoning subsystem carries out reasoning by using a trained deep neural network model, and the input of the reasoning subsystem is a low-resolution hyperspectral image and a misaligned paired high-resolution RGB image in an actual application scene. In each reasoning process, the reasoning subsystem does not need to be trained repeatedly, and the same deep neural network model is used each time.
Advantageous effects
Compared with the prior art, the invention has the following advantages:
1. the invention is used as an end-to-end solution, and can directly use the input low-resolution hyperspectral image and the high-resolution RGB reference image in the use stage without explicit intermediate steps and manual intervention.
2. The invention can realize the spatial super-resolution of the low-resolution hyperspectral image by utilizing the captured misaligned high-resolution RGB image without special equipment and only by using a hyperspectral camera, an RGB camera and a necessary fixing device.
Drawings
FIG. 1 is a schematic diagram of a core algorithm model of the method of the present invention.
FIG. 2 is a schematic diagram of the system of the present invention.
Detailed Description
For better illustrating the objects and advantages of the present invention, the following description will be made with reference to the accompanying drawings and examples.
The traditional hyperspectral fusion super-resolution method based on the RGB image usually uses the prior constraint of manual design and predicts the complete spectrum information of each pixel point of the RGB image according to the spectrum characteristics of the hyperspectral image. In recent years, the development of deep learning techniques has led to the development of this class of methods towards data-driven. The existing super-resolution technology based on hyperspectral fusion models the mapping relation from RGB information to spectral information by designing a specific deep neural network, and trains and fits the mapping relation by using a large amount of data. However, both methods are based on the assumption that the RGB image is perfectly aligned with the spectral image. This assumption greatly limits their application in view of the difficulty of achieving full alignment in real life.
According to the method for fusing the hyperspectral super-resolution based on the non-aligned RGB images, based on the deep learning theory, the high-resolution RGB reference image is used for assisting the spatial super-resolution of the low-resolution hyperspectral image, and the RGB images are not required to be completely aligned with the hyperspectral image. As shown in fig. 1.
The present embodiment includes a network construction phase, a training phase, and a use phase.
Step 1: constructing a neural network;
firstly, respectively constructing a depth RGB image feature extractor and a hyperspectral image feature extractor aiming at an RGB image and a hyperspectral image based on a deep learning theory; respectively extracting multi-level features of the RGB reference image and the hyperspectral image by using the two feature extractors;
then, aligning the RGB reference image with the multi-level features of the hyperspectral image by using a multi-level depth optical flow estimation network;
after the aligned RGB image features and the hyperspectral image features are obtained, a depth self-adaptive feature decoder is constructed, the aligned features are decoded, and a high-resolution hyperspectral image is reconstructed;
and 2, step: a training stage;
using the processed non-aligned hyperspectral fusion data set, iteratively training trainable parameters of the neural network constructed in the step 1 and storing the trainable parameters;
then, aligning the image pair based on an alignment algorithm of SIFT and RANSAC, and synthesizing a synthesized RGB image corresponding to the hyperspectral image by using a spectral response function; performing color matching on the synthetic RGB image and the non-aligned reference RGB image by using a histogram-based color matching algorithm; down-sampling the acquired high-resolution hyperspectral image to obtain a synthesized low-resolution hyperspectral image; taking the processed data as training data;
and step 3: a using stage, predicting a corresponding high-resolution hyperspectral image according to the input hyperspectral image and the RGB reference image by using the model parameters obtained in the training stage in the step 1;
firstly, inputting a synthesized low-resolution hyperspectral image and a matched RGB image into a deep neural network to obtain a predicted high-resolution hyperspectral image; then, calculating a mean square error loss function of the predicted image and a real high-resolution image of the training scene; then, calculating the gradient of each node of the deep neural network by using a back propagation technology, and updating network parameters by using a parameter optimizer; the updating is performed repeatedly using each sample in the data set until the loss falls below a set threshold.
Examples
The fusion hyperspectral super-resolution system based on the non-aligned RGB images disclosed by the embodiment comprises a data acquisition subsystem, a data processing subsystem, a training subsystem and an inference subsystem. As shown in fig. 2.
The data acquisition subsystem is used for acquiring the paired non-aligned high-resolution RGB image and the low-resolution hyperspectral image. These data will be used for training.
Optionally, the data acquisition subsystem comprises a hyperspectral camera, an RGB camera and a camera fixture, wherein the hyperspectral camera and the RGB camera are fixed on the fixture in parallel. By adjusting the angle and the focal length of the two cameras, clear images containing the same scene can be shot.
The data processing subsystem is used for processing the misaligned image acquired by the data acquisition subsystem. Specifically, the processing content may include: and aligning the image pair by using an alignment algorithm based on SIFT (Scale-invariant feature transform) and RANSAC (RANdom SAmple Consensus) and synthesizing a synthesized RGB image corresponding to the hyperspectral image by using a spectral response function. Color matching is performed on the composite RGB image and the non-aligned reference RGB image using a histogram-based color matching algorithm. And performing down-sampling on the acquired high-resolution hyperspectral image to obtain a synthesized low-resolution hyperspectral image. And taking the processed data as training data.
The training subsystem trains the deep neural network model by using the training data processed by the data processing subsystem. Specifically, firstly, a low-resolution hyperspectral image and a paired RGB image are synthesized and input into a deep neural network, and a predicted high-resolution hyperspectral image is obtained. Then, the predicted image and the real high-resolution image of the training scene are used for calculating a mean square error loss function. After the calculation is finished, calculating the gradient of each node of the deep neural network by using a back propagation technology, and then updating the network parameters by using a parameter optimizer. The updating is performed repeatedly using each sample in the data set until the loss falls below a set threshold.
The reasoning subsystem carries out reasoning by using a trained deep neural network model, and the input of the reasoning subsystem is a low-resolution hyperspectral image and a misaligned paired high-resolution RGB image in an actual application scene. In each reasoning process, the reasoning subsystem does not need to be trained repeatedly, and the same deep neural network model is used each time.
The connection relationship among the above-mentioned component systems is: the output end of the data acquisition subsystem is connected with the input end of the data processing subsystem, and the data processing subsystem is responsible for processing the data of the data acquisition subsystem and the data processing subsystem. The output end of the data processing subsystem is connected with the input end of the training subsystem, and the output end of the data processing subsystem receives the data provided by the training subsystem to complete model training. The output end of the training subsystem is connected with the input end of the reasoning subsystem, and the model trained by the training subsystem is used for reasoning during actual deployment.
The working process of the system is as follows:
step 1: using a data acquisition subsystem, misaligned pairs of hyperspectral and RGB reference images are acquired.
Step 2: and processing the data acquired by the data acquisition subsystem by using the data processing subsystem, carrying out primary alignment, cutting out a public area, carrying out normalization and formatting, and making into a data set.
And step 3: and sending the data set into a training subsystem, and training a hyperspectral fusion super-resolution network model of the non-aligned RGB images based on optical flow alignment.
And 4, step 4: and the inference module carries out inference prediction on the hyperspectral image of the actual scene and the RGB reference image by using the trained model to obtain a predicted high-resolution hyperspectral image.
The above detailed description is intended to illustrate the objects, aspects and advantages of the present invention, and it should be understood that the above detailed description is only exemplary of the present invention and is not intended to limit the scope of the present invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (3)
1. A fusion hyperspectral super-resolution method based on non-aligned RGB images is characterized in that,
the image to be processed is called a low-resolution hyperspectral image, and the image after super-resolution is called a relative high-resolution hyperspectral image; the high-resolution RGB image and the high-resolution hyperspectral image have the same resolution; the resolution difference between the high resolution and the low resolution depends on the scaling factor specified when the model runs;
the method comprises the following steps of taking a low-resolution hyperspectral image and a corresponding high-resolution RGB reference image as input, and not requiring that the RGB reference image and the hyperspectral image are completely aligned;
step 1: constructing a neural network;
firstly, respectively constructing a depth RGB image feature extractor and a hyperspectral image feature extractor aiming at an RGB image and a hyperspectral image based on a deep learning theory; respectively extracting multi-level features of the RGB reference image and the hyperspectral image by using the two feature extractors;
then, aligning the RGB reference image with the multi-level features of the hyperspectral image by using a multi-level depth optical flow estimation network;
after the aligned RGB image features and the hyperspectral image features are obtained, a depth self-adaptive feature decoder is constructed, the aligned features are decoded, and a high-resolution hyperspectral image is reconstructed;
step 2: a training stage;
using the processed non-aligned hyperspectral fusion data set, iteratively training trainable parameters of the neural network constructed in the step 1 and storing the trainable parameters;
then, aligning the image pair based on an alignment algorithm of SIFT and RANSAC, and synthesizing a synthesized RGB image corresponding to the hyperspectral image by using a spectral response function; performing color matching on the synthetic RGB image and the non-aligned reference RGB image by using a histogram-based color matching algorithm; down-sampling the acquired high-resolution hyperspectral image to obtain a synthesized low-resolution hyperspectral image; taking the processed data as training data;
and step 3: a using stage, predicting a corresponding high-resolution hyperspectral image according to the input hyperspectral image and the RGB reference image by using the model parameters obtained in the training stage in the step 1;
firstly, inputting a synthesized low-resolution hyperspectral image and a matched RGB image into a deep neural network to obtain a predicted high-resolution hyperspectral image; then, calculating a mean square error loss function of the predicted image and a real high-resolution image of a training scene; then, calculating the gradient of each node of the deep neural network by using a back propagation technology, and updating network parameters by using a parameter optimizer; the updating is performed repeatedly using each sample in the data set until the loss falls below a set threshold.
2. A fusion hyperspectral super-resolution system based on non-aligned RGB images is characterized by comprising a data acquisition subsystem, a data processing subsystem, a training subsystem and an inference subsystem;
the data acquisition subsystem is used for acquiring a matched non-aligned high-resolution RGB image and a matched low-resolution hyperspectral image, and the data are used for training;
the data processing subsystem is used for processing the misaligned image acquired by the data acquisition subsystem and taking the processed data as training data;
the training subsystem trains the deep neural network model by using the training data processed by the data processing subsystem;
the reasoning subsystem carries out reasoning by using a trained deep neural network model, and inputs the low-resolution hyperspectral image and the misaligned paired high-resolution RGB image in an actual application scene; in each reasoning process, the reasoning subsystem does not need to be trained repeatedly, and the same deep neural network model is used each time;
the output end of the data acquisition subsystem is connected with the input end of the data processing subsystem, and the data processing subsystem is responsible for processing the data of the data acquisition subsystem and the data processing subsystem; the output end of the data processing subsystem is connected with the input end of the training subsystem, and the output end of the training subsystem receives the data provided by the data processing subsystem to complete model training; the output end of the training subsystem is connected with the input end of the reasoning subsystem, and the model trained by the training subsystem is used for reasoning during actual deployment.
3. The fused hyperspectral super-resolution system based on non-aligned RGB images of claim 2, wherein the data acquisition subsystem comprises a hyperspectral camera, an RGB camera and a camera fixing device;
the hyperspectral camera and the RGB camera are fixed on the camera fixing device in parallel; by adjusting the angle and the focal length of the two cameras, clear images containing the same scene can be shot.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210401740.0A CN114972022B (en) | 2022-04-18 | 2022-04-18 | Fusion hyperspectral super-resolution method and system based on unaligned RGB image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210401740.0A CN114972022B (en) | 2022-04-18 | 2022-04-18 | Fusion hyperspectral super-resolution method and system based on unaligned RGB image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114972022A true CN114972022A (en) | 2022-08-30 |
| CN114972022B CN114972022B (en) | 2024-06-07 |
Family
ID=82978259
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210401740.0A Active CN114972022B (en) | 2022-04-18 | 2022-04-18 | Fusion hyperspectral super-resolution method and system based on unaligned RGB image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114972022B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116433551A (en) * | 2023-06-13 | 2023-07-14 | 湖南大学 | High-resolution hyperspectral imaging method and device based on double-light-path RGB fusion |
| WO2024066118A1 (en) * | 2022-09-27 | 2024-04-04 | 同济大学 | Plastic debris and microplastics detection method based on rgb and hyperspectral image fusion |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111598138A (en) * | 2020-04-24 | 2020-08-28 | 山东易华录信息技术有限公司 | Neural network learning image identification method and device |
| US20200302249A1 (en) * | 2019-03-19 | 2020-09-24 | Mitsubishi Electric Research Laboratories, Inc. | Systems and Methods for Multi-Spectral Image Fusion Using Unrolled Projected Gradient Descent and Convolutinoal Neural Network |
| CN112184560A (en) * | 2020-12-02 | 2021-01-05 | 南京理工大学 | Hyperspectral image super-resolution optimization method based on deep closed-loop neural network |
| JPWO2022064901A1 (en) * | 2020-09-28 | 2022-03-31 |
-
2022
- 2022-04-18 CN CN202210401740.0A patent/CN114972022B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200302249A1 (en) * | 2019-03-19 | 2020-09-24 | Mitsubishi Electric Research Laboratories, Inc. | Systems and Methods for Multi-Spectral Image Fusion Using Unrolled Projected Gradient Descent and Convolutinoal Neural Network |
| CN111598138A (en) * | 2020-04-24 | 2020-08-28 | 山东易华录信息技术有限公司 | Neural network learning image identification method and device |
| JPWO2022064901A1 (en) * | 2020-09-28 | 2022-03-31 | ||
| CN112184560A (en) * | 2020-12-02 | 2021-01-05 | 南京理工大学 | Hyperspectral image super-resolution optimization method based on deep closed-loop neural network |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024066118A1 (en) * | 2022-09-27 | 2024-04-04 | 同济大学 | Plastic debris and microplastics detection method based on rgb and hyperspectral image fusion |
| CN116433551A (en) * | 2023-06-13 | 2023-07-14 | 湖南大学 | High-resolution hyperspectral imaging method and device based on double-light-path RGB fusion |
| CN116433551B (en) * | 2023-06-13 | 2023-08-22 | 湖南大学 | High-resolution hyperspectral imaging method and device based on double-light-path RGB fusion |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114972022B (en) | 2024-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Jiang et al. | ATMFN: Adaptive-threshold-based multi-model fusion network for compressed face hallucination | |
| CN110415199B (en) | Multispectral remote sensing image fusion method and device based on residual learning | |
| CN109064405A (en) | A kind of multi-scale image super-resolution method based on dual path network | |
| Li et al. | Deep learning methods in real-time image super-resolution: a survey | |
| CN114972022B (en) | Fusion hyperspectral super-resolution method and system based on unaligned RGB image | |
| De Lutio et al. | Learning graph regularisation for guided super-resolution | |
| CN109636721B (en) | Video super-resolution method based on countermeasure learning and attention mechanism | |
| Yu et al. | Hallucinating unaligned face images by multiscale transformative discriminative networks | |
| CN110070517B (en) | Blurred image synthesis method based on degradation imaging mechanism and generation countermeasure mechanism | |
| Liu et al. | Exploit camera raw data for video super-resolution via hidden markov model inference | |
| Raghavendra et al. | Comparative evaluation of super-resolution techniques for multi-face recognition using light-field camera | |
| Huang et al. | Neighbor2neighbor: A self-supervised framework for deep image denoising | |
| Zhai et al. | A comprehensive review of deep learning-based real-world image restoration | |
| CN113902646A (en) | Remote sensing image pan-sharpening method based on depth layer feature weighted fusion network | |
| CN113066030A (en) | Multispectral image panchromatic sharpening method and system based on space-spectrum fusion network | |
| Shangguan et al. | Learning cross-video neural representations for high-quality frame interpolation | |
| Duan et al. | Multifocus image fusion using superpixel segmentation and superpixel-based mean filtering | |
| Wen et al. | SVBRDF Recovery from a Single Image with Highlights Using a Pre‐trained Generative Adversarial Network | |
| Jiang et al. | From less to more: Spectral splitting and aggregation network for hyperspectral face super-resolution | |
| Karim et al. | MTDFusion: A Multilayer Triple Dense Network for Infrared and Visible Image Fusion | |
| Tu et al. | RGTGAN: Reference-Based Gradient-Assisted Texture-Enhancement GAN for Remote Sensing Super-Resolution | |
| Guo et al. | Stereo cross-attention network for unregistered hyperspectral and multispectral image fusion | |
| CN109584194A (en) | Hyperspectral image fusion method based on convolution variation probabilistic model | |
| Jiang et al. | Context-patch based face hallucination via thresholding locality-constrained representation and reproducing learning | |
| Baccarelli et al. | Twinned Residual Auto-Encoder (TRAE)—A new DL architecture for denoising super-resolution and task-aware feature learning from COVID-19 CT images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |