CN115496819B - Rapid coding spectral imaging method based on energy concentration characteristic - Google Patents
Rapid coding spectral imaging method based on energy concentration characteristic Download PDFInfo
- Publication number
- CN115496819B CN115496819B CN202211442874.3A CN202211442874A CN115496819B CN 115496819 B CN115496819 B CN 115496819B CN 202211442874 A CN202211442874 A CN 202211442874A CN 115496819 B CN115496819 B CN 115496819B
- Authority
- CN
- China
- Prior art keywords
- information
- spectral
- image
- coding
- rgb image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 19
- 238000000701 chemical imaging Methods 0.000 title claims abstract description 14
- 230000003595 spectral effect Effects 0.000 claims abstract description 43
- 239000011159 matrix material Substances 0.000 claims abstract description 16
- 238000003384 imaging method Methods 0.000 claims abstract description 12
- 238000001228 spectrum Methods 0.000 claims abstract description 11
- 238000005070 sampling Methods 0.000 claims abstract description 10
- 238000000605 extraction Methods 0.000 claims abstract description 6
- 238000003780 insertion Methods 0.000 claims abstract description 5
- 230000037431 insertion Effects 0.000 claims abstract description 5
- 230000003287 optical effect Effects 0.000 claims description 4
- 230000009977 dual effect Effects 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 230000003247 decreasing effect Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract 1
- 238000012163 sequencing technique Methods 0.000 abstract 1
- 230000006870 function Effects 0.000 description 9
- 230000000875 corresponding effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 2
- 238000010187 selection method Methods 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010835 comparative analysis Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
- G01J3/28—Investigating the spectrum
- G01J3/2823—Imaging spectrometer
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The application relates to a rapid coding spectral imaging method based on energy concentration characteristics, which comprises the following steps: acquiring spectral information and RGB image information based on a dual-optical-path imaging system; establishing a sampling sequence of the spectrum information coding template based on the high-low sequence obtained by multiplying the spatial information in the unit region of the RGB image by the coding template; establishing a DCT coding template based on the DCT matrix energy concentration characteristic and the sampling sequence, and coding and reconstructing the spectral information; a spectral image enhancement network is established based on a PMS-Net characteristic extraction module and a CMMI information insertion function, and the spatial detail quality of the low-resolution reconstructed spectral image is improved through the abundant spatial information of the RGB image. The method and the device have the advantages that the corresponding sequence of the spectrum codes is obtained based on the sequencing of the RGB image space information, and more space and spectrum information is reserved under the undersampling condition; and RGB image information is fused to achieve the high-quality spectral imaging effect.
Description
Technical Field
The application relates to the field of computer image processing, in particular to a rapid coding spectrum imaging method based on energy concentration characteristics.
Background
The currently widely applied spectral imaging technology can simultaneously obtain two-dimensional spatial information and spectral dimension information of a substance, and each pixel can form corresponding spectral information, so that the accuracy and efficiency of identifying and distinguishing the substance are improved, and the technology is widely applied to the fields of food safety, medical diagnosis, detection remote sensing and the like.
However, in the existing spectral imaging technology level, it is difficult to obtain a hyperspectral image directly and quickly, and some reasons are that the stability of the equipment used in the system is insufficient, and the confidence of the result obtained by the algorithm used is low, so the existing spectral imaging quality is low; meanwhile, as the resolution ratio of the picture is positively correlated with the coding times, the more the coding times are, the higher the resolution ratio is, but the time spent on collecting data is correspondingly increased, the imaging quality is improved to a certain extent by multiple coding in part of spectral imaging technologies, but the time cost is high, and the universality and the high efficiency are not realized.
Disclosure of Invention
In order to solve the above technical problem, the present application provides a fast coding spectral imaging method based on energy concentration characteristics, including the following steps:
the method comprises the following steps: acquiring spectral information and RGB image information based on a dual-light-path imaging system;
step two: establishing a sampling sequence of the spectrum information coding template based on the high-low sequence obtained by multiplying the spatial information in the unit region of the RGB image by the coding template;
step three: establishing a DCT coding template based on the DCT matrix energy concentration characteristic and the sampling sequence, and coding and reconstructing the spectral information;
step four: a spectral image enhancement network is established based on a PMS-Net characteristic extraction module and a CMMI information insertion function, and the spatial detail quality of the low-resolution reconstructed spectral image is improved through the abundant spatial information of the RGB image.
Further, the dual-light-path imaging system in the first step comprises a DMD micro-mirror array group, the DMD micro-mirror array group respectively connects the light path with the spectrum camera and the RGB camera through array transformation, and the DMD micro-mirror array group comprises a semi-reflective lens.
The DCT matrix coding in step three is shown as formula 1:
where N is the size of the image and A is the DCT matrix.
In step four, the enhancement network includes a loss function, as shown in equation 2:
in the formula, the first step is that,Mthe result is output by the network as the total number of samples
Live image of ground
As a function of the loss between as network constraints.
Has the advantages that: (1) high quality imaging: on the basis of the original imaging method, RGB image information is fused, so that spectral information of an undersampled area which originally appears is highly supplemented, and high-quality imaging is achieved; (2) the treatment is convenient and fast: the spectral coding corresponding sequence is obtained by the size of the RGB image space information, and the processing speed is greatly improved by matrixing coding processing.
Drawings
FIG. 1 is a flow chart of the present application;
FIG. 2 is a schematic diagram of a dual optical path imaging system in the present application;
FIG. 3 is a diagram of a spectral information processing method in the present application;
FIG. 4 is a schematic diagram of an enhanced network of the present application;
FIG. 5 is a relative error map image and a spectrally distorted image between a network output image and a ground truth image;
FIG. 6 is a graph of the spectral curve of a full-band spectral enhancement system compared to the true image curve.
Detailed Description
The present application will now be described in further detail with reference to the accompanying drawings.
As shown in fig. 2, in the first step, the dual optical path imaging system specifically includes a DMD micromirror array, a spectral camera and a color camera, wherein the resolution of the DMD micromirror array is 1920 × 1080 pixels, the pixel size can reach 7.6 μm × 7.6 μm, the coding process is implemented by transforming the DMD micromirror array, the coding measurement is performed, and the measurement of the spectrometer is implemented by controlling the on and off states. The encoding process is realized by DMD micromirror array transformation, and a gray template is obtained by overturning DMD and other operations in an experiment.
As shown in fig. 3, in the second step, the RGB image contains rich spatial information, the sampling and encoding position is selected by referring to the spatial information of the RGB image, before encoding the hyperspectral image, the sampling order of the encoding template is selected by using the spatial details of the RGB image, and then the corresponding encoding template order is selected to reconstruct the spectral image, so that more effective information is reconstructed under the condition of encoding undersampling. Specifically, in the coding mode selection method designed by us, for each line of the RGB image, we operate as shown in fig. 2. Firstly, each line of an RGB image is repeated into a matrix with the same size as an encoding template; second, the matrix and the coding template are multiplied and summed to obtain N transform domain values. The absolute values of the N transform domain values are then sorted in descending order. And finally, taking the sequence of the original coding matrix corresponding to the larger absolute value as a spectral imaging coding template. In the case of undersampling, we can reliably reconstruct the spectral image and preserve the edges and details of the image by this template selection method. When the sampling time is greatly reduced, the image quality can still be guaranteed.
In step three as shown in fig. 4, for a natural scene, the image energy is generally uniformly distributed in the spatial domain, and we use the DCT matrix as a coding template to obtain the spectral domain of each row of information of the data cube through the operations of coding and dispersing the target information. As can be seen from the spectral domain, the DCT can concentrate the image energy in the upper part of its transform domain, so that in the case of undersampling, a good reconstruction can be achieved by selecting a suitable coding position. The method takes a Discrete Cosine Transform (DCT) matrix as a coding template, the DCT is separable transform, the core of the transform is a cosine function which can converge physical information energy to low-frequency components, an image can be transformed through the DCT matrix to obtain an expression of the image in a frequency domain, and the main energy of the image is concentrated in the upper left corner of the matrix, and the lower right corner only contains a small amount of energy signals. During the encoding process, a two-dimensional DCT matrix is created as follows:
where N is the size of the image and A is the DCT matrix.
As also shown in fig. 3, in step four, the pre-operation may make information enhancement more effective by inserting HRMS information into the LRHS image through the CMMI function in consideration of the adjacent spectral information correlation. And then, the pre-operated image information is put into a feature extraction network PMS-Net for feature extraction, and then after the features are subjected to sub-pixel convolution calculation, the spectral features of the image can be extracted to the maximum extent, so that sufficient data are provided for spectral trend constraint in the subsequent spectral image enhancement process.
And in the fourth step, the spectral image and the RGB image are fused based on the deep learning neural network, and the spatial information of the RGB image is fused into the reconstructed image to obtain the image with high spatial-high spectral resolution in consideration that the RGB contains abundant spatial information and the reconstructed image under the low sampling condition lacks spatial information. The method mainly comprises the following steps: in one aspect, in the case of undersampling, the RGB image information is inserted into the undersampled reconstructed spectral image by a CMMI information insertion function that preserves the spectral information with the undersampled reconstructed spectral image and the spatial information of the RGB image. The information features are then extracted using the PMS-net module and sub-pixel convolution. In this way, spectral image features can be extracted to the maximum extent. On the other hand, after the spectral dimension of the RGB image is increased, the undersampled reconstructed spectral image and the RGB image information are subjected to violent fusion. A 1*1 convolution structure is used to add or subtract spectral channels. And finally, two parts of network structures are fused to output data, and the frequency spectrum dimension is reduced. The conclusion can be drawn from the network structure, and the network structure imposes strong constraint on the size of the output spectrum through multiple times of enhancement on the hyperspectral data. The LOSS function module is mainly used for calculating network output LOSS, the MSE function is used in a network structure to calculate the difference between data according to the three-dimensional property of hyperspectral data, and a designed LOSS function formula is as shown:
in the formula, the first step is that,Mthe result is output by the network as the total number of samples
Live image of ground
As a function of the loss between as network constraints.
Finally, as shown in fig. 5-6, we select a scene in the test data, select several bands of the scene for comparative analysis, and then plot the relative error mapping image and the spectral distortion image between the network output image and the ground truth image, which shows that the designed spectral image enhancement system achieves a visually acceptable result, and the spectral curve of the spectral image enhancement system is almost the same as the real image curve of all bands.
The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications or substitutions do not depart from the spirit and scope of the present disclosure as defined by the appended claims.
Claims (5)
1. A fast coding spectral imaging method based on energy concentration characteristics is characterized by comprising the following steps:
the method comprises the following steps: acquiring spectral information and RGB image information based on a dual-light-path imaging system;
step two: establishing a sampling sequence of the spectrum information coding template based on the high-low sequence obtained by multiplying the spatial information in the unit region of the RGB image by the coding template;
step three: establishing a DCT coding template based on the DCT matrix energy concentration characteristic and the sampling sequence, and coding and reconstructing the spectral information;
step four: RGB image information is processed through a CMMI information insertion function, the processed image information is subjected to feature extraction through a PMS-Net feature extraction module, extracted features are calculated through sub-pixel convolution, a spectrum image enhancement network is established, and the quality of spatial details of a low-resolution reconstruction spectrum image is improved through rich spatial information of the RGB image, and the method specifically comprises the following steps:
4.1: inserting RGB image information into the undersampled reconstructed spectral image through a CMMI information insertion function, and extracting information characteristics of the RGB image information by using a PMS-net module and sub-pixel convolution;
4.2: fusing the undersampled reconstructed spectral image and RGB image information, and increasing or decreasing spectral channels by using a 1*1 convolution structure;
4.3: and fusing the data output in the steps 4.1 and 4.2.
2. The fast coded spectral imaging method of claim 1, wherein said dual optical path imaging system in step one comprises a DMD micro-mirror array set, which switches the optical path to the spectral camera and the RGB camera respectively by array transformation.
3. The fast coded spectral imaging method of claim 2, wherein in step two said DMD micromirror array set comprises a semi-reflective lens.
4. The fast coded spectral imaging method of claim 1, wherein the DCT matrix coding in three steps is shown in equation 1:
where N is the size of the image and A is the DCT matrix.
5. The method of claim 1, wherein the enhancement network in step four includes a loss function, as shown in equation 2:
in the formula, the first step is that,Mthe result is output by the network as the total number of samples
Live image of ground
As a function of the loss between as network constraints.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211442874.3A CN115496819B (en) | 2022-11-18 | 2022-11-18 | Rapid coding spectral imaging method based on energy concentration characteristic |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211442874.3A CN115496819B (en) | 2022-11-18 | 2022-11-18 | Rapid coding spectral imaging method based on energy concentration characteristic |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115496819A CN115496819A (en) | 2022-12-20 |
| CN115496819B true CN115496819B (en) | 2023-03-21 |
Family
ID=85116147
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211442874.3A Active CN115496819B (en) | 2022-11-18 | 2022-11-18 | Rapid coding spectral imaging method based on energy concentration characteristic |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115496819B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118212536A (en) * | 2024-05-20 | 2024-06-18 | 南京理工大学 | Physical-guided super-resolution compression coding spectrum imaging method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109447891A (en) * | 2019-01-09 | 2019-03-08 | 北京理工大学 | A kind of high quality imaging method of the spectrum imaging system based on convolutional neural networks |
| CN110533620A (en) * | 2019-07-19 | 2019-12-03 | 西安电子科技大学 | The EO-1 hyperion and panchromatic image fusion method of space characteristics are extracted based on AAE |
| CN113899453A (en) * | 2021-08-23 | 2022-01-07 | 南京知谱光电科技有限公司 | Rapid spectral imaging system and imaging method based on sub-sampling |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104006882A (en) * | 2014-05-23 | 2014-08-27 | 南京理工大学 | Spatial modulation Hadamard transform spectrograph based on DMD and spectrum rebuilding method |
| CN111709882B (en) * | 2020-08-06 | 2022-09-27 | 南京理工大学 | Super-resolution fusion calculation method based on sub-pixel convolution and feature segmentation |
| CN113049103B (en) * | 2021-03-12 | 2022-06-07 | 西安电子科技大学 | Spectrum video acquisition method based on DMD variable coding template |
-
2022
- 2022-11-18 CN CN202211442874.3A patent/CN115496819B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109447891A (en) * | 2019-01-09 | 2019-03-08 | 北京理工大学 | A kind of high quality imaging method of the spectrum imaging system based on convolutional neural networks |
| CN110533620A (en) * | 2019-07-19 | 2019-12-03 | 西安电子科技大学 | The EO-1 hyperion and panchromatic image fusion method of space characteristics are extracted based on AAE |
| CN113899453A (en) * | 2021-08-23 | 2022-01-07 | 南京知谱光电科技有限公司 | Rapid spectral imaging system and imaging method based on sub-sampling |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115496819A (en) | 2022-12-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Arad et al. | Ntire 2022 spectral recovery challenge and data set | |
| Bashir et al. | A comprehensive review of deep learning-based single image super-resolution | |
| CN113139898B (en) | Light field image super-resolution reconstruction method based on frequency domain analysis and deep learning | |
| CN111127374B (en) | Pan-sharing method based on multi-scale dense network | |
| CN111080567A (en) | Remote sensing image fusion method and system based on multi-scale dynamic convolution neural network | |
| CN109064396A (en) | A kind of single image super resolution ratio reconstruction method based on depth ingredient learning network | |
| CN112819737B (en) | Remote sensing image fusion method of multi-scale attention depth convolution network based on 3D convolution | |
| Peng et al. | Residual pixel attention network for spectral reconstruction from RGB images | |
| CN116152120B (en) | Low-light image enhancement method and device integrating high-low frequency characteristic information | |
| Arad et al. | NTIRE 2022 spectral demosaicing challenge and data set | |
| CN108288256A (en) | Multispectral mosaic image restoration method | |
| CN110880162A (en) | Snapshot spectrum depth combined imaging method and system based on deep learning | |
| CN102982520A (en) | Robustness face super-resolution processing method based on contour inspection | |
| CN113139902A (en) | Hyperspectral image super-resolution reconstruction method and device and electronic equipment | |
| CN115496819B (en) | Rapid coding spectral imaging method based on energy concentration characteristic | |
| CN112508812A (en) | Image color cast correction method, model training method, device and equipment | |
| Fan et al. | Global sensing and measurements reuse for image compressed sensing | |
| CN114266957A (en) | Hyperspectral image super-resolution restoration method based on multi-degradation mode data augmentation | |
| CN116091916A (en) | Multi-scale hyperspectral image algorithm and system for reconstructing corresponding RGB images | |
| CN116703725A (en) | Method for realizing super resolution for real world text image by double branch network for sensing multiple characteristics | |
| CN114926337A (en) | Single image super-resolution reconstruction method and system based on CNN and Transformer hybrid network | |
| Hang et al. | Prinet: A prior driven spectral super-resolution network | |
| CN115713462A (en) | Super-resolution model training method, image recognition method, device and equipment | |
| CN115578262A (en) | Polarization image super-resolution reconstruction method based on AFAN model | |
| Song et al. | Real-scene reflection removal with raw-rgb image pairs |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |