CN115456923A - Method and system for generating hyperspectral image based on hyperspectral and multispectral image fusion - Google Patents

Method and system for generating hyperspectral image based on hyperspectral and multispectral image fusion Download PDF

Info

Publication number
CN115456923A
CN115456923A CN202211118069.5A CN202211118069A CN115456923A CN 115456923 A CN115456923 A CN 115456923A CN 202211118069 A CN202211118069 A CN 202211118069A CN 115456923 A CN115456923 A CN 115456923A
Authority
CN
China
Prior art keywords
hyperspectral
image
multispectral
hyperspectral image
fusion
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211118069.5A
Other languages
Chinese (zh)
Inventor
李树涛
胡宇龙
刘海波
佃仁伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hunan University
Original Assignee
Hunan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hunan University filed Critical Hunan University
Priority to CN202211118069.5A priority Critical patent/CN115456923A/en
Publication of CN115456923A publication Critical patent/CN115456923A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing
    • G06T2207/10036—Multispectral image; Hyperspectral image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Theoretical Computer Science (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Algebra (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a method and a system for generating a hyperspectral image based on hyperspectral and multispectral image fusion, wherein the method for generating the hyperspectral image based on hyperspectral and multispectral image fusion comprises the following steps: the input hyperspectral imageYPerforming subspace decomposition to extract a left low-rank vector U of the hyperspectral image to be generated; multispectral image to be inputZGenerating a right low-rank vector V of the hyperspectral image to be generated through the trained deep generation network; left low rank vector of hyperspectral image to be generatedUAnd right low rank vectorVFusion generation of hyperspectral imageX(ii) a The system of the invention comprises a hyperspectral image acquisition unit for acquiring an input hyperspectral imageYAnd multispectral imagesZAn optical element and an imaging sensor. The hyperspectral image fusion method can generate the hyperspectral image based on hyperspectral and multispectral image fusion, and reduces the cost and efficiency of hyperspectral image acquisition.

Description

Method and system for generating hyperspectral image based on hyperspectral and multispectral image fusion
Technical Field
The invention relates to a hyperspectral image generation technology, in particular to a method and a system for generating a hyperspectral image based on hyperspectral and multispectral image fusion.
Background
Compared with the traditional full-color image or RGB image, the hyperspectral image has more spectral bands and covers the visible light band to the short wave infrared band. Due to different material reflectivity, the abundant spectral information of the hyperspectral image is more beneficial to analyzing the physical and chemical characteristics of an object, so that the hyperspectral image is widely applied to the fields of remote sensing, medical imaging, geological exploration, face recognition and the like. At present, main hyperspectral imagers on the market have three types, namely a spectral scanning type, a swinging type and a pushing type, and due to the limitation of optical imaging hardware facilities, high-resolution hyperspectral video data is difficult to acquire quickly. On the other hand, the acquired hyperspectral video data needs to be stored and transmitted to other equipment for processing, and the acquisition processing result period is long.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: the hyperspectral image fusion method and the hyperspectral image fusion system can be used for generating a hyperspectral image based on hyperspectral and multispectral image fusion, and the cost and the efficiency of hyperspectral image acquisition are reduced.
In order to solve the technical problems, the invention adopts the technical scheme that:
a method for generating a hyperspectral image based on hyperspectral and multispectral image fusion comprises the following steps:
1) The input hyperspectral imageYPerforming subspace decomposition to extract a left low-rank vector U of the hyperspectral image to be generated; multispectral image to be inputZGenerating a right low-rank vector V of a hyperspectral image to be generated through a trained depth generation network;
2) Left low rank vector of hyperspectral image to be generatedUAnd right low rank vectorVFusion generation of hyperspectral imageX。
Optionally, the depth generation network in step 1) includes a forward operation unit and a reverse operation unit, where the forward operation unit and the reverse operation unit are both formed by sequentially cascading multiple two-dimensional convolutional layers, and the multiple two-dimensional convolutional layers of the forward operation unit are arranged from large to small in scaleThe multi-spectral image acquisition device is characterized in that the scales of a plurality of two-dimensional convolution layers of the reverse operation unit are arranged from small to large, the two-dimensional convolution layers with the same scale between the forward operation unit and the reverse operation unit are connected in a jump connection mode, the adjacent two-dimensional convolution layers in the forward operation unit and the reverse operation unit are connected through corresponding up-sampling and down-sampling operation, and the multi-spectral image acquisition device is used for acquiring multi-spectral imagesZOutputting right low-rank vector from two-dimensional convolutional layer input with maximum scale in forward operation unit and two-dimensional convolutional layer output with maximum scale in reverse operation unitV。
Optionally, each two-dimensional convolutional layer comprises a two-dimensional convolution, a two-dimensional batch normalization, a linear rectification and a downsampling in sequence.
Optionally, the loss function used for training the deep generation network in step 1) is root mean square error RMSE.
Optionally, the hyperspectral image to be input in the step 1) isYPerforming subspace decomposition specifically refers to decomposing the input hyperspectral imageYSingular value decomposition is carried out, and a left singular vector obtained by singular value decomposition is used as a left low-rank vector U of the hyperspectral image to be generated; and the input hyperspectral imageYThe function expression for performing singular value decomposition is:
Y=U 1 Σ 1 V 1
in the above formula, the first and second carbon atoms are,U 1 is a matrix of left singular vectors,Σ 1 is a diagonal matrix of singular values,V 1 is the transpose of the matrix of right singular vectors.
Optionally, the hyperspectral image in step 1)YObtaining hyperspectral images for image frames from a hyperspectral video and based on raw image frames in the hyperspectral videoYThe method comprises the following steps: performing image distortion removal operation on an original image frame in a hyperspectral video according to a distortion coefficient matrix of an imaging sensor acquired by a Zhang friend calibration method, cutting images of various spectral bands, aligning and stacking the images according to band sequence to obtain a hyperspectral imageY。
Optionally, the multispectral image in step 1)ZFrom multi-spectral videoAnd obtaining a multispectral image based on an original image frame in the multispectral videoZThe method comprises the following steps: carrying out image distortion removal operation on original image frames in the multispectral video according to a distortion coefficient matrix of an imaging sensor acquired by a Zhang-friend calibration method, and carrying out image distortion removal operation according to the distortion coefficient matrix and the hyperspectral imageYCutting out the corresponding relation of the image and the hyperspectral imageYTo obtain a multi-spectral imageZ。
In addition, the invention also provides a system for generating a hyperspectral image based on the fusion of the hyperspectral image and the multispectral image, which comprises a main mirror, a diaphragm, a light splitting device, a collimating lens, a light filter array, a micro lens array, a hyperspectral image sensor, a reflective mirror, an imaging lens and a multispectral image sensor, wherein light rays captured by the main mirror are divided into two paths after passing through the diaphragm and the light splitting device, and one path of light rays are imaged on the hyperspectral image sensor after sequentially passing through the collimating lens, the light filter array and the micro lens array so as to obtain an original image frame in a hyperspectral video; one path of light rays sequentially pass through the reflector and the imaging lens and then are imaged on the multispectral image sensor to obtain an original image frame in the multispectral video.
Furthermore, the invention also provides a system for generating a hyperspectral image based on fusion of hyperspectral and multispectral images, comprising a processing computing unit and a memory which are connected with each other, wherein the processing computing unit is programmed or configured to execute the steps of the method for generating a hyperspectral image based on fusion of hyperspectral and multispectral images.
Furthermore, the invention also provides a computer-readable storage medium, in which a computer program is stored for being programmed or configured by a processing computing unit to carry out the steps of the method for generating a hyperspectral image based on fusion of a hyperspectral and multispectral image.
Compared with the prior art, the invention has the following advantages: the invention inputs the hyperspectral imageYSingular value decomposition is carried out, and the obtained left singular vector is extracted and used as a left low-rank vector U of the hyperspectral image to be generated; multispectral image to be inputZGenerating a network to be generated through the trained deep generation networkThe right low-rank vector V of the hyperspectral image; left low rank vector of hyperspectral image to be generatedUAnd right low rank vectorVFusion generation of hyperspectral imagesXTherefore, the hyperspectral image can be generated based on the hyperspectral and multispectral image fusion, and the cost and the efficiency of hyperspectral image acquisition are reduced.
Drawings
FIG. 1 is a schematic diagram of a basic process flow of a method according to an embodiment of the present invention.
Fig. 2 is a schematic structural diagram of a deep generation network in the embodiment of the present invention.
Fig. 3 is a block diagram of a system in an embodiment of the invention.
Fig. 4 is a schematic diagram of the optical path of the system in the embodiment of the present invention.
FIG. 5 is a flowchart illustrating the operation of the hardware system according to the embodiment of the present invention.
Detailed Description
As shown in fig. 1, the method for generating a hyperspectral image based on hyperspectral and multispectral image fusion in this embodiment includes:
1) The input hyperspectral imageYSingular value decomposition is carried out, and the left singular vector obtained by extraction is taken as a left low-rank vector U of the hyperspectral image to be generated; multispectral image to be inputZGenerating a right low-rank vector V of a hyperspectral image to be generated through a trained depth generation network;
2) Left low rank vector of hyperspectral image to be generatedUAnd right low rank vectorVFusion generation of hyperspectral imageX。
As shown in fig. 2, the depth generation network in step 1) includes a forward operation unit and a reverse operation unit, the forward operation unit and the reverse operation unit are both composed of a plurality of two-dimensional convolution layers which are cascaded in sequence, and a plurality of two-dimensional convolution layer scales of the forward operation unit are arranged from large to small, a plurality of two-dimensional convolution layer scales of the reverse operation unit are arranged from small to large, the two-dimensional convolution layers with the same scale between the forward operation unit and the reverse operation unit are connected in a jumping connection mode, and the two-dimensional convolution layers adjacent to each other inside are connected through corresponding up-sampling and down-sampling operationsReceiving, multispectral imageZOutputting right low-rank vector from two-dimensional convolutional layer input with maximum scale in forward operation unit and two-dimensional convolutional layer output with maximum scale in reverse operation unitV. Referring to FIG. 2, a multispectral imageZThe scale of (1) is HW x S, where H is height, W is width, and S is the number of multispectral image bands; in this embodiment, the forward operation unit and the reverse operation unit are formed by four two-dimensional convolution layers cascaded in sequence, and the dimensions of the four two-dimensional convolution layers are HW/2 × s/2, HW/4 × s/4, HW/8 × s/8, and HW/16 × s/16, respectively.
In this embodiment, each two-dimensional convolution layer includes a two-dimensional convolution, a two-dimensional batch normalization, a linear rectification, and a down-sampling in sequence.
In this embodiment, the loss function used in the deep generation network training in step 1) is a root mean square error RMSE. In addition, in the embodiment, the self-supervision method is adopted to train the deep generation network, and the self-supervision method is adopted to train the network, so that the requirement on the data volume is low, the difficult problem of scarcity of training data is solved, and the realization is easy. Since the method for training the deep generation network by using the self-supervision method is a known training method, the details of implementation thereof are not described in detail herein.
In this embodiment, the hyperspectral image to be input in step 1) isYPerforming subspace decomposition specifically refers to decomposing the input hyperspectral imageYPerforming singular value decomposition, and taking a left singular vector obtained by singular value decomposition as a left low-rank vector U of the hyperspectral image to be generated; and the input hyperspectral imageYThe function expression for performing singular value decomposition is:
Y=U 1 Σ 1 V 1
in the above formula, the first and second carbon atoms are,U 1 is a matrix of left singular vectors,Σ 1 is a diagonal matrix of singular values,V 1 is the transpose of the matrix of right singular vectors. The input hyperspectral imageYWhen singular value decomposition is performed, a specified number is takenLAnd (4) carrying out singular value decomposition on the maximum singular value according to the formula. Method for decomposing singular valueAre conventional and therefore their details will not be described in detail here.
In this embodiment, the hyperspectral image in step 1)YObtaining hyperspectral images for image frames from a hyperspectral video and based on raw image frames in the hyperspectral videoYThe method comprises the following steps: carrying out image distortion removal operation on original image frames in the hyperspectral video according to a distortion coefficient matrix of an imaging sensor acquired by a Zhang-friend calibration method, cutting each spectral band image, aligning and stacking the spectral band images according to a band sequence to obtain a hyperspectral imageY。
In this embodiment, the multispectral image in step 1) is obtainedZObtaining multispectral images for image frames from multispectral video based on original image frames in the multispectral videoZThe method comprises the following steps: carrying out image distortion removal operation on original image frames in the multispectral video according to a distortion coefficient matrix of an imaging sensor acquired by a Zhang-friend calibration method, and carrying out image distortion removal operation according to the distortion coefficient matrix and the hyperspectral imageYThe corresponding relation is cut out and is used for high-spectrum imagesYTo obtain a multi-spectral imageZ。
In this embodiment, the left low-rank vector of the hyperspectral image to be generated in step 2)UAnd right low rank vectorVFusion generation of hyperspectral imageXCan be expressed as: x = U × V.
As shown in fig. 3 and 4, this embodiment further provides a system for generating a hyperspectral image based on the hyperspectral and multispectral image fusion described above, including a main mirror 1, a diaphragm 2, a light splitter 3, a collimating lens 4, a light filter array and a microlens array 5, a hyperspectral image sensor 6, a reflective mirror 7, an imaging lens 8 and a multispectral image sensor 9, where light captured by the main mirror 1 passes through the diaphragm 2 and the light splitter 3 and is divided into two paths, and one path of light passes through the collimating lens 4, the light filter array and the microlens array 5 in sequence and is imaged on the hyperspectral image sensor 6 to obtain an original image frame in a hyperspectral video; one path of light rays sequentially pass through the reflector 7 and the imaging lens 8 and then are imaged on the multispectral image sensor 9 to obtain an original image frame in the multispectral video. As a specific implementation manner, in this embodiment: the primary mirror 1 is a variable-focus optical lens and is used for capturing light rays emitted by a target and carrying out primary imaging at the diaphragm 2; the diaphragm 2 is an entity which plays a role in limiting light beams and limits the size of an imaging range of the primary mirror 1; the light splitting device 3 is a light splitting prism and is used for splitting the light beam limited by the diaphragm M into two identical and mutually perpendicular beams, one beam is emitted to the collimating lens 4, and the other beam is emitted to the reflector 7; the collimating lens 4 is used for maintaining the collimation of the light beams emitted to the filter array and the micro lens array; the filter array and microlens array 5 includes: the optical filter array: the optical filter can be used for selecting a required radiation waveband, wherein the optical filter array is an array formed by 63 optical filters with different selected radiation wavebands of 400-1000nm; the micro-lens array is an array consisting of 63 lenses with clear apertures and micron-sized relief depths, each micro-lens can independently perform primary imaging, and an imaging surface is positioned on a photosensitive surface of the hyperspectral image sensor 6; the hyperspectral image sensor and the panchromatic camera receive optical signals and convert the optical signals into electric signals for acquiring hyperspectral video data; the reflecting mirror 7 is used for reflecting the other light beam split by the light splitting device 3, changing the direction of the light path and enabling the light path to emit to the imaging lens 8; the imaging lens 8 is used for gathering light rays and imaging, and an imaging surface is positioned on a photosensitive surface of the multispectral image sensor 9; the multispectral image sensor 9 and the RGB camera receive optical signals and convert the optical signals into electric signals for collecting multispectral video data. The hyperspectral image sensor 6 and the multispectral image sensor 9 are connected to the processing and calculating unit through the data acquisition module, namely, the steps of the method for generating the hyperspectral image based on the hyperspectral and multispectral image fusion can be executed through the processing and calculating unit.
In addition, the present embodiment also provides a system for generating a hyperspectral image based on hyperspectral and multispectral image fusion, which includes a processing and computing unit and a memory, which are connected to each other, and the processing and computing unit is programmed or configured to execute the steps of the method for generating a hyperspectral image based on hyperspectral and multispectral image fusion. It should be noted that the processing and computing unit herein may be a single microprocessor, or may be a combination of a microprocessor and its acceleration processor. For example, the system in this embodiment further includes a small artificial intelligence computing motherboard, and adopts a CPU + GPU combination: the CPU is a Quad-core ARM A57, and the main frequency is 1.43GHz; the GPU adopts NVIDIA Maxwell architecture, has 128 NVIDIA CUDA core, and can provide Application Programming Interface (API) for AI and Computer Vision (Computer Vision). In addition, the memory of this system is for including 4GB 64 bit LPDDR4, provides 1 power source interface, 1 HDMI interface, 1 DP interface, 4 USB 3.0 Type A interfaces, 1 ethernet interface, 1 Micro B USB interface, 2 camera connectors and a plurality of interfaces that supply to develop and use, and mechanical dimensions is: 100mm X80 mm X29 mm. As shown in fig. 5, the target emits light and is captured by the primary mirror, and then subjected to high-spectrum and multi-spectrum imaging by the system for generating a hyperspectral image based on hyperspectral and multi-spectrum image fusion described above. After the imaging sensors (the hyperspectral image sensor 6 and the multispectral image sensor 9) receive a control instruction for starting acquisition sent by the processing and calculating unit, the imaging sensors enter a working mode, convert optical signals on the photosensitive surface into electric signals, and transmit hyperspectral and multispectral video data to the processing and calculating unit through a data link. The processing and calculating unit performs processing by using the method for generating a hyperspectral image based on hyperspectral and multispectral image fusion (referred to as spatial-spectral super-resolution intelligent fusion for short) in the embodiment to obtain a hyperspectral image with high resolution, and further obtain video data. And if the follow-up processing is needed, the processing and calculating unit carries out follow-up intelligent processing according to the deployed algorithm and outputs a result, and if the follow-up processing is not needed, the high-resolution hyperspectral video image data are directly output. And after receiving the control instruction for stopping collecting, which is sent by the processing and calculating unit, the imaging sensors (the hyperspectral image sensor 6 and the multispectral image sensor 9) stop working, enter a standby mode, wait for the next control instruction, and otherwise maintain the working mode all the time. In addition, a wireless communication module can be carried on the processing and calculating unit according to needs, wireless communication with an upper computer is achieved, processing results are transmitted, or image fusion processing is carried out by the upper computer to obtain a high-resolution hyperspectral image. The system for generating the hyperspectral image based on the fusion of the hyperspectral image and the multispectral image integrates the functions of hyperspectral video acquisition and intelligent processing, and is wide in application range. In conclusion, the embodiment provides a system integrating functions of hyperspectral video acquisition and intelligent processing, realizes hyperspectral video acquisition and spatial-spectral super-resolution intelligent real-time processing, and has the advantages of high acquisition speed, strong anti-noise capability and wide application range.
Furthermore, the present embodiment also provides a computer-readable storage medium, in which a computer program is stored, which is programmed or configured by the processing and computing unit to perform the steps of the aforementioned method for generating a hyperspectral image based on hyperspectral and multispectral image fusion.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processing computing unit of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing computing unit of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and the scope of the present invention is not limited to the above embodiments, and all technical solutions that belong to the idea of the present invention belong to the scope of the present invention. It should be noted that modifications and embellishments within the scope of the invention may occur to those skilled in the art without departing from the principle of the invention, and are considered to be within the scope of the invention.

Claims (10)

1. A method for generating a hyperspectral image based on hyperspectral and multispectral image fusion is characterized by comprising the following steps:
1) The input hyperspectral imageYPerforming subspace decomposition to extract a left low-rank vector U of the hyperspectral image to be generated; multispectral image to be inputZGenerating a right low-rank vector V of a hyperspectral image to be generated through a trained depth generation network;
2) Left low rank vector of hyperspectral image to be generatedUAnd right low rank vectorVFusion generation of hyperspectral imagesX。
2. The method according to claim 1, wherein the depth generation network in step 1) comprises a forward operation unit and a reverse operation unit, wherein the forward operation unit and the reverse operation unit are both composed of a plurality of two-dimensional convolution layers which are sequentially cascaded, and the plurality of two-dimensional convolution layers of the forward operation unit are arranged from large to small in scaleThe scale of a plurality of two-dimensional convolution layers of the reverse operation unit is arranged from small to large, the two-dimensional convolution layers with the same scale between the forward operation unit and the reverse operation unit are connected in a jumping connection mode, the adjacent two-dimensional convolution layers in the reverse operation unit are connected through corresponding up-sampling and down-sampling operation, and the multispectral image is formed by arranging the plurality of two-dimensional convolution layers in a small scale, wherein the two-dimensional convolution layers with the same scale between the forward operation unit and the reverse operation unit in a jumping connection mode, the two-dimensional convolution layers in the reverse operation unit are connected through corresponding up-sampling and down-sampling operationZThe two-dimensional convolutional layer with the maximum scale in the forward operation unit is input, and the two-dimensional convolutional layer with the maximum scale in the reverse operation unit is output to form a right low-rank vectorV。
3. The method according to claim 2, wherein each two-dimensional convolution layer comprises a two-dimensional convolution, a two-dimensional batch normalization, a linear rectification and a downsampling in sequence.
4. The method for generating a hyperspectral image based on fusion of the hyperspectral and multispectral images according to claim 3, wherein the loss function used for the training of the depth generation network in the step 1) is Root Mean Square Error (RMSE).
5. The method for generating hyperspectral image based on fusion of the hyperspectral and multispectral images according to claim 1, wherein the hyperspectral image input in step 1) isYPerforming subspace decomposition specifically refers to decomposing the input hyperspectral imageYPerforming singular value decomposition, and taking a left singular vector obtained by singular value decomposition as a left low-rank vector U of the hyperspectral image to be generated; and the input hyperspectral imageYThe function expression for performing singular value decomposition is:
Y=U 1 Σ 1 V 1
in the above-mentioned formula, the compound has the following structure,U 1 is a matrix of left singular vectors,Σ 1 is a diagonal matrix of singular values,V 1 is the transpose of the matrix of right singular vectors.
6. Root of herbaceous plantsThe method for generating a hyperspectral image based on fusion of the hyperspectral and multispectral images of claim 1, wherein the hyperspectral image in step 1) isYObtaining hyperspectral images for image frames from a hyperspectral video and based on raw image frames in the hyperspectral videoYThe method comprises the following steps: performing image distortion removal operation on an original image frame in a hyperspectral video according to a distortion coefficient matrix of an imaging sensor acquired by a Zhang friend calibration method, cutting images of various spectral bands, aligning and stacking the images according to band sequence to obtain a hyperspectral imageY。
7. The method according to claim 1, wherein the multispectral image of step 1) is generated based on a fusion of the hyperspectral image and the multispectral imageZFor image frames from a multispectral video, and obtaining a multispectral image based on original image frames in the multispectral videoZThe method comprises the following steps: carrying out image distortion removal operation on original image frames in the multispectral video according to a distortion coefficient matrix of an imaging sensor acquired by a Zhang-friend calibration method, and carrying out image distortion removal operation according to the distortion coefficient matrix and the hyperspectral imageYThe corresponding relation is cut out and is used for high-spectrum imagesYTo obtain a multi-spectral imageZ。
8. A system for generating a hyperspectral image based on hyperspectral and multispectral image fusion according to any one of claims 1 to 7 is characterized by comprising a main mirror (1), a diaphragm (2), a light splitting device (3), a collimating lens (4), a light filter array and a micro lens array (5), a hyperspectral image sensor (6), a reflector (7), an imaging lens (8) and a multispectral image sensor (9), wherein light captured by the main mirror (1) is divided into two paths after passing through the diaphragm (2) and the light splitting device (3), and one path of light is imaged on the hyperspectral image sensor (6) after passing through the collimating lens (4), the light filter array and the micro lens array (5) in sequence to obtain an original image frame in a hyperspectral video; one path of light rays sequentially pass through the reflector (7) and the imaging lens (8) and then are imaged on the multispectral image sensor (9) to obtain an original image frame in the multispectral video.
9. A system for generating a hyperspectral image based on fusion of a hyperspectral and multispectral image, comprising a processing computing unit and a memory connected to each other, characterized in that the processing computing unit is programmed or configured to perform the steps of the method for generating a hyperspectral image based on fusion of a hyperspectral and multispectral image as claimed in any of the claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which is adapted to be programmed or configured by a processing computing unit to perform the steps of the method of generating a hyperspectral image based on hyperspectral and multispectral image fusion according to any of claims 1 to 7.
CN202211118069.5A 2022-09-14 2022-09-14 Method and system for generating hyperspectral image based on hyperspectral and multispectral image fusion Pending CN115456923A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211118069.5A CN115456923A (en) 2022-09-14 2022-09-14 Method and system for generating hyperspectral image based on hyperspectral and multispectral image fusion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211118069.5A CN115456923A (en) 2022-09-14 2022-09-14 Method and system for generating hyperspectral image based on hyperspectral and multispectral image fusion

Publications (1)

Publication Number Publication Date
CN115456923A true CN115456923A (en) 2022-12-09

Family

ID=84303695

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211118069.5A Pending CN115456923A (en) 2022-09-14 2022-09-14 Method and system for generating hyperspectral image based on hyperspectral and multispectral image fusion

Country Status (1)

Country Link
CN (1) CN115456923A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115760670A (en) * 2023-01-06 2023-03-07 之江实验室 Unsupervised hyperspectral fusion method and unsupervised hyperspectral fusion device based on network implicit prior
CN116245779A (en) * 2023-05-11 2023-06-09 四川工程职业技术学院 Image fusion method and device, storage medium and electronic equipment
CN116385841A (en) * 2023-02-28 2023-07-04 南京航空航天大学 Multi-mode ground object target identification method based on knowledge graph

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115760670A (en) * 2023-01-06 2023-03-07 之江实验室 Unsupervised hyperspectral fusion method and unsupervised hyperspectral fusion device based on network implicit prior
CN116385841A (en) * 2023-02-28 2023-07-04 南京航空航天大学 Multi-mode ground object target identification method based on knowledge graph
CN116385841B (en) * 2023-02-28 2023-11-21 南京航空航天大学 Multi-mode ground object target identification method based on knowledge graph
CN116245779A (en) * 2023-05-11 2023-06-09 四川工程职业技术学院 Image fusion method and device, storage medium and electronic equipment
CN116245779B (en) * 2023-05-11 2023-08-22 四川工程职业技术学院 Image fusion method and device, storage medium and electronic equipment

Similar Documents

Publication Publication Date Title
CN115456923A (en) Method and system for generating hyperspectral image based on hyperspectral and multispectral image fusion
Xiong et al. Hscnn: Cnn-based hyperspectral image recovery from spectrally undersampled projections
CN104776815B (en) A kind of color three dimension contour outline measuring set and method based on Darman raster
KR102269229B1 (en) Lensless Hyperspectral Imaging Method and Apparatus Therefore
US10302491B2 (en) Imaging method and apparatus
CN111670576A (en) Light field image processing method for depth acquisition
JP2013535931A (en) Reduced image acquisition time for compression imaging devices
JP7227969B2 (en) Three-dimensional reconstruction method and three-dimensional reconstruction apparatus
EP2795575B1 (en) Integrated three-dimensional vision sensor
JP7171254B2 (en) Image processing device, imaging device, and image processing method
CN110114646A (en) The dynamic Hyper spectral Imaging of object in apparent motion
CN106404171A (en) Liquid crystal hyperspectral calculation imaging measurement device and method of three-dimensional encoding
CN111866387B (en) Depth image imaging system and method
CN114757831B (en) High-resolution video hyperspectral imaging method, device and medium based on intelligent space-spectrum fusion
US11544813B2 (en) Artificial neural network model and electronic device including the same
CN107436194A (en) A kind of high light flux real time spectrum imaging device
Marquez et al. Snapshot compressive spectral depth imaging from coded aberrations
CN114595636A (en) Monocular snapshot type depth polarization four-dimensional imaging method and system
CN113534596A (en) RGBD stereo camera and imaging method
US20220103797A1 (en) Integrated Spatial Phase Imaging
EP3143583B1 (en) System and method for improved computational imaging
Gundogan et al. Computational spectral imaging with diffractive lenses and spectral filter arrays
KR101986998B1 (en) Hyperspectral Imaging Device
CN114353946B (en) Diffraction snapshot spectrum imaging method
CN112802068A (en) Device for synchronously acquiring depth, color and spectrum images and identification system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination