CN113486906B - Mosaic space spectrum gradient direction histogram extraction method of snapshot spectrum image - Google Patents
Mosaic space spectrum gradient direction histogram extraction method of snapshot spectrum image Download PDFInfo
- Publication number
- CN113486906B CN113486906B CN202110767678.2A CN202110767678A CN113486906B CN 113486906 B CN113486906 B CN 113486906B CN 202110767678 A CN202110767678 A CN 202110767678A CN 113486906 B CN113486906 B CN 113486906B
- Authority
- CN
- China
- Prior art keywords
- spectrum
- spatial
- gradient
- neighborhood
- sfa
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001228 spectrum Methods 0.000 title claims abstract description 314
- 238000000605 extraction Methods 0.000 title claims abstract description 29
- 230000003595 spectral effect Effects 0.000 claims abstract description 94
- 238000000034 method Methods 0.000 claims abstract description 23
- 239000011159 matrix material Substances 0.000 claims description 46
- 238000013461 design Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 7
- 238000010276 construction Methods 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 9
- 238000006243 chemical reaction Methods 0.000 abstract description 4
- 230000008707 rearrangement Effects 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 7
- 238000012545 processing Methods 0.000 description 6
- 238000003384 imaging method Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000000701 chemical imaging Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000001174 ascending effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 235000021391 short chain fatty acids Nutrition 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
- 238000012306 spectroscopic technique Methods 0.000 description 1
Abstract
The invention discloses a mosaic space spectrum gradient direction histogram extraction method of a snapshot spectrum image, which comprises the steps of obtaining a spectrum filter array used during shooting of the snapshot spectrum image; constructing a spatial SFA neighborhood and a spectral SFA neighborhood of pixels in the snapshot type spectral image according to the spectral filter array; determining a spatial spectrum gradient operator of the snapshot spectrum image based on a spatial SFA neighborhood and a spectrum SFA neighborhood of pixels in the snapshot spectrum image; convolving the snapshot spectrum image by using a spatial spectrum gradient operator to obtain a mosaic spatial spectrum gradient map; building a mosaic empty spectrum gradient direction histogram based on the mosaic empty spectrum gradient map; according to the invention, by analyzing the SFA mode characteristics of the mosaic image, four mosaic spatial spectrum gradient operators are designed to extract the mosaic spatial spectrum gradient characteristics, demosaicing or rearrangement operation is not required to be carried out on the acquired original mosaic image, and the influence of spatial spectrum distortion caused by the conversion process is effectively eliminated.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a mosaic spatial spectrum gradient direction histogram extraction method of a snapshot spectrum image.
Background
Spectral imaging techniques combine the spectroscopic techniques with the imaging techniques to obtain a plurality of sequential spectral images. The spectral image records the spatial information and the spectral information of each object, and can provide the material or material information of the objects in the scene, so that the characteristics of the objects can be described with higher precision and detail, and the spectral image is widely used in various fields of military, industry, agriculture and the like. However, since the spectral image contains a large amount of information, the photographing time is long, and it is difficult to photograph a video. With the update of technology, snapshot mosaic spectrum imaging technology with high compactness, low cost and high acquisition rate has been developed, and can collect motion scenes at a video rate, thus providing possibility for video tasks. This snapshot imaging technique covers a single sensor surface with a spectral filter array (spectral filter array, SFA) that can collect multiple spectral bands in a single snapshot. The SFA is composed of a basic repeating pattern, each filtering position only allows to record spectral information of one band, and the acquired two-dimensional SFA image is called a mosaic spectral image, which may also be called a snapshot spectral image.
Currently, a series of snapshot spectral sensors with different SFAs have been developed, such as a color filter array (color filter array, CFA) in color imaging and a multispectral filter array (multi-SFA, MSFA) in multispectral/hyperspectral imaging. Since the acquired image has a mosaic structure, it is generally necessary to convert the two-dimensional raw mosaic image into a three-dimensional spectral cube by rearranging or demosaicing (e.g., WB) prior to feature extraction.
However, both of these methods may destroy the structural information of the original mosaic image, introduce spatial spectral distortion, and bring about higher calculation and storage costs.
Disclosure of Invention
The invention aims to provide a mosaic spatial spectrum gradient direction histogram extraction method of a snapshot spectrum image, which is used for directly extracting features from an original mosaic image without demosaicing or rearranging the features, so that spatial spectrum distortion is avoided.
The invention adopts the following technical scheme: a mosaic spatial spectrum gradient direction histogram extraction method of a snapshot spectrum image comprises the following steps:
acquiring a spectrum filtering array used in snapshot spectrum image shooting;
constructing a spatial SFA neighborhood and a spectral SFA neighborhood of pixels in the snapshot type spectral image according to the spectral filter array;
determining a spatial spectrum gradient operator of the snapshot spectrum image based on a spatial SFA neighborhood and a spectrum SFA neighborhood of pixels in the snapshot spectrum image;
convolving the snapshot spectrum image by using a spatial spectrum gradient operator to obtain a mosaic spatial spectrum gradient map;
and constructing a mosaic spatial spectrum gradient direction histogram based on the mosaic spatial spectrum gradient map.
Further, determining the spatial gradient operator of the snapshot spectral image includes:
determining a null spectrum horizontal gradient operator and a null spectrum vertical gradient operator according to the space SFA neighborhood;
and determining a null spectrum one-step gradient operator and a null spectrum second-order gradient operator according to the spectrum SFA neighborhood.
Further, determining the spatial horizontal gradient operator and the spatial vertical gradient operator from the spatial SFA neighborhood includes:
constructing a first spatial correlation matrix f of a spatial SFA neighborhood s And a first spectral correlation matrix f k ；
According toDetermining a spatial-spectral horizontal gradient operator, wherein f xk Is a spatial spectrum horizontal gradient operator, f sx According to a first spatial correlation matrix f s Determining;
according toDetermining a spatial-spectral horizontal gradient operator, wherein f yk Is a spatial spectrum vertical gradient operator, f sy According to a first spatial correlation matrix f s And (5) determining.
Further, determining the empty spectrum first-order gradient operator and the empty spectrum second-order gradient operator according to the spectrum SFA neighborhood comprises:
constructing a second spatial correlation matrix f of the spectrum SFA neighborhood sk And a second spectral correlation matrix f λk ；
According toDetermining a spatial spectrum one-step degree operator, wherein f k-l1 Is a spatial spectrum one-step degree operator, f λ1k According to a second spatial correlation matrix f sk Determining;
according toDetermining a spatial spectrum second order gradient operator, wherein f k-l2 Is a space spectrum second order gradient operator, f λ2k According to the second spectral correlation matrix f λk And (5) determining.
Further, convolving the snapshot spectral image with the spatial gradient operator includes:
and convolving the snapshot spectrum image by using a blank spectrum horizontal gradient operator, a blank spectrum vertical gradient operator, a blank spectrum first-step gradient operator and a blank spectrum second-step gradient operator respectively to obtain a blank spectrum horizontal gradient map, a blank spectrum vertical gradient map, a blank spectrum first-order gradient map and a blank spectrum second-order gradient map of the snapshot spectrum image.
Further, constructing a mosaic spatial spectrum gradient direction histogram based on the mosaic spatial spectrum gradient map includes:
constructing a first histogram through a spatial spectrum horizontal gradient map and a spatial spectrum vertical gradient map;
constructing a second histogram through the spatial spectrum horizontal gradient map, the spatial spectrum vertical gradient map and the spatial spectrum first-order gradient map;
constructing a third histogram through the spatial spectrum horizontal gradient map, the spatial spectrum vertical gradient map and the spatial spectrum second-order gradient map;
and connecting the first histogram, the second histogram and the third histogram in series with a third dimension to form a mosaic spatial spectrum gradient direction histogram.
Further, the spatial SFA neighborhood is a set of all pixels within an n×n region formed around the target pixel p, and does not include the target pixel p;
the spectrum SFA neighborhood is a pixel set associated with m wavebands adjacent to the associated waveband of the target pixel p;
wherein m and n are positive integers.
Another technical scheme of the invention is as follows: a mosaic spatial gradient direction histogram extraction device for a snapshot spectrum image, comprising:
the acquisition module is used for acquiring a spectrum filter array used during shooting of the snapshot spectrum image;
the construction module is used for constructing a spatial SFA neighborhood and a spectral SFA neighborhood of pixels in the snapshot type spectral image according to the spectral filter array;
the design module is used for determining a spatial spectrum gradient operator of the snapshot spectrum image based on the spatial SFA neighborhood and the spectrum SFA neighborhood of the pixels in the snapshot spectrum image;
the convolution module is used for convolving the snapshot spectrum image by using the spatial spectrum gradient operator to obtain a mosaic spatial spectrum gradient map;
the building module is used for building the mosaic empty spectrum gradient direction histogram based on the mosaic empty spectrum gradient map.
Further, determining the spatial gradient operator of the snapshot spectral image includes:
determining a null spectrum horizontal gradient operator and a null spectrum vertical gradient operator according to the space SFA neighborhood;
and determining a null spectrum one-step gradient operator and a null spectrum second-order gradient operator according to the spectrum SFA neighborhood.
Another technical scheme of the invention is as follows: the mosaic spatial gradient direction histogram extraction device of the snapshot spectrum image comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the mosaic spatial gradient direction histogram extraction method of the snapshot spectrum image is realized when the processor executes the computer program.
The beneficial effects of the invention are as follows: according to the invention, by analyzing the SFA mode characteristics of the mosaic image, the SFAN concept is provided, and four mosaic spatial spectrum gradient operators are designed according to the SFAN concept to extract the mosaic spatial spectrum gradient characteristics. Because the gradient operator is based on the SFA structural design of the mosaic spectrum sensor, but not the obtained data design, the proposed feature descriptor is not limited by different SFA modes, can be expanded to any SFA mode, does not need to demosaicing or rearrangement operation on the acquired original mosaic image, and effectively eliminates the influence of spatial spectrum distortion caused by the conversion process.
Drawings
FIG. 1 is a schematic flow chart of a mosaic spatial gradient direction histogram extraction method of a snapshot spectrum image according to an embodiment of the invention;
fig. 2 is a schematic diagram of an SFA and a snapshot spectrum image collected by the SFA according to an embodiment of the present invention.
Detailed Description
The invention will be described in detail below with reference to the drawings and the detailed description.
Gradient information has become an important descriptor for applications such as object recognition, tracking, etc., and is widely used. Many gradient features have been developed for common two-dimensional images, such as LBP, SIFT, SURF, HOG and ExHOG, etc. However, the gradient operators of these methods are designed based on spatial correlation, and thus, the original mosaic image spatial aliasing information cannot be accurately described. According to the SFA mode, adjacent pixels of the image represent spectral intensity values of different wavebands, resulting in inconsistent intensity values of the same object, and therefore, the structural properties of the original mosaic image must be considered by the construction feature extraction operator.
The present invention addresses the above-mentioned problems by providing a novel spatial feature extractor for snapshot spectral images, called a mosaic spatial gradient direction histogram (histogram oforiented mosaic spatial-spectral gradient, homsg) descriptor. Considering the characteristics of a spectral filter array (Spectral filter array, SFA), SFA neighborhood (SFA) is proposed to describe the spatial spectral gradient information, and mosaic spatial spectral gradient operators are designed accordingly, finally, a homsg descriptor is constructed to directly extract spatial spectral features from the original mosaic image. Moreover, since the gradient operators are designed based on the SFA structure of the snapshot spectrum sensor instead of the acquired data, the proposed feature descriptors can be extended to any SFA mode.
The embodiment of the invention discloses a mosaic spatial spectrum gradient direction histogram extraction method of a snapshot type spectral image, which is shown in fig. 1 and comprises the following steps:
acquiring a spectrum filtering array used in snapshot spectrum image shooting; constructing a spatial SFA neighborhood and a spectral SFA neighborhood of pixels in the snapshot spectrum image according to the spectral filter array; determining a spatial spectrum gradient operator of the snapshot spectrum image based on a spatial SFA neighborhood and a spectrum SFA neighborhood of pixels in the snapshot spectrum image; convolving the snapshot spectrum image by using the spatial spectrum gradient operator to obtain a mosaic spatial spectrum gradient map; and constructing a mosaic spatial spectrum gradient direction histogram based on the mosaic spatial spectrum gradient map.
It should be understood that the steps in the foregoing embodiments do not mean the order of execution, and the execution order of each process should be determined by the functions and internal logic of the steps, and should not constitute any limitation on the implementation process of the embodiments of the present application.
According to the invention, by analyzing the SFA mode characteristics of the mosaic image, the SFAN concept is provided, and four mosaic spatial spectrum gradient operators are designed according to the SFAN concept to extract the mosaic spatial spectrum gradient characteristics. Because the gradient operator is based on the SFA structural design of the mosaic spectrum sensor, but not the obtained data design, the proposed feature descriptor is not limited by different SFA modes, can be expanded to any SFA mode, does not need to demosaicing or rearrangement operation on the acquired original mosaic image, and effectively eliminates the influence of spatial spectrum distortion caused by the conversion process.
In an embodiment of the present invention, it is first necessary to construct an SFAN of a snapshot spectrum image. There are two forms of SFAN in the raw mosaic spectral image: a spatial SFA neighborhood, i.e., a spatial SFAN (Spa-SFAN), and a Spectral SFA neighborhood, i.e., a Spectral SFAN (Spe-SFAN).
Spa-SFAN is used to describe the spatial spectral correlation of the spatial neighborhood of the pixel, spe-SFAN is used to describe the spatial spectral correlation of the spectral neighborhood of the band to which the pixel is associated. Given a pixel p of an associated band k, its Spa-SFAN associated band set is represented by the spatial ordering of the spectral bands associated with the pixels in the neighborhood, and the Spe-SFAN associated band set is represented by the ascending order of the spectral bands in the neighborhood. As shown in fig. 2 (a), the span-SFAN of the pixel with the associated band of 4 is composed of the pixels with the associated bands of 7, 3, 19, 20, 22, 6, 10 and 8, but of course, more pixels are also possible. The Spe-SFAN is a pixel composition with associated bands of 2, 3, 5 and 6, or more (such as 1, 2, 3, 5, 6 and 7).
Further, it is known that the spatial SFA neighborhood is a set of all pixels within an n×n region formed around the target pixel p, and does not include the target pixel p; the spectrum SFA neighborhood is a pixel set associated with m wavebands adjacent to the associated waveband of the target pixel p; wherein m and n are positive integers, more preferably, m is a positive integer greater than or equal to 2, and n is a positive integer greater than or equal to 3. I.e. Spa-SFAN, focuses on the spatial spectral correlation of the n x n window around the target pixel p. The Spe-SFAN focuses on the spatial spectral correlation of the pixel p with a pre-fixed number of bands closest to the band allocated to the pixel p.
The spatial spectrum correlation of Spa-SFAN and Spa-SFAN of pixels associated with each band is analyzed respectively, and four mosaic spatial spectrum gradient operators are designed: a spatial spectrum horizontal gradient operator and a spatial spectrum vertical gradient operator based on Spa-SFAN, and a spatial spectrum one-step gradient operator and a spatial spectrum second-order gradient operator based on Spe-SFAN. Spatial and spectral correlations are represented in this embodiment using spatial and spectral distances, respectively. The spatial and spectral distances are calculated from the Euclidean distance between pixels and the center wavelength difference of the pixel associated bands, respectively.
In one embodiment of the invention, determining the spatial gradient operator of the snapshot spectral image comprises:
determining a null spectrum horizontal gradient operator and a null spectrum vertical gradient operator according to the space SFA neighborhood; and determining a null spectrum one-step gradient operator and a null spectrum second-order gradient operator according to the spectrum SFA neighborhood.
In Spa-SFAN, all pixels of the mosaic image have the same spatial correlation matrix, while pixels associated with different wavebands have different spectral correlation matrices, and for collecting K wavebands of the mosaic spectral image, there are K spectral correlation matrices in total. In the Spe-SFAN, pixels associated with the same band have the same spatial and spectral correlation matrices, while pixels associated with different bands have different spatial and spectral correlation matrices, for acquiring K bands of mosaic spectral images, there are K spatial and spectral correlation matrices in total.
More specifically, one calculation method of the spatial spectrum horizontal gradient operator and the spatial spectrum vertical gradient operator is as follows: constructing a first spatial correlation matrix f of a spatial SFA neighborhood s And a first spectral correlation matrix f k The method comprises the steps of carrying out a first treatment on the surface of the According toDetermining a spatial-spectral horizontal gradient operator, wherein f xk Is a spatial spectrum horizontal gradient operator, f sx According to a first spatial correlation matrix f s Determining; according toDetermining a spatial-spectral horizontal gradient operator, wherein f yk Is a spatial spectrum vertical gradient operator, f sy According to a first spatial correlation matrix f s And (5) determining.
In one embodiment of the invention, determining a null spectrum one-step gradient operator and a null spectrum second-order gradient operator according to a spectrum SFA neighborhood comprises:
constructing a second spatial correlation matrix f of the spectrum SFA neighborhood sk And a second spectral correlation matrix f λk The method comprises the steps of carrying out a first treatment on the surface of the According toDetermining a spatial spectrum one-step degree operator, wherein f k-l1 Is a spatial spectrum one-step degree operator, f λ1k According to a second spatial correlation matrix f sk Determining; according to->Determining a spatial spectrum second order gradient operator, wherein f k-l2 Is a space spectrum second order gradient operator, f λ2k According to the second spectral correlation matrix f λk And (5) determining.
For ease of understanding, this embodiment will be explained with respect to a Spa-SFAN of a 3×3 neighborhood and a Spe-SFAN of an 8 band neighborhood. First space correlation matrix f in Spa-SFAN s And a first spectral correlation matrix f associated with pixels of band k k And a second spatial correlation matrix f of Spe-SFAN associated with pixels of band k sk And a second spectral correlation matrix f λk The respective expressions are as follows:
first spatial correlation matrixFirst spectral correlation matrix->Second spatial correlation matrix f sk ＝{f sk1 ,f sk2 ,f sk3 ,f sk4 ,1,f sk5 ,f sk6 ,f sk7 ,f sk8 Second spectral correlation matrix f λk ＝{f λk1 ,f λk2 ,f λk3 ,f λk4 ,1,f λk5 ,f λk6 ,f λk7 ,f λk8 }；
Wherein f si ＝1/(1+d si )，i＝1,2，…，8，d si Representing the spatial distance, f, between a pixel p and its Spa-SFAN neighborhood pixel si ki ＝1/1(1+d si-k )，d si-k Is the spectral distance between the band k associated with pixel p and the band associated with the Spa-SFAN neighborhood pixel si; f (f) ski ＝1/(1+d λi )，d λi Representing the spatial distance, f, between pixel p and its Spe-SFAN neighborhood pixel λi λki ＝1/(1+d λi-k )，d λi-k Is the spectral distance between the band k associated with pixel p and the band associated with its neighborhood Spe-SFAN pixel lambdaj, i representing the spatial location of that pixel.
Finally, the spatial spectrum horizontal gradient operator f of the mosaic can be obtained as described above xk And spatial spectrum horizontal gradient operator f yk And a spatial spectrum one-step degree operator f k-l1 And spatial spectrum one-step degree operator f k-l2 Wherein, the method comprises the steps of, wherein,f λ1k ＝{f sk1 ,f sk2 ,f sk3 ,f sk4 ,0,-f sk5 ,-f sk6 ,-f sk7 ,-f sk8 }，f λ2k ＝{f λk1 ,f λk2 ,f λk3 ,f λk4 ,-D k ,f λk5 ,f λk6 ,f λk7 ,f λk8 }，/>
in one embodiment of the invention, convolving the snapshot spectral image with a spatial gradient operator includes: and convolving the snapshot spectrum image by using a blank spectrum horizontal gradient operator, a blank spectrum vertical gradient operator, a blank spectrum first-step gradient operator and a blank spectrum second-step gradient operator respectively to obtain a blank spectrum horizontal gradient map, a blank spectrum vertical gradient map, a blank spectrum first-order gradient map and a blank spectrum second-order gradient map of the snapshot spectrum image. Specifically, since the above calculated spatial spectrum horizontal gradient operator, spatial spectrum vertical gradient operator, spatial spectrum first-step gradient operator and spatial spectrum second-step gradient operator are concepts of corresponding pixels, the spatial spectrum horizontal gradient map, spatial spectrum vertical gradient map, spatial spectrum first-step gradient map and spatial spectrum second-step gradient map of each pixel are obtained after convolution calculation, and then the spatial spectrum horizontal gradient map, spatial spectrum vertical gradient map, spatial spectrum first-step gradient map and spatial spectrum second-step gradient map of the snapshot type spectrum image are obtained by combining the gradient groups of each pixel.
In one embodiment of the present invention, constructing a mosaic spatial gradient direction histogram based on a mosaic spatial gradient map comprises: constructing a first histogram through a spatial spectrum horizontal gradient map and a spatial spectrum vertical gradient map; constructing a second histogram through the spatial spectrum horizontal gradient map, the spatial spectrum vertical gradient map and the spatial spectrum first-order gradient map; constructing a third histogram through the spatial spectrum horizontal gradient map, the spatial spectrum vertical gradient map and the spatial spectrum second-order gradient map; the first histogram, the second histogram and the third histogram are connected in series with a third dimension to form a mosaic spatial spectrum gradient direction histogram (histogram of oriented mosaic spatial-spectral gradient, HOMSSG) for the snapshot mosaic spectral image description.
Further, homsg gradient information may be used as an important descriptor for applications such as object recognition, tracking, and the like.
Aiming at the mosaic spectrum image acquired by the newly developed snapshot imaging sensor in real time, the invention creatively provides a mosaic spatial spectrum gradient direction histogram feature extraction method, which can directly extract spatial spectrum features from the original mosaic image without demosaicing or rearranging the acquired original mosaic image, effectively eliminates the influence of spatial spectrum distortion caused by a conversion process, reduces calculation and storage costs, improves the robustness and the effectiveness of feature extraction, is suitable for real-time video processing, and can improve higher precision for subsequent works such as identification and tracking.
The invention also discloses a mosaic spatial spectrum gradient direction histogram extraction device of the snapshot spectrum image, which comprises: the acquisition module is used for acquiring a spectrum filter array used during shooting of the snapshot spectrum image; the construction module is used for constructing a spatial SFA neighborhood and a spectral SFA neighborhood of pixels in the snapshot type spectral image according to the spectral filter array; the design module is used for determining a spatial spectrum gradient operator of the snapshot spectrum image based on the spatial SFA neighborhood and the spectrum SFA neighborhood of the pixels in the snapshot spectrum image; the convolution module is used for convolving the snapshot spectrum image by using the spatial spectrum gradient operator to obtain a mosaic spatial spectrum gradient map; the building module is used for building the mosaic empty spectrum gradient direction histogram based on the mosaic empty spectrum gradient map.
Specifically, determining the spatial gradient operator of the snapshot spectral image includes: determining a null spectrum horizontal gradient operator and a null spectrum vertical gradient operator according to the space SFA neighborhood; and determining a null spectrum one-step gradient operator and a null spectrum second-order gradient operator according to the spectrum SFA neighborhood.
The invention also discloses a mosaic spatial gradient direction histogram extraction device of the snapshot spectrum image, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor realizes the mosaic spatial gradient direction histogram extraction method of the snapshot spectrum image when executing the computer program.
The extraction device can be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The extraction means may include, but is not limited to, a processor, a memory. Those skilled in the art will appreciate that more or fewer components may be included, or certain components may be combined, or different components may be included, for example, input-output devices, network access devices, etc.
The processor may be a central processing unit (Central Processing Unit, CPU), it may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may in some embodiments be an internal storage unit of the extraction device, such as a hard disk or a memory of the extraction device. The memory may in other embodiments also be an external storage device of the extraction apparatus, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the extraction apparatus. Further, the memory may also comprise both an internal memory unit and an external memory device of the extraction apparatus. The memory is used to store an operating system, application programs, boot loader (BootLoader), data, and other programs, etc., such as program code for the computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.
It should be noted that, because the content of information interaction and execution process between the above devices/modules is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be referred to in the method embodiment section, and will not be described herein again.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional modules is illustrated, and in practical application, the above-described functional allocation may be performed by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional modules in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional modules are only for distinguishing from each other, and are not used for limiting the protection scope of the application. The specific working process of the modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Furthermore, the terms first, second, and the like in the description and in the claims, are used for distinguishing between the descriptions and not necessarily for indicating or implying a relative importance. Reference in the specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in another embodiment," and the like in various places throughout this specification are not necessarily all referring to the same embodiment, but mean "one or more, but not all, embodiments" unless expressly specified otherwise.
The method provided by the embodiment of the application can be applied to terminal equipment such as mobile phones, tablet computers, wearable equipment, vehicle-mounted equipment, augmented reality (augmented reality, AR)/Virtual Reality (VR) equipment, notebook computers, ultra-mobile personal computer (UMPC), netbooks, personal digital assistants (personal digital assistant, PDA) and the like, and the embodiment of the application does not limit the specific types of the terminal equipment.
For example, the terminal device may be a Station (ST) in a WLAN, a cellular telephone, a cordless telephone, a Session initiation protocol (Session InitiationProtocol, SIP) telephone, a wireless local loop (Wireless Local Loop, WLL)) station, a personal digital assistant (Personal Digital Assistant, PDA) device, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, an in-vehicle device, a car networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite radio, a wireless modem card, a television Set Top Box (STB), a customer premise equipment (customer premise equipment, CPE) and/or other devices for communicating over a wireless system and a next generation communication system, e.g. a mobile terminal in a 5G network or a mobile terminal in a future evolved public land mobile network (Public Land Mobile Network, PLMN) network, etc.
By way of example, but not limitation, when the terminal device is a wearable device, the wearable device may also be a generic name for applying wearable technology to intelligently design daily wear, developing wearable devices, such as glasses, gloves, watches, apparel, shoes, and the like. The wearable device is a portable device that is worn directly on the body or integrated into the clothing or accessories of the user. The wearable device is not only a hardware device, but also can realize a powerful function through software support, data interaction and cloud interaction. The generalized wearable intelligent device comprises full functions, large size, and complete or partial functions which can be realized independent of a smart phone, such as a smart watch or a smart glasses, and is only focused on certain application functions, and needs to be matched with other devices such as the smart phone for use, such as various smart bracelets, smart jewelry and the like for physical sign monitoring.
Claims (3)
1. The mosaic spatial spectrum gradient direction histogram extraction method of the snapshot spectrum image is characterized by comprising the following steps of:
acquiring a spectrum filtering array used in snapshot spectrum image shooting;
constructing a spatial SFA neighborhood and a spectral SFA neighborhood of pixels in the snapshot spectrum image according to the spectral filter array;
determining a spatial spectrum gradient operator of the snapshot spectrum image based on a spatial SFA neighborhood and a spectrum SFA neighborhood of pixels in the snapshot spectrum image;
convolving the snapshot spectrum image by using the spatial spectrum gradient operator to obtain a mosaic spatial spectrum gradient map;
building a mosaic empty spectrum gradient direction histogram based on the mosaic empty spectrum gradient map;
determining the spatial gradient operator of the snapshot spectrum image comprises:
determining a null spectrum horizontal gradient operator and a null spectrum vertical gradient operator according to the space SFA neighborhood;
determining a null spectrum one-step gradient operator and a null spectrum second-order gradient operator according to the spectrum SFA neighborhood;
determining a spatial spectrum horizontal gradient operator and a spatial spectrum vertical gradient operator according to the spatial SFA neighborhood comprises:
constructing a first spatial correlation matrix f of the spatial SFA neighborhood s And a first spectral correlation matrix f k The method comprises the steps of carrying out a first treatment on the surface of the First spatial correlation matrixFirst spectral correlation matrix->
According toDetermining a spatial-spectral horizontal gradient operator, wherein f xk Is a spatial spectrum horizontal gradient operator, f sx According to the first spatial correlation matrix f s Determination of->
According toDetermining a spatial-spectral vertical gradient operator, wherein f yk Is a spatial spectrum vertical gradient operator, f sy According to the first spatial correlation matrix f s Determination of->
Determining a null spectrum one-step gradient operator and a null spectrum second-order gradient operator according to the spectrum SFA neighborhood comprises:
constructing a second spatial correlation matrix f of the spectrum SFA neighborhood sk And a second spectral correlation matrix f λk The method comprises the steps of carrying out a first treatment on the surface of the Second spatial correlation matrix f sk ＝{f sk1 ,f sk2 ,f sk3 ,f sk4 ,1,f sk5 ,f sk6 ,f sk7 ,f sk8 Second spectral correlation matrix f λk ＝{f λk1 ,f λk2 ,f λk3 ,f λk4 ,1,f λk5 ,f λk6 ,f λk7 ,f λk8 -a }; wherein f si ＝1/(1+d si )，i＝1,2，…，8，d si Representing the spatial distance, f, between a pixel p and its Spa-SFAN neighborhood pixel si ki ＝1/1(1+d si-k )，d si-k Is the spectral distance between the band k associated with pixel p and the band associated with the Spa-SFAN neighborhood pixel si; f (f) ski ＝1/(1+d λi )，d λi Representing the spatial distance, f, between pixel p and its Spe-SFAN neighborhood pixel λi λki ＝1/(1+d λi-k )， dλi-k Is the spectral distance between the band k associated with pixel p and the band associated with its neighborhood Spe-SFAN pixel λi, i representing the spatial position of the pixel;
according toDetermining the spatial spectrum one-step degree operator, wherein f k-l1 Is a spatial spectrum one-step degree operator, f λ1k According to the second spatial correlation matrix f sk Determining f λ1k ＝{f sk1 ,f sk2 ,f sk3 ,f sk4 ,0,-f sk5 ,-f sk6 ,-f sk7 ,-f sk8 }；
According toDetermining the spatial spectrum second order gradient operator, wherein f k-l2 Is a space spectrum second order gradient operator, f λ2k According to the second spectrum correlation matrix f λk Determining f λ2k ＝{f λk1 ,f λk2 ,f λk3 ,f λk4 ,-D k ,f λk5 ,f λk6 ,f λk7 ,f λk8 }，
Convolving the snapshot spectral image with the spatial gradient operator includes:
convolving the snapshot spectrum image by using the empty spectrum horizontal gradient operator, the empty spectrum vertical gradient operator, the empty spectrum first-order gradient operator and the empty spectrum second-order gradient operator respectively to obtain an empty spectrum horizontal gradient map, an empty spectrum vertical gradient map, an empty spectrum first-order gradient map and an empty spectrum second-order gradient map of the snapshot spectrum image;
the building of the mosaic spatial spectrum gradient direction histogram based on the mosaic spatial spectrum gradient map comprises the following steps:
constructing a first histogram through the spatial spectrum horizontal gradient map and the spatial spectrum vertical gradient map;
constructing a second histogram through the spatial spectrum horizontal gradient map, the spatial spectrum vertical gradient map and the spatial spectrum first-order gradient map;
constructing a third histogram through the spatial spectrum horizontal gradient map, the spatial spectrum vertical gradient map and the spatial spectrum second-order gradient map;
the first histogram, the second histogram and the third histogram are connected in series in third dimension to form a mosaic spatial spectrum gradient direction histogram;
the spatial SFA neighborhood is a set of all pixels within an n x n region formed around the target pixel p and does not include the target pixel p;
the spectrum SFA neighborhood is a pixel set associated with m wave bands adjacent to the associated wave band of the target pixel p;
wherein m and n are positive integers.
2. A mosaic spatial gradient direction histogram extraction device for a snapshot spectrum image, comprising:
the acquisition module is used for acquiring a spectrum filter array used during shooting of the snapshot spectrum image;
the construction module is used for constructing a spatial SFA neighborhood and a spectral SFA neighborhood of pixels in the snapshot type spectral image according to the spectral filter array;
the design module is used for determining a spatial spectrum gradient operator of the snapshot spectrum image based on a spatial SFA neighborhood and a spectrum SFA neighborhood of pixels in the snapshot spectrum image;
the convolution module is used for convolving the snapshot spectrum image by using the spatial spectrum gradient operator to obtain a mosaic spatial spectrum gradient map;
the building module is used for building a mosaic spatial spectrum gradient direction histogram based on the mosaic spatial spectrum gradient map;
determining the spatial gradient operator of the snapshot spectrum image comprises:
determining a null spectrum horizontal gradient operator and a null spectrum vertical gradient operator according to the space SFA neighborhood;
determining a null spectrum one-step gradient operator and a null spectrum second-order gradient operator according to the spectrum SFA neighborhood;
determining a spatial spectrum horizontal gradient operator and a spatial spectrum vertical gradient operator according to the spatial SFA neighborhood comprises:
constructing a first spatial correlation matrix f of the spatial SFA neighborhood s And a first spectral correlation matrix f k The method comprises the steps of carrying out a first treatment on the surface of the First spatial correlation matrixFirst spectral correlation matrix->
According toDetermining a spatial-spectral horizontal gradient operator, wherein f xk Is a spatial spectrum horizontal gradient operator, f sx According to the first spatial correlation matrix f s Determination of->
According toDetermining a spatial-spectral vertical gradient operator, wherein f yk Is a spatial spectrum vertical gradient operator, f sy According to the first spatial correlation matrix f s Determination of->
Determining a null spectrum one-step gradient operator and a null spectrum second-order gradient operator according to the spectrum SFA neighborhood comprises:
constructing a second spatial correlation matrix f of the spectrum SFA neighborhood sk And a second spectral correlation matrix f λk The method comprises the steps of carrying out a first treatment on the surface of the Second spatial correlation matrix f sk ＝{f sk1 ,f sk2 ,f sk3 ,f sk4 ,1,f sk5 ,f sk6 ,f sk7 ,f sk8 Second spectral correlation matrix f λk ＝{f λk1 ,f λk2 ,f λk3 ,f λk4 ,1,f λk5 ,f λk6 ,f λk7 ,f λk8 -a }; wherein f si ＝1/(1+d si )，i＝1,2，…，8，d si Representing the spatial distance, f, between a pixel p and its Spa-SFAN neighborhood pixel si ki ＝1 / 1(1+d si-k )，d si-k Is the spectral distance between the band k associated with pixel p and the band associated with the Spa-SFAN neighborhood pixel si; f (f) ski ＝1/(1+d λi )，d λi Representing pixel p and Spe-SFAN neighborhood pixel lambda i Spatial distance between f λki ＝1/(1+d λi-k )，d λi-k Is the spectral distance between the band k associated with pixel p and the band associated with its neighborhood Spe-SFAN pixel λi, i representing the spatial position of the pixel;
according toDetermining the spatial spectrum one-step degree operator, wherein f k-l1 Is a spatial spectrum one-step degree operator, f λ1k According to the second spatial correlation matrix f sk Determining f λ1k ＝{f sk1 ,f sk2 ,f sk3 ,f sk4 ,0,-f sk5 ,-f sk6 ,-f sk7 ,-f sk8 }；
According toDetermining the spatial spectrum second order gradient operator, wherein f k-l2 Is a space spectrum second order gradient operator, f λ2k According to the second spectrum correlation matrix f λk Determining f λ2k ＝{f λk1 ,f λk2 ,f λk3 ,f λk4 ,-D k ,f λk5 ,f λk6 ,f λk7 ,f λk8 }，
Convolving the snapshot spectral image with the spatial gradient operator includes:
convolving the snapshot spectrum image by using the empty spectrum horizontal gradient operator, the empty spectrum vertical gradient operator, the empty spectrum first-order gradient operator and the empty spectrum second-order gradient operator respectively to obtain an empty spectrum horizontal gradient map, an empty spectrum vertical gradient map, an empty spectrum first-order gradient map and an empty spectrum second-order gradient map of the snapshot spectrum image;
the building of the mosaic spatial spectrum gradient direction histogram based on the mosaic spatial spectrum gradient map comprises the following steps:
constructing a first histogram through the spatial spectrum horizontal gradient map and the spatial spectrum vertical gradient map;
constructing a second histogram through the spatial spectrum horizontal gradient map, the spatial spectrum vertical gradient map and the spatial spectrum first-order gradient map;
constructing a third histogram through the spatial spectrum horizontal gradient map, the spatial spectrum vertical gradient map and the spatial spectrum second-order gradient map;
the first histogram, the second histogram and the third histogram are connected in series in third dimension to form a mosaic spatial spectrum gradient direction histogram;
the spatial SFA neighborhood is a set of all pixels within an n x n region formed around the target pixel p and does not include the target pixel p;
the spectrum SFA neighborhood is a pixel set associated with m wave bands adjacent to the associated wave band of the target pixel p;
wherein m and n are positive integers.
3. A mosaic spatial gradient direction histogram extraction apparatus for a snapshot spectrum image, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a mosaic spatial gradient direction histogram extraction method for a snapshot spectrum image as claimed in claim 1 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110767678.2A CN113486906B (en) | 2021-07-07 | 2021-07-07 | Mosaic space spectrum gradient direction histogram extraction method of snapshot spectrum image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110767678.2A CN113486906B (en) | 2021-07-07 | 2021-07-07 | Mosaic space spectrum gradient direction histogram extraction method of snapshot spectrum image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113486906A CN113486906A (en) | 2021-10-08 |
| CN113486906B true CN113486906B (en) | 2024-01-09 |
Family
ID=77941095
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110767678.2A Active CN113486906B (en) | 2021-07-07 | 2021-07-07 | Mosaic space spectrum gradient direction histogram extraction method of snapshot spectrum image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113486906B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016012980A1 (en) * | 2014-07-24 | 2016-01-28 | Ecole Polytechnique Federale De Lausanne (Epfl) | Compact multifunctional system for imaging spectroscopy |
| CN110211044A (en) * | 2019-05-24 | 2019-09-06 | 北京理工大学 | Multispectral imaging method and device based on demosaicing algorithms and principal component analysis |
| CN110458766A (en) * | 2019-07-11 | 2019-11-15 | 天津大学 | A kind of fast illuminated high spectrum image demosaicing methods |
| GB202008371D0 (en) * | 2020-06-03 | 2020-07-15 | King S College London | Method and system for joint demosaicking and spectral signature estimation |
-
2021
- 2021-07-07 CN CN202110767678.2A patent/CN113486906B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016012980A1 (en) * | 2014-07-24 | 2016-01-28 | Ecole Polytechnique Federale De Lausanne (Epfl) | Compact multifunctional system for imaging spectroscopy |
| CN110211044A (en) * | 2019-05-24 | 2019-09-06 | 北京理工大学 | Multispectral imaging method and device based on demosaicing algorithms and principal component analysis |
| CN110458766A (en) * | 2019-07-11 | 2019-11-15 | 天津大学 | A kind of fast illuminated high spectrum image demosaicing methods |
| GB202008371D0 (en) * | 2020-06-03 | 2020-07-15 | King S College London | Method and system for joint demosaicking and spectral signature estimation |
Non-Patent Citations (2)
| Title |
|---|
| Object Tracking in Hyperspectral-Oriented Video with Fast Spatial-Spectral Features;Lulu Chen et al.;Remote Sens.;全文 * |
| 光谱滤光片阵列进展;赵永强等;激光与光电子学进展;第57卷(第19期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113486906A (en) | 2021-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110827200B (en) | Image super-resolution reconstruction method, image super-resolution reconstruction device and mobile terminal | |
| CN108681994B (en) | Image processing method and device, electronic equipment and readable storage medium | |
| CN113536864A (en) | Gesture recognition method and device, computer readable storage medium and terminal equipment | |
| WO2014164158A1 (en) | Reducing object detection time by utilizing space localization of features | |
| CN108711144B (en) | Augmented reality method and device | |
| CN111369450B (en) | Method and device for removing mole marks | |
| US11948280B2 (en) | System and method for multi-frame contextual attention for multi-frame image and video processing using deep neural networks | |
| CN113507558B (en) | Method, device, terminal equipment and storage medium for removing image glare | |
| WO2023000907A1 (en) | Method and apparatus for determining spectral image, terminal, and storage medium | |
| Zhang et al. | Cloud detection method using CNN based on cascaded feature attention and channel attention | |
| CN115082310B (en) | Super-resolution reconstruction method, device and processing equipment for satellite video | |
| CN115526815A (en) | Image processing method and device and electronic equipment | |
| CN110458766B (en) | Snapshot hyperspectral image demosaicing method | |
| CN115578590A (en) | Image identification method and device based on convolutional neural network model and terminal equipment | |
| CN113486906B (en) | Mosaic space spectrum gradient direction histogram extraction method of snapshot spectrum image | |
| CN114049491A (en) | Fingerprint segmentation model training method, fingerprint segmentation device, fingerprint segmentation equipment and fingerprint segmentation medium | |
| CN116468917A (en) | Image processing method, electronic device and storage medium | |
| CN116109535A (en) | Image fusion method, device and computer readable storage medium | |
| CN112184843B (en) | Redundant data removing system and method for image data compression | |
| WO2022115996A1 (en) | Image processing method and device | |
| CN113313002A (en) | Multi-mode remote sensing image feature extraction method based on neural network | |
| CN113674331A (en) | Image alignment method and apparatus, electronic device, and computer-readable storage medium | |
| US10757390B2 (en) | Method for obtaining at least one sub-aperture image being associated with one view | |
| Liu et al. | Visual attention based hyperspectral imagery visualization | |
| CN111275632B (en) | Multispectral image sharpening method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |