WO2021113791A1 - Heat assisted detection and ranging based on spectropolarimetric imaging - Google Patents
Heat assisted detection and ranging based on spectropolarimetric imaging Download PDFInfo
- Publication number
- WO2021113791A1 WO2021113791A1 PCT/US2020/063521 US2020063521W WO2021113791A1 WO 2021113791 A1 WO2021113791 A1 WO 2021113791A1 US 2020063521 W US2020063521 W US 2020063521W WO 2021113791 A1 WO2021113791 A1 WO 2021113791A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- spectral
- frames
- imaging system
- scene
- spectral information
- Prior art date
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 46
- 238000001514 detection method Methods 0.000 title description 11
- 230000003595 spectral effect Effects 0.000 claims abstract description 83
- 238000000034 method Methods 0.000 claims abstract description 62
- 239000000463 material Substances 0.000 claims abstract description 38
- 230000005855 radiation Effects 0.000 claims abstract description 16
- 238000007619 statistical method Methods 0.000 claims abstract description 11
- 230000010287 polarization Effects 0.000 claims description 33
- 230000004044 response Effects 0.000 claims description 10
- 238000002834 transmittance Methods 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 claims description 2
- 230000009466 transformation Effects 0.000 claims description 2
- 238000001228 spectrum Methods 0.000 description 17
- 108010014173 Factor X Proteins 0.000 description 7
- 238000013459 approach Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000007613 environmental effect Effects 0.000 description 5
- 230000001419 dependent effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000005457 Black-body radiation Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 230000003292 diminished effect Effects 0.000 description 1
- 238000003331 infrared imaging Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
- G01J3/28—Investigating the spectrum
- G01J3/2823—Imaging spectrometer
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
- G01J3/02—Details
- G01J3/10—Arrangements of light sources specially adapted for spectrometry or colorimetry
- G01J3/108—Arrangements of light sources specially adapted for spectrometry or colorimetry for measurement in the infrared range
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
- G01J3/28—Investigating the spectrum
- G01J3/447—Polarisation spectrometry
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J4/00—Measuring polarisation of light
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
- G01J5/48—Thermography; Techniques using wholly visual means
- G01J5/485—Temperature profile
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
- G01J5/59—Radiation pyrometry, e.g. infrared or optical thermometry using polarisation; Details thereof
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/21—Polarisation-affecting properties
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
- G01N21/31—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry
- G01N21/35—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry using infrared light
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2431—Multiple classes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/143—Sensing or illuminating at different wavelengths
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/54—Extraction of image or video features relating to texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/758—Involving statistics of pixels or of feature values, e.g. histogram matching
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
- G01J3/28—Investigating the spectrum
- G01J2003/283—Investigating the spectrum computer-interfaced
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
- G01J3/28—Investigating the spectrum
- G01J2003/283—Investigating the spectrum computer-interfaced
- G01J2003/2833—Investigating the spectrum computer-interfaced and memorised spectra collection
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
- G01J2005/0077—Imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J4/00—Measuring polarisation of light
- G01J4/04—Polarimeters using electric detection means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10036—Multispectral image; Hyperspectral image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
Definitions
- the present disclosure generally relates to automated heat radiation analysis, and in particular, to a computer vision method of creating textures in infrared thermal images.
- LIDAR light detection and ranging
- LIDAR s effectiveness falls rapidly with distance. Additionally, with a growing number of autonomous vehicles, LIDAR detection becomes cumbersome as the same object may be illuminated with multiple LIDARs.
- Thermal images provide completely passive approach to detection and ranging.
- the lack of texture in thermal images leads to lack of discernible object features to the eye or to a computer system. This is a serious issue in using infrared thermal images for autonomous navigation.
- a method of generating object surface texture in thermal infrared images is disclosed.
- the method includes receiving heat radiation from a scene by a spectropolarimetric imaging system adapted to generate a plurality of spectral frames associated with the scene.
- the method also includes generating the plurality of spectral frames associated with the scene, each frame having a plurality of pixels.
- the method includes for each pixel from the generated plurality of spectral frames, extracting spectral information associated with the scene, including pixel- specific temperature representing an object’s temperature, and thermal texture factor representing the object’s texture.
- the method includes for each of a plurality of materials having a specific emissivity in a library, generating reference spectral information as a function of temperature and thermal texture.
- the method includes matching the extracted spectral information for each pixel from the generated plurality of spectral frames to the generated reference spectral information using a statistical method to minimize the associated variation, and extracting spectral metadata from the matched reference spectral information for the associated material based on the match.
- the plurality of spectral frames from the spectropolarimetric imaging system are each generated by applying a plurality of associated bandpass filters to the spectropolarimetric imaging system and passing the heat radiation therethrough.
- the extracted spectral information associated with the scene from the spectropolarimetric imaging system for each pixel from the generated plurality of spectral frames is based on where Ni represents output of the spectropolarimetric imaging system for each application of the associated bandpass filter i, v represents frequency,
- T iv represents transmittance curve of each pixel of the plurality of spectral frames for discretized frequency bands between i ? min and v max for each application of the bandpass filter
- Av represents width of discretized frequency band
- the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library is obtained from: where, K B is the Boltzmann constant, h is the Planck constant, c is the speed of light, m represents a material chosen from the library,
- T represents the temperature of the target
- T 0 represents the ambient temperature
- e vm represents the emissivity of the chosen material
- X represents thermal texture factor
- the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library includes a family of spectral curves i) based on a plurality of temperatures and ii) for each temperature of the plurality of temperatures, based on variation of thermal texture factor (X), wherein the thermal texture factor is a variable between 0 and 1.
- the matching of the extracted spectral information for each pixel from the spectropolarimetric imaging system from the generated plurality of spectral frames to the generated reference spectral information is based on matching
- the statistical method includes sum of least mean squares between the S v and S vm meeting a predetermined threshold.
- the statistical method includes a minimum least mean squares between the S v and S vm .
- the spectropolarimetric imaging system is further adapted to generate a plurality of polarization frames associated within the scene.
- the method further includes generating the plurality of polarization frames associated with the scene, each frame having a plurality of pixels.
- the plurality of linear polarization frames from the spectropolarimetric imaging system includes liner polarization at 0°, 45°, 90°, and -45°, thereby generating Io, Us, ho, and L45 frames.
- for each pixel from the generated plurality of polarization frames further extracting spectral information associated with the scene based on the polarization angles ( S vp ).
- the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library includes a family of spectral curves ( S vmp ) i) based on a plurality of temperatures, ii) for each temperature of the plurality of temperatures, based on variation of thermal texture factor (X), wherein the thermal texture factor is a variable between 0 and 1, and for each thermal texture factor (X), based on variation of polarization angle including 0°, 45°, 90°, and -45°
- the statistical method includes sum of least mean squares between the S vp and S vmp meeting a predetermined threshold.
- the statistical method includes a minimum least mean squares between the S vp and S vmp .
- FIG. 1 is a high level schematic diagram of a HADAR system, including a spectropolarimetric imaging system, according to the present disclosure.
- FIG. 2 is a high-level schematic of outputs of the spectropolarimetric imaging system of FIG. 1.
- FIG. 3 is a flowchart of a method for determining metadata for each pixel of the spectropolarimetric imaging system of FIG. 1, including an antighosting method for determining spectral metadata.
- FIG. 4 is a flowchart for the antighosting method of FIG. 3.
- FIG. 5a is a flowchart presenting specific steps for generating spectral metadata, including a matching method to a predetermined library.
- FIG. 5b is a flowchart depicting one example method for matching to the predetermined library.
- FIGs. 6a -6d represents graphs of spectra (FIGs. 6a, 6c, 6d) and B which is black body radiation (FIG. 6b) vs. frequency in order to demonstrate the steps of matching.
- FIGs. 7a-7d are example images where standard infrared images (FIGs. 7a and 7c), are processed according to the present disclosure, thereby generating surface textures for enhanced visualization (FIGs. 7b and 7d).
- FIG. 8 is a graph of spectra vs. frequency including additional resolution owing to degrees of polarization.
- the term “about” can allow for a degree of variability in a value or range, for example, within 10%, within 5%, or within 1% of a stated value or of a stated limit of a range.
- the term “substantially” can allow for a degree of variability in a value or range, for example, within 90%, within 95%, or within 99% of a stated value or of a stated limit of a range.
- the present disclosure provides a novel approach for object ranging that can be used in a variety of applications including autonomous vehicle navigation.
- This novel approach is based on detection of heat signature of objects, near and far away.
- the present disclosure describes a heat assisted detection and ranging (HADAR) approach which is based on capturing heat radiation - the intrinsic heat signature of a body - and can provide the unique spectral fingerprint for tactical semantic segmentation of scenes.
- HADAR heat assisted detection and ranging
- LWIR long wavelength infrared
- FIG. 1 a high level schematic diagram of a HADAR system 100 is shown.
- the HADAR system 100 is adapted to receive heat signature in the form of IR radiation from objects and direct the radiation to a spectropolarimetric imaging system 104, which is adapted to generate a plurality of frames 106, each including a plurality of pixels. Each pixel is associated with metadata, discussed below.
- a pixel and its associated metadata is referred to herein as a thermal voxel which includes metadata information about temperature, shown as block 108, emissivity spectrum e(w ), shown as block 110, degree of linear polarization (DoLP) and angle of linear polarization (AoLP), shown as block 112, and thermal texture factor X (a concept introduced to identify the intrinsic object thermal photons (signal) vs. extrinsic environmental thermal photons (noise) in the spectral domain entering the spectropolarimetric imaging system 104 which is focused on a target), identified as block 114.
- thermal texture factor X a concept introduced to identify the intrinsic object thermal photons (signal) vs. extrinsic environmental thermal photons (noise) in the spectral domain entering the spectropolarimetric imaging system 104 which is focused on a target
- a processing system either local to the HADAR system 100 or remote therefrom (e.g., a cloud-based system), detects the object, as shown in block 116; and further determines the range (i.e., distance) of the object from the HADAR system 100, as shown in block 118.
- a processing system either local to the HADAR system 100 or remote therefrom (e.g., a cloud-based system) detects the object, as shown in block 116; and further determines the range (i.e., distance) of the object from the HADAR system 100, as shown in block 118.
- the optimal feature for HADAR arises from the spatio-temporal dependence (x, y, z, t) of precisely these mentioned thermal voxel properties.
- Temperature and spectral emissivity are intrinsic properties of the thermal voxel whereas the DoLP, AoLP and thermal texture factor X involves a subtle interplay of intrinsic and extrinsic thermal photons.
- Intrinsic photons governed by spectral emissivity are thermally emitted by the target while extrinsic photons are thermally emitted by the environment then reflected off the object and reach the camera which is focused on the object.
- a method In order to determine temperature associated with each thermal voxel, a method according to the present disclosure begins with a first estimate of an average environmental temperature (assumed to be a global constant for all pixels) through an on-board thermometer or GPS -assisted weather data. [0042] In order to determine temperature of each pixel as part of the metadata of the thermal voxel, following the decoupling of intrinsic and extrinsic signals of every pixel, in a first iteration the method of the present disclosure identifies the hottest object and coldest object in the scene. In the second iteration, the environmental temperature is updated locally for every pixel but keeping it within the hot/cold bounds of these two values since the thermal noise from nearby objects dominates the scene.
- a clustering approach is exploited from unsupervised learning to de-noise the data and guide the scene analysis through publicly available atmospheric models (e.g., MODTRAN). It is also possible to segment the pixels according to noise class and identify global vs. local noisy variations in emissivity/temperature.
- MODTRAN publicly available atmospheric models
- FIG. 2 is a high- level schematic of outputs of the spectropolarimetric imaging system 104.
- this type of spectropolarimetric imaging system 104 includes a hyperspectral camera or other types of cameras known to a person having ordinary skill in the art.
- the spectropolarimetric imaging system 104 provides two types of output: 1) polarization frames as shown in the frames 202; and 2) spectral frames, as shown in the frames 204. These polarization and spectral frames (202 and 204) are based on a plurality of different settings.
- the polarization frames 202 include transmitted light from the spectropolarimetric imaging system 104 providing linearly polarized light at 0 degree, thereby generating a raw polarized image frame (Io frame).
- the spectropolarimetric imaging system 104 is adapted to provide linear polarization at 45°.
- a raw polarized image frame (I45 frame) is thus obtained.
- the spectropolarimetric imaging system 104 is further adapted to provide a linear polarization at 90°.
- a raw polarized image frame (1 ⁇ 2 frame) is thus obtained.
- the spectropolarimetric imaging system 104 is further adapted to provide linear polarization at 135° (-45°).
- the spectral frames 204 include transmitted light from the spectropolarimetric imaging system 104 providing spectral frames at a plurality of different spectral frequencies based on application of a plurality of bandpass filters each with a bandwidth (v / Vh). According to one embodiment, nine bandpass filters are applied each providing a spectral frame. These example-only spectral frames are shown in the frames 204. [0045] Reference is now made to FIG. 3, where a method 300 of determining metadata for each pixel is shown.
- polarization tunability is shown with respect to polarization frames 304 (including, as discussed above according to one embodiment, fames h, Us, ho, and L45 ).
- DoLP and AoLP are then assigned based on these mathematical operations (see below) as shown in blocks 314 and 316. From these stokes parameters, the DoLP map and AoLP map are calculated from the three Stokes parameters maps.
- spectral frames are obtained from the spectropolarimetric imaging system 104, as discussed above (i.e., a plurality, e.g., 9, bandpass filters are applied to the spectropolarimetric imaging system 104, each generating a bandlimited response from the scene).
- a plurality e.g. 9, bandpass filters are applied to the spectropolarimetric imaging system 104, each generating a bandlimited response from the scene.
- the radiation spectrum is reconstructed using an antighosting algorithm, discussed further below, from which in blocks 326, 328, and 330 the remaining metadata (T, e(w), and Thermal Lightning Factor X) are determined.
- the antighosting algorithm, referenced in block 324, is further disclosed in FIG. 4.
- an antighosting method 400 is presented. Initially, in block
- a plurality of bandlimited bandpass filters are applied to the incoming heat radiation.
- nine such bandpass filters are applied.
- Each of these filters results in a frame including intensity between a lower bandpass level and an upper bandpass level.
- the frame includes intensity for vu and Vhi , where vu is the lower bandpass level, and vm is the upper bandpass level.
- these spectral image frames are shown in block 404.
- the dark noise of the spectropolarimetric imaging system 104 see FIG.
- the transmittance curve T iv of each spectral filter i is characterized, which is a function of frequency (v).
- This transmittance curve T iv measurement is performed once using a spectrometer (i.e., frequency/wavelength bands that are generated each time a bandpass filter is applied. Measurements are made by identifying a band of frequency that passes through the filter.
- the response curve Z v of each sensor pixel is characterized.
- response of each pixel with respect to the frequency range can be ascertained.
- the response is usually provided by the manufacturer of the spectropolarimetric imaging system 104 (see FIG. 2), or the response be can characterized by a spectrometer, as known to a person having ordinary skill in the art.
- the incident spectrum S v for each i is calculated. The relationship between S v and the other parameters discussed above is provided below:
- Ni / Z v T iv S v dv, where Ni is the output of the spectropolarimetric imaging system 104 (see FIG. 2),
- T iv is the transmittance curve of each raw spectral frames
- heat radiation is expressed in terms of a pseudo-3-dimensional curve S vm which is dependent on and thermal factor X and T as provided in the equation below.
- Thermal factor X is a parameter between 0 and 1 which defines surface texture of a material.
- T represents the temperature of the target
- T 0 represents the ambient temperature
- e vm represents the emissivity of the chosen material
- X represents thermal texture factor
- one example algorithm is provided showing a method 550 for matching the obtained spectrum S v against the library spectrum S vm .
- the method 550 begin in block 552 with the obtained spectrum S v .
- a material from the library is chosen representing the material in the scene which the heat images are obtained from.
- the material spectrum S vm is governed by the equation provided above, as provided in block 554.
- These two spectra represent three dimensional curves, as discussed above.
- the two spectra are compared with each other, as provided in block 556. Using a comparison technique, e.g., least-square-error, as shown in block 558, a distinction between these two spectra is determined.
- This distinction is compared to a threshold, as shown in query 560. If the threshold is not met, then the method 550 chooses another material, as provided in block 562 and returns to block 554. If the threshold is met, then a match is ascertained and the method 550 proceeds to identifying the remaining metadata, as shown in block 564.
- S vm for all materials in the library are determined.
- comparisons between S vm and S v can be made and using a least square error method (sum of the least square errors in the aforementioned comparison) the minimum of least squares is chosen as a positive match to a selected material in the library.
- FIGs. 6a-6d To demonstrate these techniques, example graphs are provided in FIGs. 6a-6d.
- the spectrum S v of the image from the scene is shown as a function of frequency in FIG. 6a (see block 416 of FIG. 4).
- B which is black body radiation
- S vm for different materials is obtained by plotting S vm as a function of frequency for different temperatures and different X values.
- the temperature can be chosen based on a priori knowledge of the ambient temperature. As discussed X values range between 0 and 1.
- FIGs. 7a-7d two images are shown using standard infrared imaging (FIGs. 7a and 7c), while FIGs. 7b and 7d are processed images utilizing the methods described above thereby generating surface textures for enhanced visualization ⁇
- spectral metadata are used to establish a match between the family of spectral curves of a material in the library based on temperature and thermal texture factor
- additional accuracy can be achieved by adding linear polarization data into the mix.
- emissivity, and B block body radiation
- each family of curves is not only dependent on temperature, frequency but also on degree of polarization. Referring to FIG. 8, an example of this additional resolution based on degree of polarization is shown. With this added resolution, a better match can be made to the spectral information from the scene. For example, in FIG.
- a family of curves are provided for a plurality of polarizations angles. For example, according to one embodiment, if there are 4 polarization angles (i.e., 0, 45, 90, and -45), there would be four spectral curves. This additional parameter, allows added accuracy by a factor equal to the number of angles of polarization.
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Biochemistry (AREA)
- Immunology (AREA)
- Pathology (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Biology (AREA)
- Radiation Pyrometers (AREA)
Abstract
A method of generating object surface texture in thermal infrared images is disclosed which includes receiving heat radiation from a scene by a spectropolarimetric imaging system, generating a plurality of spectral frames associated with the scene, each frame having a plurality of pixels, for each pixel from the generated plurality of spectral frames, extracting spectral information associated with the scene, including pixel-specific temperature representing an object's temperature, and thermal texture factor representing the object's texture, for each of a plurality of materials having a specific emissivity in a library, generating reference spectral information as a function of temperature and thermal texture, matching the extracted spectral information for each pixel from the generated plurality of spectral frames to the generated reference spectral information using a statistical method to minimize the associated variation, and extracting spectral metadata from the matched reference spectral information for the associated material based on the match.
Description
HEAT ASSISTED DETECTION AND RANGING BASED ON
SPECTROPOLARIMETRIC IMAGING
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] The present patent application is related to and claims the priority benefit of U.S. Provisional Patent Application Serial No. 62/944,671, filed December 06, 2019, the contents of which are hereby incorporated by reference in its entirety into the present disclosure.
STATEMENT REGARDING GOVERNMENT FUNDING
[0002] This invention was made with government support under N66001-17-1-4048 awarded by the Defense Advanced Research Projects Agency. The government has certain rights in the invention.
TECHNICAU FIEUD
[0003] The present disclosure generally relates to automated heat radiation analysis, and in particular, to a computer vision method of creating textures in infrared thermal images.
BACKGROUND
[0004] This section introduces aspects that may help facilitate a better understanding of the disclosure. Accordingly, these statements are to be read in this light and are not to be understood as admissions about what is or is not prior art.
[0005] Autonomous vehicle navigation technologies are becoming more commonplace in certain vehicles. Object detection and ranging (distance of object from the vehicle) is of vital importance for these technologies. However, object detection and ranging proves to be specially challenging given the wide-ranging obstacles (e.g., road signs, road objects, other vehicles, etc.), harsh environments (bright daylight as well as pitch blackness of dark nights), and unknown terrains encountered in scenarios such as off-road driving.
[0006] The technology of choice for detection and ranging is LIDAR (light detection and ranging). LIDAR is based on illuminating an object by a laser and measuring laser return times and wavelength. These parameters can be used to generate a three dimensional (3D) map of the object and its distance from the vehicle (i.e., source of the illuminating laser). However,
LIDAR’ s effectiveness falls rapidly with distance. Additionally, with a growing number of autonomous vehicles, LIDAR detection becomes cumbersome as the same object may be illuminated with multiple LIDARs.
[0007] An alternative technology involves use of passive 3D vision, which use optical (visible) stereovision; where cost-effective red-green-blue (RGB) cameras are used for scene analysis. This approach however suffers from challenges associated with stereovision, where the errors accumulated in ranging (i.e. depth estimation) increase quadratically with distance. Furthermore, there exist no systematic procedures for target recognition or semantic segmentation for applications such as off-road navigation.
[0008] Thermal images provide completely passive approach to detection and ranging. However, the lack of texture in thermal images leads to lack of discernible object features to the eye or to a computer system. This is a serious issue in using infrared thermal images for autonomous navigation.
[0009] Therefore, there is an unmet need for improving and creating textures in an infrared thermal image to address object detection, classification, and ranging for autonomous vehicle navigation.
SUMMARY
[0010] A method of generating object surface texture in thermal infrared images is disclosed.
The method includes receiving heat radiation from a scene by a spectropolarimetric imaging system adapted to generate a plurality of spectral frames associated with the scene. The method also includes generating the plurality of spectral frames associated with the scene, each frame having a plurality of pixels. Furthermore, the method includes for each pixel from the generated plurality of spectral frames, extracting spectral information associated with the scene, including pixel- specific temperature representing an object’s temperature, and thermal texture factor representing the object’s texture. Additionally, the method includes for each of a plurality of
materials having a specific emissivity in a library, generating reference spectral information as a function of temperature and thermal texture. Furthermore, the method includes matching the extracted spectral information for each pixel from the generated plurality of spectral frames to the generated reference spectral information using a statistical method to minimize the associated variation, and extracting spectral metadata from the matched reference spectral information for the associated material based on the match.
[0011] According to one embodiment of the method, the plurality of spectral frames from the spectropolarimetric imaging system are each generated by applying a plurality of associated bandpass filters to the spectropolarimetric imaging system and passing the heat radiation therethrough.
[0012] According to one embodiment of the method, the extracted spectral information associated with the scene from the spectropolarimetric imaging system for each pixel from the generated plurality of spectral frames is based on
where Ni represents output of the spectropolarimetric imaging system for each application of the associated bandpass filter i, v represents frequency,
Tiv represents transmittance curve of each pixel of the plurality of spectral frames for discretized frequency bands between i ?min and vmax for each application of the bandpass filter, Av represents width of discretized frequency band,
Zv represents response curve of each pixel of the plurality of the spectral frames, is a transformation matrix for each application of the bandpass filter, and where Sv represents the extracted spectral information associated with the scene, where sv = (M i - Ni.
[0013] According to one embodiment of the method, the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library is obtained from:
where, KB is the Boltzmann constant, h is the Planck constant, c is the speed of light, m represents a material chosen from the library,
T represents the temperature of the target,
T0 represents the ambient temperature, evm represents the emissivity of the chosen material, and
X represents thermal texture factor.
[0014] According to one embodiment of the method, the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library includes a family of spectral curves i) based on a plurality of temperatures and ii) for each temperature of the plurality of temperatures, based on variation of thermal texture factor (X), wherein the thermal texture factor is a variable between 0 and 1.
[0015] According to one embodiment of the method, the matching of the extracted spectral information for each pixel from the spectropolarimetric imaging system from the generated plurality of spectral frames to the generated reference spectral information is based on matching
S i; tO Svm.
[0016] According to one embodiment of the method, the statistical method includes sum of least mean squares between the Sv and Svm meeting a predetermined threshold.
[0017] According to one embodiment of the method, the statistical method includes a minimum least mean squares between the Sv and Svm.
[0018] According to one embodiment of the method, the spectropolarimetric imaging system is further adapted to generate a plurality of polarization frames associated within the scene.
[0019] According to one embodiment the method further includes generating the plurality of polarization frames associated with the scene, each frame having a plurality of pixels.
[0020] According to one embodiment of the method, the plurality of linear polarization frames from the spectropolarimetric imaging system includes liner polarization at 0°, 45°, 90°, and -45°, thereby generating Io, Us, ho, and L45 frames.
[0021] According to one embodiment of the method, for each pixel from the generated plurality of polarization frames, further extracting spectral information associated with the scene based on the polarization angles ( Svp ).
[0022] According to one embodiment of the method, the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library includes a family of spectral curves ( Svmp ) i) based on a plurality of temperatures, ii) for each temperature of the plurality of temperatures, based on variation of thermal texture factor (X), wherein the thermal texture factor is a variable between 0 and 1, and for each thermal texture factor (X), based on variation of polarization angle including 0°, 45°, 90°, and -45°
[0023] According to one embodiment of the method, the statistical method includes sum of least mean squares between the Svp and Svmp meeting a predetermined threshold.
[0024] According to one embodiment of the method, the statistical method includes a minimum least mean squares between the Svp and Svmp .
BRIEF DESCRIPTION OF DRAWINGS
[0025] FIG. 1 is a high level schematic diagram of a HADAR system, including a spectropolarimetric imaging system, according to the present disclosure.
[0026] FIG. 2 is a high-level schematic of outputs of the spectropolarimetric imaging system of FIG. 1.
[0027] FIG. 3 is a flowchart of a method for determining metadata for each pixel of the spectropolarimetric imaging system of FIG. 1, including an antighosting method for determining spectral metadata.
[0028] FIG. 4 is a flowchart for the antighosting method of FIG. 3.
[0029] FIG. 5a is a flowchart presenting specific steps for generating spectral metadata, including a matching method to a predetermined library.
[0030] FIG. 5b is a flowchart depicting one example method for matching to the predetermined library.
[0031] FIGs. 6a -6d represents graphs of spectra (FIGs. 6a, 6c, 6d) and B which is black body radiation (FIG. 6b) vs. frequency in order to demonstrate the steps of matching.
[0032] FIGs. 7a-7d are example images where standard infrared images (FIGs. 7a and 7c), are processed according to the present disclosure, thereby generating surface textures for enhanced visualization (FIGs. 7b and 7d).
[0033] FIG. 8 is a graph of spectra vs. frequency including additional resolution owing to degrees of polarization.
DETAILED DESCRIPTION
[0034] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of this disclosure is thereby intended.
[0035] In the present disclosure, the term “about” can allow for a degree of variability in a value or range, for example, within 10%, within 5%, or within 1% of a stated value or of a stated limit of a range.
[0036] In the present disclosure, the term “substantially” can allow for a degree of variability in a value or range, for example, within 90%, within 95%, or within 99% of a stated value or of a stated limit of a range.
[0037] The present disclosure provides a novel approach for object ranging that can be used in a variety of applications including autonomous vehicle navigation. This novel approach is based on detection of heat signature of objects, near and far away. Towards this end, the present disclosure describes a heat assisted detection and ranging (HADAR) approach which is based on capturing heat radiation - the intrinsic heat signature of a body - and can provide the unique spectral fingerprint for tactical semantic segmentation of scenes. Additionally, as infrared heat radiation in the 8-14 micron range - long wavelength infrared, (LWIR) - is omnipresent and can be exploited at day or night.
[0038] In order to use heat signature as a primary source of information for autonomous vehicle navigation, several challenges must be addressed: 1) Ghosting (i.e., since heat radiation is omnipresent, the signal is cluttered with environmental thermal signals that cause diminished
features or textures in thermal images, thereby necessitating new algorithms that distinguish useful target information from the environmental heat signatures that swamp or clutter the scene); and 2) passive ranging accuracy and 3D vision capabilities of IR cameras suffer from the errors fundamental to stereomatching, discussed below.
[0039] Referring to FIG. 1, a high level schematic diagram of a HADAR system 100 is shown. The HADAR system 100 is adapted to receive heat signature in the form of IR radiation from objects and direct the radiation to a spectropolarimetric imaging system 104, which is adapted to generate a plurality of frames 106, each including a plurality of pixels. Each pixel is associated with metadata, discussed below. A pixel and its associated metadata is referred to herein as a thermal voxel which includes metadata information about temperature, shown as block 108, emissivity spectrum e(w ), shown as block 110, degree of linear polarization (DoLP) and angle of linear polarization (AoLP), shown as block 112, and thermal texture factor X (a concept introduced to identify the intrinsic object thermal photons (signal) vs. extrinsic environmental thermal photons (noise) in the spectral domain entering the spectropolarimetric imaging system 104 which is focused on a target), identified as block 114. With the above-identified thermal data associated with each thermal voxel, a processing system (not shown) either local to the HADAR system 100 or remote therefrom (e.g., a cloud-based system), detects the object, as shown in block 116; and further determines the range (i.e., distance) of the object from the HADAR system 100, as shown in block 118. Each of these blocks is described below in detail. [0040] The optimal feature for HADAR arises from the spatio-temporal dependence (x, y, z, t) of precisely these mentioned thermal voxel properties. Temperature and spectral emissivity are intrinsic properties of the thermal voxel whereas the DoLP, AoLP and thermal texture factor X involves a subtle interplay of intrinsic and extrinsic thermal photons. Intrinsic photons governed by spectral emissivity are thermally emitted by the target while extrinsic photons are thermally emitted by the environment then reflected off the object and reach the camera which is focused on the object.
[0041] In order to determine temperature associated with each thermal voxel, a method according to the present disclosure begins with a first estimate of an average environmental temperature (assumed to be a global constant for all pixels) through an on-board thermometer or GPS -assisted weather data.
[0042] In order to determine temperature of each pixel as part of the metadata of the thermal voxel, following the decoupling of intrinsic and extrinsic signals of every pixel, in a first iteration the method of the present disclosure identifies the hottest object and coldest object in the scene. In the second iteration, the environmental temperature is updated locally for every pixel but keeping it within the hot/cold bounds of these two values since the thermal noise from nearby objects dominates the scene. A clustering approach is exploited from unsupervised learning to de-noise the data and guide the scene analysis through publicly available atmospheric models (e.g., MODTRAN). It is also possible to segment the pixels according to noise class and identify global vs. local noisy variations in emissivity/temperature.
[0043] To better elucidate these techniques, reference is now made to FIG. 2, which is a high- level schematic of outputs of the spectropolarimetric imaging system 104. There are several possibilities for this type of spectropolarimetric imaging system 104, including a hyperspectral camera or other types of cameras known to a person having ordinary skill in the art. In any of these cases, the spectropolarimetric imaging system 104 provides two types of output: 1) polarization frames as shown in the frames 202; and 2) spectral frames, as shown in the frames 204. These polarization and spectral frames (202 and 204) are based on a plurality of different settings. For example, according to one embodiment, the polarization frames 202 include transmitted light from the spectropolarimetric imaging system 104 providing linearly polarized light at 0 degree, thereby generating a raw polarized image frame (Io frame). Next, the spectropolarimetric imaging system 104 is adapted to provide linear polarization at 45°. A raw polarized image frame (I45 frame) is thus obtained. Continuing, the spectropolarimetric imaging system 104 is further adapted to provide a linear polarization at 90°. A raw polarized image frame (½ frame) is thus obtained. Next, the spectropolarimetric imaging system 104 is further adapted to provide linear polarization at 135° (-45°). A raw polarized image frame (I -45 frame) is thus obtained. These example-only four polarized image frames are shown in the frames 202. [0044] Next, according to one embodiment, the spectral frames 204 include transmitted light from the spectropolarimetric imaging system 104 providing spectral frames at a plurality of different spectral frequencies based on application of a plurality of bandpass filters each with a bandwidth (v/ Vh). According to one embodiment, nine bandpass filters are applied each providing a spectral frame. These example-only spectral frames are shown in the frames 204.
[0045] Reference is now made to FIG. 3, where a method 300 of determining metadata for each pixel is shown. First, As discussed above, two types of tunability is carried out by the spectropolarimetric imaging system 104, including polarization and spectral tunability. First, the polarization tunability is shown with respect to polarization frames 304 (including, as discussed above according to one embodiment, fames h, Us, ho, and L45 ).
[0046] Next mathematical operations are performed on these raw polarized frames (Io, Us, ho, and I-45 ) as shown in block 306 and these three Stokes parameter maps (SO, SI, S2 ) are calculated from these operation as shown in Block 308 and provided below.
S0 = h + ho, SI = Io - ho, and S2 = Us - 1-45.
[0047] Next mathematical operations are performed on these Stokes parameter maps (SO, SI,
S2) as shown in blocks 310 and 312 and DoLP and AoLP are then assigned based on these mathematical operations (see below) as shown in blocks 314 and 316. From these stokes parameters, the DoLP map and AoLP map are calculated from the three Stokes parameters maps.
[0048] Thus two of the metadata (AoLP and DoLP) are obtained based on the operations of half of the flowchart of the method 300. With continued reference to FIG. 2, the method 300 proceeds to determine the remainder of the metadata (i.e., T, emissivity: e(w), and thermal texture Factor X). Initially, in block 320 spectral frames are obtained from the spectropolarimetric imaging system 104, as discussed above (i.e.,, a plurality, e.g., 9, bandpass filters are applied to the spectropolarimetric imaging system 104, each generating a bandlimited response from the scene). The method of obtaining the spectral frames is further discussed with reference to FIG. 3. Next, in block 324 the radiation spectrum is reconstructed using an antighosting algorithm, discussed further below, from which in blocks 326, 328, and 330 the remaining metadata (T, e(w), and Thermal Lightning Factor X) are determined. The antighosting algorithm, referenced in block 324, is further disclosed in FIG. 4.
[0049] With reference to FIG. 4, an antighosting method 400 is presented. Initially, in block
402, as discussed above, a plurality of bandlimited bandpass filters are applied to the incoming heat radiation. In one embodiment, nine such bandpass filters are applied. Each of these filters results in a frame including intensity between a lower bandpass level and an upper bandpass level. In other words, for example, for i = 1 (i.e., bandpass filter number 1, according to the
above example), the frame includes intensity for vu and Vhi , where vu is the lower bandpass level, and vm is the upper bandpass level. With reference back to FIG. 4, these spectral image frames are shown in block 404. Next in block 406, the dark noise of the spectropolarimetric imaging system 104 (see FIG. 1) and self-radiation of any bandpass filters are calibrated out, so that output of the spectropolarimetric imaging system 104 (see FIG. 1) is purely caused by incident heat radiation Sv. To achieve this noise removal and calibration, a black body source, known to a person having ordinary skill in the art, is used to measure noise response from the spectropolarimetric imaging system 104 (see FIG. 2).
[0050] Next, in block 408, the transmittance curve Tiv of each spectral filter i is characterized, which is a function of frequency (v). This transmittance curve Tiv measurement is performed once using a spectrometer (i.e., frequency/wavelength bands that are generated each time a bandpass filter is applied. Measurements are made by identifying a band of frequency that passes through the filter.
[0051] Next, in block 410, the response curve Zv of each sensor pixel is characterized. At a first level of approximation, a constant response curve can be assumed for each pixel across a range of frequencies (e.g., for i = 1, across vu and t¾i). Alternatively, response of each pixel with respect to the frequency range can be ascertained. The response is usually provided by the manufacturer of the spectropolarimetric imaging system 104 (see FIG. 2), or the response be can characterized by a spectrometer, as known to a person having ordinary skill in the art. Next in block 412, the incident spectrum Sv for each i is calculated. The relationship between Sv and the other parameters discussed above is provided below:
Ni = / ZvTivSvdv, where Ni is the output of the spectropolarimetric imaging system 104 (see FIG. 2),
Tiv is the transmittance curve of each raw spectral frames,
Zv is the response curve. To solve the above equation, as shown in block 414, each frequency band is discretized into a plurality of discrete frequency segments. By doing this discretization, the above equation can be rewritten as:
1. From the above relationship, spectrum Sv can be obtained as Sv = (Miv)~1Ni, as shown in block 416. Once the spectrum Sv has been obtained, then other metadata can be extracted, as discussed below. Towards this end, reference is made to FIG. 5a and the method 500. The method 500 begins with the obtained Sv in block 502. According to one embodiment of the present disclosure, in order to find the other metadata, the reconstructed Sv is compared to a library of material M, as shown in block 504. For each material in the library having a specific emissivity (e(w)), heat radiation is expressed in terms of a pseudo-3-dimensional curve Svm which is dependent on and thermal factor X and T as provided in the equation below. Thermal factor X is a parameter between 0 and 1 which defines surface texture of a material.
Svm = enthBn + (1 — evm)XBy, as shown in block 506,
where, KB is the Boltzmann constant, h is the Planck constant, c is the speed of light, m represents a material chosen from the library,
T represents the temperature of the target,
T0 represents the ambient temperature, evm represents the emissivity of the chosen material, and
X represents thermal texture factor.
[0052] Once a material with the above spectrum Svm has been matched to the obtained spectrum Sv, the other metadata are obtained, as shown in blocks 508, 510, and 512. From the thermal texture factor X, a map constituting an antighosting image can be generated.
[0053] With reference to FIG. 5b, one example algorithm is provided showing a method 550 for matching the obtained spectrum Sv against the library spectrum Svm. The method 550 begin in block 552 with the obtained spectrum Sv. A material from the library is chosen representing the material in the scene which the heat images are obtained from. The material spectrum Svm is governed by the equation provided above, as provided in block 554. These two spectra ( Sv and Svm ) represent three dimensional curves, as discussed above. The two spectra are compared with
each other, as provided in block 556. Using a comparison technique, e.g., least-square-error, as shown in block 558, a distinction between these two spectra is determined. This distinction is compared to a threshold, as shown in query 560. If the threshold is not met, then the method 550 chooses another material, as provided in block 562 and returns to block 554. If the threshold is met, then a match is ascertained and the method 550 proceeds to identifying the remaining metadata, as shown in block 564.
[0054] According to another approach, Svm for all materials in the library are determined. Next, comparisons between Svm and Sv can be made and using a least square error method (sum of the least square errors in the aforementioned comparison) the minimum of least squares is chosen as a positive match to a selected material in the library.
[0055] To demonstrate these techniques, example graphs are provided in FIGs. 6a-6d. First, the spectrum Sv of the image from the scene is shown as a function of frequency in FIG. 6a (see block 416 of FIG. 4). Next, for different temperature B (which is black body radiation) is determined and plotted vs. frequency, as shown in FIG. 6b (see block 554 in FIG. 5b) for different temperatures. Bv at T=273 °K is By . Next, Svmfor different materials is obtained by plotting Svm as a function of frequency for different temperatures and different X values. The temperature can be chosen based on a priori knowledge of the ambient temperature. As discussed X values range between 0 and 1. These graphs as generated for each material (mi ... mn). Once these Svm graphs are generated, the Sv from the scene is compared in a manner discussed according to FIG. 5b or an alternate plan as discussed above and a match is obtained for T, X, and a material (i.e., e).
[0056] Referring to FIGs. 7a-7d, two images are shown using standard infrared imaging (FIGs. 7a and 7c), while FIGs. 7b and 7d are processed images utilizing the methods described above thereby generating surface textures for enhanced visualization·
[0057] While in the above-described method, spectral metadata are used to establish a match between the family of spectral curves of a material in the library based on temperature and thermal texture factor, additional accuracy can be achieved by adding linear polarization data into the mix. Towards this end, it should be appreciated that emissivity, and B (block body radiation) are not only dependent on frequency but also dependent on linear polarization. Therefore, according to one embodiment, each family of curves is not only dependent on
temperature, frequency but also on degree of polarization. Referring to FIG. 8, an example of this additional resolution based on degree of polarization is shown. With this added resolution, a better match can be made to the spectral information from the scene. For example, in FIG. 8, instead of just one spectral curve for temperature T = 350°K and X = 0.1, a family of curves are provided for a plurality of polarizations angles. For example, according to one embodiment, if there are 4 polarization angles (i.e., 0, 45, 90, and -45), there would be four spectral curves. This additional parameter, allows added accuracy by a factor equal to the number of angles of polarization.
[0058] Those having ordinary skill in the art will recognize that numerous modifications can be made to the specific implementations described above. The implementations should not be limited to the particular limitations described. Other implementations may be possible.
Claims
1. A method of generating object surface texture in thermal infrared images, comprising: receiving heat radiation from a scene by a spectropolarimetric imaging system adapted to generate a plurality of spectral frames associated within the scene; generating the plurality of spectral frames associated with the scene, each frame having a plurality of pixels; for each pixel from the generated plurality of spectral frames, extracting spectral information associated with the scene, including pixel-specific temperature representing an object’s temperature, and thermal texture factor representing the object’s texture; for each of a plurality of materials having a specific emissivity in a library, generating reference spectral information as a function of temperature and thermal texture; matching the extracted spectral information for each pixel from the generated plurality of spectral frames to the generated reference spectral information using a statistical method to minimize the associated variation; and extracting spectral metadata from the matched reference spectral information for the associated material based on the match.
2. The method of claim 1, wherein the plurality of spectral frames from the spectropolarimetric imaging system are each generated by applying a plurality of associated bandpass filters to the spectropolarimetric imaging system and passing the heat radiation therethrough.
3. The method of claim 1, wherein the extracted spectral information associated with the scene from the spectropolarimetric imaging system for each pixel from the generated plurality of spectral frames is based on
where Ni represents output of the spectropolarimetric imaging system for each application of the associated bandpass filter i, v represents frequency,
Tiv represents transmittance curve of each pixel of the plurality of spectral frames for discretized frequency bands between i ?min and vmax for each application of the bandpass filter, Av represents width of discretized frequency band,
4. The method of claim 1, wherein the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library is obtained from:
¾m — ½¾ h (1 Evm)XBv ,
where, KB is the Boltzmann constant, h is the Planck constant, c is the speed of light, m represents a material chosen from the library,
T represents the temperature of the target,
T0 represents the ambient temperature, evm represents the emissivity of the chosen material, and
X represents thermal texture factor.
5. The method of claim 4, wherein the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library includes a family of spectral curves i) based on a plurality of temperatures and ii) for each temperature of the plurality of temperatures, based on variation of thermal texture factor (X), wherein the thermal texture factor is a variable between 0 and 1.
6. The method of claim 5, wherein the matching of the extracted spectral information for each pixel from the spectropolarimetric imaging system from the generated plurality of spectral frames to the generated reference spectral information is based on matching Sv to Svm.
7. The method of claim 6, wherein the statistical method includes sum of least mean squares between the Sv and Svm meeting a predetermined threshold.
8. The method of claim 6, wherein the statistical method includes a minimum least mean squares between the Sv and Svm.
9. The method of claim 1, wherein the spectropolarimetric imaging system is further adapted to generate a plurality of polarization frames associated within the scene.
10. The method of claim 9, further comprising: generating the plurality of polarization frames associated with the scene, each frame having a plurality of pixels.
11. The method of claim 10, wherein the plurality of linear polarization frames from the spectropolarimetric imaging system includes liner polarization at 0°, 45°, 90°, and -45°, thereby generating Io, Us, ho, and I -45 frames.
12. The method of claim 11, for each pixel from the generated plurality of polarization frames, further extracting spectral information associated with the scene based on the polarization angles ( Svp ).
13. The method of claim 12, wherein the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library includes a family of spectral curves ( Svmp ) i) based on a plurality of temperatures, ii) for each temperature of the plurality of temperatures, based on variation of thermal texture factor (X), wherein the thermal texture factor is a variable between 0 and 1, and for each thermal texture factor (X), based on variation of polarization angle including 0°, 45°,
90°, and -45°.
14. The method of claim 13, wherein the statistical method includes sum of least mean squares between the Svp and Svmp meeting a predetermined threshold.
15. The method of claim 13, wherein the statistical method includes a minimum least mean squares between the Svp and Svmp .
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP20895563.3A EP4070054A4 (en) | 2019-12-06 | 2020-12-06 | Heat assisted detection and ranging based on spectropolarimetric imaging |
CA3158728A CA3158728A1 (en) | 2019-12-06 | 2020-12-06 | Heat assisted detection and ranging based on spectropolarimetric imaging |
US17/778,842 US20230341265A1 (en) | 2019-12-06 | 2020-12-06 | Heat assisted detection and ranging based on spectropolarimetric imaging |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201962944671P | 2019-12-06 | 2019-12-06 | |
US62/944,671 | 2019-12-06 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021113791A1 true WO2021113791A1 (en) | 2021-06-10 |
Family
ID=76222684
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2020/063521 WO2021113791A1 (en) | 2019-12-06 | 2020-12-06 | Heat assisted detection and ranging based on spectropolarimetric imaging |
Country Status (4)
Country | Link |
---|---|
US (1) | US20230341265A1 (en) |
EP (1) | EP4070054A4 (en) |
CA (1) | CA3158728A1 (en) |
WO (1) | WO2021113791A1 (en) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6064510A (en) * | 1998-02-05 | 2000-05-16 | Carnegie Mellon University | Spectro-polarimetric imager |
US20110261191A1 (en) * | 2009-12-17 | 2011-10-27 | Raytheon Company | Textured pattern sensing and detection, and using a charge-scavenging photodiode array for the same |
US20120183175A1 (en) * | 2009-05-07 | 2012-07-19 | Institut D'optique | Method for identifying a scene from multiple wavelength polarized images |
US8659656B1 (en) * | 2010-10-12 | 2014-02-25 | The Boeing Company | Hyperspectral imaging unmixing |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102073039B (en) * | 2010-12-06 | 2014-08-06 | 中国科学院遥感应用研究所 | Thermal infrared hyperspectral emissivity simulation method and system |
-
2020
- 2020-12-06 EP EP20895563.3A patent/EP4070054A4/en active Pending
- 2020-12-06 WO PCT/US2020/063521 patent/WO2021113791A1/en unknown
- 2020-12-06 US US17/778,842 patent/US20230341265A1/en active Pending
- 2020-12-06 CA CA3158728A patent/CA3158728A1/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6064510A (en) * | 1998-02-05 | 2000-05-16 | Carnegie Mellon University | Spectro-polarimetric imager |
US20120183175A1 (en) * | 2009-05-07 | 2012-07-19 | Institut D'optique | Method for identifying a scene from multiple wavelength polarized images |
US20110261191A1 (en) * | 2009-12-17 | 2011-10-27 | Raytheon Company | Textured pattern sensing and detection, and using a charge-scavenging photodiode array for the same |
US8659656B1 (en) * | 2010-10-12 | 2014-02-25 | The Boeing Company | Hyperspectral imaging unmixing |
Non-Patent Citations (3)
Title |
---|
RIETJENS JEROEN H. H., CAMPO JOCHEN, CHANUMOLU ANANTHA, SMIT MARTIJN, NALLA RAJ, FERNANDEZ CRISTINA V., DINGJAN JOS, VAN AMERONGEN: "Expected performance and error analysis for SPEXone, a multi-angle channeled spectropolarimeter for the NASA PACE mission", POLARIZATION SCIENCE AND REMOTE SENSING IX, SPIE, 6 September 2019 (2019-09-06) - 15 August 2019 (2019-08-15), XP055834360, ISBN: 978-1-5106-2958-5, DOI: 10.1117/12.2530729 * |
See also references of EP4070054A4 * |
SMITH C. H., WRIGHT C. M., AITKEN D. K., ROCHE P. F., HOUGH J. H.: "Studies in mid-infrared spectropolarimetry -- II. An atlas of spectra", MONTHLY NOTICES OF THE ROYAL ASTRONOMICAL SOCIETY., LONDON, GB, vol. 312, no. 2, 21 February 2000 (2000-02-21), GB, pages 327 - 361, XP055834361, ISSN: 0035-8711, DOI: 10.1046/j.1365-8711.2000.03158.x * |
Also Published As
Publication number | Publication date |
---|---|
CA3158728A1 (en) | 2021-06-10 |
EP4070054A4 (en) | 2023-12-27 |
EP4070054A1 (en) | 2022-10-12 |
US20230341265A1 (en) | 2023-10-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6586430B2 (en) | Estimation of vehicle position | |
Bernstein et al. | Quick atmospheric correction code: algorithm description and recent upgrades | |
Uss et al. | Maximum likelihood estimation of spatially correlated signal-dependent noise in hyperspectral images | |
Eismann et al. | Airborne hyperspectral detection of small changes | |
CN105043552A (en) | Colorimetric temperature measurement system display and calibration method | |
Wierzbicki et al. | Method of radiometric quality assessment of NIR images acquired with a custom sensor mounted on an unmanned aerial vehicle | |
Li et al. | DIM moving target detection using spatio-temporal anomaly detection for hyperspectral image sequences | |
Jiang et al. | A practical method utilizing multi-spectral LiDAR to aid points cloud matching in SLAM | |
US20230341265A1 (en) | Heat assisted detection and ranging based on spectropolarimetric imaging | |
Kwan | Target Detection and Classification Improvements using Contrast Enhanced 16-bit Infrared Videos | |
Acito et al. | Robust technique for anomalous change detection in airborne hyperspectral imagery based on automatic and adaptive band selection | |
JP2005208023A (en) | Target-detecting apparatus | |
CN103792005A (en) | Wide-band two-dimension dispersion high-speed imaging spectrometer | |
Weyermann et al. | Spectral angle mapper (SAM) for anisotropy class indexing in imaging spectrometry data | |
Liang et al. | Deep fusion of heterogeneous sensor modalities for the advancements of ADAS to autonomous vehicles | |
Hubbard et al. | Detection of disturbed earth using hyperspectral LWIR imaging data | |
Shi et al. | Research on the dehazing effect of orthogonal polarization method based on atmospheric scattering model and considering extinction ratio parameter | |
Zhang et al. | Multi-Sensor Fusion on Automated Guided Vehicle | |
Bartlett et al. | Anomaly detection with varied ground sample distance utilizing spectropolarimetric imagery collected using a liquid crystal tunable filter | |
UrRehman et al. | Inspection on infrared-based image processing | |
Gorin et al. | New method to optimize atmospheric correction for hyperspectral thermal imaging | |
Sheeny de Moraes | All-weather object recognition using radar and infrared sensing | |
Gibney | Polarimetric phenomenology in the reflective regime: A case study using polarized hyperspectral data | |
Martin et al. | Comparison of bad pixel replacement techniques for LWIR hyperspectral imagery | |
Macfarlane et al. | Object detection and classification in aerial hyperspectral imagery using a multivariate hit-or-miss transform |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20895563 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 3158728 Country of ref document: CA |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2020895563 Country of ref document: EP Effective date: 20220706 |