US20140044340A1 - 3D Radiometry - Google Patents
3D Radiometry Download PDFInfo
- Publication number
- US20140044340A1 US20140044340A1 US13/568,546 US201213568546A US2014044340A1 US 20140044340 A1 US20140044340 A1 US 20140044340A1 US 201213568546 A US201213568546 A US 201213568546A US 2014044340 A1 US2014044340 A1 US 2014044340A1
- Authority
- US
- United States
- Prior art keywords
- imaged
- radiometric
- scene
- viewing angles
- computer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 claims abstract description 29
- 238000004590 computer program Methods 0.000 claims abstract description 10
- 230000005855 radiation Effects 0.000 claims abstract description 8
- 230000003287 optical effect Effects 0.000 claims description 13
- 238000005206 flow analysis Methods 0.000 claims description 5
- 238000012937 correction Methods 0.000 abstract description 5
- 238000001931 thermography Methods 0.000 description 20
- 238000003384 imaging method Methods 0.000 description 8
- 238000004458 analytical method Methods 0.000 description 5
- 230000003595 spectral effect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000005259 measurement Methods 0.000 description 4
- 238000001757 thermogravimetry curve Methods 0.000 description 4
- 230000006399 behavior Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000009826 distribution Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 239000007787 solid Substances 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 230000004907 flux Effects 0.000 description 2
- 238000003331 infrared imaging Methods 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 241000935974 Paralichthys dentatus Species 0.000 description 1
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 1
- XHCLAFWTIXFWPH-UHFFFAOYSA-N [O-2].[O-2].[O-2].[O-2].[O-2].[V+5].[V+5] Chemical compound [O-2].[O-2].[O-2].[O-2].[O-2].[V+5].[V+5] XHCLAFWTIXFWPH-UHFFFAOYSA-N 0.000 description 1
- 230000002547 anomalous effect Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000005670 electromagnetic radiation Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000009413 insulation Methods 0.000 description 1
- 239000003595 mist Substances 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000010845 search algorithm Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 229910052710 silicon Inorganic materials 0.000 description 1
- 239000010703 silicon Substances 0.000 description 1
- 239000000779 smoke Substances 0.000 description 1
- 238000010561 standard procedure Methods 0.000 description 1
- 238000002076 thermal analysis method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 229910001935 vanadium oxide Inorganic materials 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01K—MEASURING TEMPERATURE; MEASURING QUANTITY OF HEAT; THERMALLY-SENSITIVE ELEMENTS NOT OTHERWISE PROVIDED FOR
- G01K17/00—Measuring quantity of heat
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
- G01J5/02—Constructional details
- G01J5/026—Control of working procedures of a pyrometer, other than calibration; Bandwidth calculation; Gain control
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
- G01J5/0022—Radiation pyrometry, e.g. infrared or optical thermometry for sensing the radiation of moving bodies
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
- G01J5/10—Radiation pyrometry, e.g. infrared or optical thermometry using electric radiation detectors
- G01J5/20—Radiation pyrometry, e.g. infrared or optical thermometry using electric radiation detectors using resistors, thermistors or semiconductors sensitive to radiation, e.g. photoconductive devices
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
- G01J5/48—Thermography; Techniques using wholly visual means
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
- G01J5/80—Calibration
Definitions
- the present invention relates to novel methods for modeling a three-dimensional (3D) environment in order to correct radiometric measurements on the basis of inferred viewing angles, distances, and surface emissivities.
- radiometry shall refer generally to any measurement of the radiance of electromagnetic radiation within a specified spectral band.
- Randomness is the electromagnetic power emitted per unit area of a surface per unit solid angle and is typically measured in units of W/m 2 /sr.
- Specific radiance is the radiance per unit wavelength, given, say, in W/m 2 /sr/ ⁇ m.
- Radiance is related to the effective temperature of a surface (the product of its temperature and emissivity) in a manner expressed by Planck's law, which, when integrated over all spectral energies, yields the well-known T 4 -dependence of the Stefan-Boltzmann law.
- Planck's law which, when integrated over all spectral energies, yields the well-known T 4 -dependence of the Stefan-Boltzmann law.
- measurements over finite spectral energy ranges dictated, say, by band-limited infrared radiation detection require explicit evaluation of the integral of Planck's law over the band concerned, typically approximated using one of the forms of the Sakuma-Hattori equation, described in Sakuma et al., Establishing a practical temperature standard by using a narrow - band radiation thermometer with a silicon detector, Temperature: Its Measurement and Control in Science and Industry, vol. 5, pp. 421-27 (1982), which is incorporated herein by reference, or by other approxim
- thermoogram An image (as defined below) representing the spatial distribution (in a space of any dimensionality) of surface temperature may be referred to as a “thermogram.” “Thermography,” as used herein and in any appended claims, shall denote the study, by imaging means, of temperature distributions in structures or regions, for example, in buildings.
- “Infrared thermography” shall refer to thermography performed in whole, or in part, within the infrared portion of the electromagnetic spectrum, and more particularly, within the so-called “thermal infrared” portion of the spectrum, which need not be precisely defined herein, insofar as the term is used without limitation herein.
- Long-wave infrared (LWIR) detection [ ⁇ 8-14 ⁇ m] is typically employed for mapping temperatures near typical ambient terrestrial temperatures ( ⁇ 300° K.), because emittance (radiance integrated over solid angle) is maximized in that range, and because transmission through mist and smoke is considered superior to that of other spectral ranges.
- detector may be used herein comprehensively, and interchangeably with the term “sensor,” with either term applying either to a single detector element or to an array of sensors or detectors, whether sensitive to the flux of impinging photons, or whether sensitive to temperature in radiative equilibrium with a distant source, such as a bolometer.
- An array of detectors at the focal plane of an optical system may also be referred to, herein, as a camera.
- three-dimensional (or 3D) radiometry shall refer to radiometry, as previously defined, that takes into account the three-dimensional nature of a scene in order to measure the radiance to infer a derived quantity such as a temperature, associated with a particular surface comprising a portion of the scene.
- drive-by thermography is defined as the imaging of urban environments by scanning them from a vehicle on the street and the assignment of temperatures to points on the surfaces of the scene.
- Radiometry entails the imaging of radiance over a specified range of the infrared in order to infer surface temperature maps of complex objects.
- anomalous hot spots, or regions of large thermal gradients are identified in images and related to physical locations on structures in order to identify radiant energy leakage, for example.
- thermography arise because the mapping of detected radiance to surface temperature is not a simple one, especially when the geometry of the emitting surface is complex. At a minimum, a distance to the emitting surface must be known or assumed, and, similarly, an emissivity must be assumed or otherwise ascertained. That is why Cho et al., 3 D Thermal Modeling for Existing Buildings using Hybrid LIDAR System, Computing in Civil Engineering, pp. 552-58 (2011), incorporated herein by reference, teaches the concomitant application of a second modality (LIDAR, in that case) in order to associate a distance with distinct points associated with the imaged scene.
- LIDAR second modality
- aerial thermography is essentially 2D thermography, where ad hoc corrections are made to account for the failure of the assumptions of 2D radiometry.
- the “view factor” F 12 as defined between two infinitesimal surface elements is a geometric function describing the space angle subtended by one differential area dA 1 with respect to a second differential area dA 2 , as well-known and thoroughly covered by Modest, Radiative Heat Transfer (2d ed., 2003), pp. 145-49, for example.
- the view factor is given by
- F 12 cos ⁇ ⁇ ⁇ 1 ⁇ cos ⁇ ⁇ ⁇ 2 ⁇ ⁇ ⁇ S 2 ⁇ ⁇ A 2 .
- ⁇ i is the angle between a line connecting differential elements dA i and the normals to the respective elements
- S is the distance between the elements.
- ⁇ i are assumed to be identically zero.
- the emitting surface is deemed parallel to the sensing plane, and at a fixed distance.
- 2D thermography assumes a constant emissivity for all points on a long wave infrared image.
- embodiments of the present invention teach a novel approach to radiometry as it relates to inferring temperature based on the measured radiance in complex 3D environments and correcting for non-Lambertian behaviors of such environments, as experienced when performing drive-by imaging.
- methods are provided for deriving temperature information with respect to surfaces imaged radiometrically by means of a radiometric camera in motion with respect to the imaged surfaces. Such methods have steps of:
- the step of modeling the 3D structure of the scene is performed using optical flow analysis.
- the step of acquiring a time sequence of frames may be performed by means of an infrared radiometric camera. There may be an additional step of correcting the radiometric data to account for background radiation impinging upon the imaged surfaces.
- acquisition and storage of radiometric data may be performed by means of a camera disposed on a vehicle traversing the imaged surfaces at street level.
- An additional step may include calibrating an infrared camera in order to acquire the radiometric data.
- a computer program product for deriving temperature information with respect to surfaces that are imaged radiometrically by means of a camera in motion with respect to the imaged surfaces.
- the computer program product has a computer usable medium containing a computer readable program code that includes:
- a computer code module for storing a time sequence of images of the surfaces at a plurality of positions
- FIG. 1 is a flowchart depicting steps in practice of an embodiment of the present invention
- FIG. 2 depicts, in false color, a thermogram of a building based on drive-by radiometry in accordance with an embodiment of the present invention.
- image shall refer to any multidimensional representation, whether in tangible or otherwise perceptible form, or otherwise, whereby a value of some characteristic (such as radiance, brightness, temperature, etc.) is associated with each of a plurality of locations corresponding to dimensional coordinates of an object in physical space, though not necessarily mapped one-to-one thereonto.
- some characteristic such as radiance, brightness, temperature, etc.
- the graphic display of the spatial distribution of some field, either scalar or vectorial, such as brightness or color constitutes an image. So, also, does an array of numbers in a computer memory or holographic medium.
- imaging refers to the rendering of a stated physical characteristic in terms of one or more images.
- object shall refer to a tangible, non-transient, physical object capable of being rendered as an image.
- the object may be a building or other structure, for example.
- scene encompasses one or more objects within a field of view of a sensor.
- Such methods typically employ a sensor, such as an array of LWIR microbolometers, to acquire a time sequence of radiometric images of a scene as the sensor is moved past the scene along a linear trajectory or otherwise.
- a sensor such as an array of LWIR microbolometers
- an uncooled detector is preferred, on grounds of lifetime and economy of operation, and, more particularly, a vanadium oxide detector, such as model UC640-17, available from DRS Technologies of Parsippany, N.J., is preferred.
- a photometric or other non-radiometric detector is employed, a thermal calibration is performed, as known in the art.
- a detector such as the one described above by way of example, is mounted on a vehicle that typically travels on urban streets. It is to be understood that the drive-by applications described herein are presented by way of example, and that principles in accordance with the present invention may be employed in many contexts, including airborne imagery, for piloted aircraft or autonomous vehicles.
- Data from the detector are acquired and recorded (in step 101 ) as a time sequence of radiometric images taken from distinct angles, at a rate optimized for the application.
- a frame rate of 30 frames per second is employed, as governed by the detector frame capture rate.
- the 3D structure of the scene is then modeled ( 103 ) using techniques of optical flow modeling, as described, for example, by Horn et al., Determining Optical Flow, Artificial Intelligence, vol. 17, pp. 185-203 (1981), which is incorporated herein by reference, and in many subsequent references.
- Any optical flow algorithm employed to obtain a 3D model of the imaged scene is within the scope of the present invention. It is to be understood that 3D modeling may also be achieved other than by optical flow analysis, within the scope of the present invention.
- a preferred method of 3D modeling based on optical flow is described by Farnebburg, Two - frame motion estimation based on polynomial expansion, Image Analysis , vol. 2749, pp. 363-70 (2003), incorporated herein by reference.
- a polynomial expansion function is used to define an approximate neighborhood of each pixel, and, within each neighborhood, a search algorithm minimizes the difference in displacement between a projection of the first frame and the second frame.
- step 103 pixel motion is translated into a depth map ( 104 ) using techniques well-known in the art.
- the depth map is used to create an overlaying solid mesh map, which, in turn, is used to infer the viewing angle and distances from respective surfaces to the detector ( 105 ).
- the actual size of the remote object is readily obtained using the physical size of elements of the focal plane detector array and simple geometry.
- the normalized surface areas of objects in the scene can be estimated ( 107 ) based on images at different viewing angles obtained in successive frames of the same time sequence of images.
- a LWIR camera system In cases where a LWIR camera system is used to obtain thermal images, it must be calibrated, in step 109 .
- Various standard radiometric calibration procedures may be employed, such as taught by Liebmann et al., Infrared Calibration Development at Fluke Corporation Hart Scientific Division, Proc. Soc. Photo-Optical Instrumentation Engineers (2008), incorporated herein by reference. From such calibration procedures, known in the art, the effective detector emissivity and gain may be derived, deriving, in particular, a mapping of signal output (pixel intensity) to observed radiance.
- a directional correction factor is introduced to correct for situations where observed emissions are non-uniform since, as is well-known, the emissivity of a surface will vary with view angle, representing excursions from isotropic emissivity and Lambertian cosine-law reflectivity.
- the non-Lambertian behavior of emissivity has been studied since Schmidt et al., (1935). Accordingly, an empirical directional correct factor g( ⁇ ) is applied (in step 111 ) to account for non-Lambertian behavior of non-metallic surfaces.
- the radiance attributed to a surface of emissivity ⁇ r may be expressed in the following form:
- the temperature map, or thermogram, once generated, is provided to a user in a perceptible form, such as a visual display, as shown in FIG. 2 , for example.
- the disclosed methods of infrared thermography are implemented as a computer program product for use with a computer system.
- Such implementations may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium.
- the medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., microwave, infrared or other transmission techniques).
- the series of computer instructions embodies all or part of the functionality previously described herein with respect to the system.
- Such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies. It is expected that such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web). Of course, some embodiments of the invention may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments of the invention are implemented as entirely hardware, or entirely software (e.g., a computer program product).
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Chemical & Material Sciences (AREA)
- Engineering & Computer Science (AREA)
- Combustion & Propulsion (AREA)
- Radiation Pyrometers (AREA)
- Image Processing (AREA)
Abstract
Description
- The present invention relates to novel methods for modeling a three-dimensional (3D) environment in order to correct radiometric measurements on the basis of inferred viewing angles, distances, and surface emissivities.
- Definitions: As used herein and in any appended claims, the term “radiometry” shall refer generally to any measurement of the radiance of electromagnetic radiation within a specified spectral band.
- “Radiance” is the electromagnetic power emitted per unit area of a surface per unit solid angle and is typically measured in units of W/m2/sr. “Spectral radiance” is the radiance per unit wavelength, given, say, in W/m2/sr/μm.
- Radiance is related to the effective temperature of a surface (the product of its temperature and emissivity) in a manner expressed by Planck's law, which, when integrated over all spectral energies, yields the well-known T4-dependence of the Stefan-Boltzmann law. However, measurements over finite spectral energy ranges dictated, say, by band-limited infrared radiation detection, require explicit evaluation of the integral of Planck's law over the band concerned, typically approximated using one of the forms of the Sakuma-Hattori equation, described in Sakuma et al., Establishing a practical temperature standard by using a narrow-band radiation thermometer with a silicon detector, Temperature: Its Measurement and Control in Science and Industry, vol. 5, pp. 421-27 (1982), which is incorporated herein by reference, or by other approximation methods.
- An image (as defined below) representing the spatial distribution (in a space of any dimensionality) of surface temperature may be referred to as a “thermogram.” “Thermography,” as used herein and in any appended claims, shall denote the study, by imaging means, of temperature distributions in structures or regions, for example, in buildings.
- “Infrared thermography” shall refer to thermography performed in whole, or in part, within the infrared portion of the electromagnetic spectrum, and more particularly, within the so-called “thermal infrared” portion of the spectrum, which need not be precisely defined herein, insofar as the term is used without limitation herein. Long-wave infrared (LWIR) detection [˜8-14 μm] is typically employed for mapping temperatures near typical ambient terrestrial temperatures (˜300° K.), because emittance (radiance integrated over solid angle) is maximized in that range, and because transmission through mist and smoke is considered superior to that of other spectral ranges.
- The term “detector” may be used herein comprehensively, and interchangeably with the term “sensor,” with either term applying either to a single detector element or to an array of sensors or detectors, whether sensitive to the flux of impinging photons, or whether sensitive to temperature in radiative equilibrium with a distant source, such as a bolometer. An array of detectors at the focal plane of an optical system may also be referred to, herein, as a camera.
- The term “three-dimensional (or 3D) radiometry,” as used herein and in any appended claims, shall refer to radiometry, as previously defined, that takes into account the three-dimensional nature of a scene in order to measure the radiance to infer a derived quantity such as a temperature, associated with a particular surface comprising a portion of the scene.
- The term “drive-by thermography,” as used herein and in any appended claims, is defined as the imaging of urban environments by scanning them from a vehicle on the street and the assignment of temperatures to points on the surfaces of the scene.
- One class of application of radiometry entails the imaging of radiance over a specified range of the infrared in order to infer surface temperature maps of complex objects. In applications of this sort, anomalous hot spots, or regions of large thermal gradients, are identified in images and related to physical locations on structures in order to identify radiant energy leakage, for example.
- Complications in thermography arise because the mapping of detected radiance to surface temperature is not a simple one, especially when the geometry of the emitting surface is complex. At a minimum, a distance to the emitting surface must be known or assumed, and, similarly, an emissivity must be assumed or otherwise ascertained. That is why Cho et al., 3D Thermal Modeling for Existing Buildings using Hybrid LIDAR System, Computing in Civil Engineering, pp. 552-58 (2011), incorporated herein by reference, teaches the concomitant application of a second modality (LIDAR, in that case) in order to associate a distance with distinct points associated with the imaged scene. Similarly, Yang et al., Fusion of camera images and laser scans for wide baseline 3D scene alignment in urban environments, J. Photogram. Remote Sensing (2011), doi:10.1016/j.isprsjprs.2011.09.004, albeit not in a thermographic context, employs information separately derived from a laser scanner in order to build a third dimension into an otherwise two-dimensional image.
- Aerial thermography, as applied to urban scenes, for example, by Allinson, Evaluation of aerial thermography to discriminate loft insulation in residential housing, (U. Nottingham, 2007) to the roofs of Nottingham, and by Meier et al., Determination of persistence effects in spatio-temporal patterns of upward long-wave radiation flux density from an urban courtyard by means of time-sequential thermography, Remote Sensing of Environment, pp. 21-34 (2010) to the courtyards of Berlin, suffers from many acknowledged uncertainties, and is, essentially, uncalibrated and uncalibratable. Despite discussion by Schmidt et al., Über die Richtungsverteilung der Wärmestrahlung von Oberflächen. Forschung auf dem Gebiet des Ingenieurwesens A, vol. 6, pp. 175-83 (1935) (hereinafter, Schmidt et al., (1935)), of emissivity variation with viewing angle, the roof pitch (and, thus, sky view factor), and the effects of non-Lambertian emission (variation of emissivity with viewing angle of regard) can only be estimated in the aggregate using current aerial thermography techniques. The techniques of aerial thermography that are known in the art have no bearing on the problem of drive-by thermography, in fact, they are entirely irrelevant to street-level applications.
- As practiced, aerial thermography is essentially 2D thermography, where ad hoc corrections are made to account for the failure of the assumptions of 2D radiometry. In particular, the “view factor” F12 as defined between two infinitesimal surface elements, is a geometric function describing the space angle subtended by one differential area dA1 with respect to a second differential area dA2, as well-known and thoroughly covered by Modest, Radiative Heat Transfer (2d ed., 2003), pp. 145-49, for example. Rigorously, the view factor is given by
-
- where θi is the angle between a line connecting differential elements dAi and the normals to the respective elements, and S is the distance between the elements. In 2D radiometry, since one cannot know, or account for, the relative orientation of surfaces, θi are assumed to be identically zero. Thus, for example, the emitting surface is deemed parallel to the sensing plane, and at a fixed distance. Moreover, 2D thermography assumes a constant emissivity for all points on a long wave infrared image.
- Previous forays into 3D thermography have adopted one or more of a number of stratagems: they have addressed isolated, static objects (thereby obviating many geometrical complexity issues), or have exploited the inherent stereoscopic characteristic of multiple thermal cameras, or, else, have combined the data of a thermal camera with data of one or more digital visible images in order to infer 3D characteristics of a scene. 3D thermography of isolated and static objects is described by Kapoor et al., Comparison of Techniques for the 3D Modeling and Thermal Analysis, APEGA 2010, pp. 163-173 (2010), and by Shao, Detecting Sources of Heat Loss in Residential Buildings from Infrared Imaging, (MIT Undergraduate Thesis, 2011), both of which are incorporated herein by reference.
- It is desirable, however, to provide a mechanism for accurate (real time, nearly real-time, or post processing) analysis of radiometric data obtained in an urban environment on a drive-by basis. The techniques of radiometric analysis, employed to date, however, are far too cumbersome for this purpose, employing either imaging by multiple cameras and/or imaging modalities or explicit distance-measuring modalities, or else invoking averaged calibration factors rather than object- and scene-specific complex geometries. The resolution of the drive-by thermography problem has had to await the novel techniques claimed herein, and described for the first time in the present patent document, and in Phan, Automated Rapid Thermal Imaging Systems Technology, (MIT Ph.D. Dissertation, 2012, unpublished as of the present filing), which is filed herewith as an Appendix, and which is incorporated herein by reference.
- Accordingly, embodiments of the present invention teach a novel approach to radiometry as it relates to inferring temperature based on the measured radiance in complex 3D environments and correcting for non-Lambertian behaviors of such environments, as experienced when performing drive-by imaging. In accordance with certain embodiments of the invention, methods are provided for deriving temperature information with respect to surfaces imaged radiometrically by means of a radiometric camera in motion with respect to the imaged surfaces. Such methods have steps of:
-
- a. acquiring and storing a time sequence of frames of radiometric data including a plurality of radiometric images from distinct angles of a scene containing the surfaces;
- b. modeling a 3D structure of the scene;
- c. inferring viewing angles and distances to the imaged surfaces based on the modeled 3D structure;
- d. calculating normalized surface areas of the imaged surfaces based on the inferred viewing angles;
- e. correcting emissivities of the imaged surfaces based on the inferred viewing angles; and
- f. converting the radiometric data to a perceptible temperature map of the imaged surfaces.
- In some embodiments of the invention, the step of modeling the 3D structure of the scene is performed using optical flow analysis. In certain embodiments of the invention, the step of acquiring a time sequence of frames may be performed by means of an infrared radiometric camera. There may be an additional step of correcting the radiometric data to account for background radiation impinging upon the imaged surfaces.
- In further embodiments of the present invention, acquisition and storage of radiometric data may be performed by means of a camera disposed on a vehicle traversing the imaged surfaces at street level.
- An additional step, in accordance with yet further embodiments, may include calibrating an infrared camera in order to acquire the radiometric data.
- In accordance with alternate embodiments of the present invention, a computer program product is provided for deriving temperature information with respect to surfaces that are imaged radiometrically by means of a camera in motion with respect to the imaged surfaces. The computer program product has a computer usable medium containing a computer readable program code that includes:
- a computer code module for storing a time sequence of images of the surfaces at a plurality of positions;
-
- a computer code module modeling a 3D structure of the scene based upon optical flow analysis of the plurality of radiometric images;
- a computer code module for inferring viewing angles and distances to the imaged surfaces based on the modeled 3D structure;
- a computer code module for calculating normalized surface areas of the imaged surfaces based on the inferred viewing angles;
- a computer code module for correcting emissivities of the imaged surfaces based on the inferred viewing angles; and
- a computer code module for converting the radiometric data to a perceptible temperature map of the imaged surfaces.
- The present patent or application file contains at least one drawing executed in color. Copies of this patent with color drawing(s) will be provided by the Patent and Trademark Office upon request and payment of necessary fee.
- The foregoing features of the invention will be more readily understood from the following detailed description, considered with reference to the accompanying drawings, in which:
-
FIG. 1 is a flowchart depicting steps in practice of an embodiment of the present invention; -
FIG. 2 depicts, in false color, a thermogram of a building based on drive-by radiometry in accordance with an embodiment of the present invention. - Further definitions. As used in this description and the accompanying claims, the following term shall have the meanings indicated, unless the context otherwise requires:
- The term “image” shall refer to any multidimensional representation, whether in tangible or otherwise perceptible form, or otherwise, whereby a value of some characteristic (such as radiance, brightness, temperature, etc.) is associated with each of a plurality of locations corresponding to dimensional coordinates of an object in physical space, though not necessarily mapped one-to-one thereonto. Thus, for example, the graphic display of the spatial distribution of some field, either scalar or vectorial, such as brightness or color, constitutes an image. So, also, does an array of numbers in a computer memory or holographic medium. Similarly, “imaging” refers to the rendering of a stated physical characteristic in terms of one or more images.
- The term “object” shall refer to a tangible, non-transient, physical object capable of being rendered as an image. The object may be a building or other structure, for example. The term “scene” encompasses one or more objects within a field of view of a sensor.
- A conceptual breakthrough by the present inventors, leading to embodiments of the invention described herein, was the insight that it is possible to infer three-dimensional data with respect to objects in an imaged scene using the self-same infrared imaging device used for radiometric analysis. This enables drive-by thermography, since analysis of the data may be performed efficiently and in a scalable and inexpensive way.
- Methods in accordance with preferred embodiments of the present invention are now described with reference to the flowchart of
FIG. 1 . Such methods typically employ a sensor, such as an array of LWIR microbolometers, to acquire a time sequence of radiometric images of a scene as the sensor is moved past the scene along a linear trajectory or otherwise. Typically, in accordance with preferred embodiments of the invention, an uncooled detector is preferred, on grounds of lifetime and economy of operation, and, more particularly, a vanadium oxide detector, such as model UC640-17, available from DRS Technologies of Parsippany, N.J., is preferred. Where a photometric or other non-radiometric detector is employed, a thermal calibration is performed, as known in the art. - A detector, such as the one described above by way of example, is mounted on a vehicle that typically travels on urban streets. It is to be understood that the drive-by applications described herein are presented by way of example, and that principles in accordance with the present invention may be employed in many contexts, including airborne imagery, for piloted aircraft or autonomous vehicles.
- Data from the detector are acquired and recorded (in step 101) as a time sequence of radiometric images taken from distinct angles, at a rate optimized for the application. In a preferred embodiment of the invention, a frame rate of 30 frames per second is employed, as governed by the detector frame capture rate.
- The 3D structure of the scene is then modeled (103) using techniques of optical flow modeling, as described, for example, by Horn et al., Determining Optical Flow, Artificial Intelligence, vol. 17, pp. 185-203 (1981), which is incorporated herein by reference, and in many subsequent references. Furukawa et al., Accurate, Dense, and Robust Multi-View Stereopsis, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, pp. 1362-76 (2010), also incorporated herein by reference, showed that a single imaging modality, such as a camera, may be used to obtain a time sequence derived from multiple viewing angles relative to a single scene, in order to derive three-dimensional image models of the scene. Any optical flow algorithm employed to obtain a 3D model of the imaged scene is within the scope of the present invention. It is to be understood that 3D modeling may also be achieved other than by optical flow analysis, within the scope of the present invention.
- A preferred method of 3D modeling based on optical flow is described by Farnebäck, Two-frame motion estimation based on polynomial expansion, Image Analysis, vol. 2749, pp. 363-70 (2003), incorporated herein by reference. In accordance with Farneback, a polynomial expansion function is used to define an approximate neighborhood of each pixel, and, within each neighborhood, a search algorithm minimizes the difference in displacement between a projection of the first frame and the second frame.
- From the motion flow modeling of
step 103, pixel motion is translated into a depth map (104) using techniques well-known in the art. The depth map is used to create an overlaying solid mesh map, which, in turn, is used to infer the viewing angle and distances from respective surfaces to the detector (105). - Once the viewing angle and distance to the surface have been calculated, the actual size of the remote object is readily obtained using the physical size of elements of the focal plane detector array and simple geometry. In particular, the normalized surface areas of objects in the scene can be estimated (107) based on images at different viewing angles obtained in successive frames of the same time sequence of images.
- In cases where a LWIR camera system is used to obtain thermal images, it must be calibrated, in
step 109. Various standard radiometric calibration procedures may be employed, such as taught by Liebmann et al., Infrared Calibration Development at Fluke Corporation Hart Scientific Division, Proc. Soc. Photo-Optical Instrumentation Engineers (2008), incorporated herein by reference. From such calibration procedures, known in the art, the effective detector emissivity and gain may be derived, deriving, in particular, a mapping of signal output (pixel intensity) to observed radiance. - In order to impute a radiation temperature to a surface, its emissivity must be known or assumed. In accordance with embodiments of the present invention, a directional correction factor is introduced to correct for situations where observed emissions are non-uniform since, as is well-known, the emissivity of a surface will vary with view angle, representing excursions from isotropic emissivity and Lambertian cosine-law reflectivity. The non-Lambertian behavior of emissivity has been studied since Schmidt et al., (1935). Accordingly, an empirical directional correct factor g(θ) is applied (in step 111) to account for non-Lambertian behavior of non-metallic surfaces.
- In order to include the effects of background radiation and reflections from nearby objects impinging upon an imaged surface, the radiance attributed to a surface of emissivity εr may be expressed in the following form:
-
S r(T 0)=εr {tilde over (S)}(T 0)+(1−εr){tilde over (S)}(T b), - where Sr(T0) is the observed radiance, {tilde over (S)} is the total radiance of a blackbody at the temperature T0 of the surface in question, (1−εr) is the reflection of the surface, and Tb is the background temperature. While, if the temperatures of the surface and the background are substantially the same, the correction for background temperature is small, in cases where the emissivity is low and the background temperature exceeds the surface temperature, the background may become a critical factor in determining the observed signal at the detector. Once the observed scene has been modeled in 3D and emissivity corrections applied, the measured radiance, as derived from multiple drive-by views and subsequently processed, may be converted to a temperature map (113) using standard techniques. An example of a 3D radiometry-based thermogram of a structure, obtained by applying a method in accordance with
FIG. 1 , is shown inFIG. 2 . Temperatures range from cooler (blue) to warmer (red). - The temperature map, or thermogram, once generated, is provided to a user in a perceptible form, such as a visual display, as shown in
FIG. 2 , for example. - While embodiments of the present invention have described a motion-based approach to reconstructing the 3D environment, it is to be understood that them may also be supplemented by alternative methods such as LIDAR, stereo vision, etc.
- In preferred embodiments of the present invention, the disclosed methods of infrared thermography are implemented as a computer program product for use with a computer system. Such implementations may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. The medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., microwave, infrared or other transmission techniques). The series of computer instructions embodies all or part of the functionality previously described herein with respect to the system. Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies. It is expected that such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web). Of course, some embodiments of the invention may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments of the invention are implemented as entirely hardware, or entirely software (e.g., a computer program product).
Claims (8)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/568,546 US8818079B2 (en) | 2012-08-07 | 2012-08-07 | 3D radiometry |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/568,546 US8818079B2 (en) | 2012-08-07 | 2012-08-07 | 3D radiometry |
Publications (2)
Publication Number | Publication Date |
---|---|
US20140044340A1 true US20140044340A1 (en) | 2014-02-13 |
US8818079B2 US8818079B2 (en) | 2014-08-26 |
Family
ID=50066227
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/568,546 Expired - Fee Related US8818079B2 (en) | 2012-08-07 | 2012-08-07 | 3D radiometry |
Country Status (1)
Country | Link |
---|---|
US (1) | US8818079B2 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150009335A1 (en) * | 2013-07-08 | 2015-01-08 | Flir Systems Ab | Facilitating improved calibration of captured infrared data values by an ir imaging system in a thermography arrangement |
US20150299952A1 (en) * | 2012-10-31 | 2015-10-22 | Valmet Automation Oy | Method and apparatus for monitoring web |
US10819905B1 (en) * | 2019-09-13 | 2020-10-27 | Guangdong Media Kitchen Appliance Manufacturing Co., Ltd. | System and method for temperature sensing in cooking appliance with data fusion |
US20210142058A1 (en) * | 2019-11-08 | 2021-05-13 | Msg Entertainment Group, Llc | Providing visual guidance for presenting visual content in a venue |
US11775707B1 (en) | 2022-10-25 | 2023-10-03 | Istari, Inc. | Interconnected digital engineering and certification ecosystem |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE102016218295A1 (en) | 2016-09-23 | 2018-03-29 | Robert Bosch Gmbh | Method for non-contact determination of a two-dimensional temperature information and infrared measurement system |
DE102016218291A1 (en) | 2016-09-23 | 2018-03-29 | Robert Bosch Gmbh | Method for non-contact determination of a two-dimensional temperature information and infrared measurement system |
JP7286159B2 (en) | 2016-09-29 | 2023-06-05 | シグニファイ ホールディング ビー ヴィ | Depth cues by thermal sensing |
DE102019113691B4 (en) | 2019-05-22 | 2022-09-15 | Universität Kassel | Method and device for generating a three-dimensional thermogram |
US10819923B1 (en) | 2019-11-19 | 2020-10-27 | Waymo Llc | Thermal imaging for self-driving cars |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7769205B2 (en) * | 2006-11-28 | 2010-08-03 | Prefixa International Inc. | Fast three dimensional recovery method and apparatus |
US8152366B2 (en) * | 2008-01-23 | 2012-04-10 | University Of Delaware | Estimation of subsurface thermal structure using sea surface height and sea surface temperature |
-
2012
- 2012-08-07 US US13/568,546 patent/US8818079B2/en not_active Expired - Fee Related
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7769205B2 (en) * | 2006-11-28 | 2010-08-03 | Prefixa International Inc. | Fast three dimensional recovery method and apparatus |
US8152366B2 (en) * | 2008-01-23 | 2012-04-10 | University Of Delaware | Estimation of subsurface thermal structure using sea surface height and sea surface temperature |
Non-Patent Citations (1)
Title |
---|
J. A. Voogt, "Image representation of complete Urban surface temperatures", Geocarto International Centre, Vol. 15, No. 3, September 2000. * |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150299952A1 (en) * | 2012-10-31 | 2015-10-22 | Valmet Automation Oy | Method and apparatus for monitoring web |
US20150009335A1 (en) * | 2013-07-08 | 2015-01-08 | Flir Systems Ab | Facilitating improved calibration of captured infrared data values by an ir imaging system in a thermography arrangement |
US9681066B2 (en) * | 2013-07-08 | 2017-06-13 | Flir Systems Ab | Facilitating improved calibration of captured infrared data values by an IR imaging system in a thermography arrangement |
US10819905B1 (en) * | 2019-09-13 | 2020-10-27 | Guangdong Media Kitchen Appliance Manufacturing Co., Ltd. | System and method for temperature sensing in cooking appliance with data fusion |
US20210142058A1 (en) * | 2019-11-08 | 2021-05-13 | Msg Entertainment Group, Llc | Providing visual guidance for presenting visual content in a venue |
US11023729B1 (en) * | 2019-11-08 | 2021-06-01 | Msg Entertainment Group, Llc | Providing visual guidance for presenting visual content in a venue |
US11647244B2 (en) | 2019-11-08 | 2023-05-09 | Msg Entertainment Group, Llc | Providing visual guidance for presenting visual content in a venue |
US11775707B1 (en) | 2022-10-25 | 2023-10-03 | Istari, Inc. | Interconnected digital engineering and certification ecosystem |
Also Published As
Publication number | Publication date |
---|---|
US8818079B2 (en) | 2014-08-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8818079B2 (en) | 3D radiometry | |
US11796454B2 (en) | Gas leak emission quantification with a gas cloud imager | |
US10416076B2 (en) | Quantifying gas in passive optical gas imaging | |
US9804031B2 (en) | Apparatus and method to calculate energy dissipated from an object | |
Vidas et al. | 3D thermal mapping of building interiors using an RGB-D and thermal camera | |
CN105606222B (en) | A kind of measuring device and measuring method of flame three-dimensional temperature field | |
Meier et al. | Atmospheric correction of thermal-infrared imagery of the 3-D urban environment acquired in oblique viewing geometry | |
CN108562363A (en) | A kind of infrared signature transient state temperature field accurate measurement method | |
JP7334879B2 (en) | Correlating Thermal Satellite Image Data to Generate High Spatial Resolution Heatmaps | |
KR102071879B1 (en) | Aerosol distribution measuring system by using sky image | |
González-Jorge et al. | Single image rectification of thermal images for geometric studies in façade inspections | |
Lewis et al. | Integrated thermal infrared imaging and structure-from-motion photogrammetry to map apparent temperature and radiant hydrothermal heat flux at Mammoth Mountain, CA, USA | |
Zander | Surface temperature measurements in hypersonic testing using digital single-lens reflex cameras | |
CN110675448A (en) | Ground light remote sensing monitoring method, system and storage medium based on civil aircraft | |
Daakir et al. | Improvement of photogrammetric accuracy by modeling and correcting the thermal effect on camera calibration | |
Oreifej et al. | Automatic generation of 3d thermal maps of building interiors | |
CN114495416A (en) | Fire monitoring method and device based on unmanned aerial vehicle and terminal equipment | |
Dlesk et al. | Point cloud generation of a building from close range thermal images | |
Mohammed et al. | The effect of polynomial order on georeferencing remote sensing images | |
KR101322801B1 (en) | System and method for setting emissivity of infrared thermal vision camera using reference pattern | |
KR102220654B1 (en) | Correction System of Measurement Temperature | |
Molnár et al. | Practical application possibilities for 3D models using low-resolution thermal images | |
CN113256493A (en) | Thermal infrared remote sensing image reconstruction method and device | |
Dlesk et al. | Usage of photogrammetric processing of thermal images for civil engineers | |
Robinson et al. | 3D thermography for improving temperature measurements in thermal vacuum testing |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MASSACHUSETTS INSTITUTE OF TECHNOLOGY, MASSACHUSET Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PHAN, LONG N.;JESNECK, JONATHAN LEE;SARMA, SANJAY;REEL/FRAME:028741/0767 Effective date: 20120806 |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2551) Year of fee payment: 4 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20220826 |