WO2008127360A2

WO2008127360A2 - Real time threat detection system

Info

Publication number: WO2008127360A2
Application number: PCT/US2007/081056
Authority: WO
Inventors: John William Foster
Original assignee: Thermal Matrix, Inc.
Priority date: 2006-10-11
Filing date: 2007-10-11
Publication date: 2008-10-23
Also published as: WO2008127360A3

Abstract

A real-time threat detection system incorporating a plurality of sensors adapted to detect radiation across the majority of the electromagnetic spectrum. The system also includes an aided or automatic target recognition module which compares the data from the sensors against known radiation signatures and issues an alert when an anamolous signature is detected. The system further includes an operator station which displays sensor information allowing the operator to intervene. The sensors detect radiation which is normally emitted by persons or other bodies and display areas to the operator where normal emissions are blocked.

Description

REAL TIME THREAT DETECTION SYSTEM

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Non-Provisional Application of co-pending U.S. Provisional Application No. 60/829,041 , filed October 1 1 , 2006, which is incorporated herein by reference.

STATEMENT OF GOVERNMENT INTEREST

This invention was made with Government support under Grant No. W15P7T-06-L-9613 awarded by The Department of Defense. The Government, therefore, has certain rights in the invention.

FIELD OF INVENTION

This invention relates to surveillance, specifically the detection of Improvised Explosive Devices (IEDs).

SUMMARY OF INVENTION

The inventive system provides enhanced surveillance capabilities by incorporating multiple sensors, including thermal imaging devices, millimeter wave sensors and terahertz sensors in a comprehensive architecture capable of detecting a wide array of articles, such as improvised explosive devices. The system is designed to provide soldiers, law enforcement and other first responders with a rapidly deployable and reconfigurable tool to enhance security.

BRIEF DESCRIPTION OF THE DRAWINGS

For a fuller understanding of the invention, reference should be made to the following detailed description, taken in connection with the accompanying drawings, in which:

FIG. 1 is a representative image of the electromagnetic spectrum.

FIG. 2 is a block diagram of the inventive system.

FIG. 3 is an image of an illustrative display from the THz sub-module showing a THz image and real-time image.

FIG. 4 is an image of an illustrative display from the mmW sub-module.

FIG. 5 is an image of an illustrative display from the IR sub-module. FIG. 6 is an image of an illustrative display from the THz sub-module.

FIG. 7 is an image of an alternate display from the THz sub-module wherein areas of blocked THz radiation are shown on a real-time image.

FIG. 8 is an image of an illustrative display from the IR sub-module.

FIG. 9 is a display showing the comparison of known (or expected) thermal signatures compared to an abnormal thermal signature.

FIG. 10 is an alternate display showing the comparison of known (or expected) thermal signatures compared to an abnormal thermal signature.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The invention includes a counter-improvised explosive device (or "CIED") system-of-systems solution. In one embodiment, the invention provides a threat detection system adapted to detect threats posed by pedestrian suicide bombers and the like. In one embodiment, the threat detection system includes a suite of sensors adapted to detect radiation spanning the majority of the electromagnetic spectrum. The system possesses an open architecture, ensuring the continued integration of optimum sensor technology to meet current and future requirements. A preferred embodiment also utilizes an aided target recognition (ATR) module to perform real-time or near real-time aided pedestrian suicide bomber detection using sensor data. The system also includes an operator station adapted to display alerts in response to threats detected by the ATR module.

The system provides standoff threat detection, allowing a threat to be recognized from a safe distance. Moreover, the invention provides security forces the ability to detect at standoff ranges and with a high degree of certainty, concealed explosive devices and weapons worn under the target's apparel.

The invention includes a plurality of sensors since no existing sensor can perform the task of detecting explosive devices and weapons under all environmental conditions. The invention's "system-of-systems" approach integrates sensors and increases each technology's capabilities exponentially. The operator station adds a heightened capability in detecting anomalies, explosives and weapons. The invention also utilizes aided target recognition sub- modules for all sensors.

The use of sensors across the electromagnetic spectrum (FIG. 1 ) allows the invention to non- invasively detect whether or not a person poses a threat at distances exceeding 30 meters. The invention operates over several wave bands and all sensors are passive in nature and conform to all related safety regulations. Integrated System

A preferred embodiment, shown in FIG. 2, includes an integrated suite of sensors, preferrably mounted on a mobile platform capable of providing electric power and mobility. As shown in FIG. 2, threat detection system 10 includes a plurality of sensors (THz sensor 20, mmW sensor 22 and IR sensor 24), and visible spectrum camera 26 (/.e.CCTV) communicatively coupled to a processing unit (30) on which resides the aided target recognition module. The aided target recognition module further includes THz sensor sub-module 32, mmW sensor sub-module 34 and IR sensor sub-module 36. The aided target recognition module displays information from the sensors and sub-modules on operator unit 40.

This embodiment, however, represents only an illustrative configuration. For example, the invention can be concealed for covert observation and detection in target areas. In contrast, it can be openly displayed as a deterrent, while being used for overt observation and threat detection. Sensors used with the invention are preferably capable of penetrating through walls and can be easily incorporated into architectural features without compromising effectiveness.

Sensors

The threat detection system integrates multiple sensors that target different bands in the electromagnetic spectrum. Where previous threat detection systems rely on sensors or approaches focusing on one band, the present invention covers a majority of the electromagnetic spectrum. Each sensor detects emissions within a specific band and has its own capabilities based on the characteristics of that portion of the spectrum. By integrating the sensors into a cohesive whole, the invention mitigates the limitation of any sensor by harnessing the capabilities of other sensors.

Most clothing is transparent to radiation in the THz and mmW bands. Objects (weapons or bomb devices) that are worn beneath one's apparel will block the body's natural radiation and results in a signature that can be detected much as a lamp shade blocks the light from a light bulb. Under most circumstances, large objects such as bomb devices or protective vests worn under clothing present a detectable thermal signature which can be detected using thermal, or IR, sensors.

Terahertz (THz) Sensor

In one embodiment the invention utilizes sensors adapted to detect radiation in the 5-30 meter range (i.e. the T4000 passive THz sensor developed by ThruVision, United Kingdom). The use of THz energy to detect weapons and bomb devices hidden under clothing is a new development. The THz sensor works by detecting incoming THz energy in the sensor field of view. In one embodiment, a display application converts the signal to a digital image by assigning a value of 256 to the points of highest THz emissions and a value of 0 to the points of lowest emissions. This establishes the contrast limits and the remaining points in the field of view are normalized to fit between these limits. A look-up table assigns color values to the points, to make up a color image.

Several color palettes are available to provide multiple appearances to the image. The display application displays the images in real time and in full motion video. The THz digital image can be used by an Automatic or Aided Target Recognition (or "ATR"), discussed herein.

Referring now to FIG. 3, the operator station displays window 50 which is divided into THz image 52a and visible spectrum image 54. THz image 52a displays the information take from the THZ imager, whereas visible spectrum image 54 displays the real-time image captured by the camera (i.e. CCTV). In the example shown in FIG. 3, two target subjects (60, 62) are shown at a range of 30 meters. The target subject on the right (62) is not carrying any suspicious items that block THz radiation, thereby presenting a normal THz signature (72). The target subject on the left (60) is wearing a simulated bomb vest which blocks the THz radiation and causes the detectable signature (70). The THz image 52a is bore sighted with the visible spectrum camera. Box (56) in the center of visible image 54 indicates the THz sensor field of view. As FIG. 3 shows, the THz sensor can non-invasively detect an abnormal THz signature. In a preferred embodiment, the THz sensor is a passive sensor which emits no radiation.

Millimeter Wave Imager

In another embodiment, the invention incorporates a passive millimeter wave (or "mmW") sensor, such as the device developed by Brijot Imaging Systems, lnc in Orlando, Florida. The sensor of a preferred embodiment operates in the 80 - 100 gigahertz band of the electromagnetic spectrum. Preferably the mmW seonsor does not use the passive illumination of the sky to illuminate metal which may be hidden under clothing. The mmW sensor uses millimeter wave imagers to detect suspicious objects made of metal, plastic, ceramic, composite, liquid, and other materials on a person from a standoff distance of 1 to 5 meters.

In the embodiment shown in FIG. 4, the mmW sensor images targets at standoff ranges (preferably 5 meters to 3 meters). As shown in FIG. 4, the operator station displays window 50 which is separated into full motion window 54 (which displays a real image) and mmW image 52b. In the embodiment shown in FIG. 3, target 60 is carrying a weapon 58 under the shirt, which is clearly detected at a range of 3 meters. Here, visible spectrum image 54 is an integrated CCTV camera that is bore sighted with the mmW imager. The raw mmW image 52b is shown next to modified mmW image 52c which uses algorithms to locate and draw indicator box 56a around the hidden object. Real time indicator box 56 is then overlayed on visible spectrum image 54 to show where weapon 58 is hidden on target 60.

Infrared/Thermal Imagers

The most robust and mature sensors used by the invention are in the infrared band. The infrared, or thermal imagers, operate in both the 3-5 μm band and in the 8-12 μm band. These sensors detect temperature and temperature differential on the clothing surface.

FIG. 5 depicts three subjects (60, 62, 64), all wearing simulated bomb devices (58) under clothing, as viewed with thermal imagers. An illustrative IR sensor which is suitable for use with the invention includes, but is not limited to, the RECON III™, a tripod mounted sensor witht the ability to pan and tilt, manufactured by FLIR™. In a preferred embodiment, the IR sensor has two IR cameras: an un-cooled camera in the 8-12 μm bands; and a more sensitive camera in the 3-5 μm bands which uses a cooled InSB with 640 x 480 pixel resolutions. Another illustrative IR sensor is the Axsys™ FieldPro™ 5x which has a 3-5 μm band cooled InSB IR sensor with a continuous 5x optical zoom.

In a preferred embodiment, the invention includes a long range camera, such as a bore sighted CCTV camera, pole mounted to provide the ability to detect threats in excess of 200 meters.

Aided Target Recognition Module

Automatic Target Recognition is the machine function of detecting, classifying, recognizing, and identifying an object without human interaction. Aided Target Recognition (or "ATR") is a subset in which a human interacts with the system and makes some of the decisions. Herein, the invention utilizes an Aided Target Recognition module in conjunction with the suite of sensors, although the use of automatic target recognition is contemplated. As used herein, however, "ATR" means "Aided Target Recognition".

ATR Module for the THz Sensor

The invention includes a THz sensor sub-module adapted to receive and process information from the THz sensors. The THz sensor sub-module of a preferred embodiment receives data from the visible spectrum camera (such as a CCTV camera) and the THz to produce two images: a THz image and a CCTV camera image. In normal operation, the THz sub-module displays the information on the operator station. Once displayed the combined image is viewed on the operator station by a trained operator who uses his or her judgment in combination with the inventive system to detect hidden weapons or bombs. For example, shown in FIG. 6, three targets (60, 62, 64) approach the sensor array. The THz sub-module displays image 50 which comprises THz image 52a and CCTV image 54. The rightmost of the three targets (64) has a hidden explosive vest. Accordingly the THz signatures (70, 72, 74) are displayed and the signature(74) of the rightmost target (64) conceals his THz signature.

Here, the THz sub-module performs several functions. First the THz sub-module bore-sights the THz image with the CCTV image and overlays the THz image over the CCTV image. Second the THZ sub-module applies a binary filter to the data so that only two colors are represented rather than the normal 256. All areas where the THz energy is below the binary contrast setting are shown in a predetermined color (i.e. blue). All areas where the THz energy exceeds the contrast setting is shown as transparent. The result is an image as shown in FIG. 7.

The target subject (60) of FIG. 7 is wearing a surrogate suicide bomber vest. As the THz energy emitted by the human subject is above the contrast threshold, his image (54) is visible through the transparent color of the high energy area. But the THz energy emitted by the human is blocked where he is wearing the bomb device; so that area (59a)is shown in blue as a low energy area. Any weapon, device, or object that blocks the human's THz energy is shown as a solid area which overlays the spot on the human subject where that object is located. The area within the THz imager field of view surrounding the human subject (an area of signal clutter) is also shown as a solid area (59b) as it is an area of relatively low THz energy.

In a preferred embodiment, the operator station includes controls which will allow the operator to control the contrast threshold in real time. In this fashion the operator can precisely adjust the parameters that define the region that is represented in blue and the region that is represented as transparent.

ATR module for the mmW Sensor

The mmW sensor, as its name suggests, uses mmW energy to image suspicious objects made of metal, plastic, ceramic, composite, liquid, and other materials present on the target person from a stand-off distance. The mMW sub-module pinpoints the location of those objects using detection indicator boxes on a full motion video image of the subject. The mmW sub-module module can detect and locate large or small objects that pose a significant threat. The mmW sub-module creates a representation of the human subject using either the mmW energy or the full motion video image. The mmW sub-module calculates the percent contrast within the frame of the representative image. If the contrast levels fall below specified thresholds, then the mmW sub-module classifies this as detection. Detections can be further classified as a "warning" or as a more serious "alert" based on the strength of the detection and size of the detected object. The user can customize the detection thresholds.

ATR module for the Thermal Sensor

Under many circumstances, thermal imagery contains a thermally significant signature which can be used to identify suicide bombers. Each of the various components of a bomb device has a unique emissivity. This variation in emissivity creates a characteristic thermal contrast which can be used to detect the presence of the device. The thermal sensor sub-module uses real time cueing methodology that maximizes the detection value of this characteristic thermal contrast.

IG. 8 shows the side view of a target (60) wearing a suicide bomb device (58). The image on the left (50) is the normal image and the image on the right (50a) has been distorted along the horizontal axis in order to intensify the resolution of the thermal contrast.

The thermal sensor sub-module analyzes the thermal contrast along each vertical pixel line and creates a histogram of the thermal contrast which is shown in FIG. 9. The average of the vertical thermal signature is shown as the dark line (90) on the far right. This average is compared with the envelope of a normal signature (92) which is more continuous and does not have the characteristic chaotic slope. See FIG. 10 where these envelopes are compared side by side.

This analysis has been confirmed in a post-processing modality. In a preferred embodiment, the post processing algorithm above is used by the thermal sensor sub-module in a real time / near real time embedded algorithm which is directly inserted into the ATR module data acquisition configuration. While the implementation of any algorithm in real time poses a challenge, especially a real time video analysis program, the STR module reduces the quantity of data so as not to fundamentally affect the post data processing analysis model.

Data processing generally requires a minimum of 10 to 100 times more memory requirement and processing power than data acquisition. The ATR module uses embedded algorithm development and integration for data visualization, processing and analysis in real time or near real time with minimum latency. As the result, the ATR module is capable of ingesting and caching a high volume of data (upwards of 2GB/s). This high bandwidth capability is extensively exploited for improved processing efficiency of the ATR module, thus enabling the real time algorithm to focus on segmentation and insertion of the cueing algorithm into each and every step appropriate along the data acquisition circuit path, and achieving the truly integrated software and hardware solution.

ATR Module Fusion In a preferred embodiment, the operator station displays a fused interface whereby two signatures are fused onto one image by the ATR module. For example, the ATR module uses a visible spectrum camera, such as a video image camera, to create a wide angle field of view image of the scene of interest.

Onto that image, the ATR module overlays the bore-sighted narrow field of view THz sensor and IR sensor images. The operator, therefore, views only one image and sees the results of the suite of sensors. Detection of threats is shown as color icons or areas portrayed directly on the image of the person who poses the threat. At a minimum, the THz and Visible imagery are fused. The sensors are preferably disposed on pan and tilt units so that the operator, or an automatic tracking program, can continue to track the person of interest.

Operator Display Unit

The operator display unit of the invention shows the images of the suite of sensors. In one embodiment, the operator display unit displays data from two to eight sensors on dual 19" screen monitors. As the operator moves the mouse curser over an image, that image instantaneously expands (i.e. by a factor of four) in size and the controls for that sensor are transferred to the operator's joy stick and keyboard. The operator will be able to control the brightness, contrast, pan and tilt of each camera and the zoom of multiple cameras.

Operators are trained to identify individuals who possess weapons or explosive devices. Images from all sensors are shown in real time with all capable of 30 frames per second video input at all times. The operator can "mouse-over" the image of interest and see more detail and to control that sensor and then immediately shift to another sensor for more detail.

The operator display is preferably an open architecture so that future sensors and improvements to existing sensors can quickly and easily be incorporated into the display. The operator station is built as to allow the system to easily mature as sensors and algorithms improve. Alerts and threat levels are shown on the images and incorporated into the display which is always in the field of view of the operator.

It will be seen that the advantages set forth above, and those made apparent from the foregoing description, are efficiently attained and since certain changes may be made in the above construction without departing from the scope of the invention, it is intended that all matters contained in the foregoing description or shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.

It is also to be understood that the following claims are intended to cover all of the generic and specific features of the invention herein described, and all statements of the scope of the invention which, as a matter of language, might be said to fall therebetween. Now that the invention has been described,

Claims

What is claimed is:

1. A threat detection system, comprising:

a first sensor adapted to detect radiation in a first band of the electromagnetic spectrum;

a second sensor adapted to detect radiation in a second band of the electromagnetic spectrum;

a third sensor adapted to detect radiation in a third band of the electromagnetic spectrum;

a camera; and

an aided target recognition module in electrical communication with the camera, first, second and third sensors.

2. The threat detection system of claim 1 , wherein the first band of the electromagnetic spectrum is the terahertz band, the second band of the electromagnetic spectrum is the millimeter wave band and the third band of the electromagnetic spectrum is the infrared band.

3. The threat detection system of claim 1 , wherein the first sensor is a terahertz sensor.

4. The threat detection system of claim 3, wherein the aided target recognition module is adapted to:

capture a THz image from the terahertz sensor;

capture a visible spectrum image from the camera;

overlay the THz image on the visible spectrum image; and

establish a contrast setting having a predetermined color set;

wherein the color set of the contrast setting is less than the color set of the visible spectrum image;

wherein all areas of the THz image below the contrast setting are shown over the visible spectrum image in a color selected from the color set of the contrast setting; and wherein all areas of the THz image above the contrast setting are rendered as transparent.

5. The threat detection system of claim 4, wherein the THz energy emitted by a human is above the contrast threshold; and wherein the areas where THz energy is blocked is below the contrast threshold.

6. The threat detection system of claim 4, wherein the color set of the contrast setting is binary.

7. The threat detection system of claim 6, wherein the color selected from the color set of the contrast setting is blue.

8. The threat detection system of claim 1 , wherein the second sensor is a millimeter wave sensor.

9. The threat detection system of claim 8, wherein the aided target recognition module is adapted to:

capture a millimeter wave image of a subject; and

detect an area of contrast within the millimeter wave image.

10. The threat detection system of claim 9, wherein the aided target recognition module is adapted to:

capture a visible spectrum image of the subject from the camera;

merge the millimeter wave image with the visible spectrum image; and

identify a position on the subject correlating to the area of contrast on the millimeter wave image.

1 1. The threat detection system of claim 10, wherein the aided target recognition module is adapted to highlight an area on the merged image and the visible spectrum image correlating to the area of contrast on the millimeter wave image.

12. The threat detection system of claim 10, wherein the visible spectrum image of the subject is full motion video.

13. The threat detection system of claim 1 , wherein the third sensor is an infrared sensor.

14. The threat detection system of claim 13, wherein the aided target recognition module is adapted to:

capture a thermal image of a subject;

identify an area of thermal contrast within the image;

create a thermal contrast profile;

compare the thermal contrast profile to a predetermined thermal signature; and

detect variations between the thermal contrast profile and the predetermined thermal signature.

15. The threat detection system of claim 14, wherein the resolution of the area of thermal contrast within the thermal image is intensified by altering the thermal image along at least one axis.

16. The threat detection system of claim 14, wherein the aided target recognition module is further adapted to:

analyze each pixel line of the thermal contrast along at least one axis to create a histogram; and

store the thermal contrast histogram in the thermal contrast profile.