US20130002858A1

US20130002858A1 - Mechanisms for Conserving Power in a Compressive Imaging System

Info

Publication number: US20130002858A1
Application number: US13/534,528
Authority: US
Inventors: Robert F. Bridge; James M. Tidman; Lenore McMackin; Sujoy Chatterjee
Original assignee: InView Tech Corp
Current assignee: InView Tech Corp
Priority date: 2011-06-28
Filing date: 2012-06-27
Publication date: 2013-01-03
Also published as: WO2013003485A1; US9160914B2; US20130002715A1; US20130002968A1

Abstract

A system and method for conserving power in compressive imaging. An optical subsystem separates an incident light stream into a primary light stream and a secondary light stream. The primary light stream is modulated with a sequence of spatial patterns by a light modulator. The modulated light stream is sensed by a first light sensing device. The secondary light stream is sensed by a second light sensing device. The signal(s) produced by the second light sensing device may be monitored to determine when to turn on power to the light modulator. Thus, the light modulator may remain off when not needed. In an alternative implementation, a light sensing device is used to sense the light reflected from the light modulator in its power-off state. The signal(s) produced by that light sensing device may be monitored to determine when to turn on power to the light modulator.

Description

RELATED APPLICATION DATA

This application claims the benefit of priority to U.S. Provisional Application No. 61/502,153, filed on Jun. 28, 2011, entitled “Various Compressive Sensing Mechanisms”, invented by Tidman, Weston, Bridge, McMackin, Chatterjee, Woods, Baraniuk and Kelly, which is hereby incorporated by reference in its entirety as though fully and completely set forth herein.

FIELD OF THE INVENTION

The present invention relates to the field of compressive imaging, and more particularly, to mechanisms for conserving power in a compressive imaging system.

DESCRIPTION OF THE RELATED ART

According to Nyquist theory, a signal x(t) whose signal energy is supported on the frequency interval [−B,B] may be reconstructed from samples {x(nT)} of the signal x(t), provided the rate f_S=1/T_Sat which the samples are captured is sufficiently high, i.e., provided that f_Sis greater than 2B. Similarly, for a signal whose signal energy is supported on the frequency interval [A,B], the signal may be reconstructed from samples captured with sample rate greater than B−A. A fundamental problem with any attempt to capture a signal x(t) according to Nyquist theory is the large number of samples that are generated, especially when B (or B−A) is large. The large number of samples is taxing on memory resources and on the capacity of transmission channels.
Nyquist theory is not limited to functions of time. Indeed, Nyquist theory applies more generally to any function of one or more real variables. For example, Nyquist theory applies to functions of two spatial variables such as images, to functions of time and two spatial variables such as video, and to the functions used in multispectral imaging, hyperspectral imaging, medical imaging and a wide variety of other applications. In the case of an image I(x,y) that depends on spatial variables x and y, the image may be reconstructed from samples of the image, provided the samples are captured with sufficiently high spatial density. For example, given samples {I(nΔx,mΔy)} captured along a rectangular grid, the horizontal and vertical densities 1/Δx and 1/Δy should be respectively greater than 2B_xand 2B_y, where B_xand B_yare the highest x and y spatial frequencies occurring in the image I(x,y). The same problem of overwhelming data volume is experienced when attempting to capture an image according to Nyquist theory. The modem theory of compressive sensing is directed to such problems.
Compressive sensing relies on the observation that many signals (e.g., images) of practical interest are not only band-limited but also sparse or approximately sparse when represented using an appropriate choice of transformation, for example, a transformation such as a Fourier transform, a wavelet transform or a discrete cosine transform (DCT). A signal vector v is said to be K-sparse with respect to a given transformation T when the transformation of the signal vector, Tv, has no more than K non-zero coefficients. A signal vector v is said to be sparse with respect to a given transformation T when it is K-sparse with respect to that transformation for some integer K much smaller than the number L of components in the transformation vector Tv.
A signal vector v is said to be approximately K-sparse with respect to a given transformation T when the coefficients of the transformation vector, Tv, are dominated by the K largest coefficients (i.e., largest in the sense of magnitude or absolute value). In other words, if the K largest coefficients account for a high percentage of the energy in the entire set of coefficients, then the signal vector v is approximately K-sparse with respect to transformation T. A signal vector v is said to be approximately sparse with respect to a given transformation T when it is approximately K-sparse with respect to the transformation T for some integer K much less than the number L of components in the transformation vector Tv.
Given a sensing device that captures images with N samples per image and in conformity to the Nyquist condition on spatial rates, it is often the case that there exists some transformation and some integer K very much smaller than N such that the transform of each captured image will be approximately K sparse. The set of K dominant coefficients may vary from one image to the next. Furthermore, the value of K and the selection of the transformation may vary from one context (e.g., imaging application) to the next. Examples of typical transforms that might work in different contexts include the Fourier transform, the wavelet transform, the DCT, the Gabor transform, etc.
Compressive sensing specifies a way of operating on the N samples of an image so as to generate a much smaller set of samples from which the N samples may be reconstructed, given knowledge of the transform under which the image is sparse (or approximately sparse). In particular, compressive sensing invites one to think of the N samples as a vector v in an N-dimensional space and to imagine projecting the vector v onto each vector in a series of M vectors {R(i): i=1, 2, . . . , M} in the N-dimensional space, where M is larger than K but still much smaller than N. Each projection gives a corresponding real number s(i), e.g., according to the expression
s(i)=<v,R(i)>,
where the notation <v,R(i)> represents the inner product (or dot product) of the vector v and the vector R(i). Thus, the series of M projections gives a vector U including M real numbers: U_i=s(i). Compressive sensing theory further prescribes methods for reconstructing (or estimating) the vector v of N samples from the vector U of M real numbers. For example, according to one method, one should determine the vector x that has the smallest length (in the sense of the L₁norm) subject to the condition that ΦTx=U, where Φ is a matrix whose rows are the transposes of the vectors R(i), where T is the transformation under which the image is K sparse or approximately K sparse.
Compressive sensing is important because, among other reasons, it allows reconstruction of an image based on M measurements instead of the much larger number of measurements N recommended by Nyquist theory. Thus, for example, a compressive sensing camera would be able to capture a significantly larger number of images for a given size of image store, and/or, transmit a significantly larger number of images per unit time through a communication channel of given capacity.
As mentioned above, compressive sensing operates by projecting the image vector v onto a series of M vectors. As discussed in U.S. Pat. No. 8,199,244 (issued Jun. 12, 2012, invented by Baraniuk et al.) and illustrated in FIG. 1, an imaging device (e.g., camera) may be configured to take advantage of the compressive sensing paradigm by using a digital micromirror device (DMD) 40. An incident lightfield 10 passes through a lens 20 and then interacts with the DMD 40. The DMD includes a two-dimensional array of micromirrors, each of which is configured to independently and controllably switch between two orientation states. Each micromirror reflects a corresponding portion of the incident light field based on its instantaneous orientation. Any micromirrors in a first of the two orientation states will reflect their corresponding light portions so that they pass through lens 50. Any micromirrors in a second of the two orientation states will reflect their corresponding light portions away from lens 50. Lens 50 serves to concentrate the light portions from micromirrors in the first orientation state onto a photodiode (or photodetector) situated at location 60. Thus, the photodiode generates a signal whose amplitude at any given time represents a sum of the intensities of the light portions from the micromirrors in the first orientation state.
The compressive sensing is implemented by driving the orientations of the micromirrors through a series of spatial patterns. Each spatial pattern specifies an orientation state for each of the micromirrors. The output signal of the photodiode is digitized by an A/D converter 70. In this fashion, the imaging device is able to capture a series of measurements {s(i)} that represent inner products (dot products) between the incident light field and the series of spatial patterns without first acquiring the incident light field as a pixelized digital image. The incident light field corresponds to the vector v of the discussion above, and the spatial patterns correspond to the vectors R(i) of the discussion above.
The incident light field may be modeled by a function I(x,y,t) of two spatial variables and time. Assuming for the sake of discussion that the DMD comprises a rectangular array, the DMD implements a spatial modulation of the incident light field so that the light field leaving the DMD in the direction of the lens 50 might be modeled by
{I(nΔx,mΔy,t)*M(n,m,t)}
where m and n are integer indices, where I(nΔx,mΔy,t) represents the portion of the light field that is incident upon that (n,m)^thmirror of the DMD at time t. The function M(n,m,t) represents the orientation of the (n,m)^thmirror of the DMD at time t. At sampling times, the function M(n,m,t) equals one or zero, depending on the state of the digital control signal that controls the (n,m)^thmirror. The condition M(n,m,t)=1 corresponds to the orientation state that reflects onto the path that leads to the lens 50. The condition M(n,m,t)=0 corresponds to the orientation state that reflects away from the lens 50.
The lens 50 concentrates the spatially-modulated light field
{I(nΔx,mΔy,t)*M(n,m,t)}
onto a light sensitive surface of the photodiode. Thus, the lens and the photodiode together implement a spatial summation of the light portions in the spatially-modulated light field:
$S (t) = \sum_{n, m} I (n Δ x, m Δ y, t) M (n, m, t) .$
Signal S(t) may be interpreted as the intensity at time t of the concentrated spot of light impinging upon the light sensing surface of the photodiode. The A/D converter captures measurements of S(t). In this fashion, the compressive sensing camera optically computes an inner product of the incident light field with each spatial pattern imposed on the mirrors. The multiplication portion of the inner product is implemented by the mirrors of the DMD. The summation portion of the inner product is implemented by the concentrating action of the lens and also the integrating action of the photodiode.
In a compressive sensing (CS) device such as that described above, component devices such as the DMD, the A/D converter and the photodiode consume electrical power. In an application where continuous operation over a period of time is desired, it would save power to shut down power to one or more of these devices when there is nothing interesting in the field of view. However, when one or more of these devices is turned off, the above-described CS device is effectively blind, unable to detect when an object of interest appears in the field of view, and thus, when to turn itself on. Thus, there exists a need for mechanisms for operating a CS device in a low power mode that enables the detection of the appearance of an object of interest in the field of view.

SUMMARY

In some embodiments, a system may include an optical subsystem, a light modulation unit, a first light sensing device, a second light sensing device and a control unit.
The optical subsystem may be configured to receive an incident light stream and to separate the incident light stream into a first light stream and a second light stream. The first light stream is provided to the light modulation unit while the second light stream is supplied to the second light sensing device.
The light modulation unit includes a plurality of light reflecting elements whose orientations are independently controllable. Each of the light reflecting elements is configured to controllably switch between two active orientation states when the light modulation unit is powered on (i.e., is in the power-on state). When the light modulation unit is powered on, it is configured to modulate the first light stream with a time sequence of spatial patterns to obtain a modulated light stream.
The first light sensing device is configured to receive the modulated light stream and to generate a first data stream in response to the modulated light stream when the light modulation unit is powered on. The first data stream may include samples of intensity of the modulated light stream, e.g., samples of intensity as a function of time, or, samples of intensity as a function of time and cross-sectional position within the modulated light stream, or, samples of intensity as a function of time and wavelength.
The second light sensing device is configured to receive the second light stream and to generate a second data stream in response to the second light stream at least when the light modulation unit is powered off. The second data stream may include samples of intensity of the second light stream, e.g., samples of intensity as a function of time, or, samples of intensity as a function of time and cross-sectional position within the second light stream, or, samples of intensity as a function of time and wavelength.
The control unit may be configured to monitor the second data stream when the light modulation unit is powered off in order to detect a light variation event in the second data stream. The light variation event indicates a variation of light in the second light stream and may be indicative of the entrance or appearance of an object of interest into the field of view. The control unit may turn on power to the light modulation unit in response to determining that a trigger condition is satisfied. The trigger condition may include detection of the light variation event. The trigger condition may also include one or more other contributing sub-conditions. By maintaining the light modulation unit is a power-off state until the trigger condition is satisfied, the system is able to conserve power.
In some embodiments, a system may include a light modulation unit, a first light sensing device, a second light sensing device and a control unit.
The light modulation unit includes a plurality of light reflecting elements whose orientations are independently controllable. Each of the light reflecting elements is configured to assume a neutral orientation state when the light modulation unit is powered off, and to controllably switch between two active orientation states when the light modulation unit is powered on. The light reflecting elements are configured to modulate an incident light stream with a time sequence of spatial patterns when the light modulation unit is powered on (i.e., is in the power-on state). The action of modulating the incident light stream produces a modulated light stream. Furthermore, the light reflecting elements are configured to reflect the incident light stream from the neutral orientation state when the light modulation unit is powered off in order to produce a neutral-state light stream.
The first light sensing device is configured to receive the modulated light stream when the light modulation unit is powered on, and to generate a first data stream in response to the modulated light stream.
The second light sensing device is configured to receive the neutral-state light stream when the light modulation unit is powered off, and to generate a second data stream in response to the neutral-state light stream.
The control unit is configured to monitor the second data stream when the light modulation unit is powered off in order to detect a light variation event in the second data stream. The light variation event indicates a variation of light in the neutral-state light stream. The control unit may turn on power to the light modulation unit in response to determining that a trigger condition is satisfied. The trigger condition may include the detection of the light variation event in the second data stream.
Various additional embodiments are described in U.S. Provisional Application No. 61/502,153, filed on Jun. 28, 2011, entitled “Various Compressive Sensing Mechanisms”.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when the following detailed description of the preferred embodiments is considered in conjunction with the following drawings.

FIG. 1 illustrates a compressive sensing camera according to the prior art.

FIG. 2A illustrates one embodiment of a system 100 that is operable to capture compressive imaging samples and also samples of background light level. (LMU is an acronym for “light modulation unit”. MLS is an acronym for “modulated light stream”. LSD is an acronym for “light sensing device”.)

FIG. 2B illustrates an embodiment of system 100 that includes a processing unit 150.

FIG. 2C illustrates an embodiment of system 100 that includes an optical subsystem 105 to focus received light L onto the light modulation unit 110.

FIG. 2D illustrates an embodiment of system 100 that includes an optical subsystem 117 to direct or focus or concentrate the modulated light stream MLS onto the light sensing device 130.

FIG. 2E illustrates an embodiment where the optical subsystem 117 is realized by a lens 117L.

FIG. 2F illustrates an embodiment of system 100 that includes a control unit that is configured to supply a series of spatial patterns to the light modulation unit 110.

FIG. 3A illustrates system 200, where the light modulation unit 110 is realized by a plurality of mirrors (collectively referenced by label 110M).

FIG. 3B shows an embodiment of system 200 that includes the processing unit 150.

FIG. 4 shows an embodiment of system 200 that includes the optical subsystem 117 to direct or focus or concentrate the modulated light stream MLS onto the light sensing device 130.

FIG. 5A shows an embodiment of system 200 where the optical subsystem 117 is realized by the lens 117L.

FIG. 5B shows an embodiment of system 200 where the optical subsystem 117 is realized by a mirror 117M and lens 117L in series.

FIG. 5C shows another embodiment of system 200 that includes a TIR prism pair 107.

FIG. 6 illustrates one embodiment of a system 600 that operates in a low power mode until a light variation event is detected, whereupon it turns on power to the light modulation unit 620.

FIG. 7 illustrates an embodiment of system 600 that includes a transmitter 660.

FIG. 7B illustrates an embodiment where system 600 transmits compressively-acquired measurements to a remote system for remote image (or image sequence) reconstruction.

FIG. 8 illustrates an embodiment of system 600 where the light sensing device 640 is realized by a motion sensor.

FIG. 9 illustrates an embodiment of system 600 that includes a motion sensor in addition to the

light sensing devices

630 and 640.

FIG. 10 illustrates an embodiment of system 600 where the optical subsystem 610 includes a TIR prism pair 610T.

FIG. 11 illustrates an embodiment of system 600 where the optical subsystem 610 includes a beam splitter 610S and a TIR prism pair 610T.

FIG. 12 illustrates an embodiment of system 600 including two light sensing devices downstream from the light modulation unit 620.

FIG. 13 illustrates an embodiment of system 600 including a diffraction grating to separate the modulated light stream into a zero order beam B_oand a higher order beam B₁. Those beams are separately sensed by

light sensing device

630M and 630C.

FIG. 14 illustrates an embodiment of system 600 where the modulated light stream MLS is sensed by a single-element detector 630SE, and the light stream S₂is sensed by a sensor array 640SA.

FIG. 15A illustrates the incident light stream L being transmitted and reflected at an interface 1510 of the TIR prism pair 610T, and illustrates the path of the transmitted stream Y when the light modulation unit 620D in the powered-off state, according to one embodiment.

FIG. 15B illustrates the path of the modulated light stream MLS when the light modulation unit 620D is in the powered-on state, according to one embodiment.

FIG. 15C illustrates the path of a complementary modulated light stream MLS′ when the light modulation unit 620D is in the powered-on state, according to one embodiment.

FIG. 16 illustrates an embodiment of a method 1600 for operating a compressive sensing system in a manner that conserves power.

FIG. 17 illustrates one embodiment of a system 1700 that monitors the neutral-state light stream that is reflected from a light modulation unit 1720 when the light modulation unit is in the power-off state.

FIG. 18A illustrates an example of the reflection of the incident light stream L from the light reflecting elements of the light modulation unit when the light modulation unit is in the power-off state.

FIG. 18B illustrates how, in one embodiment, the modulated light stream MLS is generated by the light modulation unit 1720 when the light modulation unit is in the power-on state. The modulated light stream comprises portions of the incident light stream reflected by mirrors in the 1 state (as opposed to the 0 state).

FIG. 19 illustrates one embodiment of system 1700 where the light modulation unit 1720 is tilted with respect to the incident light stream L.

FIG. 20 illustrates one embodiment of system 1700 including a motion sensor 1765 in addition to the

light sensors

1730 and 1740.

FIG. 21 illustrates one embodiment of system 1700 including a dual TIR prism 1710 that generates a neutral-state light stream NLS when the light modulation unit 1720 is powered off, and generates a modulated light stream MLS when the light modulation unit is powered on.

FIG. 22A illustrates, according to one embodiment, the generation of the neutral-state light stream NLS by means of a representative reflection from a representative micromirror RM when the light modulation unit is powered off.

FIG. 22B illustrates, according to one embodiment, the generation of the modulated light stream MLS by means of a representative reflection from a representative micromirror RM (in the one state) when the light modulation unit is powered on.

FIG. 23 illustrates one embodiment of a method 2300 for operating a compressive sensing device that monitors the neutral-state light stream produced from the light modulation unit when the light modulation unit is powered off.

FIG. 24A illustrates one embodiment of a system 2400 that turns on power to a light modulation unit in response to the detection of a disturbance from a motion sensor.

FIG. 24B illustrates an embodiment of system 2400 as a camera device.

FIG. 24C illustrates an embodiment of system 2400 as a separate camera subsystem 2400A and a motion detection subsystem 2400B.

FIG. 25 illustrates one embodiment of a compressive imaging system 2400 including one or more detector channels.

FIG. 26 illustrates one embodiment of a compressive imaging system 2500 where separate portions of the modulated light stream MLS are delivered to respective light sensing devices.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and are herein described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Terminology

A memory medium is a non-transitory medium configured for the storage and retrieval of information. Examples of memory media include: various kinds of semiconductor-based memory such as RAM and ROM; various kinds of magnetic media such as magnetic disk, tape, strip and film; various kinds of optical media such as CD-ROM and DVD-ROM; various media based on the storage of electrical charge and/or any of a wide variety of other physical quantities; media fabricated using various lithographic techniques; etc. The term “memory medium” includes within its scope of meaning the possibility that a given memory medium might be a union of two or more memory media that reside at different locations, e.g., on different chips in a system or on different computers in a network.
A computer-readable memory medium may be configured so that it stores program instructions and/or data, where the program instructions, if executed by a computer system, cause the computer system to perform a method, e.g., any of a method embodiments described herein, or, any combination of the method embodiments described herein, or, any subset of any of the method embodiments described herein, or, any combination of such subsets.
A computer system is any device (or combination of devices) having at least one processor that is configured to execute program instructions stored on a memory medium. Examples of computer systems include personal computers (PCs), workstations, laptop computers, tablet computers, mainframe computers, server computers, client computers, network or Internet appliances, hand-held devices, mobile devices, personal digital assistants (PDAs), tablet computers, computer-based television systems, grid computing systems, wearable computers, computers implanted in living organisms, computers embedded in head-mounted displays, computers embedded in sensors forming a distributed network, etc.
A programmable hardware element (PHE) is a hardware device that includes multiple programmable function blocks connected via a system of programmable interconnects. Examples of PHEs include FPGAs (Field Programmable Gate Arrays), PLDs (Programmable Logic Devices), FPOAs (Field Programmable Object Arrays), and CPLDs (Complex PLDs). The programmable function blocks may range from fine grained (combinatorial logic or look up tables) to coarse grained (arithmetic logic units or processor cores).
As used herein, the term “light” is meant to encompass within its scope of meaning any electromagnetic radiation whose spectrum lies within the wavelength range [λ_L, λ_U], where the wavelength range includes the visible spectrum, the ultra-violet (UV) spectrum, infrared (IR) spectrum and the terahertz (THz) spectrum. Thus, for example, visible radiation, or UV radiation, or IR radiation, or THz radiation, or any combination thereof is “light” as used herein.
In some embodiments, a computer system may be configured to include a processor (or a set of processors) and a memory medium, where the memory medium stores program instructions, where the processor is configured to read and execute the program instructions stored in the memory medium, where the program instructions are executable by the processor to implement a method, e.g., any of the various method embodiments described herein, or, any combination of the method embodiments described herein, or, any subset of any of the method embodiments described herein, or, any combination of such subsets.
System 100 for Operating on Light
A system 100 for operating on light may be configured as shown in FIG. 2A. The system 100 may include a light modulation unit 110, a light sensing device 130 and an analog-to-digital converter (ADC) 140.
The light modulation unit 110 is configured to modulate a received stream of light L with a series of spatial patterns in order to produce a modulated light stream (MLS). The spatial patterns of the series may be applied sequentially to the light stream so that successive time slices of the light stream are modulated, respectively, with successive ones of the spatial patterns. (The action of sequentially modulating the light stream L with the spatial patterns imposes the structure of time slices on the light stream.) The light modulation unit 110 includes a plurality of light modulating elements configured to modulate corresponding portions of the light stream. Each of the spatial patterns specifies an amount (or extent or value) of modulation for each of the light modulating elements. Mathematically, one might think of the light modulation unit's action of applying a given spatial pattern as performing an element-wise multiplication of a light field vector (x_ij) representing a time slice of the light stream L by a vector of scalar modulation values (m_ij) to obtain a time slice of the modulated light stream: (m_ij)*(x_ij)=(m_ij*x_ij). The vector (m_ij) is specified by the spatial pattern. Each light modulating element effectively scales (multiplies) the intensity of its corresponding light stream portion by the corresponding scalar factor.
The light modulation unit 110 may be realized in various ways. In some embodiments, the LMU 110 may be realized by a plurality of mirrors (e.g., micromirrors) whose orientations are independently controllable. In another set of embodiments, the LMU 110 may be realized by an array of elements whose transmittances are independently controllable, e.g., as with an array of LCD shutters. An electrical control signal supplied to each element controls the extent to which light is able to transmit through the element. In yet another set of embodiments, the LMU 110 may be realized by an array of independently-controllable mechanical shutters (e.g., micromechanical shutters) that cover an array of apertures, with the shutters opening and closing in response to electrical control signals, thereby controlling the flow of light through the corresponding apertures. In yet another set of embodiments, the LMU 110 may be realized by a perforated mechanical plate, with the entire plate moving in response to electrical control signals, thereby controlling the flow of light through the corresponding perforations. In yet another set of embodiments, the LMU 110 may be realized by an array of transceiver elements, where each element receives and then immediately retransmits light in a controllable fashion. In yet another set of embodiments, the LMU 110 may be realized by a grating light valve (GLV) device. In yet another embodiment, the LMU 110 may be realized by a liquid-crystal-on-silicon (LCOS) device.
In some embodiments, the light modulating elements are arranged in an array, e.g., a two-dimensional array or a one-dimensional array. Any of various array geometries are contemplated. For example, in some embodiments, the array is a square array or rectangular array. In another embodiment, the array is hexagonal. In some embodiments, the light modulating elements are arranged in a spatially random fashion.
Let N denote the number of light modulating elements in the light modulation unit 110. In various embodiments, the number N may take a wide variety of values. For example, in different sets of embodiments, N may be, respectively, in the range [64, 256], in the range [256, 1024], in the range [1024,4096], in the range [2¹²,2¹⁴], in the range [2¹⁴,2¹⁶], in the range [2¹⁶,2¹⁸], in the range [2¹⁸,2²⁰], in the range [2²⁰,2²²], in the range [2²²,2²⁴], in the range [2²⁴,2²⁶], in the range from 2²⁶to infinity. The particular value used in any given embodiment may depend on one or more factors specific to the embodiment.
The light sensing device 130 may be configured to receive the modulated light stream MLS and to generate an analog electrical signal I_MLS(t) representing intensity of the modulated light stream as a function of time.
The light sensing device 130 may include one or more light sensing elements. The term “light sensing element” may be interpreted as meaning “a transducer between a light signal and an electrical signal”. For example, a photodiode is a light sensing element. In various other embodiments, light sensing elements might include devices such as metal-semiconductor-metal (MSM) photodetectors, phototransistors, phototubes and photomultiplier tubes.
In some embodiments, the light sensing device 130 includes one or more amplifiers (e.g., transimpedance amplifiers) to amplify the analog electrical signals generated by the one or more light sensing elements.
The ADC 140 acquires a sequence of samples {I_MLS(k)} of the analog electrical signal I_MLS(t). Each of the samples may be interpreted as an inner product between a corresponding time slice of the light stream L and a corresponding one of the spatial patterns. The set of samples {I_MLS(k)} comprises an encoded representation, e.g., a compressed representation, of an image and may be used to reconstruct the image based on any reconstruction algorithm known in the field of compressive sensing. (The image is said to be “reconstructed” because it is recognized as having previously existed, although only transiently, in the incident light stream. By use of the term “reconstruct”, we do not mean to suggest that the image has existed in stored digital form prior to the acquisition of the samples.) To reconstruct a sequence of images, the samples of the sequence {I_MLS(k)} may be partitioned into contiguous subsets, and then the subsets may be processed to reconstruct corresponding images.
In some embodiments, the samples {I_MLS(k)} may be used for some purpose other than, or in addition to, image (or image sequence) reconstruction. For example, system 100 (or some other system) may operate on the samples to perform an inference task, such as detecting the presence of a signal or object, identifying a signal or an object, classifying a signal or an object, estimating one or more parameters relating to a signal or an object, tracking a signal or an object, etc. In some embodiments, an object under observation by system 100 may be identified or classified by virtue of its sample set {I_MLS(k)} (or parameters derived from that sample set) being similar to one of a collection of stored sample sets (or parameter sets).
In some embodiments, the light sensing device 130 includes exactly one light sensing element. (For example, the single light sensing element may be a photodiode.) The light sensing element may couple to an amplifier (e.g., a TIA) (e.g., a multi-stage amplifier).
In some embodiments, the light sensing device 130 may include a plurality of light sensing elements (e.g., photodiodes). Each light sensing element may convert light impinging on its light sensing surface into a corresponding analog electrical signal representing intensity of the impinging light as a function of time. In some embodiments, each light sensing element may couple to a corresponding amplifier so that the analog electrical signal produced by the light sensing element can be amplified prior to digitization. System 100 may be configured so that each light sensing element receives, e.g., a corresponding spatial portion (or spectral portion) of the modulated light stream.
In one embodiment, the analog electrical signals produced, respectively, by the light sensing elements may be summed to obtain a sum signal. The sum signal may then be digitized by the ADC 140 to obtain the sequence of samples {I_MLS(k)}. In another embodiment, the analog electrical signals may be individually digitized, each with its own ADC, to obtain corresponding sample sequences. The sample sequences may then be added to obtain the sequence {I_MLS(k)}. In another embodiment, the analog electrical signals produced by the light sensing elements may be sampled by a smaller number of ADCs than light sensing elements through the use of time multiplexing. For example, in one embodiment, system 100 may be configured to sample two or more of the analog electrical signals by switching the input of an ADC among the outputs of the two or more corresponding light sensing elements at a sufficiently high rate.
In some embodiments, the light sensing device 130 may include an array of light sensing elements. Arrays of any of a wide variety of sizes, configurations and material technologies are contemplated. In one embodiment, the light sensing device 130 includes a focal plane array coupled to a readout integrated circuit. In one embodiment, the light sensing device 130 may include an array of cells, where each cell includes a corresponding light sensing element and is configured to integrate and hold photo-induced charge created by the light sensing element, and to convert the integrated charge into a corresponding cell voltage. The light sensing device may also include (or couple to) circuitry configured to sample the cell voltages using one or more ADCs.
In some embodiments, the light sensing device 130 may include a plurality (or array) of light sensing elements, where each light sensing element is configured to receive a corresponding spatial portion of the modulated light stream, and each spatial portion of the modulated light stream comes from a corresponding sub-region of the array of light modulating elements. (For example, the light sensing device 130 may include a quadrant photodiode, where each quadrant of the photodiode is configured to receive modulated light from a corresponding quadrant of the array of light modulating elements. As another example, the light sensing device 130 may include a bi-cell photodiode. As yet another example, the light sensing device 130 may include a focal plane array.) Each light sensing element generates a corresponding signal representing intensity of the corresponding spatial portion as a function of time. Each signal may be digitized (e.g., by a corresponding ADC, or perhaps by a shared ADC) to obtain a corresponding sequence of samples. Thus, a plurality of sample sequences are obtained, one sample sequence per light sensing element. Each sample sequence may be processed to reconstruct a corresponding sub-image. The sub-images may be joined together to form a whole image. The sample sequences may be captured in response to the modulation of the incident light stream with a sequence of M spatial patterns, e.g., as variously described above. By employing any of various reconstruction algorithms known in the field of compressive sensing, the number of pixels in each reconstructed image may be greater than (e.g., much greater than) M. To reconstruct each sub-image, the reconstruction algorithm uses the corresponding sample sequence and the restriction of the spatial patterns to the corresponding sub-region of the array of light modulating elements.
In some embodiments, the light sensing device 130 includes a small number of light sensing elements (e.g., in respective embodiments, one, two, less than 8, less than 16, less the 32, less than 64, less than 128, less than 256). Because the light sensing device of these embodiments includes a small number of light sensing elements (e.g., far less than the typical modern CCD-based or CMOS-based camera), an entity interested in producing any of these embodiments may afford to spend more per light sensing element to realize features that are beyond the capabilities of modern array-based image sensors of large pixel count, e.g., features such as higher sensitivity, extended range of sensitivity, new range(s) of sensitivity, extended dynamic range, higher bandwidth/lower response time. Furthermore, because the light sensing device includes a small number of light sensing elements, an entity interested in producing any of these embodiments may use newer light sensing technologies (e.g., based on new materials or combinations of materials) that are not yet mature enough to be manufactured into focal plane arrays (FPA) with large pixel count. For example, new detector materials such as super-lattices, quantum dots, carbon nanotubes and graphene can significantly enhance the performance of IR detectors by reducing detector noise, increasing sensitivity, and/or decreasing detector cooling requirements.
In one embodiment, the light sensing device 130 is a thermo-electrically cooled InGaAs detector. (InGaAs stands for “Indium Gallium Arsenide”.) In other embodiments, the InGaAs detector may be cooled by other mechanisms (e.g., liquid nitrogen or a Sterling engine). In yet other embodiments, the InGaAs detector may operate without cooling. In yet other embodiments, different detector materials may be used, e.g., materials such as MCT (mercury-cadmium-telluride), InSb (Indium Antimonide) and VOx (Vanadium Oxide).
In different embodiments, the light sensing device 130 may be sensitive to light at different wavelengths or wavelength ranges. In some embodiments, the light sensing device 130 may be sensitive to light over a broad range of wavelengths, e.g., over the entire visible spectrum or over the entire range [λ_L,λ_ij] as defined above.
In some embodiments, the light sensing device 130 may include one or more dual-sandwich photodetectors. A dual sandwich photodetector includes two photodiodes stacked (or layered) one on top of the other.
In one embodiment, the light sensing device 130 may include one or more avalanche photodiodes.
In one embodiment, the light sensing device 130 may include one or more photomultiplier tubes (PMTS).
In some embodiments, a filter may be placed in front of the light sensing device 130 to restrict the modulated light stream to a specific range of wavelengths or specific polarization. Thus, the signal I_MLS(t) generated by the light sensing device 130 may be representative of the intensity of the restricted light stream. For example, by using a filter that passes only IR light, the light sensing device may be effectively converted into an IR detector. The sample principle may be applied to effectively convert the light sensing device into a detector for red or blue or green or UV or any desired wavelength band, or, a detector for light of a certain polarization.
In some embodiments, system 100 includes a color wheel whose rotation is synchronized with the application of the spatial patterns to the light modulation unit. As it rotates, the color wheel cyclically applies a number of optical bandpass filters to the modulated light stream MLS. Each bandpass filter restricts the modulated light stream to a corresponding sub-band of wavelengths. Thus, the samples captured by the ADC 140 will include samples of intensity in each of the sub-bands. The samples may be de-multiplexed to form separate sub-band sequences. Each sub-band sequence may be processed to generate a corresponding sub-band image. (As an example, the color wheel may include a red-pass filter, a green-pass filter and a blue-pass filter to support color imaging.)
In some embodiments, the system 100 may include a memory (or a set of memories of one or more kinds).
In some embodiments, system 100 may include a processing unit 150, e.g., as shown in FIG. 2B. The processing unit 150 may be a digital circuit or a combination of digital circuits. For example, the processing unit may be a microprocessor (or system of interconnected of microprocessors), a programmable hardware element such as a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any combination such elements. The processing unit 150 may be configured to perform one or more functions such as image (or image sequence) reconstruction, system control, user interface, statistical analysis, and one or more inferences tasks.
The system 100 (e.g., the processing unit 150) may store the samples {I_MLS(k)} in a memory, e.g., a memory resident in the system 100 or in some other system.
In one embodiment, processing unit 150 is configured to operate on the samples {I_MLS(k)} to generate the image (or image sequence). In this embodiment, the processing unit 150 may include a microprocessor configured to execute software (i.e., program instructions), especially software for executing an image reconstruction algorithm. In one embodiment, system 100 is configured to transmit the compensated samples to some other system through a communication channel. (In embodiments where the spatial patterns are randomly-generated, system 100 may also transmit the random seed(s) used to generate the spatial patterns.) That other system may operate on the samples to reconstruct the image (or image sequence). System 100 may have one or more interfaces configured for sending (and perhaps also receiving) data through one or more communication channels, e.g., channels such as wireless channels, wired channels, fiber optic channels, acoustic channels, laser-based channels, etc.
In some embodiments, processing unit 150 is configured to use any of a variety of algorithms and/or any of a variety of transformations to perform image (or image sequence) reconstruction. System 100 may allow a user to choose a desired algorithm and/or a desired transformation for performing the image (or image sequence) reconstruction.
In some embodiments, the system 100 is configured to acquire a set Z_Mof samples from the ADC 140 so that the sample set Z_Mcorresponds to M of the spatial patterns applied to the light modulation unit 110, where M is a positive integer. The number M is selected so that the sample set Z_Mis useable to reconstruct an n-pixel image or n-pixel image sequence that represents the incident light stream, where n is a positive integer less than or equal to the number N of light modulating elements in the light modulation unit 110. System 100 may be configured so that the number M is smaller than n. Thus, system 100 may operate as a compressive sensing device. (The number of “pixels” in an image sequence is the number of images in the image sequence times the number of pixels per image, or equivalently, the sum of the pixel counts of the images in the image sequence.)
In various embodiments, the compression ratio M/n may take any of a wide variety of values. For example, in different sets of embodiments, M/n may be, respectively, in the range [0.9,0.8], in the range [0.8,0.7], in the range [0.7,0.6], in the range [0.6,0.5], in the range [0.5,0.4], in the range [0.4,0.3], in the range [0.3,0.2], in the range [0.2,0.1], in the range [0.1,0.05], in the range [0.05,0.01], in the range [0.001,0.01].
As noted above, the image reconstructed from the sample subset Z_Mmay be an n-pixel image with n≦N. The spatial patterns may be designed to support a value of n less than N, e.g., by forcing the array of light modulating elements to operate at a lower effective resolution than the physical resolution N. For example, the spatial patterns may be designed to force each 2×2 cell of light modulating elements to act in unison. At any given time, the modulation state of the four elements in a 2×2 cell will agree. Thus, the effective resolution of the array of light modulating elements is reduced to N/4. This principle generalizes to any cell size, to cells of any shape, and to collections of cells with non-uniform cell size and/or cell shape. For example, a collection of cells of size k_H×k_V, where k_Hand k_Vare positive integers, would give an effective resolution equal to N/(k_Hk_V). In one alternative embodiment, cells near the center of the array may have smaller sizes than cells near the periphery of the array.
Another way the spatial patterns may be arranged to support the reconstruction of an n-pixel image with n less than N is to allow the spatial patterns to vary only within a subset of the array of light modulating elements. In this mode of operation, the spatial patterns are null (take the value zero) outside the subset. (Control unit 120 may be configured to implement this restriction of the spatial patterns.) Light modulating elements corresponding to positions outside of the subset do not send any light (or send only the minimum amount of light attainable) to the light sensing device. Thus, the reconstructed image is restricted to the subset. In some embodiments, each spatial pattern (e.g., of a measurement pattern sequence) may be multiplied element-wise by a binary mask that takes the one value only in the allowed subset, and the resulting product pattern may be supplied to the light modulation unit. In some embodiments, the subset is a contiguous region of the array of light modulating elements, e.g., a rectangle or a circular disk or a hexagon. In some embodiments, the size and/or position of the region may vary (e.g., dynamically). The position of the region may vary in order to track a moving object. The size of the region may vary in order to dynamically control the rate of image acquisition or frame rate. In some embodiments, the size of the region may be determined by user input. For example, system 100 may provide an input interface (GUI and/or mechanical control device) through which the user may vary the size of the region over a continuous range of values (or alternatively, a discrete set of values), thereby implementing a digital zoom function. Furthermore, in some embodiments, the position of the region within the field of view may be controlled by user input.
In one embodiment, system 100 may include a light transmitter configured to generate a light beam (e.g., a laser beam), to modulate the light beam with a data signal and to transmit the modulated light beam into space or onto an optical fiber. System 100 may also include a light receiver configured to receive a modulated light beam from space or from an optical fiber, and to recover a data stream from the received modulated light beam.
In one embodiment, system 100 may be configured as a low-cost sensor system having minimal processing resources, e.g., processing resources insufficient to perform image (or image sequence) reconstruction in user-acceptable time. In this embodiment, the system 100 may store and/or transmit the samples {I_MLS(k)} so that another agent, more plentifully endowed with processing resources, may perform the image (or image sequence) reconstruction based on the samples.
In some embodiments, system 100 may include an optical subsystem 105 that is configured to modify or condition the light stream L before it arrives at the light modulation unit 110, e.g., as shown in FIG. 2C. For example, the optical subsystem 105 may be configured to receive the light stream L from the environment and to focus the light stream onto a modulating plane of the light modulation unit 110. The optical subsystem 105 may include a camera lens (or a set of lenses). The lens (or set of lenses) may be adjustable to accommodate a range of distances to external objects being imaged/sensed/captured. The optical subsystem 105 may allow manual and/or digital control of one or more parameters such as focus, zoom, shutter speed and f-stop.
In some embodiments, system 100 may include an optical subsystem 117 to direct the modulated light stream MLS onto a light sensing surface (or surfaces) of the light sensing device 130.
In some embodiments, the optical subsystem 117 may include one or more lenses, and/or, one or more mirrors.
In some embodiments, the optical subsystem 117 is configured to focus the modulated light stream onto the light sensing surface (or surfaces). The term “focus” implies an attempt to achieve the condition that rays (photons) diverging from a point on an object plane converge to a point (or an acceptably small spot) on an image plane. The term “focus” also typically implies continuity between the object plane point and the image plane point (or image plane spot); points close together on the object plane map respectively to points (or spots) close together on the image plane. In at least some of the system embodiments that include an array of light sensing elements, it may be desirable for the modulated light stream MLS to be focused onto the light sensing array so that there is continuity between points on the light modulation unit LMU and points (or spots) on the light sensing array.
In some embodiments, the optical subsystem 117 may be configured to direct the modulated light stream MLS onto the light sensing surface (or surfaces) of the light sensing device 130 in a non-focusing fashion. For example, in a system embodiment that includes only one photodiode, it may not be so important to achieve the “in focus” condition at the light sensing surface of the photodiode since positional information of photons arriving at that light sensing surface will be immediately lost.
In one embodiment, the optical subsystem 117 may be configured to receive the modulated light stream and to concentrate the modulated light stream into an area (e.g., a small area) on a light sensing surface of the light sensing device 130. Thus, the diameter of the modulated light stream may be reduced (possibly, radically reduced) in its transit from the optical subsystem 117 to the light sensing surface (or surfaces) of the light sensing device 130. For example, in some embodiments, the diameter may be reduced by a factor of more than 1.5 to 1. In other embodiments, the diameter may be reduced by a factor of more than 2 to 1. In yet other embodiments, the diameter may be reduced by a factor of more than 10 to 1. In yet other embodiments, the diameter may be reduced by factor of more than 100 to 1. In yet other embodiments, the diameter may be reduced by factor of more than 400 to 1. In one embodiment, the diameter is reduced so that the modulated light stream is concentrated onto the light sensing surface of a single light sensing element (e.g., a single photodiode).
In some embodiments, this feature of concentrating the modulated light stream onto the light sensing surface (or surfaces) of the light sensing device allows the light sensing device to sense at any given time the sum (or surface integral) of the intensities of the modulated light portions within the modulated light stream. (Each time slice of the modulated light stream comprises a spatial ensemble of modulated light portions due to the modulation unit's action of applying the corresponding spatial pattern to the light stream.)
In some embodiments, the modulated light stream MLS may be directed onto the light sensing surface of the light sensing device 130 without concentration, i.e., without decrease in diameter of the modulated light stream, e.g., by use of photodiode having a large light sensing surface, large enough to contain the cross section of the modulated light stream without the modulated light stream being concentrated.
In some embodiments, the optical subsystem 117 may include one or more lenses. FIG. 2E shows an embodiment where optical subsystem 117 is realized by a lens 117L, e.g., a biconvex lens or a condenser lens.
In some embodiments, the optical subsystem 117 may include one or more mirrors. In one embodiment, the optical subsystem 117 includes a parabolic mirror (or spherical mirror) to concentrate the modulated light stream onto a neighborhood (e.g., a small neighborhood) of the parabolic focal point. In this embodiment, the light sensing surface of the light sensing device may be positioned at the focal point.
In some embodiments, system 100 may include an optical mechanism (e.g., an optical mechanism including one or more prisms and/or one or more diffraction gratings) for splitting or separating the modulated light stream MLS into two or more separate streams (perhaps numerous streams), where each of the streams is confined to a different wavelength range. The separate streams may each be sensed by a separate light sensing device. (In some embodiments, the number of wavelength ranges may be, e.g., greater than 8, or greater than 16, or greater than 64, or greater than 256, or greater than 1024.) Furthermore, each separate stream may be directed (e.g., focused or concentrated) onto the corresponding light sensing device as described above in connection with optical subsystem 117. The samples captured from each light sensing device may be used to reconstruct a corresponding image (or image sequence) for the corresponding wavelength range. In one embodiment, the modulated light stream is separated into red, green and blue streams to support color (R,G,B) measurements. In another embodiment, the modulated light stream may be separated into IR, red, green, blue and UV streams to support five-channel multi-spectral imaging: (IR, R, G, B, UV). In some embodiments, the modulated light stream may be separated into a number of sub-bands (e.g., adjacent sub-bands) within the IR band to support multi-spectral or hyper-spectral IR imaging. In some embodiments, the number of IR sub-bands may be, e.g., greater than 8, or greater than 16, or greater than 64, or greater than 256, or greater than 1024. In some embodiments, the modulated light stream may experience two or more stages of spectral separation. For example, in a first stage the modulated light stream may be separated into an IR stream confined to the IR band and one or more additional streams confined to other bands. In a second stage, the IR stream may be separated into a number of sub-bands (e.g., numerous sub-bands) (e.g., adjacent sub-bands) within the IR band to support multispectral or hyper-spectral IR imaging.
In some embodiments, system 100 may include an optical mechanism (e.g., a mechanism including one or more beam splitters) for splitting or separating the modulated light stream MLS into two or more separate streams, e.g., where each of the streams have the same (or approximately the same) spectral characteristics or wavelength range. The separate streams may then pass through respective bandpass filters to obtain corresponding modified streams, where each modified stream is restricted to a corresponding band of wavelengths. Each of the modified streams may be sensed by a separate light sensing device. (In some embodiments, the number of wavelength bands may be, e.g., greater than 8, or greater than 16, or greater than 64, or greater than 256, or greater than 1024.) Furthermore, each of the modified streams may be directed (e.g., focused or concentrated) onto the corresponding light sensing device as described above in connection with optical subsystem 117. The samples captured from each light sensing device may be used to reconstruct a corresponding image (or image sequence) for the corresponding wavelength band. In one embodiment, the modulated light stream is separated into three streams which are then filtered, respectively, with a red-pass filter, a green-pass filter and a blue-pass filter. The resulting red, green and blue streams are then respectively detected by three light sensing devices to support color (R,G,B) acquisition. In another similar embodiment, five streams are generated, filtered with five respective filters, and then measured with five respective light sensing devices to support (IR, R, G, B, UV) multi-spectral acquisition. In yet another embodiment, the modulated light stream of a given band may be separated into a number of (e.g., numerous) sub-bands to support multi-spectral or hyper-spectral imaging.
In some embodiments, system 100 may include an optical mechanism for splitting or separating the modulated light stream MLS into two or more separate streams. The separate streams may be directed to (e.g., concentrated onto) respective light sensing devices. The light sensing devices may be configured to be sensitive in different wavelength ranges, e.g., by virtue of their different material properties. Samples captured from each light sensing device may be used to reconstruct a corresponding image (or image sequence) for the corresponding wavelength range.
In some embodiments, system 100 may include a control unit 120 configured to supply the spatial patterns to the light modulation unit 110, as shown in FIG. 2F. The control unit may itself generate the patterns or may receive the patterns from some other agent. The control unit 120 and the ADC 140 may be controlled by a common clock signal so that ADC 140 can coordinate (synchronize) its action of capturing the samples {I_MLS(k)} with the control unit's action of supplying spatial patterns to the light modulation unit 110. (System 100 may include clock generation circuitry.)
In some embodiments, the control unit 120 may supply the spatial patterns to the light modulation unit in a periodic fashion.
The control unit 120 may be a digital circuit or a combination of digital circuits. For example, the control unit may include a microprocessor (or system of interconnected of microprocessors), a programmable hardware element such as a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any combination such elements.
In some embodiments, the control unit 120 may include a random number generator (RNG) or a set of random number generators to generate the spatial patterns or some subset of the spatial patterns.
In some embodiments, system 100 is battery powered. In some embodiments, the system 100 includes a set of one or more solar cells and associated circuitry to derive power from sunlight.
In some embodiments, system 100 includes its own light source for illuminating the environment or a target portion of the environment.
In some embodiments, system 100 may include a display (or an interface configured for coupling to a display) for displaying reconstructed images or image sequences, e.g., video sequences.
In some embodiments, system 100 may include one or more input devices (and/or, one or more interfaces for input devices), e.g., any combination or subset of the following devices: a set of buttons and/or knobs, a keyboard, a keypad, a mouse, a touch-sensitive pad such as a trackpad, a touch-sensitive display screen, one or more microphones, one or more temperature sensors, one or more chemical sensors, one or more pressure sensors, one or more accelerometers, one or more orientation sensors (e.g., a three-axis gyroscopic sensor), one or more proximity sensors, one or more antennas, etc.
Regarding the spatial patterns that are used to modulate the light stream L, it should be understood that there are a wide variety of possibilities. In some embodiments, the control unit 120 may be programmable so that any desired set of spatial patterns may be used.
In some embodiments, the spatial patterns are binary valued. Such an embodiment may be used, e.g., when the light modulating elements are two-state devices. In some embodiments, the spatial patterns are n-state valued, where each element of each pattern takes one of n states, where n is an integer greater than two. (Such an embodiment may be used, e.g., when the light modulating elements are each able to achieve n or more modulation states). In some embodiments, the spatial patterns are real valued, e.g., when each of the light modulating elements admits a continuous range of modulation. (It is noted that even a two-state modulating element may be made to effectively apply a continuous range of modulation by duty cycling the two states during modulation intervals.)
The spatial patterns may belong to a set of measurement vectors that is incoherent with a set of vectors in which the image is approximately sparse (“the sparsity vector set”). (See “Sparse Signal Detection from Incoherent Projections”, Proc. Int. Conf. Acoustics, Speech Signal Processing—ICASSP, May 2006, Duarte et al.) Given two sets of vectors A={a_i} and B={b_i} in the same N-dimensional space, A and B are said to be incoherent if their coherence measure μ(A,B) is sufficiently small. Assuming that the vectors {a_i} and the vectors {b_i} have unit L²norm, the coherence measure is defined as:
$μ (A, B) = \max_{i, j} \langle 〈 a_{i}, b_{j} 〉 \rangle .$
The number of compressive sensing measurements (i.e., samples of the sequence {I_MLS(k)} needed to reconstruct an N-pixel image that accurately represents the scene being captured is a strictly increasing function of the coherence between the measurement vector set and the sparsity vector set. Thus, better compression can be achieved with smaller values of the coherence. (The measurement vector set may also be referred to herein as the “measurement pattern set”. Likewise, the sparsity vector set may also be referred to herein as the “sparsity pattern set”.)
In some embodiments, the measurement vector set may be based on a code. Any of various codes from information theory may be used, e.g., codes such as exponentiated Kerdock codes, exponentiated Delsarte-Goethals codes, run-length limited codes, LDPC codes, Reed Solomon codes and Reed Muller codes.
In some embodiments, the measurement vector set corresponds to a randomized or permuted basis, where the basis may be, for example, the Discrete Cosine Transform (DCT) basis or Hadamard basis.
In some embodiments, the spatial patterns may be random or pseudo-random patterns, e.g., generated according to a random number generation (RNG) algorithm using one or more seeds. In some embodiments, the elements of each pattern are generated by a series of Bernoulli trials, where each trial has a probability p of giving the value one and probability 1−p of giving the value zero. (For example, in one embodiment p=½.) In some embodiments, the elements of each pattern are generated by a series of draws from a Gaussian random variable.)
The system 100 may be configured to operate in a compressive fashion, where the number of the samples {I_MLS(k)} captured by the system 100 is less than (e.g., much less than) the number of pixels in the image (or image sequence) to be reconstructed from the samples. In many applications, this compressive realization is very desirable because it saves on power consumption, memory utilization and transmission bandwidth consumption. However, non-compressive realizations are contemplated as well.
In some embodiments, the system 100 is configured as a camera or imager that captures information representing an image (or a series of images) from the external environment, e.g., an image (or a series of images) of some external object or scene. The camera system may take different forms in different application domains, e.g., domains such as visible light photography, infrared photography, ultraviolet photography, high-speed photography, low-light photography, underwater photography, multi-spectral imaging, hyper-spectral imaging, etc. In some embodiments, system 100 is configured to operate in conjunction with (or as part of) another system, e.g., in conjunction with (or as part of) a microscope, a telescope, a robot, a security system, a surveillance system, a fire sensor, a node in a distributed sensor network, etc.
In some embodiments, system 100 is configured as a spectrometer.
In some embodiments, system 100 is configured as a multi-spectral or hyper-spectral imager.
In some embodiments, system 100 may also be configured to operate as a projector. Thus, system 100 may include a light source, e.g., a light source located at or near a focal point of optical subsystem 117. In projection mode, the light modulation unit 110 may be supplied with an image (or a sequence of images), e.g., by control unit 120. The light modulation unit may receive a light beam generated by the light source, and modulate the light beam with the image (or sequence of images) to obtain a modulated light beam. The modulated light beam exits the system 100 and is displayed on a display surface (e.g., an external screen).
In one embodiment, the light modulation unit 110 may receive the light beam from the light source and modulate the light beam with a time sequence of spatial patterns (from a measurement pattern set). The resulting modulated light beam exits the system 100 and is used to illuminate the external scene. Light reflected from the external scene in response to the modulated light beam is measured by a light sensing device (e.g., a photodiode). The samples captured by the light sensing device comprise compressive measurements of external scene. Those compressive measurements may be used to reconstruct an image or image sequence as variously described above.
In some embodiments, system 100 includes an interface for communicating with a host computer. The host computer may send control information and/or program code to the system 100 via the interface. Furthermore, the host computer may receive status information and/or compressive sensing measurements from system 100 via the interface.
In one realization 200 of system 100, the light modulation unit 110 may be realized by a plurality of mirrors, e.g., as shown in FIG. 3A. (The mirrors are collectively indicated by the label 110M.) The mirrors 110M are configured to receive corresponding portions of the light L received from the environment, albeit not necessarily directly from the environment. (There may be one or more optical elements, e.g., one or more lenses along the input path to the mirrors 110M.) Each of the mirrors is configured to controllably switch between at least two orientation states. In addition, each of the mirrors is configured to (a) reflect the corresponding portion of the light onto a sensing path 115 when the mirror is in a first of the two orientation states and (b) reflect the corresponding portion of the light away from the sensing path when the mirror is in a second of the two orientation states.
In some embodiments, the mirrors 110M are arranged in an array, e.g., a two-dimensional array or a one-dimensional array. Any of various array geometries are contemplated. For example, in different embodiments, the array may be a square array, a rectangular array, a hexagonal array, etc. In some embodiments, the mirrors are arranged in a spatially-random fashion.
The mirrors 110M may be part of a digital micromirror device (DMD). For example, in some embodiments, one of the DMDs manufactured by Texas Instruments may be used.
The control unit 120 may be configured to drive the orientation states of the mirrors through the series of spatial patterns, where each of the patterns of the series specifies an orientation state for each of the mirrors.
The light sensing device 130 may be configured to receive the light portions reflected at any given time onto the sensing path 115 by the subset of mirrors in the first orientation state and to generate an analog electrical signal I_MLS(t) representing a cumulative intensity of the received light portions as function of time. As the mirrors are driven through the series of spatial patterns, the subset of mirrors in the first orientation state will vary from one spatial pattern to the next. Thus, the cumulative intensity of light portions reflected onto the sensing path 115 and arriving at the light sensing device will vary as a function time. Note that the term “cumulative” is meant to suggest a summation (spatial integration) over the light portions arriving at the light sensing device at any given time. This summation may be implemented, at least in part, optically (e.g., by means of a lens and/or mirror that concentrates or focuses the light portions onto a concentrated area as described above).
System realization 200 may include any subset of the features, embodiments and elements discussed above with respect to system 100. For example, system realization 200 may include the optical subsystem 105 to operate on the incoming light L before it arrives at the mirrors 110M, e.g., as shown in FIG. 3B.
In some embodiments, system realization 200 may include the optical subsystem 117 along the sensing path as shown in FIG. 4. The optical subsystem 117 receives the light portions reflected onto the sensing path 115 and directs (e.g., focuses or concentrates) the received light portions onto a light sensing surface (or surfaces) of the light sensing device 130. In one embodiment, the optical subsystem 117 may include a lens 117L, e.g., as shown in FIG. 5A.
In some embodiments, the optical subsystem 117 may include one or more mirrors, e.g., a mirror 117M as shown in FIG. 5B. Thus, the sensing path may be a bent path having more than one segment. FIG. 5B also shows one possible embodiment of optical subsystem 105, as a lens 105L.
In some embodiments, there may be one or more optical elements intervening between the optical subsystem 105 and the mirrors 110M. For example, as shown in FIG. 5C, a TIR prism pair 107 may be positioned between the optical subsystem 105 and the mirrors 110M. (TIR is an acronym for “total internal reflection”.) Light from optical subsystem 105 is transmitted through the TIR prism pair and then interacts with the mirrors 110M. After having interacted with the mirrors 110M, light portions from mirrors in the first orientation state are reflected by a second prism of the pair onto the sensing path 115. Light portions from mirrors in the second orientation state may be reflected away from the sensing path.
Separating Light Before Modulation to Detect Light Variation when the Modulator is Powered Off
In one set of embodiments, a system 600 may be configured as illustrated in FIG. 6. System 600 may include an optical subsystem 610, a light modulation unit 620, a light sensing device 630, a light sensing device 640 and a control unit 650. (Furthermore, system 600 may include any subset of the features, embodiments and elements discussed above with respect to system 100 and system realization 200 and discussed below with respect to system 1700.) System 600 may be configured to compressively acquire image information from a received stream of light. However, in order to save power, the light modulation unit 620 may be maintained in a powered-off state until the occurrence of a light variation event. The light variation event is indicative of the potential presence of an object of interest in the field of view or scene under observation. Furthermore, after turning on power to the light modulation unit, power may be saved by selectively, not continuously, performing image reconstruction and/or transmission of the compressively-acquired information. For example, the data stream acquired by the light sensing device 630 may be monitored to detect object motion and/or the occurrence of a signal of interest in the field of view. The reconstruction and/or transmission processes may be invoked in response to such conditions. The reconstruction algorithm may be computationally intensive. Thus, the system may save a considerable amount of power, e.g., battery life, by running the reconstruction algorithm selectively, not continuously. Similarly, the system may save power by using the transmitter selectively (when needed), not continuously. Furthermore, the system may save power by transmitting compressively-acquired measurements to a remote computer and invoking execution of the reconstruction algorithm on the remote computer instead of executing the reconstruction algorithm itself.
In some embodiments, the system 600 may be configured as a surveillance system or security monitoring system.
The optical subsystem 610 may be configured to receive an incident light stream L and to separate the incident light stream into a light stream S₁and a light stream S₂. The light stream S₁is supplied to the light modulation unit 620, and the light stream S₂is supplied to the light sensing device 640. The separation may be performed in any of a wide variety of ways, using any of a wide variety of optical components or combination of optical components. In some embodiments, the separation is performed so that each of the light streams S₁and S₂has the same field of view into the external environment. For example, the separation may be performed so that each ray in the incident light stream is split into two parts, one part entering the light stream S₁and the other part entering the light stream S₂.
In some embodiments, the optical subsystem 610 may separate the incident light stream so that the light streams S₁and S₂are spectrally similar to the incident light stream L but of lower power than the incident light stream. (Two light streams are said to be spectrally similar when the wavelength spectrum of one is a scalar multiple of the wavelength spectrum of the other.)
In some embodiments, the optical subsystem 610 may separate the incident light stream so that the light stream S₁and/or the light stream S₂is (are) spectrally different from incident light stream L. For example, the light stream S₂may be restricted to the IR band while the incident light stream L is a broadband stream including visible light as well as IR light.
In some embodiments, the optical subsystem 610 may separate the incident light stream so that the light streams S₁and S₂are spectrally different from each other. In some embodiments, the optical subsystem 610 may separate the incident light stream so that the light streams S₁and S₂are restrictions of the incident light stream L to different wavelength bands.
In some embodiments, the optical subsystem 610 may include one or more optical devices such as beam splitters, prisms, TIR prisms, diffraction gratings, mirrors (e.g., partially transmitting mirrors), etc.
In some embodiments, the optical subsystem 610 may be configured in any of the various ways described in connection with “optical subsystem 1310” of U.S. patent application Ser. No. 13/193,553, filed on Jul. 28, 2011, entitled “Determining Light Level Variation in Compressive Imaging by Injecting Calibration Patterns into Pattern Sequence”, which is hereby incorporated by reference in its entirety.
The light modulation unit 620 may include a plurality of light modulating elements (e.g., as variously described above in connection with light modulation unit 110). In some embodiments, the light modulation unit 620 includes a plurality of light reflecting elements whose orientations are independently controllable, e.g., as variously described above in connection with mirrors 110M. The light modulation unit 620 may be, e.g., a digital micromirror device (DMD). When the light modulation unit is powered on (i.e., is in the power-on state), each of the light reflecting elements is configured to controllably switch between two active orientation states. Furthermore, when it is powered on, the light modulation unit is configured to modulate the light stream S₁with a time sequence of spatial patterns to obtain a modulated light stream MLS.
The light sensing device 630 may be configured to receive the modulated light stream MLS and to generate a data stream D₁in response to the modulated light stream when the light modulation unit 620 is powered on. In some embodiments, the light sensing device 630 may include only one light sensing element and an analog-to-digital converter, e.g., as variously described above. In other embodiments, the light sensing device 630 may include a plurality (or an array) of light sensing elements and one or more analog-to-digital converters, e.g., as variously described above. In some embodiments, the light sensing device 630 may include a focal plane array (FPA).
One or more devices (e.g., optical devices) may intervene on the optical path between the light modulation unit 620 and the light sensing device 630, to focus, direct or concentrate the modulated light stream MLS onto the light-sensing surface(s) of the light sensing device 630, e.g., as variously described above.
The light sensing device 640 may be configured to receive the light stream S₂and to generate a data stream D₂in response to the light stream S₂at least when the light modulation unit 620 is powered off (i.e., is in the powered-off state). In some embodiments, the light sensing device 640 may include only one light sensing element (such as a photodiode) and an analog-to-digital converter (e.g., as variously described above). In other embodiments, the light sensing device 640 includes a plurality (or array) of light sensing elements and one or more analog-to-digital converters. For example, in one embodiment, the light sensing device 640 may include a focal plane array (FPA).
One or more devices (e.g., optical devices) may intervene on the optical path between the optical subsystem 610 and the light sensing device 640 to focus, direct or concentrate the light stream S₂onto the light-sensing surface(s) of the light sensing device 640, e.g., as variously described above.
The control unit 650 may be configured to monitor the data stream D₂when the light modulation unit is powered off in order to detect a light variation event in the data stream D₂. The light variation event indicates a variation of light in the light stream S₂, e.g., a temporal variation having sufficient strength and/or a variation having certain spatial and/or spectral properties. (“Sufficiently strong” means, e.g., that the magnitude of the light variation is greater than a predetermined threshold.) The control unit 650 may be further configured to turn on power to the light modulation unit 620 in response to determining that a trigger condition is satisfied. The trigger condition may include the detection of the light variation event. (In some embodiments, the trigger condition may involve one or more other conditions in addition to the detection of the light variation event. In other embodiments, the trigger condition is the same thing as the detection of the light variation event.) By maintaining the light modulation unit in a power-off state until the occurrence of a light variation event, system 600 conserves power when there is likely nothing of interest occurring in the scene under observation.
The control unit 650 may be realized in any of various forms. For example, the control unit may include a microprocessor (or system of interconnected microprocessors), one or more programmable hardware elements such as field-programmable gate arrays (FPGAs), custom-designed digital circuitry such as one or more application specific integrated circuits (ASICs), or any combination such elements.
In some embodiments, the control unit 650 may be configured to inject a sequence of measurement patterns into the time sequence of spatial patterns after turning on power to the light modulation unit, and to execute a reconstruction algorithm on a subset of the data stream D₁(i.e., a subset that corresponds to the sequence of measurement patterns) in order to obtain an image or image sequence. By saying that the sequence of measurement patterns is “injected” into the time sequence of spatial patterns, we mean to imply that the time sequence of spatial patterns is free to include spatial patterns other than the measurement patterns. For example, the time sequence of spatial patterns may include calibration patterns and/or bright-spot search patterns in addition to the measurement patterns. Thus, it is not necessary that the time sequence of spatial patterns be entirely composed of measurement patterns. The reconstructed image or image sequence represents the scene under observation. The measurement patterns may be configured as variously described above. (See the above discussion of the “measurement vector set”.)
In some embodiments, the system 600 may also include a transmitter 660, e.g., as shown in FIG. 7. The control unit 650 may be configured to inject a sequence of measurement patterns into the time sequence of spatial patterns after turning on power to the light modulation unit 620, and to direct the transmitter 660 to transmit a subset of the data stream D₁onto a transmission channel, i.e., a subset of the data stream D₁that corresponds to the sequence of measurement patterns. (The samples forming the subset are acquired in response to the application of the sequence of measurement patterns to the light stream S₁by the light modulation unit 620.)
In some embodiments, the transmitter 660 may be configured to transmit the subset of the data stream D₁(or the entirety of the data stream D₁) to a remote system 670. The remote system may be equipped with processing resources (e.g., one or more processors configured to execute program instructions) to perform image (or image sequence) reconstruction based on the subset. Thus, system 600 may save power by not performing the reconstruction itself.
In different embodiments, the transmitter 660 may be configured for transmission over respectively different kinds of communication channel. For example, in some embodiments, the transmitter transmits electromagnetic signals (e.g., radio signals or optical signals) through a wireless or wired channel. In one embodiment, the transmitter transmits electromagnetic signals through an electrical cable. In another embodiment, the transmitter transmits electromagnetic waves through free space (e.g., the atmosphere). In yet another embodiment, the transmitter transmits through free space or through an optical fiber using modulated light signals or modulated laser signals. In yet another embodiment, the transmitter transmits acoustic signals through an acoustic medium, e.g., a body of water. The transmitter may be any type of transmitter known in the art of telecommunications.
In some embodiments, system 600 also includes a receiver as well as a transmitter to permit two-way communication with one or more other parties.
In some embodiments, the light sensing device 640 may include a single light sensing element and an analog-to-digital converter (ADC), e.g., as variously described above in connection with light sensing device 130 and ADC 140. The light sensing element is configured to generate an analog electrical signal representing intensity of the light stream S₂as a function of time. The ADC is configured to capture a sequence of samples of the analog electrical signal. (Each sample represents the intensity of the light stream S₂at a corresponding time.) The data stream D₂may include the sequence of samples. The control unit 650 may monitor the sequence of samples (or, one or more time derivatives of the sequence of samples) to detect the light variation event. For example, the control unit may apply a low pass filter to the sequence of samples (to attenuate high-frequency noise) and then compute a time derivative of the filtered signal. When the absolute value (or the square) of the time derivative exceeds a detection threshold, the control unit may declare that the light variation event has occurred.
In some embodiments, the light variation event is interpreted as a variation in the total intensity of the light stream S₂.
In some embodiments, the light sensing device 640 may be configured in any of the various ways described in connection with “light sensing device 1320” of U.S. patent application Ser. No. 13/193,553, filed on Jul. 28, 2011, entitled “Determining Light Level Variation in Compressive Imaging by Injecting Calibration Patterns into Pattern Sequence”. See especially FIGS. 13A through 17B and the corresponding textual description in that patent application.
In some embodiments, the light sensing device 640 includes a light sensing element and analog electrical circuitry. The light sensing element is configured to generate an analog electrical signal representing intensity of the light stream S₂as a function of time. The analog electrical circuitry operates on the analog electrical signal and performs the function of motion detection. The analog electrical circuitry may generate a decision signal that represents at any given time whether or not the motion is present in the field of view represented by the light stream S₂.
In some embodiments, the light sensing device 640 may include a plurality (e.g., an array) of light sensing elements, each configured to receive a corresponding spatial portion of the light stream S₂. (For example, the plurality of light sensing elements may be arranged to cover a cross section of the light stream S₂. Each light sensing element thus receives the portion of the light stream S₂that impinges upon its surface.) For each of the light sensing elements, the light sensing device 640 may be configured to capture a corresponding sequence of samples representing intensity over time of the corresponding spatial portion of the light stream S₂. The data stream D₂may include these sequences of samples.
In one embodiment, the light sensing device 640 may include an M×N array of light sensing elements, where M is a positive integer, N is a positive integer and the product MN is greater than or equal to two. Each of the MN light sensing elements captures samples for a corresponding region of the light stream S₂. For example, in the case M=N=2, each light sensing element may capture samples for a corresponding quadrant of the light stream S₂.
In one embodiment, the control unit 650 may detect the light variation event by: (a) computing a weighted combination of the sequences of intensity samples (captured from the respective light sensing elements) to obtain a composite signal, and (b) comparing the absolute value of the time derivative of the composite signal to a detection threshold. For example, the sample sequence(s) corresponding to one half of the field of view may be weighted positively while the sample sequence(s) corresponding to other half may be weighted negatively. As another example, the field of view may be divided into four quadrants. (In this example, the light sensing device 640 includes at least four light sensing elements.) Sample sequences from the northeast and southwest quadrants may be weighted positively while sample sequences from the northwest and southeast quadrants may be weighted negatively.
In another embodiment, the control unit 650 may detect the light variation event by: computing a time derivative (or a smoothed time derivative) of each of the sample sequences to obtain a corresponding time derivative signal, and determine if the absolute value (or square) of at least one of the time derivative signals exceeds a predetermined threshold.
In yet another embodiment, the data stream D₂may include a sequence of frames, with each frame including a sample from each of the light sensing elements of the light sensing device 640. The control unit 650 may monitor the data stream D₂by analyzing the sequence of frames. For example, the control unit 650 may detect the light variation event by computing motion vectors between successive frames of the frame sequence in a manner that is used in the MPEG video encoding algorithm. An average magnitude (or a statistic) of the motion vectors may be compared to a threshold. The light variation event may be declared when the threshold is exceeded.
In some embodiments, the light sensing device 640 may be configured as variously described in U.S. patent application Ser. No. 13/197,304, filed on Aug. 3, 2011, entitled “Decreasing Image Acquisition Time for Compressive Imaging Devices”, which is hereby incorporated by reference in its entirety. For example, light sensing device 640 may include the “light sensing device 630” and the “sampling subsystem 640” described in that patent application.
In some embodiments, the light sensing device 640 may include a plurality (e.g., an array) of light sensing elements, each configured to receive a corresponding spectral portion of the light stream S₂. Furthermore, for each of the light sensing elements, the light sensing device 640 may be configured to capture a corresponding sequence of intensity samples representing intensity over time of the corresponding spectral portion of the light stream S₂. The data stream D₂may include these sequences of samples.
One or more optical devices may intervene along the optical path between the optical subsystem 610 and the light sensing device 640 in order to spatially separate the light stream S₂into a set or continuous distribution of wavelength components so that different members of the set or different portions of the continuous distribution impinge upon corresponding ones of the light sensing elements. The one or more optical devices may include devices such as diffraction gratings, prisms, optical filters, mirrors, etc. In one embodiment, the one or more optical devices include a diffraction grating.
In one embodiment, the light sensing device 640 may include three light sensing elements for three-channel color measurement (e.g., RGB measurement), with each light sensing element capturing a corresponding one of the three colors. In another embodiment, the light sensing device 640 may include two or more light sensing elements corresponding to different subbands in the infrared band.
In one embodiment, the light sensing device 640 may be (or include) a spectrometer.
In some embodiments, the light sensing device 640 may be (or include) a motion sensor 640M, e.g., as illustrated in FIG. 8. The motion sensor may be (or include) any of a wide variety of known devices for motion sensing or motion detection. The motion sensor is configured to generate the data stream D₂in response to the light stream S₂. For example, the motion sensor may generate an analog electrical signal in response to the light stream S₂and digitize the analog electrical signal in order to obtain a sequence of samples of the analog electrical signal. When an object enters into the field of view, the analog electrical signal and the data stream D₂(the sequence of samples) exhibit a disturbance. The above-described light variation event may be interpreted as being this disturbance. The control unit may detect the disturbance by detecting the occurrence of a pulse of sufficient amplitude in the analog electrical signal or in the data stream D₂. In one embodiment, the control unit 650 may detect the disturbance by comparing the absolute value of the sampled signal (the sequence of samples) or its time derivative to a threshold. The light variation event occurs when the threshold is exceeded. (Computational algorithms for detecting motion-induced disturbances in the signal(s) generated by motion sensors are well known in the prior art, and thus, need not be elaborated here.)
In some embodiments, the motion sensor 640M may include built-in circuitry for detecting the disturbance. In these embodiments, the built-in circuitry may assert a signal in response to the detection of the disturbance and inject the signal into the data stream D₂. The control unit 650 recognizes that the disturbance has been detected when it receives the signal from the data stream D₂.
Objects of interest may be known to emit light (electromagnetic energy) in a particular wavelength band. Thus, in some embodiments, the motion sensor 640M may be sensitive to light in that wavelength band. For example, in one embodiment, the motion sensor may include a passive infrared sensor. Various objects of interest (human beings, automobiles, aircraft, buildings, etc.) are known to emit infrared radiation. Thus, the entrance of such an object into the field of view will be accompanied by a disturbance (e.g., a step) in the infrared band.
In some embodiments, system 600 may include a motion sensor 645 in addition to the light sensing device 640, e.g., as shown in FIG. 9. In these embodiments, the optical subsystem 610 may be configured to separate the incident light stream L into three output streams including the light stream S₁, the light stream S₂and the light stream S₃. As described above, the light stream S₁is provided to the light modulation unit 620, and the light stream S₂is provided to the light sensing device 640. The light stream S₃is provided to the motion sensor 645. The motion sensor 645 is configured to receive the light stream S₃and generate a sequence of samples D₃in response to the light stream S₃(e.g., similar to what is described above in connection with motion sensor 640M). The control unit 650 may be configured to monitor the sequence of samples D₃, e.g., to detect a motion-induced disturbance in the sequence of samples. The above-described trigger condition (used to turn on power to the light modulation unit 620) may include the condition of detecting this motion-induced disturbance and the condition of detecting the light variation event. By combining the two detection conditions, the system 600 may be more immune to false alarms. (A false alarm is the situation where the system decides to turn on power to the light modulation unit when there is actually no object of interest in the field of view.)
The optical subsystem 610 may separate the incident light stream L into three output streams (S₁, S₂and S₃) in any of various ways, using any combination of optical devices. The streams S₁, S₂and S₃may be spectrally similar or dissimilar to the incident light stream L. Furthermore, the streams S₁, S₂and S₃may be spectrally similar or dissimilar to each other. For example, in one embodiment, the streams S₂and S₂are restricted to the infrared band, while stream S₁includes light in the visible spectrum.
In some embodiments, the light sensing device 630 may include a single light sensing element and an analog-to-digital converter (ADC), e.g., as described above in connection with light sensing device 130 and ADC 140. The light sensing element may be configured to generate an analog electrical signal representing intensity of the modulated light stream MLS as a function of time. For example, the light sensing element may be a photodiode. The ADC is configured to capture a sequence of samples of the analog electrical signal. (Each sample represents the intensity of the modulated light stream at a corresponding time.) The data stream D₁may include this sequence of samples.
In some embodiments, the light sensing device 630 may include a plurality (e.g., an array) of light sensing elements, each configured to receive a corresponding spatial portion of the modulated light stream MLS. For each of the light sensing elements, the light sensing device 630 is configured to capture a corresponding sequence of intensity samples representing intensity over time of the corresponding spatial portion. The data stream D₁may include these sequences of intensity samples captured from the light sensing elements. When image reconstruction is performed on the data stream D₁, the sample sequence captured from each light sensing element may be used to reconstruct a corresponding subimage that is representative of a corresponding subregion of the scene under observation. A complete image, representing the whole of the scene under observation, may be generated by joining together (concatenating) the subimages. For more information on how to reconstruct an image (or sequence of images) from a data stream obtained from a set of parallel light sensing elements, please see U.S. patent application Ser. No. 13/197,304, filed on Aug. 3, 2011, entitled “Decreasing Image Acquisition Time for Compressive Imaging Devices”.
In some embodiments, the light sensing device 630 may be configured as variously described in U.S. patent application Ser. No. 13/197,304. For example, light sensing device 630 may include the “light sensing device 630” and the “sampling subsystem 640” described in that patent application.
In some embodiments, the control unit 650 may be configured to monitor the data stream D₁after turning on power to the light modulation unit 620 in order to detect object movement within a field of view corresponding to the incident light stream L (i.e., within the scene under observation). The system 600 may conserve power by disabling the reconstruction and/or transmission of compressively-acquired measurements until object movement is detected. While monitoring the data stream D₁for object movement, the control unit may direct the light modulation unit 620 to apply one or more spatial patterns having all one values, i.e., spatial patterns where all the light reflecting elements are set to the orientation state that reflects light to the light sensing device 630. Alternatively, the control unit may direct the light modulation unit to apply spatial patterns from a measurement pattern set that is incoherent relative to the sparsity pattern set in which the signal is compressible (or sparse). The “signal” referred to here may be interpreted as the generic image carried by the incident light stream L.
In some embodiments, the control unit 650 may employ any known algorithm for the detection of object movement.
In one embodiment, the control unit 650 may add the sequences of intensity samples (captured from the light sensing elements of light sensing device 630) to obtain a total intensity signal. The control unit may compare the absolute value of the total intensity signal or its time derivative to a motion detection threshold. The control unit may declare the presence of object movement when the threshold is exceeded.
In one embodiment, the control unit 650 may compute a weighted combination of the sequences of intensity samples (captured from the light sensing elements of light sensing device 630) to obtain a composite signal and then compare the absolute value of the composite signal or its time derivative to a motion detection threshold. The control unit may declare the presence of object movement when the threshold is exceeded. For example, the sample sequence(s) corresponding to one half of the field of view may be weighted positively while the sample sequence(s) corresponding to other half may be weighted negatively. As another example, the field of view may be divided into four quadrants. (In this example, the light sensing device 630 includes at least four light sensing elements.) Sample sequences from the northeast and southwest quadrants may be weighted positively while sample sequences from the northwest and southeast quadrants may be weighted negatively.
In another embodiment, the control unit 650 may detect object movement by: computing a time derivative (or a smoothed time derivative) of each of the sample sequences to obtain a corresponding time derivative signal, and determine if the absolute value (or square) of at least one of the time derivative signals exceeds a predetermined threshold.
In yet another embodiment, the data stream D₁may include a sequence of frames, with each frame including a sample from each of the light sensing elements of the light sensing device 630. The control unit 650 monitor the data stream D₁by analyzing the sequence of frames. For example, the control unit 650 may detect object movement by computing motion vectors between successive frames of the frame sequence in a manner that is used in the MPEG algorithm. Then an average magnitude (or a statistic) of the motion vectors may be compared to a threshold. Object movement is declared to be occurring when the threshold is exceeded.
In response to detecting object movement within the field of view, the control unit 650 may be configured to execute a reconstruction algorithm, e.g., any of the reconstruction algorithms described above, or any of the reconstruction algorithms known in the field of compressive sensing. The reconstruction algorithm may be configured to compute an image or image sequence based on at least a subset of the data stream D₁. (The image or image sequence represents the scene under observation.) The “subset” of the data stream may be a subset starting at the time the movement was detected, or, at the time the light variation event was detected. To implement the latter alternative, the control unit may buffer the data stream D₁in a memory, thus allowing access to past values of the data stream D₁.
In some embodiments, the system 600 may also include a transmitter, e.g., as variously described above. In these embodiments, the control unit 650 may be configured to direct the transmitter to transmit at least a subset of the data stream D₁onto a communication channel in response detecting object movement within the field of view. As above, the “subset” that is transmitted may be a subset starting at (or near) the time the object movement was detected, or, at the time the light variation event was detected.
If the spatial patterns being used while monitoring for object movement are not measurement patterns, the control unit 650 may inject measurement patterns into the sequence of spatial patterns upon the detection of object movement. The above-described reconstruction or transmission may be based on a subset of the data stream D₁corresponding to the measurement patterns, i.e., acquired in response to the measurement patterns.
In some embodiments, the control unit 650 may be configured to turn off power to the light modulation unit 620 in response to an expiration of a predetermined amount of time in which no object movement is detected in the monitoring of the data stream D₁.
In some embodiments, the control unit 650 may be configured to continue to monitor the data stream D₂after the light modulation unit is powered on, and to turn off power to the light modulation unit in response to an expiration of a predetermined amount of time in which no light variation is detected in the continued monitoring of the data stream D₂.
In some embodiments, the control unit 650 may be configured to monitor the data stream D₁after turning on power to the light modulation unit in order to detect a signal (within the incident light stream) having significant energy within a signal class of interest. (“Significant energy” may be interpreted, e.g., as more than a threshold amount of energy.) System 600 may conserve power by disabling the reconstruction and/or transmission of compressively-acquired measurements until the signal is detected.
The signal class of interest may be represented by a sparse subset of a dictionary of signals. In some embodiments, the signal class of interest may be defined by the occurrence of certain patterns of movement or certain geometric shapes within the incident light field, certain patterns of spatial and/or temporal correlation, certain spectral properties or certain trajectories of spectral variation. For example, when observing a doorway, events of interest may be associated with the opening and closing of the door. As another example, when observing a flight of stairs, events of interest may be associated with movements along the flight of stairs.
In some embodiments, while monitoring the data stream D₁for a signal in the signal class of interest, the control unit 650 may inject measurement patterns (e.g., as variously described above) into the sequence of spatial patterns. Alternatively, the control unit may inject one or more all-ones patterns into the spatial pattern sequence.
For information on how to detect a signal within a given signal class based on compressively acquired samples, please see U.S. patent application Ser. No. 12/091,340, filed Oct. 25, 2006, entitled “Method and Apparatus for On-Line Compressed Sensing”; and U.S. patent application Ser. No. 12/091,069, filed Oct. 25, 2006, entitled “Method and Apparatus for Signal Detection, Classification and Estimation from Compressive Measurements”. These patent applications are hereby incorporated by reference in their entireties.
In one embodiment, the control unit 650 may perform a correlation computation between the data stream D₁and a known template (or set of templates) to determine a correlation value. If the correlation value exceeds a given threshold, the control unit may declare that the signal within the signal class has occurred.
In response to detecting a signal in the signal class of interest, the control unit 650 may inject measurement patterns into the sequence of spatial patterns (or continue to inject measurement patterns if it has already been so doing).
In some embodiments, the control unit 650 may be configured to execute a reconstruction algorithm in response to detecting a signal having significant energy within the signal class of interest. The reconstruction algorithm may be designed to compute an image or image sequence (that represents the external scene) based on at least a subset of the data stream D₁, e.g., a subset of the data stream D₁starting at the time the signal is detected, or, a subset starting a predetermined amount of time before the signal is detected. (To implement the latter alternative, the system 600 may buffer the data stream D₁in a memory.) Furthermore, the subset of the data stream D₁is preferably a subset corresponding to the injected measurement patterns.
In some embodiments, the system 600 may include a transmitter, e.g., as variously described above. The control unit 650 may be configured to direct the transmitter to transmit at least a subset of the data stream D₁onto a communication channel in response to detecting a signal having significant energy within the signal class of interest. As above, the “subset” of the data stream D₁that is transmitted may start at the time the signal is detected, or, a predetermined amount of time before the signal is detected. Furthermore, the subset of the data stream D₁is preferably a subset corresponding to the injected measurement patterns.
In one embodiment, the signal class of interest may be derived during a training phase by supplying various instances of an event of interest to the system 600, and capturing the data stream D₁generated by the light sensing device 630 in response to each instance. The captured data may then be analyzed to compute a subset of the dictionary that is highly correlated with the instances.
In some embodiments, the light sensing device 630 may include a plurality of light sensing elements, each configured to receive a corresponding spectral portion of the modulated light stream MLS. For each of the light sensing elements, the light sensing device 630 is configured to capture a corresponding sequence of intensity samples representing intensity over time of the corresponding spectral portion of the modulated light stream. The data stream D₁may include these sequences of intensity samples captured from the light sensing elements. Thus, the data stream D₁may be interpreted as a sequence of spectral intensity vectors. The elements of each vector may represent the intensity of corresponding wavelength components or wavelength bands in the modulated light stream MLS.
One or more optical devices may intervene along the optical path between the light modulation unit 620 and the light sensing device 630 in order to spatially separate the modulated light stream MLS into a set or continuous distribution of wavelength components so that different members of the set or different portions of the continuous distribution impinge upon corresponding ones of the light sensing elements. For example, the one or more optical devices may include devices such as diffraction gratings, prisms, optical filters, mirrors, etc.
In some embodiments, the light sensing device 630 may be (or include) a spectrometer.
In some embodiments, the control unit 650 may be configured to monitor the data stream D₁in order to detect a spectral signature of interest in the modulated light stream MLS. For example, in a system targeted for the observation of vehicles, the control unit may monitor the data stream D₁for spectral signatures characteristic of the exhaust gases generated by one or more kinds of vehicles. In a system targeted for the surveillance of human beings, the control unit may monitor the data stream for the presence of infrared emissions or certain characteristic patterns of IR emission. In a system targeted for chemical plume detection, the control unit may search the data stream for the presence of any of a predetermined set of spectral patterns corresponding to one or more chemical plumes of interest. In a system targeted for astronomical observation, the control unit may monitor the data stream for the presence of a spectral pattern of interest to the user, e.g., the emission spectra of certain elements or combinations of elements.
While monitoring the data stream D₁for a spectral signature of interest, the control unit 650 may direct the light modulation unit 620 to apply one or more spatial patterns having all one values, i.e., spatial patterns where all the light reflecting elements are set to the orientation state that reflects light to the light sensing device 630. (In some embodiments, the spatial patterns may be patterns that take the one value within a subregion of the field of view and the zero value outside that subregion.) Alternatively, the control unit may direct the light modulation unit to apply spatial patterns from a measurement pattern set, i.e., a pattern set that is incoherent relative to the sparsity pattern set in which the signal is compressible (or sparse), thus enabling the reconstruction of a separate image for each spectral portion of the modulated light stream from the corresponding sequence of intensity samples.
In some embodiments, the spectral signature detection may be performed using a prior art detection algorithm.
In some embodiments, the control unit may be configured to execute a reconstruction algorithm in response to detecting the spectral signature of interest, e.g., as variously described above. The reconstruction algorithm may be configured to compute an image or image sequence (that represents the external scene) based on at least a subset of the data stream D₁. The “subset” of the data stream D₁may be a subset starting at the time the spectral signature of interest is detected, or, a predetermined amount of time before the spectral signal detection. (The latter option may be implemented by buffering the data stream D₁in memory, thus allowing access to past values of the data stream D₁.)
The control unit 650 may inject measurement patterns into the sequence of spatial patterns to be used by the light modulation unit 620 in response to detecting the spectral signature of interest. If the control unit has already been injecting measurement patterns while monitoring for the spectral signature of interest, it may continue to do so after detecting that spectral signature. The “subset” of the data stream D₁used to perform the reconstruction is preferably a subset that corresponds to the measurement patterns (although not necessarily a contiguous subset in time since other kinds of patterns may be interspersed with the measurement patterns in the sequence of spatial patterns).
An image reconstructed by the control unit 650 may include a plurality of component images corresponding respectively to a plurality wavelength bands. Each component image is reconstructed from a subset of the intensity samples captured from a corresponding one of the light sensing elements, and represents a spectrally-limited view of the external scene, i.e., a view that is limited to the portion of the electromagnetic spectrum captured by the corresponding light sensing element. (Recall that each light sensing element receives a corresponding spectral portion of the modulated light stream.) For example, in the case where the modulated light stream is separated into red, green and blue components, and the light sensing device has three light sensing elements for respectively capturing those three components, the control unit may reconstruct a multi-spectral image having three component images, i.e., a red image based on samples from the red light sensing element, a green image based on samples from the green light sensing element, and a blue image based on samples from the blue light sensing element. This example naturally extends to any number of components and any set of wavelength bands. The multispectral image may have any number of components images, depending on the number of light sensing elements.
In some embodiments, the system 600 may also include a transmitter, e.g., as variously described above. The control unit 650 may be configured to direct the transmitter to transmit at least a subset of the data stream _Dlonto a communication channel. The subset may be as variously discussed above.
In some embodiments, the control unit 650 may be configured to search for a region within the field of view that contains a bright object (or an object that is bright than the general background) such as the sun or a reflection of the sun from a shiny surface in the field of view. The search may be performed after turning on power to the light modulation unit 620. The search may include injecting a sequence of search patterns into the sequence of spatial patterns, and selecting a next search pattern to be injected into the sequence of spatial patterns based on a portion of the data stream D₁that corresponds to one or more previous ones of the search patterns that have already been injected into the sequence of spatial patterns. (For more information on how to conduct such a search, please refer to U.S. patent application Ser. No. 13/207,276, filed on Aug. 10, 2011, entitled “Dynamic Range Optimization in a Compressive Imaging System”, which is hereby incorporated by reference in its entirety.) Once the region corresponding to the bright object has been determined, the control unit 650 may mask out (i.e., remove) the bright object by masking the spatial patterns supplied to the light modulation unit 620. In particular, each of the spatial patterns may be masked so that it is set to zero (or perhaps, attenuated) within the bright object region but unmodified outside the bright object region. Thus, portions of the modulated light stream MLS corresponding to the bright object region do not reach (or reach with attenuated intensity) the light sensing device 630 while portions of the modulated light stream corresponding to the exterior of the bright object region are modulated as they would have been without the masking. This masking process may be applied to a sequence of measurements patterns. The subset of the data stream D₁corresponding to the masked measurement patterns may be used to reconstruct an image (or image sequence) corresponding to the exterior of the bright object region.
In one alternative embodiment, the control unit may mask out (i.e., remove) the set complement of the bright object region, again by masking the spatial patterns. In particular, each of the spatial patterns is set to zero (or perhaps, attenuated) outside the bright object region and unmodified inside the bright object region. Thus, portions of the modulated light stream outside the bright object region do not reach the light sensing device 630, while portions of the modulated light stream inside the bright object region are modulated as they would have been without the masking. This masking process may be applied to a sequence of measurements patterns. The subset of the data stream D₁corresponding to the masked measurement patterns may be used to reconstruct an image (or image sequence) corresponding to the interior of the bright object region.
In some embodiments, system 600 may also include an audio sensor such as a microphone or an array of microphones. The control unit 650 may be configured to monitor the signal(s) generated by the audio sensor to detect an audio event of interest, e.g., an occurrence of an audio feature or signal of interest. In one embodiment, the control unit may detect an increase in audio power. In another embodiment, the control unit may detect an occurrence of an audio spectrum (or spectrogram) belonging to a signal class of interest, e.g., an audio spectrum (or spectrogram) characteristic of the human voice. The detection of the audio event may be used to further qualify the trigger condition. (Recall that the trigger condition determines whether to turn on power to the light modulation unit 620). For example, the control unit may require both the light variation event and the audio event occur in order to assert the trigger condition.
In some embodiments, the system 600 may include one or more sensors such as chemical sensors, pressure sensors, proximity sensors, radiation sensors, particle detectors (e.g., Geiger counters), smoke detectors and vibration sensors. The sensing signals generated by any combination of such sensors may be used to qualify the trigger condition.
In some embodiments, the system 600 may include a light source for illuminating the external scene. For example, the light source may be powered on in response to detection of the light variation event.
In some embodiments, the optical subsystem 610 may be (or include) a TIR prism pair 610T, e.g., as shown in FIG. 10. (TIR is an acronym for “total internal reflection”.) The TIR prism pair splits the incident light stream L into the light streams S₁and S₂at one or more of its internal faces, i.e., the faces where the two prisms come into close proximity. The light stream S₁proceeds to the light modulation unit 620 where it experiences modulation with the time sequence of spatial patterns. The modulated light stream MLS returns to the TIR prism pair where it experience total internal reflection at one of the internal faces. After total internal reflection, the modulated light stream exits the TIR prism pair onto an optical path leading to the light sensing device 630.
In other embodiments, the optical subsystem 610 may be (or include) a dual TIR prism.
For more information on how to use TIR prisms and dual TIR prisms in compressive imaging systems, please refer to U.S. patent application Ser. No. 13/207,900, filed on Aug. 11, 2011, entitled “TIR Prism to Separate Incident Light and Modulated Light in Compressive Imaging Device”, which is hereby incorporated by reference in its entirety.
In some embodiments, the system embodiment of FIG. 9 may be realized as shown in FIG. 11, where the optical subsystem 610 includes a beam splitter 610S and the TIR prism pair 610T. The beam splitter 610S splits in the incident light stream L into the light stream S₂and an intermediate stream K. The TIR prism pair 610T splits the intermediate stream K into the light stream S₁and the light stream S₃. The light stream S₁proceeds to the light modulation unit 620 while the light stream S₃proceeds to the motion sensor 645.
In some embodiments, the system 600 may be configured as shown in FIG. 12. In these embodiments, the modulated light stream MLS may be separated by an optical subsystem 625 into light stream T_Aand T_B. The light stream T_Ais supplied to a light sensing device 630A; and the light stream T_Bis supplied to a light sensing device 630B. Light sensing device 630A generates a data stream D_Ain response to the light stream T_A, e.g., as variously described above in connection with light sensing device 630. Similarly, light sensing device 630B generates a data stream D_Bin response to the light stream T_B, e.g., as variously described above in connection with light sensing device 630. The control unit 650 may monitor the data stream D₂and determine when to turn on power to the light modulation unit 620. The control unit 650 may also monitor the data stream D_Aand/or the data stream D_Bin order to decide when to invoke reconstruction on the data stream D_Aand/or the data stream D_B, or when to invoke transmission of the data stream D_Aand/or the data stream D_B.
In some embodiments, light sensing device 630A may include one or more light sensing elements, each configured to convert a corresponding spatial portion of the light stream T_Ainto a corresponding sequence of intensity samples; and light sensing device 630B may include one or more light sensing elements, each configured to convert a corresponding spectral portion of the light stream T_Binto a corresponding sequence of intensity samples.
FIG. 13 shows an embodiment of system 600 having light sensing devices 630C, 630M and 640F. The incident light stream L passes through a camera lens 605, and enters the TIR prism pair 610T. The TIR prism pair separates the light stream L into light streams S₁and S₂. The light stream S₁is supplied to the digital micromirror device 620D. The light stream S₂is sensed by a focal plane array 640F (e.g., a visible band FPA) having any desired number of light sensing elements. The focal plane array 640F converts the light stream S₂into a data stream D₂, which is provided to the control unit 650.
The light stream S₁is modulated by the DMD 620D. The resulting modulated light stream MLS returns to the TIR prism pair 610T, experiences total internal reflection at an internal surface of the TIR prism pair, exits the TIR prism pair, and is supplied to a diffraction grating 625DG. The diffraction grating is configured to separate the modulated light stream MLS into a beam B₀and a beam B₁. The beam B₀is the beam of zeroth diffraction order. Thus, there is no angular separation of wavelength components in the beam B₀. The beam B₁is a beam of non-zero diffraction order (e.g., first order). Thus, the wavelength components in the beam B₁are spread out in angle.
The focusing lens 627FL directs the beam B₀to an imaging lens 627IL, which images the beam B₀onto the detector 630M. The detector 630M includes an array of light sensing elements. Each element receives a corresponding spatial portion of the beam B₀and generates a sequence of intensity samples representing the intensity of the spatial portion over time. The data stream D_Mgenerated by the detector 630M includes the sequences of intensity samples captured from the light sensing elements. The data stream D_Mis provided to the control unit 650.
The focusing lens 627FL also directs the beam B₁to the light sensing device 630C. The light sensing device 630C may be a multiband SWIR sensor array. Each element of the array senses a different portion of the SWIR spectrum. (SWIR is an acronym for short wavelength infrared.) Each element generates a sequence of intensity samples that represents intensity of the corresponding spectral portion over time. The data stream D_Cgenerated by the light sensing device 630C includes the sequences of intensity samples captured from the array elements. The data stream D_Cis provided to the control unit 650.
In some embodiments, the light sensing device 630C may include an array light sensing elements (e.g., photodiodes), where the spectral sensitivity of the light sensing elements varies within the array. In these embodiments, it is possible to make spectrum measurements even without a diffractive optical element (such as diffraction grating 625DG) in front of the array.
The control unit 650 may be configured to monitor the data stream D₂to decide when to turn on power to the DMD 620D. Furthermore, the control unit 650 may monitor the data streams D_Cand/or D_M, e.g., as variously described above, in order to decide when to invoke further actions such as image (or image sequence) reconstruction on the data stream D_Cand/or the data stream D_M, and/or, transmission of the data streams D₂, D_Cand D_Mor any subset of those data streams.
The control unit 650 may reconstruct an image (or image sequence) from the data stream D_Mby reconstructing a plurality of subimages (subimage sequences) from the respective sample sequences of the data stream D_M, e.g., as variously described above. Furthermore, the control unit may reconstruct an image (or image sequence) from the data stream D_Cby reconstructing a plurality of component images (component image sequences) from the respective intensity samples of the data stream D_C, e.g., as variously described above.
The control unit 650 may transmit the data stream D_M(or a portion thereof) and/or the data stream D_C(or a portion thereof). The control unit 650 of FIG. 13 may be configured to perform any of the various operations described above, or any combination of those operations.
FIG. 14 illustrates an embodiment of system 600 having light sensing elements 630SE and 640SA. The action of the TIR prism pair 610T in this embodiment is similar to what has been described above in connection with FIG. 10 or FIG. 13. The light sensing device 640SA may be a sensor array, e.g., visible light sensor array or a SWIR sensor array. The light sensing device 630SE may be a single-element detector such as a photodiode. The focusing lens 627FL directs, concentrates or focuses the modulated light stream MLS onto the light sensing device 630SE, e.g., as variously described above.
FIGS. 15A and 15B illustrate the TIR prism pair 610T and the DMD 620D in two different states. FIG. 15B illustrates the state where the DMD 620D is off, and thus, the micromirrors assume a neutral orientation state. A representative one of the micromirrors is labeled “RM”. FIG. 15B illustrates the state where the DMD 620D is on, and thus, each of the micromirrors takes one of two possible active orientation states. (The number of micromirrors shown in the figures is not meant to be limiting. Indeed, in most applications, the number of micromirrors would be much larger.)
In FIG. 15A, the incident light stream L enters a first prism 1505 of the TIR prism pair 610T, and experiences transmission and reflection from the interface 1510 between the two prisms. The reflection produces a reflected light stream X, and the transmission produces a transmitted light stream Y. The transmitted light stream Y proceeds through the second prism 1515 of the TIR prism pair, exits the second prism, is reflected from the micromirrors in the powered-off state, i.e., their neutral orientation state. The reflected stream Y′ returns to the second prism, and experiences Fresnel reflection at the interface 1510, thereby producing the light stream Z. In some embodiments, the light sensing device 640 may be configured to receive and sense the light stream X. The light stream X is continuously available, i.e., available whether the DMD 620D is on or off. In some embodiments, the light sensing device 640 may be configured to receive and sense the light stream Z. The light stream Z is available only when the DMD 620 is off (When the DMD is on, the micromirrors are switching between the two active orientation states. See FIG. 15B and the corresponding discussion below.) In some embodiments, one light sensing device may be configured to receive and sense the light stream X, and another light sensing device may be configured to receive and sense the light stream Z.
In FIG. 15B, the DMD 620D is on. Thus, the micromirrors are being digitally switched between the two active orientation states according to the spatial patterns that are being supplied to the DMD. The two active orientation states are
different from the neutral orientation state. For example, the active orientation states may be +φ and −φ degrees relative to the normal of the plane of the micromirror array, where φ is a positive constant fixed by the design of the DMD. (The neutral orientation state may be zero degrees). FIG. 15B shows a snapshot in time. As described above, the incident light stream L enters the first prism 1505 of the TIR prism pair 610T, experiences reflection and transmission at the interface 1510, thus producing reflected light stream X and transmitted light stream Y. The transmitted light stream Y exits the second prism 1515 of the TIR prism pair, and experiences reflection from the micromirrors. The modulated light stream MLS comprises the portions of the transmitted light stream Y that are reflected at any given time by the subset of micromirrors in the first active orientation state (the one state). The representative mirror RM is shown as being in the first active orientation state. The modulated light stream MLS returns to the second prism 1515 of the TIR prism pair, experiences total internal reflection at the interface 1510, and exits the second prism onto an optical path leading to one or more light sensing devices (e.g., the light sensing device 630). Note that the subset of micromirrors residing in the first active orientation state at any given time depends on the current spatial pattern. That subset may vary anywhere from the null set (for the all-zeros spatial pattern) to the entirety of the micromirror array (for the all-ones spatial pattern).
Comparing FIGS. 15A and 15B, note that the modulated light stream MLS exits the second prism 1515 at a different beam angle than the light stream Z. Thus, these two light streams may be separately sensed using two light sensing devices.
As shown in FIG. 15C, the subset of micromirrors in the second active orientation state also reflect light from the incident light stream L to form a complementary modulated light stream MLS′. (FIG. 15C shows a representative micromirror RM′ in the second active orientation state.) The stream MLS′ returns to the TIR prism pair 610T, but exits the TIR prism pair along a path different from the light stream X. Thus, the light sensing device 640 may sense the light stream X without substantial interference from the stream MLS′.
In some embodiments, a surveillance network may include a plurality of sensing nodes, each sensing node being an instance of system 600. Each sensing node may transmit its compressive-acquired information to a central processing node. The central processing node may be configured to perform reconstruction on the compressively-acquired information from each sensing node, or, on a selected subset of the sensing nodes. In one embodiment, each sensing node may be configured to alert other sensing nodes (e.g., neighboring sensor nodes) when it detects the light variation event or when the trigger condition becomes true or when it detects object movement or when it detects a signal belonging to a signal class of interest, e.g., by transmitting messages to those other sensing nodes.
In one set of embodiments, a method 1600 for operating an imaging system or a light acquisition system may involve the operations shown in FIG. 16. (Furthermore, method 1600 may include any subset of the features, embodiments and elements discussed above with respect to system 100, system realization 200 and system 600.) Method 1600 may be performed by system 600 as variously described above.
At 1610, an incident light stream may be separated into at least a first light stream and a second light stream, e.g., as variously described above.
At 1620, the first light stream may be supplied to a light modulation unit (e.g., the light modulation unit 620 as described above). The light modulation unit is configured to modulate the first light stream with a time sequence of spatial patterns when it is powered on, e.g., as variously described above. The modulation of the first light stream produces a modulated light stream MLS.
At 1630, a first data stream is generated in response to the modulated light stream MLS when the light modulation unit is powered on. The first data stream may be generated as variously described above.
At 1640, a second data stream is generated in response to the second light stream at least when the light modulation unit is powered off, e.g., as variously described above.
At 1650, the second data stream is monitored when the light modulation unit is powered off in order to detect a light variation event in the second data stream, where the light variation event indicates a variation of light in the second light stream.
Operation 1660 includes turning on power to the light modulation unit in response to determining that a trigger condition is satisfied. The trigger condition includes detection of the light variation event in the second data stream.
In some embodiments, the method 1600 also includes performing the following operations after turning on power to the light modulation unit: (a) injecting a sequence of measurement patterns into the time sequence of spatial patterns; and (b) executing a reconstruction algorithm on a subset of the first data stream corresponding to the sequence of measurement patterns in order to obtain an image or image sequence.
In some embodiments, the method 1600 also includes performing the following operations after turning on power to the light modulation unit: (a) injecting a sequence of measurement patterns into the time sequence of spatial patterns; and (b) directing a transmitter to transmit a subset of the first data stream onto a transmission channel, where the subset of the first data stream corresponds to the sequence of measurement patterns.
In some embodiments, the action 1640 of generating the second data stream includes converting the second light stream into an analog electrical signal and capturing a sequence of samples of the analog electrical signal, where the second data stream includes the sequence of samples.
In some embodiments, the action 1640 of generating the second data stream includes converting spatial portions of the second light stream into respective sequences of samples, where the second data stream includes the sequences of samples.
In some embodiments, the action 1640 of generating the second data stream includes converting spectral portions of the second light stream into respective sequences of samples, where the second data stream includes the sequences of samples.
In some embodiments, the action 1640 of generating the second data stream is performed by a motion sensor (or motion detector).
In some embodiments, the action 1610 of separating the incident light stream includes separating the incident light stream into three output streams including the first light stream, the second light stream and a third light stream. The third light stream may be converted into a sequence of samples using a motion sensor. The sequence of samples may be monitored to detect a disturbance in the sequence of samples, e.g., as variously described above. The trigger condition may include detection of this disturbance in addition to the detection of the light variation event.
In some embodiments, the action 1630 of generating the first data stream includes converting spatial portions of the modulated light stream into respective sequences of intensity samples, where the first data stream includes the sequences of intensity samples. The first data stream may be monitored after turning on power to the light modulation unit in order to detect object movement within a field of view corresponding to the incident light stream.
In some embodiments, in response detecting object movement within the field of view, a reconstruction algorithm may be executed in order to compute an image or image sequence (representing the external scene) based on at least a subset of the first data stream.
In some embodiments, in response detecting object movement within the field of view, a transmitter is directed to transmit at least a subset of the first data stream onto a communication channel.
In some embodiments, the method 1600 may also include turning off power to the light modulation unit in response to an expiration of a predetermined amount of time in which no object movement is detected in the monitoring of the first data stream.
In some embodiments, the method 1600 may include monitoring the first data stream after turning on power to the light modulation unit in order to detect a signal within the incident light stream having significant energy within a signal class of interest.
In some embodiments, the method 1600 may also include executing a reconstruction algorithm in response to detecting a signal having significant energy within the signal class of interest. The reconstruction algorithm computes an image or image sequence (representing the external scene) based on at least a subset of the first data stream.
In some embodiments, the method 1600 may also include directing a transmitter to transmit at least a subset of the first data stream onto a communication channel in response to detecting a signal having significant energy within the signal class of interest.
In some embodiments, the action 1630 of generating the first data stream includes converting spectral portions of the modulated light stream into respective sequences of intensity samples, where the first data stream includes the sequences of intensity samples. The method 1600 may also include monitoring the first data stream in order to detect a spectral signature of interest in the modulated light stream. In one embodiment, a reconstruction algorithm may be executed in response to detecting the spectral signature of interest. The reconstruction algorithm computes an image or image sequence based on at least a subset of the first data stream. In another embodiment, a transmitter may be directed to transmit at least a subset of the first data stream onto a communication channel in response to detecting the spectral signature of interest.
In some embodiments, method 1600 may also include searching for a bright object (or relatively bright object) within a field of view of the incident light field after turning on power to the light modulation unit. The search may include: injecting a sequence of search patterns into the sequence of spatial patterns; and selecting a next search pattern to be injected into the sequence of spatial patterns based on a portion of the first data stream that corresponds to one or more previous search patterns that have already been injected into the sequence of spatial patterns.
Detecting Light Level Variation from Neutral-State Mirrors
In one set of embodiments, a system 1700 may be configured as illustrated in FIG. 17. The system 1700 may include a light modulation unit 1720, a light sensing device 1730, a light sensing device 1740 and a control unit 1750. (Furthermore, the system 1700 may include any subset of the features, embodiments and elements described above in connection with system 100, system realization 200 and system 600.)
The light modulation unit 1720 includes a plurality of light reflecting elements whose orientations are independently controllable. Each of the light reflecting elements is configured to assume a neutral orientation state when the light modulation unit 1720 is powered off, e.g., as shown in FIG. 18A. (In some embodiments, the neutral orientation state is a state where the light reflecting elements are approximately parallel to an underlying substrate.) Furthermore, each of the light reflecting elements is configured to controllably switch between two active orientation states when the light modulation unit is powered on (i.e., is in the power-on state). FIG. 18B shows a snapshot in time where half the light reflecting elements are in one of the two active orientation states, and the remaining half of the light reflective elements are in the other of the two active orientation states; the light reflecting element RM is in the “one state” that reflects light to the light sensing device 1730 (or, more generally, onto an optical path that leads to the light sensing device 1730); the light reflecting element RM′ is in the “zero state” that reflects light away from light sensing device 1730 and light sensing device 1740.
Note that FIGS. 18A and 18B shows a one-dimensional array of light modulating elements for clarity of illustration. However, in many embodiments of system 1700, the array is two dimensional. Furthermore, the number of light reflecting elements shown in FIGS. 18A and 18B is not meant to be limiting. Indeed, in many embodiments of system 1700, the number of light reflecting elements is much larger.
In some embodiments, the light modulation unit 1720 includes the set of mirrors 110M, as variously described above.
In some embodiments, the light modulation unit 1720 is a digital micromirror device (DMD).
The light reflecting elements are configured to modulate the incident light stream L with a time sequence of spatial patterns when the light modulation unit 1720 is powered on. The action of modulating the incident light stream produces the modulated light stream MLS. The modulated light stream MLS includes portions of the incident light stream L that are reflected from any of the light reflecting elements in the one state. See the representative reflection of a light portion from light reflecting element RM in FIG. 18B. Moreover, the light reflecting elements are configured to reflect the incident light stream L from the neutral orientation state when the light modulation unit is powered off, e.g., as shown in FIG. 18A. This reflection of the incident light stream from the light reflecting elements in the neutral orientation state produces the neutral-state light stream NLS.
The light sensing device 1730 is configured to receive the modulated light stream MLS when the light modulation unit is powered on, and to generate the data stream D₁in response to the modulated light stream. The light sensing device 1730 may be realized by the light sensing device 630 as variously described above.
The light sensing device 1740 is configured to receive the neutral-state light stream NLS when the light modulation unit is powered off, and to generate the data stream D₂in response to the neutral-state light stream. The light sensing device 1740 may be realized by the light sensing device 640 as variously described above.
In some embodiments, the light reflecting elements are arranged on a plane, and the light modulation unit 1720 is configured so that an angle of incidence θ_incof the incident light stream L on the plane is not equal to zero, e.g., as shown in FIG. 19. (The vector n is a normal to the plane.) This condition guarantees that the neutral-state light stream can be separated from the incident light stream L. For example, in different embodiments, the angle of incidence θ_incmay be, respectively, greater than or equal to 2 degrees, greater than or equal to 4 degrees, greater than or equal to 8 degrees, greater than or equal to 16 degrees, greater than or equal to 20 degrees, greater than or equal to 24 degrees, greater than or equal to 28 degrees, greater than or equal to 32 degrees.
The control unit 1750 may be configured to monitor the data stream D₂when the light modulation unit 1720 is powered off in order to detect a light variation event in the data stream D₂, where the light variation event indicates a variation of light (e.g., as variously described above) in the neutral-state light stream NLS. The control unit 1750 is further configured to turn on power to the light modulation unit 1720 in response to determining that a trigger condition is satisfied, where the trigger condition includes detection of the light variation event in the data stream D₂, e.g., as variously described above. The control unit 1750 may be realized by control unit 650 as variously described above.
In some embodiments, system 1700 may include an active cooling system for cooling the light sensing device 1730.
In some embodiments, the control unit 1750 may turn on power to the active cooling system in response to detecting the light variation event. In other embodiments, the active cooling system may remain on continuously, e.g., as long as power is supplied to system 1700.
In some embodiments, the light sensing device 1730 is actively cooled, and the light sensing device 1740 is not actively cooled.
In some embodiments, the control unit 1750 is configured to monitor the data stream D₁after turning on power to the light modulation unit in order to detect a signal in the incident light stream having sufficient energy within a signal class of interest, e.g., as variously described above.
In some embodiments, the control unit is configured to execute a reconstruction algorithm in response to detecting a signal having significant energy within the signal class of interest, e.g., as variously described above. The reconstruction algorithm is configured to compute an image or image sequence based on at least a subset of the data stream D₁.
In some embodiments, the system 1700 also includes a transmitter, e.g., as variously described above. The control unit is configured to direct the transmitter to transmit at least a subset of the data stream D₁onto a communication channel in response to detecting a signal having significant energy within the signal class of interest.
In some embodiments, the control unit 1750 is configured to monitor the data stream D₂while the light modulation unit is powered off in order to detect a signal in the neutral-state light stream having sufficient energy within a signal class of interest, e.g., as variously described above. This signal may be identified with the above-described light variation event. Alternatively, the detection of this signal may be combined with the above-described light variation event to determine the trigger condition (for turning on power to the light modulation unit).
In some embodiments, the light sensing device 1740 includes a light sensing element and an analog-to-digital converter (ADC), e.g., as variously described above. The light sensing element is configured to generate an analog electrical signal representing intensity of the neutral-state light stream NLS as a function of time. The ADC is configured to capture a sequence of samples of the analog electrical signal. The data stream D₂may include this sequence of samples. In one embodiment, the light sensing element generates the analog electrical signal so that the analog electrical signal represents intensity of the neutral-state light stream NLS in a restricted wavelength band, e.g., in the infrared band.
In some embodiments, the light sensing device 1740 includes a plurality of light sensing elements (e.g., as variously described above), where each of the light sensing elements is configured to receive a corresponding spatial portion of the neutral-state light stream NSL. For each of the light sensing elements, the light sensing device 1740 is configured to capture a corresponding sequence of samples representing intensity over time of the corresponding spatial portion of the neutral-state light stream NLS. The data stream D₂may includes these sample sequences.
In one embodiment, the light sensing device 1740 may include a focal plane array (FPA).
In some embodiments, the light sensing device 1740 includes a plurality of light sensing elements (e.g., as variously described above), where each of the light sensing elements is configured to receive a corresponding spectral portion of the neutral-state light stream NLS. For each of the light sensing elements, the light sensing device 1740 is configured to capture a corresponding sequence of intensity samples representing intensity of the corresponding spectral portion of the neutral-state light stream. The data stream D₂may include these sample sequences.
In some embodiments, the light sensing device 1740 includes (or is) a motion sensor, e.g., as variously described above. The motion sensor may be configured to generate the data stream D₂in response to the neutral-state light stream. In one embodiment, the motion sensor may be (or include) a passive infrared (IR) sensor.
In some embodiments, the system 1700 also includes an optical subsystem 1760 and a motion sensor 1765, e.g., as shown in FIG. 20. The optical subsystem 1760 is configured to receive an input light stream X (from the external scene) and separate the input light stream X into two output streams including the incident light stream L and a secondary light stream S. In some embodiments, the optical subsystem 1760 may be realized by optical subsystem 610 as variously described above. In some embodiments, the secondary light stream S may be restricted to a particular wavelength band, e.g., the infrared band.
The motion sensor 1765 may be configured to receive the secondary light stream S and generate a sequence of samples D₃in response to the secondary light stream. The control unit 1750 may be configured to monitor the sequence of samples D₃in order to detect a disturbance in the sequence of samples D₃. The trigger condition (that determines when to turn on power to the light modulation unit 1720) may also include the detection of this disturbance. For example, the trigger condition may be the logical AND of the detection of the light variation event and the detection of the disturbance in the sequence of samples D₃.
In some embodiments, the system 1700 may include a dual TIR prism 1710, e.g., as shown in FIG. 21. The dual TIR prism is configured so that the incident light stream L enters at a face K1, experiences total internal reflection at face K2, exits the dual TIR prism at face K3 and is incident on the light modulation unit 1720, whereupon its course depends on whether the light modulation unit 1720 is in the powered-on state or the powered-off state.
If the light modulation unit is in the powered-off state, the incident light stream L is reflected by the light reflecting elements in the neutral state, thereby producing the neutral-state light stream NLS, which enters the dual TIR prism at face K3, experiences total internal reflection at face K4, and exits the dual TIR prism at face K5. FIG. 22A shows a representative light reflecting element RM (greatly enlarged for the sake of illustration) in the neutral orientation state.
Alternatively, if the light modulation unit is in the powered-on state, the incident light stream L is modulated (as variously described above) to obtain the modulated light stream MLS. The modulated light stream MLS enters the dual TIR prism at face K3, passes through faces K2 and K4, and exits at face K6. FIG. 22B shows a representative light reflecting element RM in the one state. (Recall that the modulated light stream MLS is composed of portions of the incident light stream that are reflected at any given time by any light reflecting elements in the one state.)
In one set of embodiments, a method 2300 may include the operations shown in FIG. 23. Furthermore, method 2300 may include any subset of the features, embodiments and elements described above in connection with system 100, system realization 200, system 600, method 1600 and system 1700.
At 2310, an incident light stream L is supplied to a light modulation unit (e.g., the light modulation unit 1720 or the light modulation unit 620D as variously described above). The light modulation unit includes a plurality of light reflecting elements whose orientations are independently controllable. Each of the light reflecting elements takes a neutral orientation state when the light modulation unit is powered off.
At 2315, the incident light stream L is reflected from the plurality of light reflecting elements when the light modulation unit is powered off to obtain a neutral-state light stream NLS, e.g., as variously described above. Since the light modulation unit is powered off, the incident light stream is reflected from the light reflecting elements in the neutral orientation state.
At 2320, a neutral-state data stream is generated (e.g., as variously described above) in response to the neutral-state light stream NLS when the light modulation unit is powered off.
At 2325, the neutral-state data stream is monitored when the light modulation unit is powered off in order to detect a light variation event in the neutral-state data stream. The light variation event indicates a variation of light in the neutral-state light stream, e.g., as variously described above.
Operation 2330 involves turning on power to the light modulation unit in response to determining that a trigger condition is satisfied, e.g., as variously described above. The trigger condition includes the condition of detecting the light variation event in the neutral-state data stream.
After turning on power to the light modulation unit, the incident light stream is modulated with a time sequence of spatial patterns to obtain a modulated light stream MLS (as indicated at 2335), e.g., as variously described above.
At 2340, an on-state data stream is generated in response to the modulated light stream MLS, e.g., as variously described above.
In some embodiments, the method 2300 may also include monitoring the on-state data stream after turning on power to the light modulation unit in order to detect a signal in the incident light stream having sufficient energy within a signal class of interest, e.g., as variously described above.
In some embodiments, the method 2300 may also include executing a reconstruction algorithm in response to detecting a signal having significant energy within the signal class of interest. The reconstruction algorithm is configured to compute an image or image sequence representing the external scene based on at least a subset of the on-state data stream.
In some embodiments, the method 2300 may also include directing a transmitter to transmit at least a subset of the on-state data stream onto a communication channel in response to detecting a signal having significant energy within the signal class of interest. The transmitter may rest in a powered off state when not being used to transmit data.
In some embodiments, the action 2320 of generating the neutral-state data stream includes converting the neutral-state light stream into an analog electrical signal and capturing a sequence of samples of the analog electrical signal, e.g., as variously described above.
In some embodiments, the action 2320 of generating the neutral-state data stream includes converting spatial portions of the neutral-state light stream into respective sequences of intensity samples, where the neutral-state data stream includes the sequences of intensity samples.
In some embodiments, the action 2320 of generating the neutral-state data stream includes converting spectral portions of the neutral-state light stream into respective sequences of intensity samples, where the neutral-state data stream includes the sequences of intensity samples.
In some embodiments, the action 2320 of generating the neutral-state data stream is performed by a motion sensor.
In some embodiments, the method 2300 also includes: separating an input light stream into two output streams including the incident light stream and a secondary light stream; converting the secondary light stream into a sequence of samples using a motion sensor; and monitoring the sequence of samples to detect a disturbance in the sequence of samples, where the trigger condition also includes the condition of detecting the disturbance.
In one set of embodiments, a compressive imaging system 2400 may be configured as shown in FIG. 24. System 2400 may include the light modulation unit 2420, a light sensing device 2430, a motion sensor 2435 and control unit 2450. (Furthermore, system 2400 may include any subset of the features, embodiments and elements described above in connection with system 100, system realization 200, system 600, method 1600, system 1700 and method 2300.) System 2400 conserves power by maintaining the light modulation unit in a power-off state until motion is detected in the scene under observation.
The light modulation unit 2420 may be configured to receive a light stream L₁from an optical input path P₁. The light modulation unit includes a plurality of light reflecting elements whose orientations are independently controllable. Each of the light reflecting elements is configured to controllably switch between two active orientation states when the light modulation unit is powered on. The light modulation unit, when it is powered on, is configured to modulate the light stream L₁with a time sequence of spatial patterns to obtain a modulated light stream MLS. The light modulation unit 2420 may be realized by light modulation unit 110 or mirrors 110M or light modulation unit 620 as variously described above.
The light sensing device 2430 may be configured to receive the modulated light stream MLS and to generate a data stream D₁in response to the modulated light stream when the light modulation unit is powered on. The light sensing device 2430 may be realized by the light sensing device 630 as variously described above. In some embodiments, light sensing device 2430 may include light sensing device 130 and analog-to-digital converter 140 as described above in connection with system 100.
The motion sensor 2440 may be configured to receive a light stream L₂from an optical input path P₂, and to generate a data stream D₂in response to the light stream L₂at least when the light modulation unit is powered off. The optical input paths P₁and P₂are independent optical paths having separate apertures into the environment. The motion sensor 2440 may be any known type of motion sensor or motion detector. In one embodiment, motion sensor 2440 is a passive infrared (PIR) sensor.
The control unit 2450 is configured to monitor the data stream D₂when the light modulation unit 2420 is powered off in order to detect a disturbance in the data stream D₂. The disturbance indicates object motion in the field of view of the light stream L₂. (For example, the disturbance may be a pulse indicating entry of an object into the field of view.) The control unit 2450 is configured to turn on power to the light modulation unit 2420 in response to detection of the disturbance. Control unit 2450 may incorporate any subset of the features, embodiments and elements described above in connection with control unit 650.
In some embodiments, the motion sensor 2440 includes built-in circuitry to detect the disturbance in the analog domain (as described above in connection with motion sensor 640M). In these embodiments, the motion sensor 2440 may assert a signal when the disturbance has been detected. That signal may be injected into the data stream D₂so the control unit 2450 can responsively turn on power to the light modulation unit 2420.
In some embodiments, system 2400 may be realized as a camera device 2400CM, e.g., as shown in FIG. 24B. The optical input paths P₁and P₂have separate input ports 2422 and 2442 on the housing 2405 of the camera device. The input ports admit light from the external environment. The input ports and optical paths may be configured to capture light from the same general field of view into the environment.
In alternative embodiments, system 2400 may include a camera subsystem 2400A and a motion detection subsystem 2400B, e.g., as shown in FIG. 24C. The two subsystems are physically separate units and are configured to communicate through a communication channel 2460. The optical input path P₁has the input port INP₁on the housing H_CSof the camera subsystem 2400A. The optical input path P₂has an input port INP₂on the housing H_MDSof the motion detection subsystem 2400B. The camera subsystem 2400A and the motion detection subsystem 2400B may be positioned and oriented so that their respective light input ports capture light from the same general region of the environment, e.g., from a scene under observation.
The transmitter 2445 transmits the data stream D₂over the communication channel 2460 to the receiver 2447. The receiver 2447 receives the data stream D₂and forwards it to the control unit 2450, where it is processed as described above. The transmitter and receiver may be configured to communicate over the communication channel 2460 in any of various ways, e.g., by means of electromagnetic signals (such as radio waves or optical signals).
In some embodiments, system 2400 may also include the optical subsystem 610 and the light sensing device 640 as described above in connection with system 600. Control unit 2450 may be configured to perform any of the operations described above in connection with control unit 650.
In some embodiments, system 2400 may also include the light sensing device 1740 described above, which is configured to sense neutral-state light reflected from the light modulation unit when it is powered off. Control unit 2450 may be configured to perform any of the operations described above in connection with control unit 1750.
Compressive Imaging System 2500
In one set of embodiments, a compressive imaging system 2500 may be configured as shown in FIG. 25. The compressive imaging (CI) system may include an optical system 2510, a spatial light modulator 2515, a set 2520 of one or more photodetectors, a set 2525 of one or more amplifiers (i.e., one amplifier per detector), a set 2530 of analog-to-digital converters (one ADC per detector), and a processing element 2540.
The optical system 2510 focuses an incident light stream onto the spatial light modulator 2515, e.g., as variously described above. See the discussion above regarding optical subsystem 105. The incident light stream carries an image (or a spectral ensemble of images) that is to be captured by the CI system in compressed form.
The spatial light modulator 2515 modulates the incident light stream with a sequence of spatial patterns to obtain a modulated light stream, e.g., as variously described above.
Each of the detectors 2520 generates a corresponding electrical signal that represents the intensity of a corresponding portion of the modulated light stream, e.g., a spatial portion or a spectral portion of the modulated light stream.
Each of the amplifiers 2525 (e.g., transimpedance amplifiers) amplifies the corresponding detector signal to produce a corresponding amplified signal.
Each of the ADCs 2530 acquires samples of the corresponding amplified signal.
The processing element 2540 may operate on the sample sets obtained by the respective ADCs to reconstruct respective images. The images may represent spatial portions or spectral slices of the incident light stream. Alternatively, or additionally, the processing element may send the sample sets to a remote system for image reconstruction.
The processing element 2540 may include one or more microprocessors configured to execute program instructions stored in a memory medium.
The processing element 2540 may be configured to control one or more other elements of the CI system. For example, in one embodiment, the processing element may be configured to control the spatial light modulator 2515, the transimpedance amplifiers 2525 and the ADCs 2530.
The processing element 2540 may be configured to perform any subset of the above-described methods on any or all of the detector channels.
Compressive Imaging System 2600
In one set of embodiments, a compressive imaging system 2600 may be configured as shown in FIG. 26. The compressive imaging system includes the light modulation unit 110 as variously described above, and also includes optical subsystem 2510, a set of L light sensing devices LSD₁through LSD_L, and a set of L signal acquisition channels C₁through C_L, where L in a positive integer.
The light modulation unit 110 receives an incident light stream and modulates the incident light stream with a sequence of spatial patterns to obtain a modulated light stream MLS, e.g., as variously described above.
The optical subsystem 2610 delivers portions (e.g., spatial portions or spectral portions) of the modulated light stream to corresponding ones of the light sensing devices LSD₁through LDS_L.
For information on various mechanisms for delivering spatial subsets of the modulated light stream to respective light sensing devices, please see U.S. patent application Ser. No. 13/197,304, filed on Aug. 3, 2011, titled “Decreasing Image Acquisition Time for Compressive Imaging Devices”, invented by Woods et al., which is hereby incorporated by reference in its entirety.
In some embodiments, the optical subsystem 2610 includes one or more lenses and/or one or more mirrors arranged so as to deliver spatial portions of the modulated light stream onto respective ones of the light sensing devices. For example, in one embodiment, the optical subsystem 2610 includes a lens whose object plane is the plane of the array of light modulating elements and whose image plane is a plane in which the light sensing devices are arranged. (The light sensing devices may be arranged in an array.)
In some embodiments, optical subsystem 2610 is configured to separate the modulated light stream into spectral components and deliver the spectral components onto respective ones of the light sensing devices. For example, optical subsystem 2610 may include a grating, a spectrometer, or a tunable filter such as a Fabry-Perot Interferometer to achieve the spectral separation.
Each light sensing device LSD_jgenerates a corresponding electrical signal v_j(t) that represents intensity of the corresponding portion MLS_jof the modulated light stream.
Each signal acquisition channel C_jacquires a corresponding sequence of samples {V_j(k)} of the corresponding electrical signal v_j(t). Each signal acquisition channel may include a corresponding amplifier (e.g., a TIA) and a corresponding A/D converter.
The sample sequence {V_j(k)} obtained by each signal acquisition channel may be used to reconstruct a corresponding sub-image which represents a spatial portion or a spectral slice of the incident light stream. The number of samples m in each sample sequence {V_j(k)} may be less than (typically much less than) the number of pixels in the corresponding sub-image. Thus, each signal acquisition channel C_jmay operate as a compressive sensing camera for a spatial portion or spectral portion of the incident light.
Each of the signal acquisition channels may include any subset of the embodiments, features, and elements described above.
In some embodiments, system 2600 may also include the optical subsystem 610 for separating the incident light stream L into two or more light streams prior to the light modulation unit 110.
The following numbered paragraphs describe various additional embodiments of systems and methods.
1. A method comprising: dispersing an incident light stream with a dispersive optical element to obtain a dispersed light stream; modulating the dispersed light stream with a time sequence of spatial patterns using a light modulator to obtain a modulated light stream; sensing the modulated light stream with a single element photodetector to acquire a sequence of measurements, wherein the measurements are acquired in response to the modulation of the dispersed light stream with the time sequence of spatial patterns, wherein the measurements represent intensity of the modulated light stream over time.
2A. The method of paragraph 1, further comprising: reconstructing a plurality of color images based on the sequence of measurements and the time sequence of spatial patterns.
2B. The method of paragraph 1, wherein the dispersive optical element comprises a hybrid TIR-dispersive optical element whose first surface acts as a dispersive surface (like a prism) that separates the image into a number of component colors that are spatially shifted by an amount equal to or greater than a few mirrors on the DMD. (TIR is an acronym for “total internal reflection”.) The calculation of the necessary dispersion of this prism is disclosed in U.S. Provisional Application No. 61/502,153, where it was proven that the dispersion requirements and the TIR requirements could be simultaneously fulfilled.
2C. The method of paragraph 1, wherein a spatial-spectral unmixing algorithm is used to separate the spectral component images from a single stream of measurements from a single detector. The algorithm applies different basis sets to the reconstruction of the same stream of data, where the different basis sets correspond to shifted versions of one another, corresponding to the dispersive shift of the images at the wavelengths of interest.
3. A method comprising: dispersing an incident light stream with a dispersive optical element to obtain a dispersed light stream; modulating the dispersed light stream with a time sequence of spatial patterns using a light modulator to obtain a modulated light stream; sensing the modulated light stream with a linear array of light sensing elements to acquire a block of measurements from each light sensing element, wherein the measurements of each block represent intensity over time of a portion of the modulated light stream captured by the corresponding light sensing element.
4. The method of paragraph 1, further comprising reconstructing a plurality of color images based on the blocks of measurements from the respective light sensing elements.
5. A system comprising: a light modulation unit to operate as an adaptive entrance pupil for an input light stream to obtain a modified light stream; a rotatable diffraction grating configured to diffract modified light stream to obtain a diffracted light stream; a detector array; an optical subsystem configured to focus different wavelength bands of the diffracted light stream onto different elements of the detector array; circuitry configured to capture measurements from each of the elements of the detector array.
6. The system of paragraph 5, further comprising: a processor configured to reconstruct a plurality of images, wherein each of the elements is reconstructed from the measurements captured from a corresponding one of the elements of the detector array.
7. A method for realizing an imaging spectrometer with the components of a non-imaging spectrometer by using a programmable entrance pupil mask. The programmable mask modulates the light so that a compressive sensing reconstruction algorithm can be applied to the wavelength-dispersed measurements at each detector in the detector array, to reconstruct an image at each measured wavelength. The programmable mask may be realized by a light modulation unit (e.g., as variously described above).
Any of the various embodiments described herein may be combined to form composite embodiments. Furthermore, any of the various features, embodiments and elements described in U.S. Provisional Application No. 61/502,153 may be combined with any of the various embodiments described herein.
The principles of the present invention are not limited to light. Various embodiments are contemplated where the signals being processed are electromagnetic waves or particle beams or seismic waves or acoustic waves or surface waves on a boundary between two fluids or gravitational waves. In each case, a space-time signal is directed to an array of signal-modulating elements whose transmittances or reflectances are individually varied so as to modulate the space-time signal with a time sequence of spatial patterns. The modulated space-time signal may be sensed by a transducer to generate an electrical signal that represents intensity of the modulated space-time signal as a function of time. The electrical signal is sampled to obtain measurements. The measurements may be processed as variously described above to reconstruct the image or image sequence carried by the original space-time signal.
Although the embodiments above have been described in considerable detail, numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Claims

1. A system comprising:

an optical subsystem configured to receive an incident light stream and to separate the incident light stream into a first light stream and a second light stream;

a light modulation unit including a plurality of light reflecting elements whose orientations are independently controllable, wherein each of the light reflecting elements is configured to controllably switch between two active orientation states when the light modulation unit is powered on, wherein the light modulation unit, when it is powered on, is configured to modulate the first light stream with a time sequence of spatial patterns to obtain a modulated light stream;

a first light sensing device configured to receive the modulated light stream and to generate a first data stream in response to the modulated light stream when the light modulation unit is powered on;

a second light sensing device configured to receive the second light stream and to generate a second data stream in response to the second light stream at least when the light modulation unit is powered off;

a control unit configured to:

monitor the second data stream when the light modulation unit is powered off in order to detect a light variation event in the second data stream, wherein the light variation event indicates a variation of light in the second light stream; and

turn on power to the light modulation unit in response to determining that a trigger condition is satisfied, wherein the trigger condition includes detection of the light variation event.

2. The system of claim 1, wherein the control unit is configured to:

inject a sequence of measurement patterns into the time sequence of spatial patterns after turning on power to the light modulation unit; and

execute a reconstruction algorithm on a subset of the first data stream corresponding to the sequence of measurement patterns in order to obtain an image or image sequence.

3. The system of claim 1, further comprising a transmitter, wherein the control unit is configured to:

direct the transmitter to transmit a subset of the first data stream onto a transmission channel, wherein the subset of the first data stream corresponds to the sequence of measurement patterns.

4. The system of claim 1, wherein the second light sensing device includes a light sensing element and an analog-to-digital converter (ADC), wherein the light sensing element is configured to generate an analog electrical signal representing intensity of the second light stream as a function of time, wherein the ADC is configured to capture a sequence of samples of the analog electrical signal, wherein the second data stream includes the sequence of samples.

5. The system of claim 1, wherein the second light sensing device includes a plurality of light sensing elements, each configured to receive a corresponding spatial portion of the second light stream, wherein, for each of the light sensing elements, the second light sensing device is configured to capture a corresponding sequence of samples representing intensity over time of the corresponding spatial portion of the second light stream, wherein the second data stream includes the sequences of samples.

6. The system of claim 1, wherein the second light sensing device includes a plurality of light sensing elements, each configured to receive a corresponding spectral portion of the second light stream, wherein, for each of the light sensing elements, the second light sensing device is configured to capture a corresponding sequence of intensity samples representing intensity over time of the corresponding spectral portion of the second light stream, wherein the second data stream includes the sequences of samples.

7. The system of claim 1, wherein the second light sensing device includes a motion sensor configured to generate the second data stream in response to the second light stream.

8. The method of claim 1, wherein the control unit is configured to continue to monitor the second data stream after the light modulation unit is powered on, and to turn off power to the light modulation unit in response to an expiration of a predetermined amount of time in which no light variation is detected in the continued monitoring of the second data stream.

9. The system of claim 1, wherein the first light sensing device includes a plurality of light sensing elements, each configured to receive a corresponding spatial portion of the modulated light stream, wherein, for each of the light sensing elements, the first light sensing device is configured to capture a corresponding sequence of intensity samples representing intensity over time of the corresponding spatial portion of the modulated light stream, wherein the first data stream includes the sequences of intensity samples.

10. The system of claim 9, wherein the control unit is further configured to monitor the first data stream after turning on power to the light modulation unit in order to detect object movement within a field of view corresponding to the incident light stream.

11. The method of claim 10, wherein the control unit, in response to detecting object movement within the field of view, is configured to execute a reconstruction algorithm, wherein the reconstruction algorithm is configured to compute an image or image sequence based on at least a subset of the first data stream.

12. The method of claim 10, further comprising a transmitter, wherein the control unit, in response detecting object movement within the field of view, is configured to direct the transmitter to transmit at least a subset of the first data stream onto a communication channel.

13. The method of claim 10, wherein the control unit is configured to turn off power to the light modulation unit in response to an expiration of a predetermined amount of time in which no object movement is detected in the monitoring of the first data stream.

14. The system of claim 1, wherein the control unit is further configured to monitor the first data stream after turning on power to the light modulation unit in order to detect a signal within the incident light stream and having significant energy within a signal class of interest.

15. The system of claim 14, wherein the control unit, in response to detecting said signal having significant energy within the signal class of interest, is configured to execute a reconstruction algorithm, wherein the reconstruction algorithm is configured to compute an image or image sequence based on at least a subset of the first data stream.

16. The system of claim 14, further comprising a transmitter, wherein the control unit, in response to detecting said signal having significant energy within the signal class of interest, is configured to direct the transmitter to transmit at least a subset of the first data stream onto a communication channel.

17. The system of claim 1, wherein the first light sensing device includes a plurality of light sensing elements, each configured to receive a corresponding spectral portion of the modulated light stream, wherein, for each of the light sensing elements, the first light sensing device is configured to capture a corresponding sequence of intensity samples representing intensity over time of the corresponding spectral portion of the modulated light stream, wherein the first data stream includes the sequences of intensity samples,

18. The system of claim 17, wherein the control unit is further configured to monitor the first data stream in order to detect a spectral signature of interest in the modulated light stream.

19. The system of claim 18, wherein the control unit is configured to execute a reconstruction algorithm in response to detecting the spectral signature of interest, wherein the reconstruction algorithm is configured to compute an image or image sequence or multispectral image based on at least a subset of the first data stream.

20. The system of claim 18, further comprising a transmitter, wherein the control unit, in response to detecting the spectral signature of interest, is configured to direct the transmitter to transmit at least a subset of the first data stream onto a communication channel.

21. The system of claim 1, wherein, after turning on power to the light modulation unit, the control unit is configured to search for a region within the field of view that contains a bright object by:

injecting a sequence of search patterns into the sequence of spatial patterns; and

selecting a next search pattern to be injected into the sequence of spatial patterns based on a portion of the first data stream that corresponds to one or more previous ones of the search patterns that have already been injected into the sequence of spatial patterns.

22. A method comprising:

separating an incident light stream into at least a first light stream and a second light stream;

supplying the first light stream to a light modulation unit, wherein the light modulation unit is configured to modulate the first light stream with a time sequence of spatial patterns when it is powered on, wherein said modulating the first light stream produces a modulated light stream;

generating a first data stream in response to the modulated light stream when the light modulation unit is powered on;

generating a second data stream in response to the second light stream at least when the light modulation unit is powered off;

monitoring the second data stream when the light modulation unit is powered off in order to detect a light variation event in the second data stream, wherein the light variation event indicates a variation of light in the second light stream; and

turning on power to the light modulation unit in response to determining that a trigger condition is satisfied, wherein the trigger condition includes detection of the light variation event in the second data stream.

23. The method of claim 22, further comprising:

performing a set of operations after turning on power to the light modulation unit, wherein the set of operations includes:

injecting a sequence of measurement patterns into the time sequence of spatial patterns; and

executing a reconstruction algorithm on a subset of the first data stream corresponding to the sequence of measurement patterns in order to obtain an image or image sequence.

24. The method of claim 22, further comprising:

directing a transmitter to transmit a subset of the first data stream onto a transmission channel, wherein the subset of the first data stream corresponds to the sequence of measurement patterns.

25. The method of claim 22, wherein said generating the second data stream includes converting the second light stream into an analog electrical signal and capturing a sequence of samples of the analog electrical signal, wherein the second data stream includes the sequence of samples.

26. The method of claim 22, wherein said generating the second data stream includes converting spatial portions of the second light stream into respective sequences of samples, wherein the second data stream includes the sequences of samples.

27. The method of claim 22, wherein said generating the second data stream includes converting spectral portions of the second light stream into respective sequences of samples, wherein the second data stream includes the sequences of samples.

28. The method of claim 22, wherein said generating the second data stream is performed by a motion sensor.

29. The method of claim 28, wherein the motion sensor includes a passive infrared sensor.

30. The method of claim 22, wherein said generating the first data stream comprises converting spatial portions of the modulated light stream into respective sequences of intensity samples, wherein the first data stream includes the sequences of intensity samples.

31. The system of claim 30, further comprising:

monitoring the first data stream after turning on power to the light modulation unit in order to detect object movement within a field of view corresponding to the incident light stream.

32. The method of claim 31, further comprising:

in response detecting object movement within the field of view, executing a reconstruction algorithm in order to compute an image or image sequence based on at least a subset of the first data stream.

33. The method of claim 31, further comprising:

in response detecting object movement within the field of view, directing a transmitter to transmit at least a subset of the first data stream onto a communication channel.

34. The method of claim 31, further comprising:

turning off power to the light modulation unit in response to an expiration of a predetermined amount of time in which no object movement is detected in the monitoring of the first data stream.

35. The method of claim 22, further comprising:

monitoring the first data stream after turning on power to the light modulation unit in order to detect a signal within the incident light stream having significant energy within a signal class of interest.

36. The method of claim 35, further comprising:

in response to detecting said signal having significant energy within the signal class of interest, executing a reconstruction algorithm, wherein the reconstruction algorithm computes an image or image sequence based on at least a subset of the first data stream.

37. The method of claim 35, further comprising:

in response to detecting said signal having significant energy within the signal class of interest, directing a transmitter to transmit at least a subset of the first data stream onto a communication channel.

38. The method of claim 22, wherein said generating the first data stream includes converting spectral portions of the modulated light stream into respective sequences of intensity samples, wherein the first data stream includes the sequences of intensity samples.

39. The method of claim 38, further comprising:

monitoring the first data stream in order to detect a spectral signature of interest in the modulated light stream.

40. The method of claim 39, further comprising:

in response to detecting the spectral signature of interest, executing a reconstruction algorithm, wherein the reconstruction algorithm computes an image or image sequence based on at least a subset of the first data stream.

41. The method of claim 39, further comprising:

in response to detecting the spectral signature of interest, directing a transmitter to transmit at least a subset of the first data stream onto a communication channel.

42. The method of claim 22, further comprising:

searching for a bright object within a field of view of the incident light field after turning on power to the light modulation unit, wherein said searching includes:

selecting a next search pattern to be injected into the sequence of spatial patterns based on a portion of the first data stream that corresponds to one or more previous search patterns that have already been injected into the sequence of spatial patterns.

43. A system comprising:

a light modulation unit including a plurality of light reflecting elements whose orientations are independently controllable, wherein each of the light reflecting elements is configured to assume a neutral orientation state when the light modulation unit is powered off, wherein each of the light reflecting elements is configured to controllably switch between two active orientation states when the light modulation unit is powered on, wherein the light reflecting elements are configured to modulate an incident light stream with a time sequence of spatial patterns when the light modulation unit is powered on, wherein the modulation produces a modulated light stream, wherein the light reflecting elements are configured to reflect the incident light stream from the neutral orientation state when the light modulation unit is powered off to produce a neutral-state light stream;

a first light sensing device configured to receive the modulated light stream when the light modulation unit is powered on, and to generate a first data stream in response to the modulated light stream;

a second light sensing device configured to receive the neutral-state light stream when the light modulation unit is powered off, and to generate a second data stream in response to the neutral-state light stream;

a control unit configured to:

monitor the second data stream when the light modulation unit is powered off in order to detect a light variation event in the second data stream, wherein the light variation event indicates a variation of light in the neutral-state light stream; and

turn on power to the light modulation unit in response to determining that a trigger condition is satisfied, wherein the trigger condition includes detection of the light variation event in the second data stream.

44. The system of claim 43, wherein, after turning on power to the light modulation unit, the control unit is configured to:

inject a sequence of measurement patterns into the time sequence of spatial patterns; and

45. The system of claim 43, further comprising a transmitter, wherein, after turning on power to the light modulation unit, the control unit is configured to:

46. The system of claim 43, wherein the first light sensing device is actively cooled, wherein the second light sensing device is not actively cooled.

47. The system of claim 43, wherein the plurality of light reflecting elements are arranged on a plane, wherein the light modulation unit is configured so that an angle of incidence of the incident light stream on the plane is not equal to zero.

48. The system of claim 43, wherein the control unit is configured to monitor the first data stream after turning on power to the light modulation unit in order to detect a signal in the incident light stream having sufficient energy within a signal class of interest.

49. The system of claim 48, wherein the control unit is configured to execute a reconstruction algorithm in response to detecting said signal having significant energy within the signal class of interest, wherein the reconstruction algorithm is configured to compute an image or image sequence based on at least a subset of the first data stream.

50. The system of claim 48, further comprising a transmitter, wherein the control unit is configured to direct the transmitter to transmit at least a subset of the first data stream onto a communication channel in response to detecting said signal having significant energy within the signal class of interest.

51. The system of claim 43, wherein the control unit is configured to monitor the second data stream after turning on power to the light modulation unit in order to detect a signal in the incident light stream having sufficient energy within a signal class of interest.

52. The system of claim 43, wherein the second light sensing device includes a light sensing element and an analog-to-digital converter (ADC), wherein the light sensing element is configured to generate an analog electrical signal representing intensity of the neutral-state light stream as a function of time, wherein the ADC is configured to capture a sequence of samples of the analog electrical signal, wherein the second data stream includes the sequence of samples.

53. The system of claim 43, wherein the second light sensing device includes a plurality of light sensing elements, each configured to receive a corresponding spatial portion of the neutral-state light stream, wherein, for each of the light sensing elements, the second light sensing device is configured to capture a corresponding sequence of samples representing intensity over time of the corresponding spatial portion of the neutral-state light stream, wherein the second data stream includes the sequences of samples captured by the light sensing elements.

54. The system of claim 43, wherein the second light sensing device includes a plurality of light sensing elements, each configured to receive a corresponding spectral portion of the neutral-state light stream, wherein, for each of the light sensing elements, the second light sensing device is configured to capture a corresponding sequence of intensity samples representing intensity of the corresponding spectral portion of the neutral-state light stream, wherein the second data stream includes the sequences of samples captured by the light sensing elements.

55. The system of claim 43, wherein the second light sensing device includes a motion sensor configured to generate the second data stream in response to the neutral-state light stream.

56. The system of claim 43, further comprising an optical subsystem and a motion sensor, wherein the optical subsystem is configured to receive an input light stream and separate the input light stream into two output streams including the incident light stream and a secondary light stream, wherein the motion sensor is configured to receive the secondary light stream and generate a sequence of samples in response to the secondary light stream, wherein the control unit is configured to monitor the sequence of samples, wherein the trigger condition also includes detection of a disturbance in the sequence of samples.

57. A method comprising:

supplying an incident light stream to a light modulation unit, wherein the light modulation unit includes a plurality of light reflecting elements whose orientations are independently controllable, wherein each of the plurality of light reflecting elements take a neutral orientation state when the light modulation unit is powered off;

when the light modulation unit is powered off, reflecting the incident light stream from the plurality of light reflecting elements to obtain a neutral-state light stream, wherein the incident light stream is reflected from the light reflecting elements in the neutral orientation state;

generating a neutral-state data stream in response to the neutral-state light stream when the light modulation unit is powered off;

monitoring the neutral-state data stream when the light modulation unit is powered off in order to detect a light variation event in the neutral-state data stream, wherein the light variation event indicates a variation of light in the neutral-state light stream;

turning on power to the light modulation unit in response to determining that a trigger condition is satisfied, wherein the trigger condition includes said detection of the light variation event in the neutral-state data stream;

after turning on power to the light modulation unit, modulating the incident light stream with a time sequence of spatial patterns to obtain a modulated light stream; and

generating an on-state data stream in response to the modulated light stream.

58. The method of claim 57, further comprising:

executing a reconstruction algorithm on at least a subset of the on-state data stream to obtain an image or image sequence.

59. The method of claim 57, further comprising:

transmitting at least a subset of the on-state data stream onto a transmission channel.

60. The method of claim 57, further comprising:

after turning on power to the light modulation unit, monitoring the on-state data stream in order to detect a signal in the incident light stream having sufficient energy within a signal class of interest.

61. The method of claim 60, further comprising:

in response to detecting said signal having significant energy within the signal class of interest, executing a reconstruction algorithm, wherein the reconstruction algorithm is configured to compute an image or image sequence based on at least a subset of the on-state data stream.

62. The method of claim 60, further comprising:

in response to detecting said signal having significant energy within the signal class of interest, directing a transmitter to transmit at least a subset of the on-state data stream onto a communication channel.

63. The method of claim 57, wherein said generating the neutral-state data stream includes converting the neutral-state light stream into an analog electrical signal and capturing a sequence of samples of the analog electrical signal.

64. The method of claim 57, wherein said generating the neutral-state data stream includes converting spatial portions of the neutral-state light stream into respective sequences of intensity samples, wherein the neutral-state data stream includes the sequences of intensity samples.

65. The method of claim 57, wherein said generating the neutral-state data stream includes converting spectral portions of the neutral-state light stream into respective sequences of intensity samples, wherein the neutral-state data stream includes the sequences of samples.

66. The method of claim 57, wherein said generating the neutral-state data stream is performed by a motion sensor.

67. The method of claim 57, further comprising:

separating an input light stream into two output streams including the incident light stream and a secondary light stream;

converting the secondary light stream into a sequence of samples using a motion sensor; and

monitoring the sequence of samples to detect a disturbance in the sequence of samples, wherein the trigger condition also includes detection of the disturbance.

68. A system comprising:

a light modulation unit configured to receive a first light stream from a first optical input path, wherein the light modulation unit includes a plurality of light reflecting elements whose orientations are independently controllable, wherein each of the light reflecting elements is configured to controllably switch between two active orientation states when the light modulation unit is powered on, wherein the light modulation unit, when it is powered on, is configured to modulate the first light stream with a time sequence of spatial patterns to obtain a modulated light stream;

a motion sensor configured to receive a second light stream from a second optical input path, and to generate a second data stream in response to the second light stream at least when the light modulation unit is powered off, wherein the first and second optical input paths are independent optical paths;

a control unit configured to:

monitor the second data stream when the light modulation unit is powered off in order to detect a disturbance in the second data stream, wherein the disturbance indicates object motion in the field of view of the second light stream; and

turn on power to the light modulation unit in response to detection of the disturbance.

69. The system of claim 68, wherein the motion sensor includes a passive infrared sensor.