US20060142981A1  Statistical modeling and performance characterization of a realtime dual camera surveillance system  Google Patents
Statistical modeling and performance characterization of a realtime dual camera surveillance system Download PDFInfo
 Publication number
 US20060142981A1 US20060142981A1 US11/360,800 US36080006A US2006142981A1 US 20060142981 A1 US20060142981 A1 US 20060142981A1 US 36080006 A US36080006 A US 36080006A US 2006142981 A1 US2006142981 A1 US 2006142981A1
 Authority
 US
 United States
 Prior art keywords
 method
 object
 camera
 plurality
 space
 Prior art date
 Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
 Abandoned
Links
Images
Classifications

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N7/00—Television systems
 H04N7/18—Closed circuit television systems, i.e. systems in which the signal is not broadcast
 H04N7/181—Closed circuit television systems, i.e. systems in which the signal is not broadcast for receiving images from a plurality of remote sources

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T7/00—Image analysis
 G06T7/20—Analysis of motion
 G06T7/277—Analysis of motion involving stochastic approaches, e.g. using Kalman filters

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T7/00—Image analysis
 G06T7/70—Determining position or orientation of objects or cameras
 G06T7/77—Determining position or orientation of objects or cameras using statistical methods

 G—PHYSICS
 G08—SIGNALLING
 G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
 G08B13/00—Burglar, theft or intruder alarms
 G08B13/18—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength
 G08B13/189—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems
 G08B13/194—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
 G08B13/196—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
 G08B13/19602—Image analysis to detect motion of the intruder, e.g. by frame subtraction
 G08B13/19604—Image analysis to detect motion of the intruder, e.g. by frame subtraction involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change

 G—PHYSICS
 G08—SIGNALLING
 G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
 G08B13/00—Burglar, theft or intruder alarms
 G08B13/18—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength
 G08B13/189—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems
 G08B13/194—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
 G08B13/196—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
 G08B13/19602—Image analysis to detect motion of the intruder, e.g. by frame subtraction
 G08B13/19608—Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position

 G—PHYSICS
 G08—SIGNALLING
 G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
 G08B13/00—Burglar, theft or intruder alarms
 G08B13/18—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength
 G08B13/189—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems
 G08B13/194—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
 G08B13/196—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
 G08B13/19617—Surveillance camera constructional details
 G08B13/19626—Surveillance camera constructional details optical details, e.g. lenses, mirrors, multiple lenses
 G08B13/19628—Surveillance camera constructional details optical details, e.g. lenses, mirrors, multiple lenses of wide angled cameras and camera groups, e.g. omnidirectional cameras, fish eye, single units having multiple cameras achieving a wide angle view

 G—PHYSICS
 G08—SIGNALLING
 G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
 G08B13/00—Burglar, theft or intruder alarms
 G08B13/18—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength
 G08B13/189—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems
 G08B13/194—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
 G08B13/196—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
 G08B13/19639—Details of the system layout
 G08B13/19641—Multiple cameras having overlapping views on a single scene

 G—PHYSICS
 G08—SIGNALLING
 G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
 G08B13/00—Burglar, theft or intruder alarms
 G08B13/18—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength
 G08B13/189—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems
 G08B13/194—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
 G08B13/196—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
 G08B13/19639—Details of the system layout
 G08B13/19641—Multiple cameras having overlapping views on a single scene
 G08B13/19643—Multiple cameras having overlapping views on a single scene wherein the cameras play different roles, e.g. different resolution, different camera type, masterslave camera

 G—PHYSICS
 G08—SIGNALLING
 G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
 G08B13/00—Burglar, theft or intruder alarms
 G08B13/18—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength
 G08B13/189—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems
 G08B13/194—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
 G08B13/196—Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
 G08B13/19639—Details of the system layout
 G08B13/19647—Systems specially adapted for intrusion detection in or around a vehicle

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N5/00—Details of television systems
 H04N5/222—Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
 H04N5/225—Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, camcorders, webcams, camera modules specially adapted for being embedded in other devices, e.g. mobile phones, computers or vehicles
 H04N5/232—Devices for controlling television cameras, e.g. remote control ; Control of cameras comprising an electronic image sensor

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T2207/00—Indexing scheme for image analysis or image enhancement
 G06T2207/30—Subject of image; Context of image processing
 G06T2207/30196—Human being; Person

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T2207/00—Indexing scheme for image analysis or image enhancement
 G06T2207/30—Subject of image; Context of image processing
 G06T2207/30232—Surveillance

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T2207/00—Indexing scheme for image analysis or image enhancement
 G06T2207/30—Subject of image; Context of image processing
 G06T2207/30241—Trajectory
Abstract
The present invention relates to a method for visually detecting and tracking an object through a space. The method chooses modules for a restricting a search function within the space to regions with a high probability of significant change, the search function operating on images supplied by a camera. The method also derives statistical models for errors, including quantifying an indexing step performed by an indexing module, and tuning system parameters. Further the method applies a likelihood model for candidate hypothesis evaluation and object parameters estimation for locating the object.
Description
 1. Field of the Invention
 The present invention relates to computer vision systems, more particularly to a system having computationally efficient realtime object detection, tracking, and zooming capabilities.
 2. Description of Prior Art
 Recent advancements in processing and sensing performances are facilitating increased development of realtime video surveillance and monitoring systems.
 The development of computer vision systems that meet application specific computational and accuracy needs are important to the deployment of reallife computer vision systems. Such a computer vision system has not yet been realized.
 Past works have addressed methodological issues and have demonstrated performance analysis of components and systems. However, it is still an art to engineer systems that meet given application needs in terms of computational speed and accuracy. The trend in the art is to emphasize statistical learning methods, more particularly Bayesian methods for solving computer vision problems. However, there still exists the problem of choosing the right statistical likelihood model and the right priors to suit the needs of an application. Moreover, it is still computationally difficult to satisfy realtime application needs.
 Sequential decomposition of the total task into manageable subtasks (with reasonable computational complexity) and the introduction of pruning thresholds is one method to solve the problem. Yet, this introduces additional problems because of the difficulty in approximating the probability distributions of observables at the final step of the system so that Bayesian inference is plausible. This approach to perceptual Bayesian is described, for example, in V. Ramesh et al., “Computer Vision Performance Characterization,” RADIUS: Image Understanding for Imagery Intelligence, edited by, O. Firschein and T. Strat, Morgan Kaufmann Publishers, San Francisco, 1997, incorporated herein by reference, and W. Mann and T. Binford, “Probabilities for Bayesian Networks in Vision,” Proceedings of the ARPA IU Workshop, 1994, Vol. 1, pp. 633643. The work done by Ramesh et al., places an emphasis on performance characterization of a system, while Mann and Binford attempted Bayesian inference (using Bayesian networks) for visual recognition. The idea of gradual pruning of candidate hypotheses to tame the computational complexity of the estimation/classification problem has been presented by Y. Amit and D. Geman, “A computational model for visual selection,” Neural Computation, 1999. However, none of the works identify how the subtasks (e.g., feature extraction steps) can be chosen automatically given an application context.
 Therefore, a need exists for a method and apparatus for a computationally efficient, realtime camera surveillance system with defined computational and accuracy constraints.
 The present invention relates to computer vision systems, more particularly to a system having computationally efficient realtime detection and zooming capabilities.
 According to an embodiment of the present invention, by choosing system modules and performing an analysis of the influence of various tuning parameters on the system a method according to the present invention performs proper statistical inference, automatically set control parameters and quantify limits of a dualcamera realtime video surveillance system. The present invention provides continuous high resolution zoomedin image of a person's head at any location in a monitored area. Preferably, an omnidirectional camera video used to detect people and to precisely control a high resolution foveal camera, which has pan, tilt and zoom capabilities. The pan and tilt parameters of the foveal camera and its uncertainties are shown to be functions of the underlying geometry, lighting conditions, background color/contrast, relative position of the person with respect to both cameras as well as sensor noise and calibration errors. The uncertainty in the estimates is used to adaptively estimate the zoom parameter that guarantees with a user specified probability, ∀, that the detected person's face is contained and zoomed within the image.
 The present invention includes a method for selecting intermediate transforms (components of the system), as well as processing various parameters in the system to perform statistical inference, automatically setting the control parameters and quantifying a dualcamera realtime video surveillance system.
 Another embodiment of the present invention relates to a method for visually locating and tracking an object through a space. The method chooses modules for a restricting a search function within the space to regions with a high probability of significant change, the search function operating on images supplied by a camera. The method also derives statistical models for errors, including quantifying an indexing step performed by an indexing module, and tuning system parameters. Further, the method applies a likelihood model for candidate hypothesis evaluation and object parameters estimation for locating the object.
 The step of choosing the plurality of modules further includes applying a calibration module for determining a static scene, applying an illuminationinvariant module for tracking image transformation, and applying the indexing module for selecting regions of interest for hypothesis generation. Further, the method can apply a statistical estimation module for estimating a number of objects and their positions and apply a foveal camera control module for estimating control parameters of a foveal camera based on location estimates and uncertainties.
 Additional modules can be applied by the method, for example, a background adaptation module for detecting and tracking the object in dynamically varying illumination situations.
 Each module is application specific based on prior distributions for imposing restrictions on a search function. The prior distributions includes for example: an object geometry model; a camera geometry model; a camera error model; and an illumination model.
 According to an embodiment of the present invention the camera is an omnicamera. Further, the object is tracked using a foveal camera.
 The method derives statistical models a number of times to achieve a given probability of misdetection and false alarm rate. The method also validates a theoretical model for the space monitored for determining correctness and closeness to reality. The indexing module selects regions with a high probability of significant change, motivated by two dimensional image priors induced by prior distributions in the space, where the space is in three dimensional.
 The method of applying a likelihood model includes estimating an uncertainty of the object's parameters for predicting a system's performance and for automating control of the system.
 In an alternative embodiment the method can be employed in an automobile wherein the space includes an interior compartment of the automobile and/or the exterior of the automobile.
 In yet another embodiment of the present invention, a computer program product is presented. The program product includes a computer program code stored on a computer readable storage medium for, for detecting and tracking objects through a space. The computer program product includes computer readable program code for causing a computer to choose modules for a restricting search functions within a context to regions with a high probability of significant change within the space. The computer program product also includes computer readable, program code for causing a computer to derive statistical models for errors, including quantifying an indexing step, and tuning system parameters. Further included is computer readable program code for causing a computer to apply a likelihood model for candidate hypothesis evaluation and object parameters estimation within the space.
 Preferred embodiments of the present invention will be described below in more detail with reference to the accompanying drawings:

FIG. 1 is a block diagram showing a method for tracking an object through a space according to one embodiment of the present invention; 
FIG. 2 is an illustration of a system of cameras for tracking a person according to one embodiment of the present invention; 
FIG. 3 is an illustration of an omniimage including the geometric relationships between elements of the system while tracking a person according to one embodiment of the present invention; 
FIG. 4 is an illustration of how uncertainties in three dimensional radial distancesinfluence foveal camera control parameters; and 
FIG. 5 is an illustration of the geometric relationship between a foveal camera and a person.  Throughout the diagrams, like labels in different figures denote like or corresponding elements or relationships. Further, the drawings are not to scale.
 The present invention solves the problems existing in the prior art described above, based on the following methods.
 System Configuration choice: According to one embodiment of the present invention, modules are chosen for an optical surveillance system, by use of context, in other words: application specific prior distributions for modules. These modules can include, for example, object geometry, camera geometry, error models and illumination models. Realtime constraints are imposed by pruning or indexing functions that restrict the search space for hypotheses. The choice of the pruning functions is derived from the application context and prior knowledge. A proper indexing function will be one that simplifies computation of the probability of false hypothesis or the probability of missing a true hypotheses as a function of the tuning constraints.
 Statistical Modeling and Performance
 Characterization: According to an aspect of the present invention, the derivation of statistical models for errors at various stages in the chosen vision system configuration assists in quantifying the indexing step. The parameters are tuned to achieve a given probability of missdetection and false alarm rate. In addition, a validation of theoretical models is performed for correctness (through MonteCarlo simulations) and closeness to reality (through real experiments).
 Hypotheses verification and parameter estimation: Bayesian estimation is preferably used to evaluate candidate hypotheses and estimate object parameters by using a likelihood model, P (measurements/hypothesis), that takes into account the effects of the preprocessing steps and tuning parameters. In addition, the uncertainty of the estimate is derived to predict system performance.
 One embodiment of the present invention includes a two camera surveillance system which continuously provides zoomedin high resolution images of the face of a person present in a room. These images represent the input to higherlevel vision modules, e.g., face recognition, compaction and eventlogging.
 In another embodiment, the present invention provides: 1) realtime performance on a lowcost PC, 2) person misdetection rate of Π_{m}, 3) person falsealarm rate of Π_{f}, 4) adaptive zooming of person irrespective of background scene structure with maximal possible zoom based on uncertainty of person attributes estimated (e.g., location in three dimensional (3D), height, etc.), with performance of the result characterized by face resolution attainable in area of face pixel region (as a function of distance, contrast between background and object, and sensor noise variance and resolution) and bias in the centering of the face. In addition, the method makes assumptions about scene structure, for example, the scene illuminate consists of light sources with similar spectrum (e.g., identical light sources in an office area), the number of people to the detected and tracked is bounded, and the probability of occlusion of persons (due to other persons) is small.
 Referring to
FIG. 2 , to continuously monitor an entire scene, the present invention uses an omnidirectional sensor including a omnicamera 205 and a parabolic mirror 206, for example, the OmniCam of S. Nayer, “Omnidirectional Video Camera,” Proceedings of the DARPA Image Understanding Workshop, Vol. 1, pp. 235242, 1997. This camera is preferably mounted below the ceiling 200 looking into the parabolic mirror located on the ceiling. The parabolic mirror 206 enables the camera 205 to see in all directions simultaneously. Note thatFIG. 2 is an illustration of one embodiment of the present invention. Other embodiments are contemplated, including, for example, different mirror alignments, alternative camera designs (including, for example, catadioptric stereo, panoramic, omni, and foveal cameras), varying the orientation of the cameras and multiple cameras systems. The present invention can be employed using a verity of cameras, calibration modules (discussed below) including a combination of real world and image measurements, compensate for different perspectives.  The present invention uses omniimages to detect and estimate the precise location of a given person's foot in the room and this information is used to identify the pan, tilt and zoom settings for a highresolution foveal camera. An omniimage is the scene as viewed from the omnicamera 205, typically in conjunction with a parabolic mirror 206, mounted preferably on the ceiling 200.
 According to one embodiment of the present invention, the choice of the various estimation steps in the system is motivated from image priors and realtime requirements. The camera control parameters, e.g., pan and tilt, are selected based on the location estimate and its uncertainty (that is derived from statistical analysis of the estimation steps) so as to center the person's head location in the foveal image. The zoom parameter is set to maximum value possible so that the camera view still encloses the persons head within the image.
 The general Bayesian formulation of the person detection and location estimation problem does not suit the realtime constraints imposed by the application. In one embodiment of the present invention, this formulation is used only after a pruning step. The pruning step rules out a majority of false alarms by designing an indexing step motivated by the two dimensional (2D) image priors (region size, shape, intensity characteristics) induced by the prior distribution in the 3D scene. The prior distributions for person shape parameters, including, for example, size, height, and his/her 3D location, are reasonably simple. These priors on the person model parameters induce 2D spatially variant prior distributions in the projections, e.g., the region parameters for a given person in the image depends on the position in the image, whose form depends on the camera projection model and the 3D object shape. In addition to shape priors, the image intensity/color priors can be used in the present invention.
 Typically, a method according to the present invention does not make assumptions about the object intensity, e.g., the homogeneity of the object since people can wear variety of clothing and the color spectrum of the light source is therefore not constrained. However, in an alternative embodiment, in a surveillance application, the background is typically assumed to be a static scene (or a slowly time varying scene) with known background statistics. Gaussian mixtures are typically used to approximate these densities. To handle shadowing and illumination changes, these distributions are computed after the calculation of an illumination invariant measure from a local region in an image. The prior distribution of the spectral components of the illuminants are assumed to have same but unknown spectral distribution. Further, the noise model for CCD sensor noise 106 can be specified. This is typically chosen to be i.i.d. zero mean Gaussian noise in each color band.
 In one embodiment of the present invention, the system preferably includes five functional modules: calibration, illuminationinvariant measure computation at each pixel, indexing functions to select sectors of interest for hypothesis generation, statistical estimation of person parameters (e.g., foot location estimation), and foveal camera control parameter estimation.
 Referring to
FIG. 1 , block diagram of the transformations applied to the input. A sensor 100, for example, an omnidirectional camera, records a scene 105, which preferably is recorded as a color image, the scene 105 is sent to input 110 as: {circumflex over (R)}(x,y),Ĝ(x,y),{circumflex over (B)}(x,y). The sensor is also subject to sensor noise 106 which will become part of the input 110.  The input 110, defined above, is transformed 115 (T:R^{3}→R^{2}), typically to compute an illumination invariant measure {circumflex over (r)}_{c}(x,y),ĝ_{c}(x,y) 120. The statistical model for the distribution of the invariant measure is influenced by the sensor noise model and the transformation T(.). The invariant measure mean (B_{o}(x,y)=(r_{b}(x,y),g_{b}(x,y))) and covariance matrix Σ_{B} _{ o }(x,y), is computed at each pixel (x,y) from several samples of R(x,y), G(x,y), B(x,y) for the reference image 121 of the static scene. A change detection measure
 {circumflex over (d)}^{2}(x,y) image 130 is obtained by computing the Mahalanobis distance 125 between the current image data values {circumflex over (r)}_{c}(x,y),ĝ_{c}(x,y) and the reference image data B_{o}(x,y). This distance image is used as input to two indexing functions P_{1}( ) 135 and P_{2}( ) 140. P_{1}( ) 135 discards the radial lines 2 by choosing hysteresis thresholding parameters 136 that satisfy a given combination of probability of false alarm and missdetection values, passing the results 137 to P_{2}( ) 140. P_{2}( ) 140 discards segments along the radial lines in the same manner, by choosing hysteresis thresholding parameters 138. The result is a set of regions with high probability of significant change 141. At this point the method employs a full blown statistical estimation technique 145 that uses the 3D model information 146, camera geometry information 147, priors 148 (including objects, shape, and 3D location), to estimate the number of objects and their positions 150. The method preferably estimates the control parameters 155 for the foveal camera based on the location estimates and uncertainties. Accordingly, the foveal camera is directed by the control parameters and hysteresis thresholding parameters, for example, a missdetection threshold.
 Additional modules are contemplated by the present invention. For example, a background adaptation module 111. To generalize the system and cover outdoor and hybrid illumination situations (indoor plus outdoor illumination) as well as slow varying changes in the static background scene, the present invention incorporates a scheme described in “Adaptive background mixture models for realtime tracking”, Chris Stauffer, W. E. L. Grimson (Proceedings of the CVPR conference, 1999), incorporated herein by reference. It can be shown qualitatively that the statistics for background pixels can be approximated by a Gamma distribution. The statistics are stable within a given time window. In the present invention the background adaptation module is fused with the system, without changing the entire analysis and algorithm. By remapping the teststatistic derived from the data, so that the cumulative density function of the remapped teststatistic approximates the cumulative density function of a Chisquare distribution. Therefore, the result of the Grimsonapproach is remapped pixelwise to obtain dĝ^{2 }in block 112, following the transform described below. By adding dĝ^{2 }(for each pixel) to the {circumflex over (d)}^{2 }value 130 (see equ. 7), a new distance image is obtained. This distance image can be input to the index function 135.
 The output of the background adaptation module 111 is also used to update the static background statistics, as shown in block 121.
 The distribution of pixels of the new distance measurement are also Chisquare distributed. The only difference is a rise in the degree of freedoms from two to three. The analysis remains the same, the thresholds are derived as described below. This is an illustration of how different modules can be fused in an existing framework without changing the statistical analysis. After reading the present invention, formulation of these additional modules will be within the purview of one ordinary skilled in the art.
 The projection model for the two cameras is discussed below with respect to
FIGS. 2 through 5 . The following geometric model parameters are denoted as: 
 H_{o }height of OmniCam above floor (inches)
 H_{f }height of foveal camera above floor (inches)
 H_{p }person's height (inches)
 R_{h }person's head radius (inches)
 R_{f }person's foot position in world coordinates (inches)
 D_{c }on floor projected distance between cameras (inches)
 p(x_{c},y_{c}) position of OmniCam center, (in omniimage) (pixel coordinates)
 r_{m }radius of parabolic mirror (in omniimage)(pixels)
 r_{h }distance person's head—(in omniimage)(pixels)
 r_{f }distance person's foot—(in omniimage) (pixels)
 η—angle between the person and the foveal camera relative to the OmniCam image center (Please see
FIG. 3 ).  2—angle between the radial line corresponding to the person and the zero reference line (please see
FIG. 3 ).
 Where capital variables are variables in 3D, and small variables are given in image coordinates. During the calibration step (combination of real world and image measurements) H_{o}, H_{f}, D_{c}, r_{m }and p(x_{c},y_{c}) are initialized and the corresponding standard deviations or tolerances are determined. In a preferred embodiment the calibration step is performed offline. Heights are typically calculated from the floor 201 up.
 Using the geometric features of an OmniCam 205, including a parabolic mirror, and under the hypothesis that the person 220 is standing upright, the relationship between r_{f }respectively r_{h }and R_{p }can be shown to be:
$\begin{array}{cc}{R}_{p}=a\text{\hspace{1em}}{H}_{o}\text{\hspace{1em}}a=2\text{\hspace{1em}}\frac{{r}_{m}{r}_{f}}{{r}_{m}^{2}{r}_{f}^{2}}\text{\hspace{1em}}\mathrm{with}& \left(1\right)\\ {R}_{p}=b\left({H}_{o}{H}_{p}\right)\text{\hspace{1em}}b=2\text{\hspace{1em}}\frac{{r}_{m}{r}_{h}}{{r}_{m}^{2}{r}_{h}^{2}}\text{\hspace{1em}}\mathrm{with}& \left(2\right)\end{array}$
Let ∀, and ∃ be the foveal camera 210 control parameters for the tilt and pan angles respectively. Further, D_{p}, the projected real world distance between the foveal camera 210 and the person 220. Assuming, the person's head is approximately located over his/her feet, and using basic trigonometry inFIGS. 2 and 3 , it can easily be seen that D_{p}, ∀, and ∃ are equal to:$\begin{array}{cc}{D}_{p}=\sqrt{{D}_{c}^{2}+{R}_{p}^{2}2{D}_{c}{R}_{p}\mathrm{cos}\left(\vartheta \right)}& \left(3\right)\\ \mathrm{tan}(\forall )=\frac{{H}_{p}{R}_{h}{H}_{f}}{{D}_{p}};\mathrm{sin}\left(\beta \right)=\frac{{R}_{p}}{{D}_{p}}\mathrm{sin}\left(\vartheta \right)& \left(4\right)\end{array}$
where θ is the angle between the person 220 and the foveal camera 210 relative to the OmniCam 205 position.  This step is the module that takes in as input, the current color image ({circumflex over (R)}(x,y),Ĝ(x,y),{circumflex over (B)}(x,y)), normalizes it to obtain ({circumflex over (r)}_{c}(x,y),ĝ_{c}(x,y)) and compares it with the background statistical model (B_{o}(x,y),Σ_{B} _{ o }(x,y)) to produce an illumination invariant measure image {circumflex over (d)}^{2}(x,y). This section illustrates the derivation of the distribution of {circumflex over (d)}^{2}(x,y) given that the input image measurements {circumflex over (R)}, Ĝ and {circumflex over (B)} are Gaussian with mean R,G,B, and identical standard deviation Φ.
 With respect to
FIG. 1 , the illumination prior assumption 116, is that the scene contains multiple light sources with the same spectral distribution with no constraint on individual intensities. To compensate for shadows which are often present in the image, the method employs a shadow invariant representation of the color data. The invariant representation is according to G. Wyszecki and W. S. Stiles “Color Science: Concepts and Methods, Quantitative Data and Formulae,” John Wiley & Son, 1982 incorporated herein by reference. Accordingly, let S═R+G+B. The illumination normalizing transform T:R^{3}→R^{2 }appropriate for the method's assumptions is:$r=\frac{R}{R+G+B},g=\frac{G}{R+G+B}.$
It can be shown that, the uncertainties in the normalized estimates {circumflex over (r)} and ĝ are dependent not only on sensor noise variance, but also on the actual true unknown values of the underlying samples (due to the nonlinearities in the transformation T(.)). Based on the assumption of a moderate signal to noise ratio (i.e., Φ<<S), the method approximates ({circumflex over (r)},ĝ)^{T }as having a normal distribution with pixeldependent covariance matrix$\begin{array}{cc}\left(\begin{array}{c}\hat{r}\\ \hat{g}\end{array}\right)=\left(\begin{array}{c}\frac{R+{\eta}_{R}}{S+{\eta}_{R}+{\eta}_{G}+{\eta}_{B}}\\ \frac{G+{\eta}_{R}}{S+{\eta}_{R}+{\eta}_{G}+{\eta}_{B}}\end{array}\right)~N\left(\left(\begin{array}{c}r\\ g\end{array}\right),\sum _{\hat{r},\hat{g}}\right)\text{\hspace{1em}}\mathrm{with}\text{}\sum _{\hat{r},\hat{g}}=\frac{{\sigma}^{2}}{{S}^{2}}\left(\begin{array}{cc}1\frac{2R}{S}+3\frac{{R}^{2}}{{S}^{2}}& \frac{R+G}{S}+3\frac{R\text{\hspace{1em}}G}{{S}^{2}}\\ \frac{R+G}{S}+3\frac{R\text{\hspace{1em}}G}{{S}^{2}}& 1\frac{2G}{S}+3\frac{{G}^{2}}{{S}^{2}}\end{array}\right)& \left(5\right)\end{array}$  The values of σ_{{circumflex over (r)},{circumflex over (r)}} ^{2}, σ_{ĝ,ĝ} ^{2}, and σ_{{circumflex over (r)},ĝ} ^{2 }are determined offline for an entire OmniCam 205 frame, e.g., for each point or pixel on the image plane 207. These points vary spatially. Note, that in the normalized space the covariance matrix for each pixel is different: Bright regions in the covariance image correspond to regions with high variance in the normalized image. These regions correspond to dark regions in RGB space.
 Since the covariance matrices in the normalized space are pixeldependent, a method according to the present invention calculates the test statistic, i.e., the Mahalanobis distance d^{2}, that provides a normalized distance measure of a current pixel being background. Let {circumflex over (μ)}_{b }be the vector of mean r_{b}, and mean g_{b }at a certain background position (mean b_{b }is redundant, due to normalization), and {circumflex over (μ)}_{c }be the corresponding vector of the current image pixel. Since
$\begin{array}{cc}\left(\begin{array}{c}{\hat{r}}_{c}{\hat{r}}_{b}\\ {\hat{g}}_{c}{\hat{g}}_{b}\end{array}\right)~N\left(\left(\begin{array}{c}{r}_{c}{r}_{b}\\ {g}_{c}{g}_{b}\end{array}\right),{\sum}_{{\hat{r}}_{c},{\hat{g}}_{c}}+{\sum}_{{\hat{r}}_{b},{\hat{g}}_{b}}\right)& \left(6\right)\end{array}$
the method can define, for each pixel, a metric d^{2 }which corresponds to the probability, that {circumflex over (μ)}_{c }is background pixel:
{circumflex over (d)} ^{2}=({circumflex over (μ)}_{b}−{circumflex over (μ)}_{c})^{T}(2Σ_{{circumflex over (r)}} _{ b } _{−ĝ} _{ b })^{−1}({circumflex over (μ)}_{b}−{circumflex over (μ)}_{c}) (7)
For background pixels, {circumflex over (d)}^{2 }is approximately χ^{2 }distributed with two degrees of freedom. For object pixels {circumflex over (d)}^{2 }happens to be noncentral χ^{2 }distributed with two degrees of freedom, and noncentrality parameter c.  To address realtime computational requirements of the application the method identifies sectored segments in the image that potentially contains people of interest. To perform this indexing step in a computational efficient manner the method defines two index functions P_{1}( ) and P_{2}( ) that are applied sequentially as shown in
FIG. 1 . Essentially P_{1}( ) and P_{2}( ) are projection operations. For instance, define {circumflex over (d)}^{2}(R,θ) as the change detection measure image in polar coordinates with coordinate system origin at the omniimage center p(x_{c},y_{c}). Then, P_{1}( ) is chosen to be the projection along radial lines to obtain {circumflex over (M)}_{θ}, the test statistic that can be used to identify changes along a given direction 2. This test statistic is justified by the fact that the object projection is approximated by a lineset (approximated as an ellipse) whose major axis passes through the omniimage center with a given length distribution that is a function of the radial foot position coordinates of the person in the omniimage. This section derives the expressions for the probabilities of false alarm and misdetection at this step as a function of the input distributions for {circumflex over (d)}^{2}(R,θ), the prior distribution for the expected fraction of the pixels along a given radial line belonging to the object, and the noncentrality parameter of {circumflex over (d)}^{2}(R,θ) in object locations.  Let L_{θ} ^{x} ^{ c } ^{,y} ^{ c }be a radial line trough p(x_{c},y_{c}) parameterized by angle 2, and {circumflex over (M)}(θ)=Σ_{r}d^{θ} ^{2}(r) denote the accumulative measure of d^{2 }values at image position p(θ,r) parameterized by angle 2 and distance r in a polar coordinate system at p(x_{c},y_{c}). Applying Canny's hysteresis thresholding technique on {circumflex over (M)}(θ), provides the sectors of significant change bounded by left and right angles 2_{1 }respectively 2_{r}. Let r_{m }be the total number of pixels along a radial line L_{θ} ^{x} ^{ c } ^{,y} ^{ c }, and k be the expected number of object pixels along this line. The distribution of k can be derived from the projection model and the 3D prior models for person height, size, and position described previously. The distribution of the cumulative measure is:
Background M_{q}˜c_{2r} _{ m } ^{2}(0) (8)
Object M_{q}˜(r_{m} −k)c_{2(r} _{ m } _{−k})^{2}(0)+kc_{2k} ^{2}(c) (9)
with cε[0 . . . inf).  To obtain a falsealarm rate for false sectors of equal or less than x_{f}% the method can set the lower threshold T_{1 }so that
∫_{0} ^{T} ^{ 1 }χ_{{circumflex over (M)}} _{ θ } ^{2}(ε)dε=1−x _{f}% (10)
To guarantee a misdetection rate of equal or less than x_{m}%, theoretically, the method can solve for an upper threshold T_{u }similarly by evaluating the distribution in object equation above. Note that k is a function of H_{p}, R_{f}, and C. Therefore, the illustrative method would need to know the distributions of H_{p}, R_{f}, and c to solve for T_{u}. Rather then make assumptions about the distribution of noncentral parameter c, the method uses LUT T_{u}(x_{m}) generated by simulations instead.  The second index function P_{2}( ) essentially takes as input the domain corresponding to the radial lines of interest and performs a pruning operation along the radial lines R. This is done by the computation of {overscore (d)}_{θ} _{ f } ^{2}(r) the integration of the values {circumflex over (d)}^{2}( ) along 2_{f}=2+π/2 (within a finite window whose size is determined by the prior density of the minor axis of the ellipse projection), for each point r on the radial line 2. The derivation of the distribution of the test statistic and the choice of the thresholds are exactly similar to the above step.
 The illustrative method derives the distributions of the {circumflex over (d)}^{2 }image measurements, and has narrowed the hypotheses for people location and attributes. The method performs a Bayes estimation of person locations and attributes. This step uses the likelihood models L({circumflex over (d)}^{2}background) and L({circumflex over (d)}^{2}object) along with 2D prior models for person attributes induced by 3D object priors P(R_{p}), P(H), P(2) and P(S). The present embodiment uses the fact that the probability of occlusion of a person is small to assert that the probability of a sector containing multiple people is small. The center angle 2_{f }of a given sector would in this instance provide the estimate of the major axis of the ellipse corresponding to the person. It is then sufficient to estimate the foot location of person along the radial line corresponding to 2_{f}. The center angle 2_{f }of the sector defines the estimate for the angular component of the foot position. The illustrative method approximates {circumflex over (θ)}_{f }to be normal distributed with unknown 2_{f }and variance σθ_{f}. 2_{f}'s are estimated as the center positions of the angular sectors given by P_{1}( ). The standard deviation of a given estimate can be determined by assuming that the width of the angular sector gives the 99 percentile confidence interval. Alternatively, this estimation can be obtained through sampling techniques.
 Given the line 2^{f }it is necessary to estimate the foot position of the person along this radial line. To find this estimate and variance of the radial foot position r_{f }the method chooses the best hypothesis for the foot position that minimizes the Bayes error. Let P(h_{i}m) denote the posterior probability to be maximized, where h_{i }denotes the ith out of multiple foot position hypotheses and m the measurements ({overscore (d)}_{θ} _{ f } ^{2}(r)), that are statistically independent; hyperscript b or o denotes background respectively object:
$\begin{array}{cc}\begin{array}{c}P\left({h}_{i}\u2758m\right)=P\left({h}_{i}^{b}\u2758{m}^{b}\right)P\left({h}_{i}^{o}\u2758{m}^{o}\right)\\ =P\left({h}_{i}^{b}\u2758{m}^{b}\right)\left(1P\left({\stackrel{\_}{h}}_{i}^{o}\u2758{m}^{o}\right)\right)\\ =\frac{p\left({m}^{b}\u2758{h}_{i}^{b}\right)P\left({h}_{i}^{b}\right)}{p\left({m}^{b}\right)}\frac{p\left({m}^{o}\right)p\left({m}^{o}\u2758{\stackrel{\_}{h}}_{i}^{o}\right)P\left({\stackrel{\_}{h}}_{i}^{o}\right)}{p\left({m}^{o}\right)}\end{array}& \left(11\right)\end{array}$
where p denotes the density function. P(h_{i}*m) becomes maximal for maximal p(m^{b}h_{i} ^{b}) and minimal p(m^{o}{overscore (h)}_{i} ^{o}), so that$\begin{array}{cc}{r}_{f}=\underset{{r}_{f}^{\prime}}{\mathrm{arg}\text{\hspace{1em}}\mathrm{max}}\mathrm{log}\left(\frac{p\left({m}^{b}\u2758{h}_{i}^{b}\right)}{p\left({m}^{o}\u2758{\stackrel{\_}{h}}_{i}^{o}\right)}\right)=\underset{{r}_{f}^{\prime}}{\mathrm{arg}\text{\hspace{1em}}\mathrm{max}}\left(\sum _{r=0}^{{r}_{f}^{\prime}1}{\stackrel{\_}{d}}_{{\theta}_{f}}^{2}\left(r\right)+\sum _{r={r}_{h}\left({r}_{f}^{\prime}\right)}^{{r}_{m}}{\stackrel{\_}{d}}_{{\theta}_{f}}^{2}\left(r\right)\sum _{r={r}_{f}}^{{r}_{h}\left({r}_{f}^{\prime}\right)1}{\stackrel{\_}{d}}_{{\theta}_{f}}^{2}\left(r\right)\right)& \left(12\right)\end{array}$  In one embodiment of the present invention, an estimate of the uncertainty in the foot position r_{f }is made. The method provides pdf's up to the latest step in the algorithm. At this point it is affordable to simulate the distribution of r_{f }and generate σ_{{circumflex over (r)}} _{ f } ^{2 }via perturbation analysis, since only few estimates with known distributions are involved in few operations. The method can approximate {circumflex over (r)}_{f }as Gaussian distributed with unknown mean {circumflex over (r)}_{f}, and variance σ_{{circumflex over (r)}} _{ f } ^{2}.
 Once the foot position P(θ_{f}, r_{f}) is known, the method can apply formula 1 through 4 above, to estimate 3D distances R_{p}, D_{p}, and foveal camera control parameter tilt ∀, pan ∃ and zoom factor z.

FIGS. 4 and 5 illustrate how uncertainties in 3D radial distance R_{p }influence the foveal camera control parameters. For the following error propagation steps the method assumes that {circumflex over (r)}_{m},{circumflex over (r)}_{p},Ĥ_{o},Ĥ_{p},Ĥ_{f}, and {circumflex over (D)}_{c }are Gaussian random variables with true unknown means r_{m}, r_{p},H_{o},H_{p},R_{h},H_{f}, and D_{c}, and variances σ_{{circumflex over (r)}} _{ m } ^{2},σ_{{circumflex over (r)}} _{ p } ^{2},σ_{Ĥ} _{ o } ^{2},σ_{Ĥ} _{ p } ^{2},σ_{{circumflex over (R)}} _{ h } ^{2}, σ_{Ĥ} _{ f } ^{2}; and σ_{{circumflex over (D)}} _{ c } ^{2 }respectively (all estimated in the calibration phase). The estimates and it's uncertainties propagate through the geometric transformations. The method produces the final results for the uncertainties in tilt ∀, and pan ∃, which were used to calculate the zoom parameter z. (for more details, and derivations of σ_{{circumflex over (R)}} _{ p } ^{2}, σ_{{circumflex over (D)}} _{ p } ^{2 }see M. Greiffenhagen and V. Ramesh, “AutoCameraMan: MultiSensor Based RealTime People Detection and Tracking System,” Technical Report, Siemens Corporate Research, Princeton, N.J., USA, November 1999.):$\begin{array}{cc}{\sigma}_{\mathrm{tan}\text{\hspace{1em}}\alpha}^{2}=\frac{{\sigma}_{{\hat{D}}_{p}}^{2}}{{D}_{p}^{4}}\left({\left({H}_{p}{R}_{h}{H}_{f}\right)}^{2}+{\sigma}_{{\hat{H}}_{p}}^{2}+{\sigma}_{{\hat{R}}_{p}}^{2}+{\sigma}_{{\hat{H}}_{f}}^{2}\right)+\frac{{\sigma}_{{\hat{H}}_{p}}^{2}+{\sigma}_{{\hat{R}}_{p}}^{2}+{\sigma}_{{\hat{H}}_{f}}^{2}}{{D}_{p}^{2}}& \left(13\right)\\ {\sigma}_{{\mathrm{sin}}^{\beta}}^{2}=\frac{{R}_{p}^{2}{\sigma}_{\hat{\upsilon}}^{2}{\mathrm{cos}}^{2}\upsilon}{{D}_{p}^{2}}+\left({\mathrm{sin}}^{2}v+{\sigma}_{\hat{v}}^{2}{\mathrm{cos}}^{2}v\right)*\left(\frac{{R}_{p}^{2}{\sigma}_{{\hat{D}}_{p}}^{2}}{{D}_{p}^{4}}+\frac{{\sigma}_{{\hat{R}}_{p}}^{2}}{{D}_{p}^{2}}+\frac{{\sigma}_{{\hat{R}}_{p}}^{2}{\sigma}_{{\hat{D}}_{p}}^{2}}{{D}_{p}^{4}}\right)& \left(14\right)\end{array}$  Given the uncertainties in the estimates, the method derives the horizontal and vertical angle of view for the foveal camera, γ_{h }respectively γ_{v}, which map directly to the zoom parameter z.
FIGS. 4 and 5 show the geometric relationships for the vertical case. Following equation provides the vertical angle of view.$\begin{array}{cc}{\gamma}_{v}=2a\text{\hspace{1em}}\mathrm{tan}\left(\frac{{\hat{R}}_{h}+{f}_{v}{\sigma}_{\mathrm{tan}\text{\hspace{1em}}\hat{\text{\hspace{1em}}\alpha}}{\hat{D}}_{p}^{\prime}}{\sqrt{{\hat{R}}_{h}^{2}+{\hat{D}}_{p}^{\mathrm{\prime 2}}}}\right)\text{\hspace{1em}}\mathrm{with}\text{\hspace{1em}}{\hat{D}}_{p}^{\prime}=\frac{{\hat{D}}_{p}}{\mathrm{cos}\text{\hspace{1em}}\alpha}& \left(15\right)\end{array}$
where factor f_{v }solves for${\int}_{0}^{\frac{\mathrm{fv}}{2}}N\left(0,1\right)d\xi =\frac{{x}_{z}}{2}\%$
given user specified confidence percentile x_{z }that the head is display in the foveal frame. Similar derivations apply for the horizontal case.  The method verifies the correctness of the theoretical expressions and approximations through extensive simulations only show plots validating expressions for illumination normalization (eqn. 5), and for foveal camera control parameters (eqn. 13, 14). This validation assumes correctness of the underlying statistical models. Validation of the models on real data is discussed below.
 The correctness of the models is verified by comparing ground truth values against module estimates for mean and variance of the running system. The following is an illustration of an embodiment of the present invention, eight positions P1P8 are marked having different radial distances and pan angles. Positions and test persons were chosen to simulate different positions, illumination, and contrast. The table for the final foveal camera control parameters is for one person. Ground truth values for the mean values were taken by measuring tilt angle a, and pan angle b by hand, and are compared against the corresponding mean of system measurements estimated from 100 trials per position and person. The variances calculated from the system estimates for pan and tilt angle are compared against the average of the corresponding varianceestimates calculated based on the analysis. The comparison between system output and ground truth demonstrates the correctness of the model assumptions in the statistical modeling process (see Table 1).
TABLE 1 Validation: First two lines shows the predicted and experimental variances for the tilt angle, respectively. The next two lines correspond to pan angle. ×10^{−5} P1 P2 P3 P4 P5 P6 P7 P8 {circumflex over ( )}2 2.1 2.12 1.57 1.4 1.35 1.31 1.31 1.32 σ tan {circumflex over (α)} {circumflex over ( )}2 2.05 2.04 1.6 1.34 1.36 1.32 1.4 1.31 σ tan {circumflex over (α)} {circumflex over (σ)}_{sin {circumflex over (β)}} ^{2} 28.9 26.1 21.3 17.9 15.3 15.2 18.4 20.1 {tilde over (σ)}_{sin {circumflex over (β)}} ^{2} 25.9 24.1 19.5 15.1 14.9 15 18.1 19.3  The performance of the running system will now be discussed. The output of the foveal camera is sufficient as input for face recognition algorithms. Illustrating how the statistical analysis is used to optimize the camera setup, equ. 13 and 14 suggest that the configuration that minimizes these uncertainties is the one with large intercamera distance D_{c }and foveal camera height H_{f }equal to the mean person eyelevel height Hp.
 The present invention is reliable in terms of detection and zooming over longtime experiments within the operational limits denoted by the outer line of the upper right contour plot.
 The setup of the system (for example, placement of foveal camera) influences precision globally and locally. Preferred directions of low uncertainties can be used to adapt the system to user defined accuracy constraints in certain areas of the room.
 In another embodiment of the present invention, a system for monitoring in and around an automobile is presented. The inventions uses an omnidirectional sensor (a standard camera plus a mirror assembly) to obtain a global view of the surroundings within and outside the automobile. The omnicamera video is used for detection and tracking of objects within and around the automobile. The concept is an extension of the methods described above with respect to tracking objects within a room. In this embodiment the system can be used to improve safety and security.
 The video analysis system can include multiple modules. For example, a calibration module where the center of the Omnicamera image is used with height information of the ceiling of the automobile to translate image coordinates to ground plane coordinates. Where a CAD model of the automobile is available, the image coordinates can be mapped to a 3D point on the interior of the automobile using this calibration step (if the automobile is not occupied). Another example is a change detection module that compares a reference map (reference image plus variation around the reference image) to current observed image map to determine a pixelbased change detection measure. This is done by transforming the color video stream into normalized color space (to deal with illumination variation). The change detection measure is used to index into a set of possible hypothesis for object positions and locations. Yet another example includes a background update module for varying background conditions (e.g. gain control change, illumination changes). A grouping module that takes the change detection measure along with a geometric model of the environment and the objects to identify likely object locations. In the current embodiment, the method provides the areas in the image corresponding to the windows and model people by upright cylinders when they are outside of the automobile. In the interior of the automobile, people can be modeled by generalized cylinders. Still another module includes an object tracking module that takes location information over time to do prediction of object locations in the subsequent time step and to reestimate their new locations. Preferably, the visualization is presented on a color liquid crystal display (LCD) panel mounted with the rearview mirror. The visualization module presents geometrically warped video of the omnicam video. This is useful for driver assistance (e.g. while the driver is backing up or when he/she is changing lanes). Other modules are contemplated by the present invention including, for example, a module that determines an approaching object's potential threat, e.g., at a higher rate of speed or from a particular direction.
 According to the automotive embodiment of the present invention, the OmniCam is a catadioptric system that includes two parts: a parabolic mirror; and a standard CCD camera looking into it. The invention is useful as a sensor for use in driver assistance. It is also useful for monitoring the surroundings when the automobile is stationary and for recording videos in the event that a person approaches the automobile and attempts to get unauthorized access. The omnicamera system can be use in conjunction with a pantilt camera to enable the capture of a zoomed up image of the persons involved. Once a person gains unauthorized access to the automobile and an alarm is triggered, a security system integrating vision, global positioning system (GPS) and mobile phone, can transmit the time, location and the face image of the person to a central security agency. In addition to the monitoring capability, the ability to present the panoramic view of the surroundings provides a method to alert the driver to potential danger in the surrounding area by visually emphasizing the region in the panoramic view. In addition, due to the mounting position of the Omnicamera, looking up into a parabolic mirror located on the ceiling of the automobile (preferably centered), parts of the surroundings that are invisible to the driver are visible in the Omniview. Thus, the driver blind spot area is significantly reduced. By evaluating the panoramic view it is possible to trigger warnings, e.g., if other cars enter a driver's blind spot. If automobile status information (speed, steering wheel position, predicted track) is combined with panoramic video processing it is possible to alert a driver to impending dangers or potential accidents.
 The present invention contemplates a system and method for tracking an object. The invention can be employed in varying circumstances, for example, video conferencing, distance learning, and security stations where a user can define an area of interest there by replacing traditional systems employing banks of monitors. The present invention also contemplates an application wherein the system is used in conjunction with a datalog for recording time and location together with images of persons present. In a datalog application the system can associate an image with recorded information upon the occurrence of an event, e.g., a person sits at a computer terminal within an area defined for surveillance. The datalog portion of the system is preferably performed by a computer, where the computer records, for example, the time, location, and identity of the subject, as well as an accompanying image. The present invention is not limited to the above applications, rather the invention can be implemented in any situations where object detection, tracking, and zooming is needed.
 Having described preferred embodiments of the present invention having computationally efficient realtime detection and zooming capabilities, it is noted that modifications and variations can be made by persons skilled in the art in light of the above teachings. It is therefore to be understood that changes may be made in the particular embodiments of the invention disclosed which are within the scope and spirit of the invention as defined by the appended claims.
Claims (14)
1. A method for visually locating and tracking an object through a space, comprising the steps of:
choosing a plurality of modules for restricting a search function within the space to a plurality of regions with a high probability of significant change, the search function operating on images supplied by a camera;
deriving statistical models for errors, including quantifying an indexing step performed by an indexing module, and tuning system parameters; and
applying a likelihood model for candidate hypothesis evaluation and object parameters estimation for locating the object.
2. The method of claim 1 , wherein the step of choosing the plurality of modules further comprises the steps of:
applying a calibration module for determining a static scene;
applying an illuminationinvariant module for tracking image transformation; and
applying the indexing module for selecting regions of interest for hypothesis generation.
3. The method of claim 2 , further comprising the steps of:
applying a statistical estimation module for estimating a number of objects and their positions; and
applying a foveal camera control module for estimating a plurality of control parameters of a foveal camera based on location estimates and uncertainties.
4. The method of claim 2 , further comprising the step of applying a background adaptation module for detecting and tracking the object in dynamically varying illumination situations.
5. The method of claim 1 , wherein each module is application specific based on a plurality of prior distributions for imposing restrictions on a search function.
6. The method of claim 5 , wherein the plurality of prior distributions comprise:
an object geometry model;
a camera geometry model;
a camera error model; and
an illumination model.
7. The method of claim 1 , wherein the camera is an omnicamera.
8. The method of claim 1 , wherein the object is tracked using a foveal camera.
9. The method of claim 1 , wherein the step of deriving statistical models is applied a plurality of times to achieve a given probability of misdetection and false alarm rate.
10. The method claim 9 , further comprising the step of validating a theoretical model for the space monitored for determining correctness and closeness to reality.
11. The method of claim 1 , wherein the indexing module selects a plurality of regions with a high probability of significant change, motivated by a plurality of two dimensional image priors induced by a plurality of prior distributions in the space, wherein the space is three dimensional.
12. The method of claim 1 , wherein the step of applying a likelihood model further comprises the step of estimating an uncertainty of the object's parameters for predicting a system's performance and for automating control of the system.
13. The method of claim 1 , employed in an automobile wherein the space monitored comprises one of an interior compartment of the automobile and an exterior of the automobile.
14. A computer program product comprising computer program code stored on a computer readable storage medium for, for locating and tracking objects through a space, the computer program product comprising:
computer readable program code for causing a computer to choose a plurality of modules for a restricting search functions within a context to a plurality of regions with a high probability of significant change within the space;
computer readable program code for causing a computer to derive statistical models for errors, including quantifying an indexing step, and tuning system parameters; and
computer readable program code for causing a computer to apply a likelihood model for candidate hypothesis evaluation and object parameters estimation for locating the object.
Priority Applications (2)
Application Number  Priority Date  Filing Date  Title 

US09/592,532 US7006950B1 (en)  20000612  20000612  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
US11/360,800 US20060142981A1 (en)  20000612  20060223  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
Applications Claiming Priority (2)
Application Number  Priority Date  Filing Date  Title 

US11/360,800 US20060142981A1 (en)  20000612  20060223  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
US11/484,994 US20070019073A1 (en)  20000612  20060712  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
Related Parent Applications (1)
Application Number  Title  Priority Date  Filing Date  

US09/592,532 Continuation US7006950B1 (en)  20000612  20000612  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
Related Child Applications (1)
Application Number  Title  Priority Date  Filing Date 

US11/484,994 Continuation US20070019073A1 (en)  20000612  20060712  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
Publications (1)
Publication Number  Publication Date 

US20060142981A1 true US20060142981A1 (en)  20060629 
Family
ID=35922912
Family Applications (4)
Application Number  Title  Priority Date  Filing Date 

US09/592,532 Active 20200901 US7006950B1 (en)  20000612  20000612  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
US11/112,930 Active 20241218 US7899209B2 (en)  20000612  20050422  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
US11/360,800 Abandoned US20060142981A1 (en)  20000612  20060223  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
US11/484,994 Abandoned US20070019073A1 (en)  20000612  20060712  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
Family Applications Before (2)
Application Number  Title  Priority Date  Filing Date 

US09/592,532 Active 20200901 US7006950B1 (en)  20000612  20000612  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
US11/112,930 Active 20241218 US7899209B2 (en)  20000612  20050422  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
Family Applications After (1)
Application Number  Title  Priority Date  Filing Date 

US11/484,994 Abandoned US20070019073A1 (en)  20000612  20060712  Statistical modeling and performance characterization of a realtime dual camera surveillance system 
Country Status (1)
Country  Link 

US (4)  US7006950B1 (en) 
Cited By (5)
Publication number  Priority date  Publication date  Assignee  Title 

US20100138094A1 (en) *  20081202  20100603  Caterpillar Inc.  System and method for accident logging in an automated machine 
US20110178703A1 (en) *  20090114  20110721  Sjoerd Aben  Navigation apparatus and method 
JP2014527655A (en) *  20110714  20141016  バイエリッシェ モートーレン ウエルケ アクチエンゲゼルシャフトＢａｙｅｒｉｓｃｈｅ Ｍｏｔｏｒｅｎ Ｗｅｒｋｅ Ａｋｔｉｅｎｇｅｓｅｌｌｓｃｈａｆｔ  Pedestrian gait recognition method and apparatus for a portable terminal 
JP2015501578A (en) *  20111014  20150115  オムロン株式会社  Method and apparatus for projective space monitoring 
US9615062B2 (en)  20101230  20170404  Pelco, Inc.  Multiresolution image display 
Families Citing this family (56)
Publication number  Priority date  Publication date  Assignee  Title 

US8352400B2 (en)  19911223  20130108  Hoffberg Steven M  Adaptive pattern recognition based controller apparatus and method and humanfactored interface therefore 
US7904187B2 (en)  19990201  20110308  Hoffberg Steven M  Internet appliance system and method 
US10298735B2 (en)  20010424  20190521  Northwater Intellectual Property Fund L.P. 2  Method and apparatus for dynamic configuration of a multiprocessor health data system 
US7146260B2 (en)  20010424  20061205  Medius, Inc.  Method and apparatus for dynamic configuration of multiprocessor system 
US7940299B2 (en) *  20010809  20110510  Technest Holdings, Inc.  Method and apparatus for an omnidirectional video surveillance system 
US7657935B2 (en) *  20010816  20100202  The Trustees Of Columbia University In The City Of New York  System and methods for detecting malicious email transmission 
JP2003223633A (en) *  20020129  20030808  Sharp Corp  Omnidirectional visual system 
JP4010444B2 (en) *  20020228  20071121  シャープ株式会社  Omnidirectional monitoring control system, omnidirectional monitoring control method and omnidirectional monitoring control program 
US7178049B2 (en)  20020424  20070213  Medius, Inc.  Method for multitasking multiple Java virtual machines in a secure environment 
US7194110B2 (en) *  20021218  20070320  Intel Corporation  Method and apparatus for tracking features in a video sequence 
US20050169415A1 (en) *  20040130  20050804  Agere Systems Inc.  Timing error recovery system 
TWI253292B (en) *  20040323  20060411  YuLin Chiang  Pano camera monitoring and tracking system and method thereof 
IL161082A (en) *  20040325  20080807  Rafael Advanced Defense Sys  System and method for automatically acquiring a target with a narrow fieldofview gimbaled imaging sensor 
US7593547B2 (en) *  20041012  20090922  Siemens Corporate Research, Inc.  Videobased encroachment detection 
US7337650B1 (en)  20041109  20080304  Medius Inc.  System and method for aligning sensors on a vehicle 
US20080291278A1 (en) *  20050405  20081127  Objectvideo, Inc.  Widearea sitebased video surveillance system 
US7583815B2 (en) *  20050405  20090901  Objectvideo Inc.  Widearea sitebased video surveillance system 
US7466842B2 (en) *  20050520  20081216  Mitsubishi Electric Research Laboratories, Inc.  Modeling low frame rate videos with bayesian estimation 
US20070076099A1 (en) *  20051003  20070405  Eyal Eshed  Device and method for hybrid resolution video frames 
US20080100473A1 (en) *  20061025  20080501  Siemens Corporate Research, Inc.  Spatialtemporal Image Analysis in Vehicle Detection Systems 
US20080272884A1 (en) *  20070503  20081106  Sybase 365, Inc.  System and Method for Enhanced Threat Alerting 
US9019381B2 (en) *  20080509  20150428  Intuvision Inc.  Video tracking systems and methods employing cognitive vision 
US20090296989A1 (en)  20080603  20091203  Siemens Corporate Research, Inc.  Method for Automatic Detection and Tracking of Multiple Objects 
DE102008049872A1 (en) *  20081001  20100429  Mobotix Ag  Safety knob for Security Cameras 
US9358924B1 (en)  20090508  20160607  Eagle Harbor Holdings, Llc  System and method for modeling advanced automotive safety systems 
US8417490B1 (en)  20090511  20130409  Eagle Harbor Holdings, Llc  System and method for the configuration of an automotive vehicle with modeled sensors 
CN102422719B (en) *  20090514  20140924  皇家飞利浦电子股份有限公司  Method for controlling lighting, lighting system and image processing device 
US8577083B2 (en)  20091125  20131105  Honeywell International Inc.  Geolocating objects of interest in an area of interest with an imaging system 
CN101719276B (en) *  20091201  20150902  北京中星微电子有限公司  A method and apparatus for detecting objects in the image 
US20110181716A1 (en) *  20100122  20110728  Crime Point, Incorporated  Video surveillance enhancement facilitating realtime proactive decision making 
US8385632B2 (en) *  20100601  20130226  Mitsubishi Electric Research Laboratories, Inc.  System and method for adapting generic classifiers for object detection in particular scenes using incremental training 
US9134399B2 (en)  20100728  20150915  International Business Machines Corporation  Attributebased person tracking across multiple cameras 
US8515127B2 (en)  20100728  20130820  International Business Machines Corporation  Multispectral detection of personal attributes for video surveillance 
US8532390B2 (en)  20100728  20130910  International Business Machines Corporation  Semantic parsing of objects in video 
WO2012048173A2 (en)  20101007  20120412  Siemens Corporation  Multisensor system for high performance and reconfigurable outdoor surveillance 
US9497388B2 (en)  20101217  20161115  Pelco, Inc.  Zooming factor computation 
US8448056B2 (en) *  20101217  20130521  Microsoft Corporation  Validation analysis of human target 
US8953039B2 (en) *  20110701  20150210  Utc Fire & Security Corporation  System and method for autocommissioning an intelligent video system 
US9070285B1 (en) *  20110725  20150630  UtopiaCompression Corporation  Passive camera based cloud detection and avoidance for aircraft systems 
US9235895B2 (en) *  20111213  20160112  Hitachi, Ltd.  Method for estimating direction of person standing still 
US9082004B2 (en)  20111215  20150714  The Nielsen Company (Us), Llc.  Methods and apparatus to capture images 
US8886392B1 (en)  20111221  20141111  Intellectual Ventures Fund 79 Llc  Methods, devices, and mediums associated with managing vehicle maintenance activities 
US8704904B2 (en)  20111223  20140422  H4 Engineering, Inc.  Portable system for high quality video recording 
WO2013131036A1 (en)  20120301  20130906  H4 Engineering, Inc.  Apparatus and method for automatic video recording 
AU2013225635B2 (en)  20120302  20171026  H4 Engineering, Inc.  Waterproof Electronic Device 
US9723192B1 (en)  20120302  20170801  H4 Engineering, Inc.  Application dependent video recording device architecture 
TWI468641B (en) *  20121109  20150111  Univ Nat Central  Time synchronization calibration method and system for image taking and coordinate reading and delay time calculation method thereof 
US10009579B2 (en)  20121121  20180626  Pelco, Inc.  Method and system for counting people using depth sensor 
US9367733B2 (en)  20121121  20160614  Pelco, Inc.  Method and apparatus for detecting people by a surveillance system 
US9639747B2 (en) *  20130315  20170502  Pelco, Inc.  Online learning method for people detection and counting for retail stores 
JP6512793B2 (en) *  20141107  20190515  キヤノン株式会社  Imaging device, surveillance camera system, control method and program for imaging device 
US9712828B2 (en) *  20150527  20170718  Indian Statistical Institute  Foreground motion detection in compressed video data 
US9721472B2 (en) *  20150922  20170801  Ford Global Technologies, Llc  Formulating lane level routing plans 
CN106888352A (en) *  20151216  20170623  中兴通讯股份有限公司  Coke pushing position determination method and device 
GB2553570A (en) *  20160909  20180314  Canon Europa Nv  Surveillance apparatus and surveillance method 
CN106791701A (en) *  20170120  20170531  国网河北省电力公司衡水供电分公司  Power line positioning routinginspection recorder 
Citations (8)
Publication number  Priority date  Publication date  Assignee  Title 

US5323470A (en) *  19920508  19940621  Atsushi Kara  Method and apparatus for automatically tracking an object 
US5434617A (en) *  19930129  19950718  Bell Communications Research, Inc.  Automatic tracking camera control system 
US5473369A (en) *  19930225  19951205  Sony Corporation  Object tracking apparatus 
US5574498A (en) *  19930925  19961112  Sony Corporation  Target tracking system 
US5953077A (en) *  19970117  19990914  Fox Sports Productions, Inc.  System for displaying an object that is not visible to a camera 
US6590999B1 (en) *  20000214  20030708  Siemens Corporate Research, Inc.  Realtime tracking of nonrigid objects using mean shift 
US6680745B2 (en) *  20001110  20040120  Perceptive Network Technologies, Inc.  Videoconferencing method with tracking of face and dynamic bandwidth allocation 
US20050259848A1 (en) *  20000204  20051124  Cernium, Inc.  System for automated screening of security cameras 
Family Cites Families (5)
Publication number  Priority date  Publication date  Assignee  Title 

US6188776B1 (en) *  19960521  20010213  Interval Research Corporation  Principle component analysis of images for the automatic location of control points 
US6188777B1 (en) *  19970801  20010213  Interval Research Corporation  Method and apparatus for personnel detection and tracking 
US6353679B1 (en) *  19981103  20020305  Compaq Computer Corporation  Sample refinement method of multiple mode probability density estimation 
US7233886B2 (en) *  20010119  20070619  Smartsignal Corporation  Adaptive modeling of changed states in predictive condition monitoring 
US7136507B2 (en) *  20031117  20061114  Vidient Systems, Inc.  Video surveillance system with rulebased reasoning and multiplehypothesis scoring 

2000
 20000612 US US09/592,532 patent/US7006950B1/en active Active

2005
 20050422 US US11/112,930 patent/US7899209B2/en active Active

2006
 20060223 US US11/360,800 patent/US20060142981A1/en not_active Abandoned
 20060712 US US11/484,994 patent/US20070019073A1/en not_active Abandoned
Patent Citations (8)
Publication number  Priority date  Publication date  Assignee  Title 

US5323470A (en) *  19920508  19940621  Atsushi Kara  Method and apparatus for automatically tracking an object 
US5434617A (en) *  19930129  19950718  Bell Communications Research, Inc.  Automatic tracking camera control system 
US5473369A (en) *  19930225  19951205  Sony Corporation  Object tracking apparatus 
US5574498A (en) *  19930925  19961112  Sony Corporation  Target tracking system 
US5953077A (en) *  19970117  19990914  Fox Sports Productions, Inc.  System for displaying an object that is not visible to a camera 
US20050259848A1 (en) *  20000204  20051124  Cernium, Inc.  System for automated screening of security cameras 
US6590999B1 (en) *  20000214  20030708  Siemens Corporate Research, Inc.  Realtime tracking of nonrigid objects using mean shift 
US6680745B2 (en) *  20001110  20040120  Perceptive Network Technologies, Inc.  Videoconferencing method with tracking of face and dynamic bandwidth allocation 
Cited By (6)
Publication number  Priority date  Publication date  Assignee  Title 

US20100138094A1 (en) *  20081202  20100603  Caterpillar Inc.  System and method for accident logging in an automated machine 
US8473143B2 (en) *  20081202  20130625  Caterpillar Inc.  System and method for accident logging in an automated machine 
US20110178703A1 (en) *  20090114  20110721  Sjoerd Aben  Navigation apparatus and method 
US9615062B2 (en)  20101230  20170404  Pelco, Inc.  Multiresolution image display 
JP2014527655A (en) *  20110714  20141016  バイエリッシェ モートーレン ウエルケ アクチエンゲゼルシャフトＢａｙｅｒｉｓｃｈｅ Ｍｏｔｏｒｅｎ Ｗｅｒｋｅ Ａｋｔｉｅｎｇｅｓｅｌｌｓｃｈａｆｔ  Pedestrian gait recognition method and apparatus for a portable terminal 
JP2015501578A (en) *  20111014  20150115  オムロン株式会社  Method and apparatus for projective space monitoring 
Also Published As
Publication number  Publication date 

US7899209B2 (en)  20110301 
US7006950B1 (en)  20060228 
US20100007740A1 (en)  20100114 
US20070019073A1 (en)  20070125 
Similar Documents
Publication  Publication Date  Title 

Radke et al.  Image change detection algorithms: a systematic survey  
Pavlidis et al.  Urban surveillance systems: from the laboratory to the commercial world  
US6956469B2 (en)  Method and apparatus for pedestrian detection  
EP1030188B1 (en)  Situation awareness system  
US7623674B2 (en)  Method and system for enhanced portal security through stereoscopy  
US6961466B2 (en)  Method and apparatus for object recognition  
US7003136B1 (en)  Planview projections of depth image data for object tracking  
CA2692424C (en)  System and process for detecting, tracking and counting human objects of interest  
US8000498B2 (en)  Moving object detection apparatus and method  
JP3603737B2 (en)  Mobile tracking method and apparatus  
US8189051B2 (en)  Moving object detection apparatus and method by using optical flow analysis  
US8224029B2 (en)  Object matching for tracking, indexing, and search  
US6542621B1 (en)  Method of dealing with occlusion when tracking multiple objects and people in video sequences  
US7940957B2 (en)  Object tracker for visually tracking object motion  
CA2575211C (en)  Apparatus and method for processing video data  
JP3279479B2 (en)  Video monitoring method and apparatus  
JP5551595B2 (en)  Runway Surveillance System and Method  
Reinders et al.  Locating facial features in image sequences using neural networks  
US8379926B2 (en)  Vision based real time traffic monitoring  
JP4355341B2 (en)  Visual tracking using the depth data  
US20070248244A1 (en)  Image surveillance/retrieval system  
US20060177097A1 (en)  Pedestrian detection and tracking with night vision  
US7274380B2 (en)  Augmented reality system  
US7929728B2 (en)  Method and apparatus for tracking a movable object  
US9854147B2 (en)  Method and system for performing adaptive image acquisition 
Legal Events
Date  Code  Title  Description 

STCB  Information on status: application discontinuation 
Free format text: ABANDONED  FAILURE TO RESPOND TO AN OFFICE ACTION 