WO2012048173A2 - Multi-sensor system for high performance and reconfigurable outdoor surveillance - Google Patents

Abstract

A high performance and reconfigurable surveillance platform for outdoor surveillance is provided. It incorporates multiple high-resolution color cameras that ensure sufficient resolution on a target of interest. Video streams are stitched into a single panoramic video. A pan/tilt/zoom (PTZ) camera is accurately registered to the panoramic view. An even higher resolution of a target of interest can be acquired using the PTZ camera using optical zoom. Cameras are synchronized with high accuracy in order to capture registered video streams. The system configuration, including camera type, desired resolution, is defined by an XML file without changing the code. The surveillance system provides multi-threaded applications through specified thread pools.

Description

MULTI-SENSOR SYSTEM FOR HIGH PERFORMANCE AND RECONFIGURABLE

OUTDOOR SURVEILLANCE

STATEMENT OF RELATED APPLICATIONS

[0001] The present application claims the benefit of U.S. Provisional Patent Application Ser. No 61/390,655 filed on October 7, 2010, which is incorporated by reference herein in its entirety.

TECHNICAL FIELD

[0002] The present invention relates to high performance and re-configurable surveillance systems.

BACKGROUND

[0003] Existing commercial outdoor surveillance systems are insufficient for high demand applications where area coverage, tracking capability and identification accuracy far exceed traditional surveillance systems. Such applications include portal-less monitoring of pathways leading toward critical infrastructures, airport security, and outdoor large crowd events. Existing commercial systems are lacking in the several senses, including lack of re- configurability, which in general requires complete reprogramming of the system when a new surveillance component is added and integrated with the system.

[0004] Accordingly, novel and improved methods and systems to create a reconfigurable surveillance system are required.

SUMMARY

[0005] In accordance with an aspect of the present invention methods and systems are provided to conduct multi-sensor, high-performance and reconfigurable surveillance which is preferably performed in an outdoor environment.

[0006] In accordance with an aspect of the present invention a surveillance system is provided for monitoring an area, comprising a panoramic camera arranged to view the area, a stereo camera arranged to view the area, a PTZ camera arranged to view the area, a processor having multiple cores that handles multiple threads that is connected the panoramic camera, the stereo camera and the PTZ camera, a plurality of plug-in components connected to the processor, each of the plurality of plug-in components being recognizable by the processor, and a memory readable by the processor, the memory storing a configuration file, wherein the processor communicates with the panoramic camera, the stereo camera, the PTZ camera and the plurality of plug-in components in accordance with the configuration file, processing the image signals from the panoramic camera, the stereo camera, the PTZ camera and the plurality of plug- in components on multiple threads using the multiple cores.

[0007] In accordance with a further aspect of the present invention a surveillance system is provided, wherein the configuration file is an XML file.

[0008] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the configuration file includes operational scheduling information.

[0009] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the configuration file specifies that stabilization be performed prior to image acquisition if stabilization is to be performed.

[0010] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein one of the plurality of plug-in components is a video stabilization plug-in device and the processor controls the video stabilization plug-in prior to image acquisition if the configuration file indicates stabilization is to be performed.

[0011] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the plurality of plug-in components includes a panoramic video stitching plug-in component.

[0012] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the plurality of plug-in components includes a video recording plug-in component.

[0013] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the plurality of plug-in components includes a video analytics plug-in component.

[0014] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the plurality of plug-in components includes a detachable display plug-in component.

[0015] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the plurality of plug-in components includes a PTZ camera foveating plug-in component. [0016] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the plurality of plug-in components includes a stereo matching plug-in component.

[0017] In accordance with yet a further aspect of the present invention a surveillance system is provided, comprising a sensor planning toolkit that can be used to determine an architecture of the system.

[0018] In accordance with yet a further aspect of the present invention a surveillance system is provided, comprising a calibration toolkit that calibrates the cameras.

[0019] In accordance with yet a further aspect of the present invention a surveillance system is provided, comprising a synchronized high resolution video capturing engine connected to the processor.

[0020] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the panoramic camera comprises three cameras arranged to provide a panoramic view of the area.

[0021] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the panoramic camera comprises four cameras arranged to provide a panoramic view of the area.

[0022] In accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the panoramic camera comprises three cameras arranged to provide a panoramic view of the area.

DESCRIPTION OF THE DRAWINGS

[0023] FIG. 1 is an illustration of a surveillance development system in accordance with an aspect of the present invention;

[0024] FIG. 2 is an illustration of thread pool performance in accordance with an aspect of the present invention;

[0025] FIG. 3 illustrates camera coverage in accordance with an aspect of the present invention;

[0026] FIG. 4 illustrates person height as a function of distance to a camera in accordance with an aspect of the present invention;

[0027] FIG. 5 illustrates stereo matching and triangulation performed in accordance with an aspect of the present invention; [0028] FIG. 6 illustrates image stitching performed in accordance with an aspect of the present invention;

[0029] FIGS. 7 and 8 illustrate a processor containing system enabled to execute instructions to perform methods in accordance with one or more aspects of the present invention.

DESCRIPTION

[0030] Many existing commercial outdoor surveillance systems are insufficient for high demand applications where area coverage, tracking capability and identification accuracy far exceed traditional surveillance systems. Such applications include portal-less monitoring of pathways leading toward critical infrastructures, airport security, and outdoor large crowd events. Existing commercial systems are lacking in the following senses:

[0031] 1. Low resolution and/or limited field of view (FOV). A standard VGA resolution (640x480) camera can either cover a large FOV with very low resolution on a specific person, or zoom in on a specific person (thus with high resolution on a person) but with very narrow FOV.

[0032] 2. Unsynchronized. For tracking fast moving objects, such as running people or a car, accurate synchronization among different cameras is crucial. Existing commercial surveillance systems usually lack such capability.

[0033] 3. Not easily reconfigurable. Existing systems usually compose of customized cameras with a few resolution selections. Augmenting these systems with different types of sensors (heterogeneous sensors) would require reprogramming of the whole system.

[0034] 4. Limited computing resource. Most applications are designed for last or previous generation of computers. They are unlikely to handle high computational demands, such as tracking everyone in a moderate crowd.

[0035] Methods and systems are provided herein in accordance with one or more aspects of the present invention that are part of a high performance and reconfigurable surveillance platform for surveillance and preferably outdoor surveillance. The above limitations of many existing systems are addressed herein in the following ways:

[0036] 1. In accordance with an aspect of the present invention a system incorporates multiple high-resolution (1024x768) color cameras, to ensure sufficient resolution on a target of interest. For extended field of view, individual video streams are stitched into a single panoramic video. In addition, a pan/tilt/zoom (PTZ) camera is accurately registered to the panoramic view, and an even higher resolution of a target of interest can be acquired using the PTZ camera using optical zoom.

[0037] 2. In accordance with a further aspect of the present invention, the cameras are synchronized with high accuracy in order to capture temporally registered video streams.

[0038] 3. In accordance with a further aspect of the present invention the system configuration, including camera type, desired resolution, et al, sensor type and image processing functionality and processing priority can be defined using a XML file without changing the code.

[0039] 4. In accordance with a further aspect of the present invention a software platform is built to provide multi-threaded applications through platform supported or developer specified thread pools. This architecture frees an algorithm developer from the limitation of using a single CPU. Highly parallel, multiple thread algorithms can be easily developed using multi-core computers (it multiple thread algorithms can be easily developed using multi-core computers (it is predicted that within the next few years, 8-core or 16-core computers will become the standard of commercial of the shelf personal computers.)

[0040] The system has been implemented in at least one embodiment and has been tested.

[0041] The high performance and reconfigurable outdoor surveillance system provided in accordance with one or more aspects of the present invention includes a software development platform. This system in a different mode of operation may be an operational system. In another embodiment the system has as a counterpart an operational system with configured software modules that match software modules that have been developed and/or tested on a development system of FIG. 1. The system of FIG. 1 can also change function, i.e. it can be used for development and then serve in an operational mode.

[0042] The software development platform creates, loads or saves a file which may be an XML file for configuration of the software/hardware modules. It also 1) thread pools for parallel computing; 2) video-driven processing pipeline for various applications (image processing or video analytics). A system overview is shown in FIG. 1.

[0043] The following provides a description of several of the components: a Software Platform, sensor planning sub-system, a calibration sub-system, a real-time video stitching subsystem, a stereo-matching sub-system, a video stabilization sub-system, a video recording sub system, a display sub system and a PTZ camera sub system. Other sub systems are possible, including a configuration sub system.

[0044] Software Platform

[0045] The software architecture is designed to accommodate the system's functionality, performance and development goals. To achieve these goals, a framework and a system are provided as an aspect of the present invention in which stages of processing (multi-camera capturing and video analytics) are performed in separate object-oriented modules.

[0046] The main application refers to the interface definitions of these modules, and the implementations of them are isolated from the main application. This approach allows simultaneous development of those modules, for instance on different systems, as well as stability from their implementation changes. For instance, one module is a camera calibration toolkit. A complete system may be implemented or designed on a development system. The development system is programmed so that it is aware and will use the camera calibration toolkit. In one embodiment, the camera may provide identifying or operational data via the interface to a configuration sub system. Such data may also be entered into the system by a system developer. Such data may include information on which port a camera interface is accessible and a location or a name of a file that contains for instance camera control data.

[0047] A developer in one embodiment of the present invention programs the system, for instance in a control section, as to how and in what order a processor will or can access the appropriate modules and how data from attached hardware and/or external communication ports is accessed. A processor has to be made aware what components are available and which modules can be appropriately used. For instance, a processor has to know that there are three cameras, before it instructs the camera calibration toolkit to calibrate the set of three cameras.

[0048] In a further embodiment components are enabled to provide identifying data. This may be helpful in some situations wherein a system can automatically configure at least some aspects. However, it may be that more than three cameras are attached to the surveillance system and that the system has several operational modules of which some will not use all available cameras in an operational situation. In such a case, a configuration that is manually provided by a developer may need to be entered in a configuration file to instruct a processor which module to use and/or which sensor to use. Not using all modules or all data streams may affect a performance of the system, for instance by increasing the performance. [0049] Once the development system has been completed, all relevant configuration data in one embodiment is stored in a file, such as an XML file. It is noted that a configuration file in one embodiment helps a control unit to administer components and software modules. It may be that the operational system has a defined number of cameras, but it is not yet known what an optimal positioning and a calibration of the cameras is. A configuration file in such a situation may not contain all data for a system to be operational. An operational system may have to go first through a positioning/calibration routine to generate its required settings using the configuration data from the configuration file.

[0050] In a further embodiment operational setting may also be included in a file developed on a development system and transferred to an operational system. Such a file may be a configuration file. For instance, an operator of an operational system may be unwilling or unable to run on-site positioning and or calibration software or other set-up software. It may be beneficial, either by off-site experimentation or by off-site simulation to develop an optimum positioning and/or configuration of a number of cameras and/or of the camera system. In such a case, the configuration file is developed off-site and contains positioning and other data that has been developed not on the operational site and is loaded into the configuration file. An operational system, in such an embodiment, will receive a configuration file that contains module and component data and also set-up data that has not been determined on-site.

[0051] In one embodiment of the present invention development and/or delivery of a configuration file and/or of the framework program that controls software modules of a composite camera/sensor surveillance system is provided to a user of a surveillance system. It is known that a surveillance camera can be selected from a plurality of manufacturers and that a camera can be selected from a wide choice of specifications. It is also known that selection and positioning of sensors and cameras may be critical for specific surveillance tasks and that in many cases selection and positioning is not performed optimally, leading to a sub-optimal performance of a surveillance task. A system and methods for multi-sensor planning are disclosed in US Patent Application Ser. No. 11/636,955 to Mittal filed on Dec. 1 1, 2006 which is incorporated herein by reference. By using maps and geometric and geographic data of the operating environment an optimal positioning of cameras and/or sensors can be achieved in the context of a specific surveillance task. [0052] US Patent Ser. No. 7,899,209 to Greiffenhagen et al. issued on March 1, 201 1 and filed on April 22, 2005 which is incorporated herein by reference, discloses selecting a module that is optimal for a specific surveillance task for a plurality of cameras.

[0053] Based on the knowledge of the required surveillance task, the geometry of the surveillance area and a required performance measure a selection is made of a number of surveillance cameras, their minimum specifications, the positioning of the cameras in the surveillance environment and the required software modules to perform the surveillance tasks with the minimum performance level. The determined preferred configuration of the surveillance system is written into a configuration file which can be an XML file. Such a configuration file in one embodiment also includes data related to a geographical position of a camera. The configuration file in a further embodiment is uploaded into an operational system.

[0054] A configuration file can be developed or created on one system (for instance a development system) and installed and used on a second system (an operational system). In one embodiment of the present invention a configuration file is a part of a control application of a surveillance system. In a first embodiment a control module applies a configuration file to configure the application as to what components are available and which modules at which time or in parallel have to be applied. The configured control application can be saved and is uploaded into memory for execution during start-up of the application. This means that once the application has been configured using the configuration file, the configuration file is no longer required to be present during future start-ups if the properly configured control application has been saved on a storage medium. In a second embodiment a configuration file becomes part of a control application. The control application is programmed to refer to the configuration file at least during start-up of the application. This requires that the configuration file is present at least during start-up of the control application.

[0055] Video data from multiple cameras are generated in a tremendous amount, and the later processing is also computing intensive. The framework and system provided herein ensures high performance for the real-time requirement. Real-time herein means in one embodiment of the present invention that a processing of previous video frames is completed before a new video frame has to be processed. In a further embodiment of the present invention the processing takes place in a substantially real-time manner. This means that displaying a result of image processing may take more than time elapsed between the last received image frame and the reception of a new image frame, but shorter than an elapse of more than 10 image frames in one embodiment of the present invention; or shorter than an elapse of more than 5 image frames in another embodiment of the present invention. For many viewers, such an elapse is too short to be noticed in video images and a processing result will appear to be substantially real-time.

[0056] A surveillance system as provided as an aspect of the present invention has at least a "virtual" panoramic camera (or a plurality of cameras of which images are stitched to form a panoramic camera), a stereoscopic camera, a PZT camera, at least two image processing modules for image processing of the camera image data or at least 4 image processing modules for image processing of the camera data in a substantially concurrent or real-time manner, a plurality of processors or a plurality of cores in one or more processors to perform real-time or substantially real-time image processing.

[0057] To further improve the performance per real-time requirement, the framework and system as an aspect of the present invention also features a thread pool, which maximally exploits multi-core computer architecture. The thread pool is flexibly configurable to use all the available CPU cores, such parallelism in the application and modules are properly taken advantage of. FIG. 2 illustrates the performance of the thread pool on a dual Quad-core PC for thread pool performance on an 8-core processor with a maximum of 24 jobs in timing tests for parallel computing using multi-thread computation.

[0058] In an embodiment of the current invention, the processors and/or cores in a system are under control of a single operating system or management functions that control the system. In a further embodiment of the present invention, the system provides a scheduling of modules to be executed, which may include a priority in assigning threads.

[0059] For instance, a system in an update is provided with an image stabilization module for a camera that is fixed to for instance a pole that is subject to strong winds. Image data from that camera has to be stabilized before it can further be processed, which requires a priority.

Such priority may be determined by a managing operating system, including an assignment to a thread or a core. Such priority may also be part of a configuration file, such as the above mentioned XML file.

[0060] Sensor planning toolkit [0061] A computational toolkit is provided for planning sensor placement. The inputs of the tool include parameters such as the camera type (chip size), focal length, camera placement (height, location, looking direction, et al.), and environment map. The outputs include coverage of the a typical pedestrian (region within which foot of a pedestrian is visible (foot plane), or the region within which head of a pedestrian is visible (head plane)), as illustrated in FIG. 3 which illustrates camera coverage for one specific camera design; person height as a function of focal length (converted to horizontal field of view (HFOV)) and image resolution, as shown in FIG. 4, which illustrates person height as a function of distance to camera.

[0062] This tool helps design the cameras and lenses to be used in a specific application such that the coverage and resolution requirements are satisfied before actual hardware procurement.

[0063] Camera Calibration Toolkit

[0064] Camera calibration is conducted by a module provided in accordance with an aspect of the present invention that instructs a processor to perform instructions including the following three steps:

[0065] 1. Internal camera calibration, to accurately estimate parameters such as focal length, camera projection center et al. This needs to be done for each camera individually.

[0066] 2. Relative external camera calibration. The geometric relationships between any pair of cameras need to be estimated, including relative rotation and translation with respect to each other.

[0067] 3. Absolute external camera calibration. The geometry of the camera rig relative to a global coordinate system to relate image projections to a pedestrian walking in the scene.

[0068] A camera calibration system based on Siemens 3D geometry technologies is implemented. Calibration errors where achieved (in terms of projection errors) between 0.4 pixels to 1.5 pixels for internal calibration (image sizes were 1024x768).

[0069] The relative and external calibration tools were also developed, involving human interaction (the most robust way tested). The final system projection error was between 0.5- 2.5 pixels.

[0070] The calibration toolkit has been designed for convenient deployment of multi- camera systems in outdoor environments. [0071] A second function of the camera calibration toolkit is to provide geometric mappings between any two cameras, with respect to 3D geometry of the environment. For example, a homography can be derived between two panoramic cameras because they share the same view point. This is important for 1) stitching panoramic views; 2) relating a 3D object to an image; 3) and target handoff between two cameras. In a further embodiment of the present invention a panoramic image is created from images from at least three cameras. In a further embodiment of the present invention a panoramic image is created from images from at least three cameras.

[0072] Stereo Matching

[0073] Stereo matching is important for accurately localizing a target in a geo-registered coordinate system. Using the camera calibration toolkit (as a geometry toolkit), a normalized cross correlation based approach for stereo matching was created. Triangulation of the matched points then provides 3D localization of the target. FIG. 5 illustrates stereo matching and triangulation to localize a pedestrian. The Z plane is the ground plane and the location of the chest of the pedestrian is reported in meters.

[0074] PTZ Camera Foveating

[0075] A PTZ camera, e.g. a SONY SNC-RZ30N camera, has multiple degrees of freedom. As an end user, it may not be clear how the internal camera motions are implemented while the Pan, Tilt and Zoom are controlled. Therefore, the exact mathematical form of the projection from the 3D world location to the 2D image location would be unknown. Also, the different brand of PTZ camera has its own way of the PTZ motion implementation. Substantial effort would be needed to find out the mathematical form. In accordance with an aspect of the present invention, a practical way to quickly calibrate the PTZ camera is provided.

[0076] Calibration steps include:

• Grid the road surface of the 3D world location;

• Generate the mapping between the PTZ's pan and tilt parameters and the road surface grid by manually controlling the PTZ camera to the grid;

• For any new 3D world location on the road surface:

-find out the minimal triangular that contains the new location;

-find out the weight parameter to the 3 vertex of the triangular; -using the same weight to the PTZ parameters; and

-the weighted sum would be the approximated the responding PTZ parameter.

[0077] When the PTZ camera is very close to other cameras, once a person's foot position is detected, it can be approximated where the center of the person's body (or other parts of the body of interest in) will be projected to the ground. The projected position will be used as the new 3D world location. The approximated PTZ parameter of the location will be used by the PTZ camera. If the PTZ camera is far away from other cameras, the use of another marker off the ground to do the calibration is needed.

[0078] If the mapping is not accurate enough for a particular application, the grid is made finer to increase the calibration accuracy.

[0079] Detachable Display

[0080] With multiple synchronized cameras running at high frame rate (30 frames/second), and additional IP PTZ camera running at about at least lOfps, the system needs to process a high volume of data in limited time. In such a high-throughput system, the display of the videos also poses a challenge. Even more challenging, one of the working scenarios with a very slow GUI involves a remote desktop connection, where the host computer processes the data and the remote computer displays the video and processing results. To prevent the display of video and processing results from dragging down the system speed, a strategy in implemented called detached display. Basically, the processing runs in a background thread and the GUI and display runs in another thread. The processing thread does not wait for the GUI to display full data; instead, it runs at its own pace for most timely processing. When the video data is ready, it notifies the GUI to fetch the video and then continues to its further processing stages. When it finishes all processing stages, it checks if the GUI thread fetched the video data. If the GUI thread took the video, it then sends the processing results to the GUI thread, and the GUI displays everything. If the GUI thread did not fetch the video data, which means the GUI is slow, the processing thread cancels the video transfer and does not send the processing results to it. The GUI essentially skips this frame. In the end, the processing and GUI threads run at different frame rate. On fast computers with a fast graphics card, they both run at full frame rate. On computers with slow graphics card, or with remote desktop, the GUI runs at a lower frame rate, but the processing thread still runs at full frame rate to provide timely threat detection. This module may be called display management. [0081] Video Stabilization

[0082] In outdoor environment, a camera system can be shaky due to effects such as wind blowing or vibration of the mounting structure (e.g., on a vehicle-borne surveillance platform). For video analytics algorithm, such as background subtraction, the induced image plane motion can be devastating. To cancel these effects, a video stabilization subsystem is required. As an aspect of the present invention and as part of the system provided herein, a software stabilization module is provided, using a sum of squared difference measure as metric and a quasi-Newton optimization algorithm was implemented as a video stabilization plug-in.

[0083] Panoramic View Stitching

[0084] The purpose of this module is the provision of tools for stitching and blending several video feeds into a common panoramic view for visualization (wide field of view situational awareness) and subsequent processing. One challenge is to ensure real-time capability while being flexible enough to accommodate a variety of projection models for the generation of the panoramic view.

[0085] Two approaches have been included to address the above:

1. Separation of the projective model into a pre-processing step and an on-line look-up process, and

2. Multi-threaded computation for the actual stitching and blending process.

[0086] FIG. 6 illustrates a video frame stitched in accordance with an aspect of the present invention.

[0087] Configuration file

[0088] It has been discussed above how a configuration file is developed. This section describes how in an embodiment of the present invention the configuration file can be used. In one embodiment of the present invention the configuration file is stored on a storage medium that is part of an operational surveillance system. Such a system is illustrated in FIG. 1 wherein the software/processor module assumes the role of Control and Management. The system has a control and management module that manages at least part of the modules or data generated by at least part of the "plug-in" modules and makes sure that all modules receive their required data. A "plug-in" module may run on its own processor or is assigned a thread in a multi-core processor. Furthermore, a module may need to be informed how many cameras and/or sensors there are, the availability of an output channel, including a channel to a display, positioning of the cameras and certain properties of the cameras, the availability of a display and characteristics of a graphics card to display surveillance data, for instance in an image overlay.

[0089] In one embodiment at a start-up situation of an operational surveillance system, a control module accesses the configuration file and reads which equipment and modules are supposed to be present. It then checks the operational system for a presence of required hardware and modules. In a further embodiment of the present invention the control module generates an error message or an alert to install or enable the required hardware or software.

[0090] In one illustrative embodiment, a moving object is identified in a specific location and needs to be tracked and potentially recognized moving outside the field of vision of a first camera. A first "plug-in" module is tracking the object and forecasts a possible next location of that object outside the field-of-vision of that object. A second "plug-in" unit has access to configuration data and determines that a second camera will cover the new expected location of the moving object. The module may also provide the required PZT settings for the second camera.

[0091] The Control and Management module receives the requests for data (for instance a request for a camera to track the object, with expected coordinates). The Control module provides the request to the module that has to address the request (for instance that can determine which camera has the best FOV and what the required PZT settings are related to a required performance level) including required data. The Control unit may provide configuration data as required. However, a module may also provided with its own copy of a configuration file, if data transfer bottlenecks need to be avoided.

[0092] A control and management module may be a program that is specifically developed for one or more surveillance tasks, such a tracking and tracing of one or more objects using stitched or 3D images and calculating specific numerical characteristics of an object.

[0093] A control and management module may be a generic program that for instance polls interfaces between plug-in modules and transports data between the plug-ins. The plug-ins may be highly autonomous and thanks to the configuration file know which of the other plug- ins may have required data or need required data. For instance a video-analytics plug-in has analyzed image data from a first camera and needs a new setting for a second camera to continue tracking of the object with a pre-determined measure of quality. The control and management module takes care of appropriate transfer of data between the modules. One result may be a change of PZT settings of the second camera, which will be instructed via the Control and Management module. Data required for the Control and Management module to connect the modules and to control the hardware are that are configuration dependent are contained in the configuration file.

[0094] In one embodiment of the present invention an operational system may have more cameras and/or cameras installed or may contain more "plug- in" modules than are administered in the configuration file. In a further embodiment of the present invention the operational system ignores equipment and/or modules that are not in the configuration file. In yet a further embodiment of the present invention the operational system can be placed in a development mode to administer not yet administered hardware or modules into the configuration file. The inclusion of a configuration file also allows an operation surveillance system to be updated in an effective manner. For instance, required increased performance of an operational surveillance system may require the replacement of a low resolution (for instance 640x480) grey-scale camera by an at least high-resolution (1024x768) color camera. One can thus develop the new system on a development system as illustrated in FIG. 1, by connecting/replacing the camera, generating a new configuration file and updating the control module and if required one or more specific plug-in modules and update the configuration file and relevant modules in the operational surveillance system.

[0095] Development system and operational system may be the same system. They may also be different systems, wherein an operational system does not have to be interrupted to do development work. Furthermore, the modular approach and using a configuration file allow development of different surveillance processes based on common modules and hardware and an easy upgrade of surveillance systems working under similar or related conditions.

[0096] All necessary interfaces of the modules and interfaces were defined, implemented and tested the corresponding modules.

[0097] Synchronized High Resolution Video Recording

[0098] A video recording systems are known. In the following, a hard disk based digital video recording method that can record multiple synchronized video channels is described. [0099] A digital video recorder (DVR) is a device that records video in a digital format to a disk drive, USB drive or other mass storage device. Although a traditional DVR could record multiple channel videos at the same time, the video data are not synchronized.

[0100] In some applications it is important that the videos from multiple cameras are synchronized before we record or process the videos. Synchronizing frames from multiple cameras allows to establishing the time correspondence among the cameras. With strict time correspondence, one can have a reliable base for spatial correspondence among the images from different cameras. Spatial correspondence is essential for a lot of multi-camera vision applications, for example, stereo matching, scene awareness and surveillance. In these applications, if synchronization is inaccurate, the performance of detection or recognition can be significantly marred.

[0101] The recording system as described provides a solution of synchronized multiple channel digital video recorder. It has a synchronization module which can synchronize for instance up to 8 channels at the same time. The maximum time difference among synchronized video can be less than 2 ms (assuming the video frame rate is 30 frames per second). Any part or all of the synchronized video can be played back at the same time.

[0102] The hardware features include:

• Single or multiple video inputs. Based on the hard-disk's limitation, the number of the inputs are up to 8;

• Network connections to control features of the recorder; and

• Connections to external storage media.

[0103] The software features include:

• Synchronization multiple video channels (up to 8 channels);

• User selectable video synchronization resolution;

• User selectable image capture rates (up to 30 frame per second);

• User selectable image resolution (up to 1024 by 768 per channel);

• User selectable compression methods determine quality of playback;

• Input, time and date stamping; and

• Archival.

[0104] Compared to the existing DVRs, one different component in the SMCDVR is the synchronization module, the multi-camera engine. The multi-camera engine in the system is responsible for camera synchronization. The multiple cameras in the system are organized into boards.

[0105] A board is a logical unit that contains one or more cameras, which can be synchronized with reasonably small effort. The board captures a synchronized frame set from its cameras when it receives a capture request. Normally, there is only one requirement when the capturing frames from the cameras onboard: the time difference tolerance. A board may reflect a physical device that offers hardware based synchronization for the cameras onboard. Depending on the nature of the board, synchronizing frames from its cameras can be very simple or more than trivial.

[0106] A multi-camera engine contains one or more boards. When the engine receives a capture request, it forward the request to each board, then inspect the timestamps of the frame sets received from the boards. If they are not synchronized, the older frame sets are discarded and new frame sets are captured from corresponding boards.

[0107] A capture request to the engine can, in accordance with an aspect of the invention, have 3 requirements as follows . (1) Intra-frameset difference. This is the difference of timestamps of the maximum and minimum of the frames from all cameras. This shows the range of the frames distributed in time. The ideal case is 0. (2) Pre-request window. This dictates that the frames must be no older than a certain time from when the capture request is received.(3) Post-request window. This dictates that the frames must be captured within a limited time from when the capture request is received.

[0108] In one embodiment of the present invention, the engine checks if all the three requirements are met for the framesets from the board. It keeps capturing from the boards and checks the received framesets, as follows:

[0109] (1) If any of the boards is stopped (due to software or hardware failure, such as video signal loss and so on), an error is raised;

[0110] (2) If a frameset goes beyond the post-request window, the capture has been carried on for too long, and an error is raised;

[0111] (3) If a frameset is beyond pre-request window, recapture from the board;

[0112] (4) If the intra-frameset difference is too big, recapture from the board that gave the oldest framesets;

[0113] (5) Otherwise, return the captured framesets to the caller. [0114] The framesets returned from the multi-camera engine is guaranteed to be within the specified requirements. If a post-request window violation error is found, the application has the option to re-issue a capture request or escalate the error for user attention and other further actions.

[0115] Systems as described herein are further illustrated in FIG. 7. A system contains a processor 2002 to execute instructions that perform the methods provided in accordance with aspects of the present invention. A processor 2002 can be a single core processor, or a multi- core processor or a plurality of processors, each of which can be a single or multi-core processor. The surveillance system has at least one camera 2000 connected to the processor 2002 wherein the processor is enabled to process image data. In addition to the at least one camera other cameras can be connected to the processor as well as additional hardware such as sensors such as proximity sensors, radar, or any other sensor that is applied in a surveillance. The processor also has access to instructions to execute which can be stored on a memory or storage device 2008. The processor also has access to a configuration file which is available from a storage or memory device 2006. The processor can display an image, including a processed surveillance image on a display 2004.

[0116] The methods as provided herein are, in one embodiment of the present invention, implemented on a system or a computer device. A system illustrated in FIG. 8 and as provided herein is enabled for receiving, processing and generating data. The system is provided with data that can be stored on a memory 1801. Data may be obtained from a sensor such as a camera or may be provided from a data source. Data may be provided on an input 1806. Such data may be image data or positional data, or any other data that is helpful in a surveillance system. The processor is also provided or programmed with an instruction set or program executing the methods of the present invention is stored on a memory 1802 and is provided to the processor 1803 (which can be a multi-processor or a multi-core processor), which executes the instructions of 1802 to process the data from 1801. Data, such as image data or any other data provided by the processor can be outputted on an output device 1804, which may be a display to display data or a data storage device. The processor also has a communication channel 1807 to receive external data from a communication device and to transmit data to an external device, including the configuration file if so desired. The system in one embodiment of the present invention has an input device 1805, which may be or include a sensor, a camera, a keyboard, a mouse, a pointing device or any other device that can generate data to be provided to processor 1803. The processor can be dedicated hardware. However, the processor can also be a CPU or any other computing device that can execute the instructions of 1802. Accordingly, the system as illustrated in FIG. 8 provides a system for data processing as part of a surveillance system resulting from at least a camera or any other data source and is enabled to execute the steps of the methods as provided herein as an aspect of the present invention.

[0117] In accordance with an aspect of the present invention, methods and systems areprovided that provide for a high-performance, reconfigurable outdoor surveillance system that encompasses the following subsystems: A real-time (30 frames per second), high throughput (over 300 Megabytes per second), and accurately synchronized (less than ¼ milliseconds) multiple sensor data acquisition sub-system for outdoor surveillance applications;

[0118] A software platform supporting multi-threaded processing, detachable display, flexible reconfiguration of sensors;

[0119] A sensor planning sub-system that assists accurate placement of cameras;

[0120] A calibration sub-system that relates the cameras to a geo-registered coordinate system;

[0121] A real-time video stitching sub-system for situational awareness in outdoor surveillance systems;

[0122] A stereo-matching sub-system for accurate measurement of the object location in a geo-registered 3D coordinate system;

[0123] A video stabilization sub-system;

[0124] A video recording subsystem; and

[0125] A Pan/Tilt/Zoom camera slaving and fixation sub-system.

[0126] In a further embodiment of the present invention, the high-performance, reconfigurable surveillance system that encompasses the subsystems operates on a multithreading system that has one or more processors or at least one processor with a plurality of cores. [0127] In a further embodiment of the present invention, the high-performance, reconfigurable surveillance system that encompasses subsystems operates in real-time or substantially in real-time.

[0128] In a further embodiment of the present invention, the high-performance, reconfigurable surveillance system that encompasses subsystems is configurable by applying a configuration file which may be an XML file.

[0129] In a further embodiment of the present invention, the high-performance, reconfigurable surveillance system that encompasses subsystems is configurable by applying a configuration file that enables a scheduling of modules and/or threads.

[0130] In summary, in accordance with an aspect of the present invention methods and systems are provided to conduct multi-sensor, high-performance and reconfigurable surveillance which is preferably performed in an outdoor environment. In accordance with an aspect of the present invention, as illustrated in FIG. 1, a surveillance system is provided for monitoring an area, comprising a panoramic camera arranged to view the area, a stereo camera arranged to view the area, a PTZ camera arranged to view the area, a processor having multiple cores that handles multiple threads that is connected the panoramic camera, the stereo camera and the PTZ camera, a plurality of plug- in components connected to the processor, each of the plurality of plug-in components being recognizable by the processor, and a memory readable by the processor, the memory storing a configuration file, wherein the processor communicates with the panoramic camera, the stereo camera, the PTZ camera and the plurality of plug-in components in accordance with the configuration file, processing the image signals from the panoramic camera, the stereo camera, the PTZ camera and the plurality of plug-in components on multiple threads using the multiple cores.

[0131] In accordance with an aspect of the present invention, the plug-in components are recognizable immediately by the processor so that the processor knows the configuration of the system. The recognition feature is easily implemented in well known fashions.

[0132] The configuration file can be a XML file. Also in summary, in accordance with yet a further aspect of the present invention a surveillance system is provided, wherein the configuration file can include operational scheduling information. The configuration file can specify that stabilization be performed prior to image acquisition if stabilization is to be performed, one of the plurality of plug-in components is a video stabilization plug-in device and the processor controls the video stabilization plug-in prior to image acquisition if the configuration file indicates stabilization is to be performed.

[0133] As shown in FIG. 1, the plurality of plug-in components can include any combination of a panoramic video stitching plug-in component, a video recording plug-in component, one or more video analytics plug-in component, a detachable display plug-in component, a PTZ camera foveating plug-in component, a stereo matching plug-in component, a sensor planning toolkit that can be used to determine an architecture of the system and a calibration toolkit.

[0134] As further illustrated in FIG. 1, the system can include a a synchronized high resolution video capturing engine connected to the processor.

[0135] The panoramic camera comprises a plurality of cameras arranged to provide a panoramic view of the area. For example, three, four or five cameras can be used to provide the panoramic camera function.

[0136] While there have been shown, described and pointed out fundamental novel features of the invention as applied to preferred embodiments thereof, it will be understood that various omissions and substitutions and changes in the form and details of the methods and systems illustrated and in its operation may be made by those skilled in the art without departing from the spirit of the invention. It is the intention, therefore, to be limited only as indicated by the scope of the claims.

Claims

1. A surveillance system for monitoring an area, comprising:

a panoramic camera arranged to view the area;

a stereo camera arranged to view the area;

a PTZ camera arranged to view the area;

a processor having multiple cores that handles multiple threads that is connected the panoramic camera, the stereo camera and the PTZ camera;

a plurality of plug-in components connected to the processor, each of the plurality of plug-in components being recognizable by the processor; and

a memory readable by the processor, the memory storing a configuration file;

wherein the processor communicates with the panoramic camera, the stereo camera, the PTZ camera and the plurality of plug- in components in accordance with the configuration file, processing the image signals from the panoramic camera, the stereo camera, the PTZ camera and the plurality of plug-in components on multiple threads using the multiple cores.

2. The surveillance system of claim 1, wherein the configuration file is an XML file.

3. The surveillance system of claim 1, wherein the configuration file includes operational scheduling information.

4. The surveillance system of claim 3, wherein the configuration file specifies that stabilization be performed prior to image acquisition if stabilization is to be performed.

5. The surveillance system of claim 4, wherein one of the plurality of plug-in components is a video stabilization plug-in device and the processor controls the video stabilization plug-in prior to image acquisition if the configuration file indicates stabilization is to be performed.

6. The surveillance system of claim 1 , wherein the plurality of plug-in components includes a panoramic video stitching plug-in component.

7. The surveillance system of claim 1, wherein the plurality of plug-in components includes a video recording plug-in component.

8. The surveillance system of claim 1, wherein the plurality of plug-in components includes a video analytics plug-in component.

9. The surveillance system of claim 1, wherein the plurality of plug-in components includes a detachable display plug-in component.

10. The surveillance system of claim 1, wherein the plurality of plug- in components includes a PTZ camera foveating plug-in component.

11. The surveillance system of claim 1, wherein the plurality of plug-in components includes a stereo matching plug-in component.

12. The surveillance system of claim 1, comprising a sensor planning toolkit that can be used to determine an architecture of the system.

13. The surveillance system of claim 12, comprising a calibration toolkit that calibrates the cameras.

14. The surveillance system of claim 13, comprising a synchronized high resolution video capturing engine connected to the processor.

15. The surveillance system of claim 1, wherein the panoramic camera comprises three cameras arranged to provide a panoramic view of the area.

16. The surveillance system of claim 1, wherein the panoramic camera comprises four cameras arranged to provide a panoramic view of the area.

17. The surveillance system of claim 1, wherein the panoramic camera comprises five cameras arranged to provide a panoramic view of the area.