WO2006109162A2

WO2006109162A2 - Distributed smart video surveillance system

Info

Publication number: WO2006109162A2
Application number: PCT/IB2006/000879
Authority: WO
Inventors: Touradj Ebrahimi; Frederic Albert Dufaux
Original assignee: Emitall Surveillance S.A.
Priority date: 2005-04-15
Filing date: 2006-04-13
Publication date: 2006-10-19
Also published as: WO2006109162A3

Abstract

A distributed smart video surveillance system (20) is disclosed which includes at least one surveillance camera (22,24,26) and a processor, for example, a remote server (27) with storage capability. The captured video data is transmitted to the remote server, where it is analyzed forming a distributed smart video surveillance system. In particular, in intrusion type applications, video data from static surveillance cameras monitoring regions of interest is machine analyzed to determine if any abnormal conditions exist. The abnormal condition is detected simply by detecting a change in a scene. As such, the system in accordance with the present is relatively accurate and minimizes false alarms. If an abnormal condition is detected, a response to the abnormal condition is automatically initiated. The system is configured so that heterogeneous clients (40,42, 44, 46) can access the system in order to view live or recorded video. In one embodiment of the invention, the system is configured to receive commands from a cell phone to start, stop, pause and resume the video surveillance.

Description

DISTRIBUTED SMART VIDEO SURVEILLANCE SYSTEM CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of US Provisional Patent Application No. 60/594,525, filed on April 15, 2005.

BACKGROUND OF THE INVENTION

1. Field of the Invention

[0002] The present invention relates to a surveillance system and more particularly to a surveillance system which includes one or more video surveillance cameras and optionally one or more other types of sensors, such as motion detectors, smoke detectors, temperature detectors and the like for automatically analyzing the video and any other data and initiating one or more responses, such as triggering an alarm, triggering an electromagnetic actuator, for example, a door or window actuator which optionally allows heterogeneous clients to monitor live and archived video surveillance sessions.

2. Description of the Prior Art

[0003] With the increase of terrorist threats in the recent years and the high level of criminality in urban areas, security remains a major public concern. Video surveillance is one approach to address this issue. Large video surveillance systems have been widely deployed for many years in strategic places, such as airports, banks, subways, city centers or private houses. However, some current systems are analog and are based on proprietary solutions. It is expected that the next generation of video surveillance systems will be digital and based on standard technologies and Internet Protocol (IP) networking.

[0004] Another expected evolution is towards smart video surveillance systems. Current systems are restricted to capture, transmit and store video sequences. Examples of such systems are disclosed in US Patent Nos. 5,517,236; 5,844,520; 6,462,774; 6,621,518; 6,747,554 and 6,757,008, hereby incorporated by reference. In a corporate or public environment, a security guard is required by users of such systems to monitor control screens in order to detect unusual or suspect situations and to set off an alarm. Such methodology is known to be unreliable. In addition, while people greatly appreciate the sense of increased security brought by a video surveillance system, user's of such video surveillance systems legitimately fear the loss of privacy which comes along with such surveillance.

[0005] In order to solve this problem, smarter video surveillance systems have been developed. An example of such a smarter video surveillance system is disclosed in US Patent No. 5,666,157, hereby incorporated by reference. In particular, the '157 patent discloses a video surveillance system which includes a video surveillance camera and a system for analyzing the video data and initiating a response. In this system, movements of individuals appearing in a video scene are analyzed by sampling individuals on a frame by frame basis and computing various parameters with respect to an individual's movements, such as speed , velocity and track. The parameters of the incoming video signal are compared with a "signature" of parameters representative of "various positions and movements of the body parts of an individual having various levels of criminal intent". If the computed parameters match a stored signature, various actions are initiated , such as generating an alarm; activating a camera to record an individual's face; broadcasting a recorded message over a loud speaker or turning on lights.

[0006] Unfortunately, there are several problems with the system disclosed in the '157 patent. First, the system requires extremely intensive computation. In particular, an abnormal condition is represented by various parameters of an individual's body parts, such as speed and acceleration. As such, various body parts of all individuals in a frame must be detected. The speed and acceleration of those body parts must be detected and compared with a signature of those parameters. Second, the system is also relatively inaccurate resulting in false alarms. Frequent false alarms are known to cause security and other personnel to ignore all alarms.

[0007] Thus, there is a need for a relatively accurate video surveillance system that can be used to detect an abnormal condition and initiate a response. There is also a need for a system for detecting an abnormal condition that is relatively less computation intensive than known systems.

SUMMARY OF THE INVENTION

[0008] Briefly, the present invention relates to a distributed smart video surveillance system which includes at least one surveillance camera and a processor, for example, at least one remote server with storage capability. The captured video data is transmitted to the remote server, where it is analyzed forming a distributed smart video surveillance system. In particular, in intrusion type applications, video data from static surveillance cameras monitoring regions of interest is machine analyzed to determine if any abnormal conditions exist. The abnormal condition is detected simply by detecting a change in a scene. As such, the system in accordance with the present is relatively accurate and minimizes false alarms. If an abnormal condition is detected, a response to the abnormal condition is automatically initiated. In an optional embodiment of the invention, the system is configured so that heterogeneous clients can access the system in order to view live or recorded video.

DESCRIPTION OF THE DRAWING

[0009] These and other advantages of the present invention will be readily understood with reference to the following specification and attached drawing wherein:

[0010] Fig. IA is an exemplary high level diagram of the distributed smart surveillance system in accordance with the present invention.

[0011] Fig. IB is an exemplary embodiment of distributed smart surveillance system illustrated in Fig. IA, implemented using multiple servers.

[0012] Fig. 1C is similar to Fig. IB, but implemented as a single server.

[0013] Fig. 2 is an exemplary block diagram for the video surveillance cameras in accordance with the present invention.

[0014] Fig. 3 is an alternative exemplary block diagram of the processor for use with the present invention illustrating the use of a single server.

[0015] Fig. 4 is an exemplary block diagram for an authorized client accessing live or archived video data in an application where portions of the video scenes have been scrambled.

[0016] Fig. 5 is similar to Fig. 4 but illustrating unauthorized client access to live or archived video data.

DETAILED DESCRIPTION

[0017] The present invention relates to a video surveillance system that includes at least one video surveillance camera and a processor. In accordance with an important aspect of the invention, the system is configured to machine analyze video scenes to determine if an abnormal condition exists. In accordance with an important aspect of the invention, the abnormal condition is detected simply by detecting a change in a scene. As such, the system in accordance with the present is relatively accurate and minimizes false alarms. In accordance with an important aspect of the invention, if an abnormal condition is detected, an automated response may be initiated. For example, upon detection of an abnormal condition, the system may trigger an alarm; turn on lights; initiate a voice response over a speaker, as well as initiate various types of electro-mechanical actuators to close or open doors and windows in a building or other responses. In accordance with one embodiment of the invention as illustrated in Fig. IA₃ the system may include at least one remote server and at least one relatively low cost video surveillance camera. Alternatively, as illustrated in Fig. IB, the system may be configured with three servers, hosted by one or more machines, including a surveillance server, an access server and a gateway server. The surveillance server is used to receive and process signals from one or more surveillance cameras. The access server may be used to allow heterogeneous clients to monitor live and archived video surveillance sessions forming a distributed smart video surveillance system. The gateway server may be used to enable access to live and archived video surveillance sessions byway of a cellular telephone.

[0018] A high level diagram of a distributed smart video surveillance system in accordance with the present invention is illustrated in Fig. IA and generally identified with the reference numeral 20. The system 20 includes a number of video surveillance cameras 22, 24 and 26 and optionally a central server 27. The system 20 may optionally include other types of sensors, such as temperature detectors 28, smoke detectors 30, motion detectors 32 and the like, positioned so as to cover an area to be monitored, and configured to communicate with the central server 27, for example, by way of a wired or wireless communication link 29, which may optionally be bi-directional. The video data from the surveillance cameras 22, 24 and 26 is encoded and the resulting bitstream is transmitted over either be wired or wireless communication link 33, which may optionally be bi-directional. Wireless networks are especially appealing for the video surveillance cameras 22, 24 and 26 as it makes it very easy to deploy and relocate video surveillance cameras 22, 24 and 26 as the surveillance needs evolve.

[0019] The system may be configured with one server 82, as illustrated in Fig. 1C or alternatively, more than more than one server, hosted by one or more machines, as illustrated in Fig. IB. In particular, as illustrated in Fig. IB, the system, generally identified with the reference numeral 80, may include a surveillance server 82, a gateway server 84 and an access server 86. As mentioned above, the surveillance server 82 is used for receiving signals from one or more surveillance cameras. For example, as illustrated in Fig. IB, one or more wireless surveillance cameras 88, 90 can be connected to the surveillance server 82 by way of a wireless access point. Hard wired surveillance cameras 94, 96 can also be connected to the surveillance server 82. Access to the system 80 may be over a public communication network, such as the Internet, or other network, by one or more heterogeneous clients by way of the access server 86 forming a distributed smart video surveillance system. As shown in Fig. IB, the heterogeneous clients may include a hard-wired desktop computer 98 as well as wireless clients, such as a wireless laptop computer 100 or PDA 102 connected to the communication network by way of a wireless access point 104. Optionally, a gateway server 84 may be provided to provide access to the system over the communication network by way of a cellular phone 106. In this way, commands to start, stop, pause or resume surveillance can be communicated to the system 80 by cell phone 106. [0020] In addition to the cameras 88, 90, 92 and 94, other types of sensors may be connected to the surveillance server 82. For example, a wireless microphone 108, a wireless motion detector 110 and a wireless motion detector 112 can be connected to the surveillance server 82 by way of the wireless access point 92. Similarly, a hard- wired microphone 120, motion detector 122 and a smoke detector 124 can be directly connected to the surveillance server 82.

[0021] In addition to surveillance, the system 80 may be configured to provide an automated response. For example, the surveillance server 82 may be configured to actuate one or more remote devices in response to detection of an abnormal condition. For example, the system 80 may be configured to provide wireless control by way of the wireless access point 92 to one or more remote devices, such as: a spotlight 114, loud speaker 116 and an electric motor 118, used as a door or window actuator. Similarly, the system 80 maybe configured so as to enable hard-wire control of a spotlight 126, a loudspeaker 128 and an electric motor 130. Control of various devices , such as spot lights and appliances which include electric motors, by way of a PC is well known in the art., for example, as disclosed in http://www.x 10.com/automation/computer kits.html ; http://www.smarthome.com/prodindex.asp?catid=2 , http://www.smarthome.com/2814ub.html.

[0022] Control of the loudspeaker 116 is also within the ordinary skill in the art. For example, a prerecorded message is stored on a PC for example in a .WAV file on MS Windows. In the case of an alert , a simple software command is used to play back the .WAV file, in a conventional manner, for example, as disclosed at http ://msdn.microsoft. com/archive/default. asp?url=/archive/en- us/directx9 m Apr 2005/directx/audiovideoplayback/audiovideoplavback.asp.

[0023] In accordance with an important aspect of the invention, the system 20, 80 is configured to machine analyze video data and detect abnormal conditions and initiate a response. As such, one or more devices, such as spot lights 34, 114, 126; loud speakers 36, 116, 128; and electro-mechanical actuators 38, 118, 130 may be coupled to the server 27, 82 by way of a wired or wireless communication link, which optionally may be bidirectional. When an abnormal condition is detected, the system 20,80 may automatically initiates a response, such as triggering an alarm; turning on spotlights 34, 114, 126; initiating a voice or other aural response over a loud speaker 36, 116, 128; or initiating an electro-mechanical actuator 38, 118, 130 to open or close a door or window or triggering an electric door lock.

[0024] The system 20, 80 may optionally be configured to additionally enable monitoring of live or archived video data by way of various homogeneous clients, such as, a desk top personal computer (PC) 40, 98; a personal digital assistant 42, 102; a lap top PC 44, 100; or a cell phone 46, 106. The homogeneous clients 40, 42, 44 ,46, 98, 100, 102, 106 are coupled to the server 27, 82 over a bi-directional communication link 48. As the bitstream is scalable, the central server 27 or surveillance server 82 can adapt the resolution and bandwidth of the delivered video depending on the performance and characteristics of the client and its network connection by way of a wired or wireless network so that mobile clients can access the system. For instance, policemen or security guards can be equipped with laptops or PDAs while on patrol. The system can also be configured so that home owners, or others, are automatically an SMS or MMS messages in the event an abnormal condition, such as an intrusion is detected. An example of such a system is disclosed in US Patent No. 6,698,021, hereby incorporated by reference.

[0025] One important aspect of the invention is that the video processing may be carried out by the central server 27 or surveillance server 82 or alternatively integrated with the cameras 22, 24 and 26, while providing access to the system 20, 82, by one or more heterogeneous clients 40, 42, 44, 46, 98, 100, 102, 106, thus forming a distributed smart video surveillance system. The distributed architecture is more flexible and cost- effective than known systems. More particularly, in such an application, as will be discussed in more detail below, the central server 27 and the surveillance server 82 receive bitstreams from the various video surveillance cameras 22, 24,, 26, 88, 90, 92 94. Each bitstream is processed by the central server 27 or surveillance server 82 in order to analyze the video content and identify region of interests in order to detect abnormal conditions and initiate an appropriate response, e.g. trigger an alarm, take countermeasures, or send SMS or MMS messages. The central server 27 and surveillance server also store the video data.

[0026] The smart distributed video surveillance system 20, 80 may be based on standard technologies. For example, various communication protocols are suitable for the various communication links 29, 33, 35 and 48. For example, the various communication links can utilize Internet Protocol (IP). In addition, the JPEG 2000 standard can be used to encode the video data in the camera. Because of the high coding efficiency of the JPEG 2000 standard, good image quality is achieved. Furthermore, the JPEG 2000 codestream supports seamless and efficient scalability, both in resolution and quality. In order to secure the surveillance data, the JPSEC standard may be used. The JPSEC secure encoding can also be used in order to protect the integrity of a region of interest or the whole scene captured. Finally, the JPWL encoding standard can be used in order to make the codestream more robust to transmission errors in the case of error-prone network connections. In addition, regions corresponding to people and goods in the scene can be scrambled in order to preserve the anonymity of under surveillance, for example, as described in International Patent Application under the Patent Cooperation Treaty, Application No. PCT/DB05/002989, filed on July 7, 2005, hereby incorporated by reference.

PROCESS LEVEL SYSTEM DESCRIPTION: CAMERA [0027] In known surveillance systems, so-called smart video cameras are used. An exemplary smart video camera is a Hawking Model No. HNC320W/NC300. These smart video surveillance cameras function as camera servers and have substantial processing power. As such, such smart video surveillance cameras are relatively are higher in cost and have higher power consumption than other known video surveillance cameras, as it performs sophisticated processing (analysis, compression, digital signature, etc.), and communication tasks. In accordance with an important aspect of the invention, the system 20, 80 distributes the computation burden to the server 27,82, which allows lower cost video surveillance cameras to be used while providing a modular and cost-effective architecture.

[0028] As a result, the cameras remain simple. It is sufficient to carry the processes illustrated in Figure 2. A simplified flow chart for a smart surveillance camera in accordance with the present invention is illustrated in Fig. 2. Video content is acquired in step 50 by a capture device, such as a surveillance camera 22, 24, 26, 88, 90, 94, 96, which may include a camera, connected to PC or processor by way of a USB port. The PC may be coupled in a wired or wireless network, such as a WiFi (also known as IEEE 822.11) network.

[0029] The scene is first acquired by a capture device, for example, a visible spectrum, near-infrared or infrared camera. The near infrared and infrared cameras allow for night operations without additional lighting. In accordance with one aspect of the invention, the camera 22, 24, 26, 88, 90, 94, 96, may be a relatively low cost conventional web cam, for example, a Quick Cam Pro 4000, as manufactured by Logitech. The PC may be a standard laptop PC with a 2.4 GHz Pentium processor. Such conventional web cams come with standard software for capturing and storing video content on a frame by frame basis.

[0030] The video may be encoded (i.e. compressed) in step 52 at the camera, for example, using the JPEG 2000 standard. However, other compression techniques can also be used, such as JPEG, MPEG, or any other proprietary compression. This allows for the use of relatively low cost off-the-shelf cameras with a lower encoding complexity.

[0031] In step 54, the compressed bitstream is transmitted to the central server 27, for example, over the communication link 33 (Fig. 1), for example, using IP protocol. As mentioned above, the communication link 33 can be wired or wireless. Wireless applications are a relatively easy and cost-efficient method to deploy cameras on an on-demand basis. .As is known in the art, a significant part of the cost associated with a video surveillance system is in the deployment and wiring of cameras. In addition, it is often desirable to install a surveillance system in a location for a limited time, for instance, during a special event. However, wireless networks are known to be subject to frequent transmission errors. .As such, the development of wireless imaging solutions robust to transmission errors is important. Accordingly Wireless JPEG 2000 (JPWL) may be used. JPWL is an extension of the baseline JPEG 2000 specification. It defines additional mechanisms to achieve the efficient transmission of JPEG 2000 content over an error-prone network. It has been shown that JPWL tools result in very significant video quality improvement in the presence of errors. In one embodiment of the invention, JPWL tools can therefore be used in order to make the codestream more robust to transmission errors and to improve the overall quality of the system. JPWL is described in detail in Dufaux et al; "JPWL:JPEG 2000 foe Wireless Applications"; Journal of SPIE Proceedings- Applications of Digital Image Processing XXVII,, Denver, Colorado, November 2004, pages 309-318, hereby incorporated by reference.

Process Level System Description: Server

[0032] In accordance with one aspect of the present invention, the central server 27 or surveillance server 82 may be used to carry the burden of processing the video content, and various management, storage and communication tasks. The central server 27 and surveillance server 82 are configured to perform the tasks as illustrated in Fig. 3 and discussed below. By shifting the burden of many of the computations video processes is performed in the server, more computational resources are available. Compressed video content from the cameras 22, 24 and 26 is first decoded, as indicated in step 56. The decoded bitstreams are then analyzed, as indicated in step 58. The video content is analyzed in step 40 to detect the occurrence of events in the scene (e.g. intrusion, presence of people). Jn accordance with an important aspect of the present invention, the detection of an abnormal condition is rather simple and requires the detection of a change in a frame or scene. As such, the system is relatively more reliable and less computation intensive than known smart video surveillance systems. The information about the objects in the scene is then passed on in order to encode the object with better quality or to scramble it, or both. As mentioned above, relying on a human operator monitoring control screens in order to set off an alarm is notoriously inefficient. Therefore, another purpose of the analysis may be to either bring to the attention of the human operator abnormal behaviors or events, or to automatically trigger alarms.

Change Detection

[0033] Various techniques can be used for detecting a change in a video scene. In the system 20, 80, virtually any detection algorithm can be used to detect faces or other information, such as cars, people's silhouettes, skin, license plates, fire, etc. instead or in conjunction with others, as well as the above described method. However, in accordance with an important aspect of the invention, the system assumes that all cameras remain static. In other words, the cameras do not move and are continuously in a static position thereby continuously monitoring the same scene. In order to reduce the complexity of the video analysis in step 58 a simple frame difference algorithm may be used. As such, the background is initially captured and stored. Regions corresponding to changes are merely obtained by taking the pixel by pixel difference between the current video frame and the stored background, and by applying a threshold. For example, the change detection may be determined by simply taking the difference between the current frame and a reference background frame and determining if the difference is greater than a threshold. For each pixel x, a difference D_n (x) = I_n(x) - B(x) is calculated, where I_n(x) is the ra-th image and B(x) is the stored background.

[0034] A change mask M(x) may be generated according to the following decision rule:

0 Otherwise where T is the threshold and M(x) is the pixel in the image being analyzed.

[0035] The threshold may be selected based on the level of illumination of the scene and the automatic gain control and white balance in the camera. The automatic gain control relates to the gain of the sensor while the white balance relates to the definition of white. As the lighting conditions change, the camera may automatically change these settings, which may affect the appearance of the captured images (e.g. they may be lighter or darker), hence adversely affecting the change detection technique. To remedy this, threshold may be adjusted upwardly or downwardly for the desired contrast.

[0036] In order to take into account changes of illumination from scene to scene, the background may be periodically updated. For instance, the background can be updated as a linear combination of the current frame and the previously stored background as set forth below.

if n=iF with i = 1, 2 (F is the period of the update)

B_n = B_n-i otherwise

Where B_n = the current background

B_n-_I = the previous background I_n = the current frame α = a constant

[0037] In order to smooth and to clean up the resulting change detection mask, a morphological filter may be applied. Morphological filters are known in the art and are described in detail in: Salembier et al , "Flat Zones Filtering Connected Operators and Filters by Reconstruction", IEEE Transactions on Image Processing, Vol. 4, No. 8, Aug. 1995, pages 1153-1160, hereby incorporated by reference. In general, morphological filters can be used to clean-up a segmentation mask by removing small segmented regions and by removing small holes in the segmented regions. Morphological operations modify the pixels in an image depending on the neighboring pixels and Boolean operations by performing logical operations on each pixel.

[0038] Two basic morphological operations are dilation and erosion. Most morphological operations are based on these two operations. Dilation is the operation which gradually enlarges the boundaries of regions in other words allows objects to expand, thus potentially filling in small holes and connecting disjoint objects. Erosion operation erodes the boundaries of regions. It allows objects to shrink while the holes within them become larger. The opening operation is the succession of two basic operations, erosion followed by dilation. When applied to a binary image, larger structures remain mostly intact, while small structures like lines or points are eliminated. It eliminates small regions, smaller than the structural element and smoothes regions' boundaries. The closing operation is the succession of two basic operations, dilation followed by erosion. When applied to a binary image, larger structures remain mostly intact, while small gaps between adjacent regions and holes smaller than the structural element are closed, and the regions' boundaries are smoothed.

Face Detection

[0039] The detection of the presence of people in a scene is one of the most relevant bits of information a video surveillance system can convey. In addition to detecting changes in a scene, the system may optionally be configured to detect objects, such as cars, people, license plates, etc. The system in accordance with the present invention may use a face detection technique based on a fast and efficient machine learning technique for object detection, for example, available from the Open Computer Vision Library, available at http://www.Sourceforge.net/projects/opencvlibrary , described in detail in Viola et al, "Rapid Object Detection Using a Boosted Cascade of Simple Features, IEEE Proceedings CVPR. Hawaii, Dec. 2001, pages 511-518 and Lienhart et al "Empirical Analysis of Detection Cascades of Boosted Classifiers for Rapid Object Detection"; MRL Technical Reports, Intel Labs, 2002.

[0040] The information about the objects in the scene is then passed on to the following modules in order to encode them with better quality and to scramble them. Another purpose of the analysis module is to either assist a human operator bringing to his attention abnormal behaviors or events, or automatically trigger alarms and take appropriate countermeasures. Note that information from the video analysis can trigger different processing in the subsequent steps. For instance, regions of interest can be encoded with higher quality, or scrambling can be applied to regions of interest while leaving the remaining of the scene in clear.

Decisions and Actions

[0041] Based on the results from the video analysis, the system may initiate a response. In one embodiment of the invention, a simple threshold on the size of the region resulting from the change detection is used to indicate an abnormal condition, such as an intrusion. When an abnormal condition is detected, the system 20, 80 can initiate a response by actuating one or more security devices, such as an alarm, turn on spotlights 34, 114, 126; broadcast voice a pre-recorded warning message over a loudspeaker 36, 116, 128;or actuate an electromechanical actuator 38, 118,130; to open or close doors and windows, for example. The system may also be used to automatically call the police or a private security service with conventional technology, such as a eDbox-103 manufactured by eDevice (http ://www. edevice.com^') which enables a PC to dial a phone call or to receive a phone call.

[0042] . Alternatively, the system can send an SMS message or an MMS message to the owner of the premises An MMS message is especially appealing and used to provide one or more images from the scene corresponding to the detected intrusion or a short video sequence. SMS, MMS and GMS are well known in the art. See for example http://en.wikipedia.org/wiki/; http://en.wikipedia.org/wiki/Multimedia Messaging Service : and http ://www. gsmworld.com/index .shtml. The owner can view these images on a cell phone and decide whether it is a true or false alarms, and whether to call the police. Several manufactures have GSM/GPRS modems which allow to send SMS/MMS from a PC, or receive SMS/MMS on a PC, for instance "Wavecom FastTrack GPRS module" or "Nokia D211 - http://europe.nokia.eom/nokia/0, 1522,,00.html?orig=/phones/nokiad211/"[O]. The system 20 can also request that the portion of the scene corresponding to an event being stored and archived on the server.

Encoding

[0043] In one embodiment of the invention, the captured video sequence is encoded in the camera 22, 24 26 , 88, 90, 94 96. Alternatively, the video data can be encoded at the central server 27 or surveillance server 82, as indicated in step 62. The video data may be encoded using the JPEG 2000 standard. The JPEG 2000 standard is the latest standard for still image coding. It is well-suited for video surveillance applications for a number of reasons. First, even though it leads to inferior coding performance compared to an inter-frame coding scheme, intra-frame coding allows for easy browsing and random access in the encoded video sequence, requires lower complexity in the encoder, and is more robust to transmission errors in an error-prone network environment. Moreover, JPEG 2000 intra-frame coding outperforms previous intra-frame coding schemes such as JPEG, and achieves a sufficient quality for a video surveillance system. JPEG 2000 also supports region of interest coding which is very useful in our application. Indeed, in video surveillance system, foreground objects are very important, but the background is nearly irrelevant. In our case, the regions detected by the analysis module can be encoded with high quality, while the remaining of the scene is coded with low quality. For instance, the face of a suspect can be encoded with high quality, hence enabling its identification, even though the video sequence is highly compressed. Seamless scalability is another very important feature of JPEG 2000. As it is based on a wavelet transform generating a multi-resolution representation, spatial scalability is immediate in JPEG 2000. As the video sequence is coded in intra-frame, namely each individual frame is independently coded using JPEG 2000, temporal scalability is also straightforward. Finally, the JPEG 2000 codestream can be build with several quality layers optimized for various bit rates. In addition, this functionality is obtained with negligible penalty cost in terms of coding efficiency. The resulting codestream then supports efficient quality scalability. This property of seamless and efficient spatial, temporal and quality scalability is essential when clients with different performance and characteristics have to access the video surveillance system.

[0044] Techniques for encoding digital video content in various compression formats including JPEG 2000 is extremely well known in the art. An example of such a compression technique is disclosed in: Skodras et al; "The JPEG 2000 Still Image Compression Standard"; IEEE Signal Processing Magazine: volume 18, Sept. 2001, pages 36-58, hereby incorporated by reference.

[0045] If the video data coming from the camera 22, 24 and 26 is already compressed, transcoding can be used in the central server 27 or the surveillance server 82 to modify the coding parameters. For instance, a JPEG 2000 bitstream can be transcoded in order to include a region of interest with a higher quality than the background. Alternatively, for compatibility with existing cameras or to reduce complexity in the camera, other compression techniques can also be used in the camera, such as JPEG. In this case, transcoding from JPEG to JPEG 2000 can take place in the server. Other coding schemes such as scalable video coding can also be used instead of JPEG 2000 for this component of the system.

Security

[0046] Secured JPEG 2000 (JPSEC) may be used to secure the video codestream, as indicated in step 64. Secured JPEG 2000 (JPSEC), for example, as disclosed in Dufaux et al; "JPSEC for Secure Imaging in JPEG 2000"; Journal of SPIE Proceedings -Applications of Digital Image Processing XXVII, Denver, Colorado, November 2004, pages 319-330, hereby incorporated by reference, may be used to secure the video codestream. The JPSEC standard extends the baseline JPEG 2000 specifications to provide a standardized framework for secure imaging, which enables the use of security tools such as content protection, data integrity check, authentication, and conditional access control. [0047] JPSEC is used in the video surveillance system in accordance with, the present invention as a tool for conditional access control. For example , pseudo-random noise can be added to selected parts of the codestream to scramble or obscure persons and objects of interest. Authorized users provided with the pseudo-random sequence can therefore remove this noise. Conversely, unauthorized users will not know how to remove this noise and consequently will only have access to a distorted image. The data to remove the noise may be communicated to authorized users by means of a key or password which describes the parameters of to generate the noise, or to reverse the scrambling and selective encryption applied.

[0048] In order to fully exploit and retain the properties of the JPEG 2000 standard, the scrambling may be selectively applied on the code-blocks composing the codestream. The system is composed of three main components: scrambling, pseudo-random number generator and an encryption algorithm. The scrambling can be performed on quantized wavelet coefficients or alternatively directly on the codestream. In the first case, the signs of the coefficients in each code-block are inverted pseudo-randomly, while in the second case, bits of the codestream are flipped. In both cases, the scrambling process is driven by a pseudo-random number generator using several seed values. To communicate the seed values to authorized users, they may be encrypted and inserted in the JPSEC codestream.

[0049] An important aspect of the system in accordance with the present invention is that it may use a conditional access control technique to preserve privacy. With such conditional access control, the distortion level introduced in specific parts of the image can be controlled. This allows for access control by resolution, quality or regions of interest in an image. Specifically, it allows for portions of the video content in a frame to be scrambled. In addition, several levels of access can be defined by using different encryption keys. For example, people and/or objects in a scene that are detected may be scrambled without scrambling the background scene. In particular, as discussed in Dufaux et al; "JPSEC for Secure Imaging in JPEG 2000"; scrambling may be selectively applied only to the code-blocks corresponding to the regions of interest. Furthermore, the amount of distortion in the protected image can be controlled by applying the scrambling to some resolution levels or quality layers. In this way, people and/or objects, such as cars, under surveillance cannot be recognized, but the remaining of the scene is clear. The encryption key can be kept under tight control for the protection of the person or persons in the scene but available to selectively enable unscrambling to enable objects and persons to be identified.

[0050] Two levels of scrambling can be used with this conditional access control technique: on the one hand the regions of interest may be scrambled and the corresponding seeds encrypted with a first key; on the other hand the whole image may scrambled and the corresponding seeds encrypted with a second key. As such, someone snooping on the system will not have access to the video data. Moreover, operators of the surveillance system, in possession of the second encryption key, will be able to view the scene but not fully recognize people and/or objects present. In such an embodiment, only persons with both encryption keys will be able view the whole scene including human faces and other objects of interest in the scene without distortion.

[0051] A JPSEC tool for data integrity may also be used to detect tampering of the codestream by an attacker, as described in detail in Dufaux et al; "Securing JPEG 2000 Compressed Images"; Journal of SPIE Proceedings- Applications of Digital Image Processing XXVI, San Diego , California, November 2003, pages 397-406, hereby incorporated by reference. A particularly efficient way to achieve this is to use a technique based on hashing and digital signatures of the codestream on a code-block basis on JPEG 2000 compressed bit streams. Despite efficient use of JPSEC in the described system, other alternative techniques for securing and authenticating video can replace the above mentioned security methods in the described system.

Storage

[0052] Encoded video data, that may be encrypted, as indicated in step 64, may be stored in memory storage at the central server 27 as indicated in step 66. The results of the video analysis in step 58 as well as the initiated responses by the system 20 may also be stored in step 66.

Transmission

[0053] The various stored data, as discussed above, may then be transmitted, as indicated in step 68, over the communication links 35 and 38 to initiate a response to an abnormal condition or allow a homogeneous client 40, 42, 44 and 46 to access the system 20. The communication links 35 and 48 may be in accordance with the JPWL standard, as discussed above.

Process Level System Description: Clients

[0054] In one embodiment of the invention, heterogeneous clients 40, 42, 44, 46, 98, 100, 102, 106 may be configured to access the central server 27 or surveillance server 82 in order to monitor the live or archived video surveillance sequences, as discussed above. As the bitstream is scalable, the central server 27 or surveillance server 82 is able to adapt the resolution and bandwidth of the delivered video depending on the performance and characteristics of the client and its network connection. Delivery can be through a wired or wireless communication link 48, as discussed above. Therefore, mobile clients can access the system 20, 80 and receive a bitstream adapted to their network conditions and terminal capabilities. As such, policemen or security guards can be equipped with wireless laptops 44, 100 or PDAs 42, 102 while on patrol. Home owners can receive a MMS with a slideshow or a short video in case of intrusion detection by way of a cell phone 46. As discussed above, the clients 40, 42, 44, 46, 98, 100, 102, 106 can connect to the central server 27 or surveillance server 82 and request a live or a pre-recorded video stream.

[0055] Figs. 4 and 5.are flow diagrams which illustrate the steps involved for a client 40, 42, 44,46, 98, 100, 102, 106 to access the system 20, 80. In the system 20, 80, the same code-stream may be transmitted to all clients 40, 42, 44, 46, 98, 100, 102, 106; regardless of their access control credentials. On the one hand, unauthorized clients may be configured so as not to possess the private key required for unscrambling the content. Therefore, in that configuration, those clients will only be able to view a distorted version of the content where private information is not identifiable. On the other hand, authorized clients, e.g. law- enforcement authorities, can unscramble the code-stream and recover the complete undistorted scene.

[0056] Fig. 4 represents a flow diagram for an authorized client 40, 42, 44 and 46, while Fig. 5 represents a flow diagram for an unauthorized client 40, 42, 44 and 46. Referring first to Fig.4, the client 40, 42, 44, 46, 98, 100, 102, 106 uses a private key, sent previously to the client 40, 42, 44,46, 98, 100, 102, 106 by the central server 27 or the surveillance server 82, to descramble the video content from the central server in step 70. Next, in step 72, the decrypted video content is decoded in step 72 and displayed in step 74. Referring to Fig. 5, unauthorized clients 40, 42, 44, 46, 98, 100, 102, 106 will not have a private key and are thus unable to decrypt the video content from the central server 27 or the surveillance server 82. The unauthorized clients 40, 42, 44, 46, 98, 100, 102, 106 will however be able to decode the encrypted video content from the central server 27 or the surveillance server 82, in step 76 and display it in step 78.

Remote Control and Operations

[0057] The video surveillance system 20, 80 can be controlled and operated remotely. For example, instructions can be sent to the central server 27 or to the gateway server 84 over the Internet or other public or private communication network, using a standard web browsers, by SMS using a cell phone, or with a remote control. In particular, commands to start, stop, pause, or resume the video surveillance can be sent remotely. It is also possible to query the status of the system 20, 80 e.g. verifying that it is functioning correctly. Finally, it is possible to remotely request a live or archived video. This video can be streamed on the Internet (wired or wireless) or sent to a cell phone by MMS or streamed to it, for example, by 3GPP, which defines MMS and video streaming on 3^rd generation mobile networks: http://www.3gpp.org/ [0058] Obviously, many modifications and variations of the present invention are possible in light of the above teachings. Thus, it is to be understood that, within the scope of the appended claims, the invention may be practiced otherwise than as specifically described above.

[0059] What is claimed and desired to be secured by a Letters Patent of the United States is:

Claims

We claim:

1. A distributed smart video surveillance system comprising: at least one video surveillance camera configured to capture video content of an area of interest, analyze said captured video content and identify objects of interest and transmit said video content over a network; at least one server coupled to said network for receiving and storing said encoded video content, said server configured to analyze said video content in order to detect an abnormal condition by detecting a change in a frame and provide an indication of said abnormal condition; and at least one client for communicating with said at least one server and accessing said video content.

2. The distributed smart video surveillance system as recited in claim 1, wherein said system is configured to enable access to live and stored video content.

3. The distributed smart video surveillance system as recited in claim 1, further including one or more sensors coupled to said at least one server.

4. The distributed smart video surveillance system as recited in claim 1, wherein said one or more sensors are hardwired to said at least one server.

5. The distributed smart video surveillance system as recited in claim 1, wherein said one or more sensors are wirelessly connected to said at least one server.

6. The distributed smart video surveillance system as recited in claim 1, further including one or more remote response devices and wherein said at least one server is configured to control said one or more remote devices upon detection of an abnormal condition.

7. The distributed smart video surveillance system as recited in claim 1, wherein said one or more remote devices are selected from the group consisting of a spotlight, a loudspeaker, or an electric motor.

8. The distributed smart video surveillance system as recited in claim 4, wherein said one or more sensors are selected from the group consisting of a microphone, a motion detector, or a smoke detector.

9. The distributed smart video surveillance system as recited in claim 4, wherein said one or more sensors are wirelessly connected to said at least one server.

10. The distributed smart video surveillance system as recited in claim 4, wherein said one or more sensors are hard wired to said at least one server.

11. The distributed smart video surveillance system as recited in claim 1, wherein said system includes a surveillance server for receiving and processing signals from said at least one video surveillance camera and an access server for enabling access from said at least one client.

12. The distributed smart video surveillance system as recited in claim 11, wherein said access server is configured to enable access of a plurality of heterogeneous clients.

13. The distributed smart video surveillance system as recited in claim 11, further including a gateway server for receiving commands from a cell phone and communicating those commands to the surveillance server to control said at least one video surveillance camera.

14. The distributed smart video surveillance system as recited in claim 1, wherein said at least one server is configured to receive commands from a cell phone to control said at least one video surveillance camera.