WO2014009290A1

WO2014009290A1 - Dual mode occupancy detection system and method

Info

Publication number: WO2014009290A1
Application number: PCT/EP2013/064311
Authority: WO
Inventors: Harshit Tiwari; Cristian Andreola
Original assignee: Osram Gmbh
Priority date: 2012-07-12
Filing date: 2013-07-05
Publication date: 2014-01-16

Abstract

The present invention relates to a novel human occupancy detection system method for use in lighting/HVAC control. The system uses the vision based algorithm to detect human head- top circles in top view videos captured by ceiling mounted camera. The color images converted to grayscale images and gradient filter are applied to find gradient vectors at each point of the image. The gradient vector then propagated back- wards to find the center candidate. The center point accumulation is used to find the unique center candidate for near circular objects like human head. A pre or post PIR based detection is done to enhance the accuracy of the occupancy detection and to avoid detection of inanimate human head like objects by the vision based technique.

Description

DUAL MODE OCCUPANCY DETECTION SYSTEM AND METHOD FIELD OF INVENTION

The present invention relates to an improved occupancy detec^¬ tion technique by fusion of inexpensive Passive Infra-Red (PIR) Sensor and occupancy detection algorithm using ceiling mounted cameras which utilizes the circularity of head shape for accurate detection. Specifically, the aim of present in^¬ vention is to increase the accuracy of the overall system by avoiding false positive detection of head-like inanimate ob^¬ jects. To this end, the heat signatures of the detected head^¬ like objects, available through PIR sensors, are used to dis- tinguish between actual heads and inanimate head-like ob- j ects .

BACKGROUND ART

Knowledge regarding the occupancy of a particular region can be useful in a variety of applications. For instance, knowl^¬ edge of occupancy within a building can be used to improve the energy efficiency by the automation of lightning, HVAC, comfort, and convenience of a building and in emergency situations to improve search and rescue efforts of first re- sponders by providing information regarding the location of occupants. Typically, building occupancy is determined based solely on data provided by sensors. These occupancy estimates may result in the generation of errors due to loss of sensor data or accumulation of errors in the sensor data over time.

Conventional vision-based detection algorithms which exploit the specific characteristics of humans, entail additional in^¬ frastructure and installation costs apart from the added the image processing complexity due to multiple networked cam^¬ eras. Ceiling mounted cameras offer a cheap alternative in terms of cost as well as computation complexity as it can be readily integrated to existing ceiling fixtures (e.g., lumi- naires, smoke detectors, airconditioner vents, etc) . However, ceiling mounted cameras pose several challenges, for example only the human head-top is visible and no information regard^¬ ing conventional human features is available. Various approaches to vision-based occupancy detection have been discussed in literature from standalone vision-based de^¬ tection techniques to a multi-modal fusion of different sen^¬ sors. In prior art, the authors have presented a technique to detect and localize human head using motion detection, Hough transform and a statistical color transform etc. Also pre^¬ sented method to find the absolute position of a person in room is presented using the focusing ring of an autofocus camera . In prior art, Gao, L.: Human detection by omni-directional camera, Semester Thesis, Department of Information Technology and Electrical Engineering, Swiss Federal Institute of Tech^¬ nology, Zurich (2011), a human detection technique using omni-directional camera is proposed. In another literature by Spinello, L., Siegwart, R. : Human detection using multimodal and multi dimentional features. In Robotics and Automation, 2008. ICRA 2008. IEEE International Conference on. (2008) 3264 -3269, a human detection method based on a Bayesian fusion approach using laser range data and camera images has been proposed. A fast and efficient HOG+SVM framework for hu^¬ man detection via reuse of features and sub-cell based inter^¬ polation is discussed in Pang, Y., Yuan, Y., Li, X., Pan, J.: Efficient hog human detection. Signal Processing 91 (2011) 773- 781.

In another published document by Karuppiah, D., Deegan, P., Araujo, E., Yang, Y., Holness, G., Zhu, Z., Lerner, B., Gru- pen, R., Riseman, E.: Software mode changes for continuous motion tracking. In: Proceedings of the First International Workshop on Self-adaptive Software. IWSAS ' 2000, Secaucus, NJ, USA, Springer-Verlag New York, Inc. (2000) 161-180, an application with a redundant sensor array, distributed spa^¬ tially over an office-like environment has been used to track and localize a human being. The system is capable of reacting at run-time to various kinds of faults namely: hardware fail^¬ ure, inadequate sensor geometries, occlusion, and bandwidth limitations. In Cucchiara, R., Prati, A., Benini, L.,

Farella, E.: T-park: ambient intelligence for security in public parks. In: Intelligent Environments, 2005, The IEEE International Workshop on (Ref . No. 2005/11059) . (2005) 243- 25, a technique for multi-camera people tracking via auto- matic calibration of camera pairs and Areas of Field Views ( AoFoVs) for consistent labelling of people has been proposed. Also, information from sensor networks distributed at the borders of the AoFoV is used to trigger algorithms to iden^¬ tify the single person.

An Integrated MultiModal Sensor Network for Video Surveil^¬ lance is presented (Andrea Prati, Roberto Vezzani, University of Modena and Reggio Emilia & Luca Benini, Elisabetta

Farella, Piero Zappi University of Bologna) to enhance video surveillance systems, multi-modal sensor integration can be a successful strategy. In this work, a computer vision system able to detect and track people from multiple cameras is in- tegrated with a wireless sensor network mounting PIR (Passive InfraRed) sensors.

Most vision-based and multi-modal detection algorithms rely on additional information for human detection such a motion cues or manually marked triggers / trip wires. In addition, most of these techniques use side-view cameras for human de^¬ tection in which many rich features of the human body such as the skin/face color (hue), shape and movement of the limbs etc. are available.

The present invention perform human detection in top-view videos using ceiling mounted cameras. The algorithm is fully automatic in the sense it does not require any additional manual inputs such as trip-wires or manual triggers for de- tection.

Various patents describe using optical triangulation to meas^¬ ure the distance of objects from a video sensor. For example, in U.S. Pat. No. US5,255,064, multiple images from a video camera are used to apply triangulation to determine the dis^¬ tance of a moving target.

In a PCT publication, WO2011151232, an optical system for occupancy sensing, and corresponding method is disclosed. The system includes a plurality of optical line sensors, each consisting of a linear array of light sensing elements; and an optical light integrating device that integrates light from rays with incidence angles subject to geometric con^¬ straints to be sensed by a light sensing element.

In an US publication, US2011267186, an optical method and system is disclosed. Therein successive images from an infra- red camera are analyzed to detect thermal characteristics of an occupant as well as movement.

In a PCT publication, WO2011091868, a system and method for 2D occupancy sensing is disclosed. The 2D occupancy system for determining a position of a user (e.g. a human, an object or an animal) in an environment according to embodiments of the invention includes a host device and a plurality of mo^¬ tion detection devices wherein the host device and the plu- rality of motion detection devices are connected through a network. Each motion detection device has a viewing angle and the viewing angle of any motion detection device overlaps with the viewing angle of at least one other motion detection device .

In another US publication, US2010262296, a system for the control of a set of powered utilities is disclosed. The sys^¬ tem comprises a first device which in turn comprises a proc^¬ essor capable of altering a state of a first powered utility. This first device further comprises a data port configured to transmit a set of messages. These messages include a trans^¬ mitted message delivered from the processor.

In yet another US publication US2004066500, occupancy detec- tion and measurement, and obstacle detection using imaging technology is disclosed. Embodiments include determining oc^¬ cupancy, or the presence of an object or person in a scene or space. If there is occupancy, the amount of occupancy is measured .

SUMMARY OF INVENTION

Energy efficiency of systems employing occupancy detection techniques depends on the accuracy of the human detection al- gorithm. Design of such accurate algorithms is a non-trivial problem with vision based detection techniques and particu^¬ larly challenging using ceiling mounted camera where only head-top is visible consistently. Herein disclosed is a novel vision-based occupancy detection algorithm using ceiling mounted cameras which utilizes the circularity of head shape for accurate detection. Specifically, in the said algorithm, modelling is carried out with head-top as an imperfect circle with the centre coordinates (x, y) and radius (r) as random variables. A 3-D distribution is built from the potential circle centre candidates along the ( x, y, r) dimensions and accumulate the centre points in 3-D space to obtain an accu^¬ rate classification. Experiments are conducted using an ex^¬ haustive video database and for varying illumination and am- bience and showed that the said algorithm achieves a detec^¬ tion accuracy of up to 96%.

In the present disclosure novel occupancy detection algorithm is introduced using ceiling mounted cameras. In such a sys- tern, human detection implies detection of human head-tops since the only recognizable feature of human body available in all views is the head-top. Due to near-spherical shape of a human head, the projection of a head on a 2-D plane is al^¬ ways near-circular in shape. This characteristic of the human head-top is used to accomplish occupancy detection in a cir^¬ cle detection-based framework.

Described herein is a system for occupancy detection techniques exploiting the centre point's distribution for head- top circles in a region. The system includes at least one ceiling mounted camera for recoding purposes and which is op^¬ erable connected a controller using the input from the said camera and configured for executing an algorithm that gener- ates occupancy detection associated with the detection of hu^¬ man head tops predicted movement of occupants within each of the plurality of segments. In another embodiment there may be a plurality of ceiling cameras or otherwise which are operable connected with each other. The algorithm is a vision-based occupancy detection algorithm using gradient back propagation approach through ceiling mounted cameras which utilizes the circularity of head shape for accurate detection. The system further includes an output operably connected to the controller to com^¬ municate the occupancy estimates generated by the algorithm. In another aspect, described herein is a method of detecting occupancy in a region exploiting the centre points distribu- tion for head-top circles in the region. The method includes modelling using occupancy detection algorithm using ceiling mounted cameras. Due to near-spherical shape of a human head, the projection of a head on a 2-D plane (e.g. the camera plane) is always near-circular in shape. The characteristics of the human head-top is used to accomplish occupancy detec^¬ tion in a circle detection-based framework. The method fur^¬ ther includes calculating model-based predictions of occu^¬ pancy within the region by detecting human head-tops by locating circles in grayscale images. The circles are detected by back propagating the gradients in the image and applying thresholds on accumulated points thereby detecting circle centres. The points are accumulated in a 3-D accumulator ar^¬ ray of size (pl,ql,r3), where pi & ql are the width and the height of the image respectively and r3 is the number of ra- dius values. The value of r3 depends upon the range of radius values used for circle detection. In another aspect, described herein is a system for detecting occupancy in a region using centre points distribution for head-top circles in a region. The system includes means for detecting each of the plurality of head top circles in a par- ticular region. The system further includes means for calculating model-based detections of occupancy within the region based on state equations that model occupancy of each region, detecting human head-tops by locating circles in grayscale images, further the circles are detected by back-propagating the gradients in the image and applying thresholds on accumu^¬ lated points thereby detecting circle centres. The points of the potential centre candidates are accumulated in a 3-D ac^¬ cumulator. After accumulation, a suitable threshold is applied to each 2-D (x,y) plane in accumulator array to detect circles of corresponding radii.

In another aspect, described herein is a computer readable storage medium encoded with a machine-readable computer pro^¬ gram code for generating occupancy detection for a region, the computer readable storage medium including instructions for causing a controller to implement the said method. The computer program includes instructions for calculating model- based detections of occupancy within the region based on state equations that model occupancy of the region, detecting human head-tops by locating circles in grayscale images, fur^¬ ther the circles are detected by back-propagating the gradi^¬ ents in the image and applying thresholds on accumulated points thereby detecting circle centres. The points of the potential centre candidates are accumulated in a 3-D accumu- lator. After accumulation, a suitable threshold is applied to each 2-D (x,y) plane in accumulator array to detect circles of corresponding radii. As it is insufficient to model the head-top as a unique cir^¬ cle due to variability of the human head in terms of its shape and size among different people, hence, it is proposed herein to use the parametric definition of a circle in terms of centre coordinates and radius (x, y, r) to model the head- top circle as an imperfect circle with x, y and r as random variables. Circle centre coordinates obtained by gradient back propagation algorithm form points in the (x, y, r) space. The circle centre points are accumulated in 3-D space to obtain the actual head-top circles. Further, the human head-tops appear as homogenous objects in top-view videos. The said property is used to distinguish head-top circles from random circles formed due to clutter in the scene. The centre-points accumulation and homogeneity criterion together yield 79% improvement in the error-rate performance of the gradient back propagation algorithm alone.

As per another exemplary embodiment a fusion of infrared and vision based technologies for increased accuracy of the de- tection rates is disclosed. A Passive Infra-Red (PIR) sensor can detect movement of any object in its Field-of-View (FOV) which is not in thermal equilibrium with its surroundings e.g. Humanbeings. PIR sensors can detect motion of human beings with sufficient accuracy. A 2-D array of infra-red sensors is used along with a camera in a dual-mode occupancy sensor. The 2-D PIR sensor array is overlaid on the grayscale image from the vision sen^¬ sor. Optics can be used to limit the FOV of each sensor in the PIR sensor array to a unique block. The number of parti^¬ tions of the overall FOV thus depends on the size of the PIR sensor array. For an MxN sensor array, we will have an MxN grid of blocks in the overall FOV. Heating signatures in these blocks will be monitored indi^¬ vidually by a dedicated PIR sensor. Upon detection of occu^¬ pancy in one or more blocks in the grid through the PIR sen^¬ sor array, vision algorithm can be used to detect if there is human presence in the specified block (s) . This is a vision sensing post PIR detection.

In another mode of fusion, involving use of PIR sensing post vision-based sensing can also be adopted. Vision based algo- rithm can be applied first to detect all head-like objects in the scene. Heat signatures of all detected objects from the vision based approach can then be obtained from the PIR sensor array. Human head-tops can be distinguished from all de^¬ tected objects by means of their heat signatures. Use of heat signatures for human detection will help avoid head-like in^¬ animate objects (e.g. helmet, black bags, dark disk-like ob^¬ jects), from being detected falsely by the vision-based algo^¬ rithm. This technique of dual mode sensing can, in principle, lead to significant improvement in detection accuracy.

Therefore also herein described an improved system for occu^¬ pancy detection exploiting the centre point's distribution for head top circles in a region, the system comprising: at least one ceiling mounted camera operably connected to a con^¬ troller configured for executing an algorithm that generates occupancy detection associated with the detection of human head tops and predicted movement of occupants within each of the plurality of regions, wherein the algorithm is a vision- based occupancy detection algorithm using gradient back propagation approach through ceiling mounted cameras and which utilizes the head shape for accurate detection; at least one Passive Infra Red (PIR) sensor configured to detect heat signatures of objects in its Field-of-View (FOV) which is not in thermal equilibrium with its surroundings; and an output operably connected to the controller and PIR sensor configured to communicate the occupancy detection and pre^¬ dicted movement of the occupants. BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS

Embodiments of the invention are illustrated by way of exam^¬ ple, and not by way of limitation, in the figures of the ac^¬ companying drawings .

Fig 1 illustrates an overview of the disclosed system. The Controller (2) is connected to a ceiling mounted video camera (1) having fish-eye lens and PIR sensors (3, 4) . The relevant PIR sensor sends occupancy data to the Controller in its field of view (FOV) . The Controller controls the switches of the lighting/HVAC to turn on or off according the occupancy detection .

Fig 2 illustrates a snapshot of test-cases used for algorithm testing

Fig 3 (a) illustrates gradient vectors emerging outside the object for circular objection along radial lines

Fig 3 (b) illustrates gradient back propagation for circular obj ects

Fig 4 illustrates circle detection using gradient back propa- gation

Fig 5 illustrates distribution of centre candidates for dif^¬ ferent values of radius

Fig 6 illustrates a 3-D volume for concentrating centre candidates along (x, y, r) dimensions;

Fig 7 illustrates homogeneity criterion gradient sums using circular integrator filters;

Fig 8 illustrates the functionality of dual-mode occupancy sensor. The 2-D grid in the left picture is a virtual grid displaying the FOV of each individual sensor in the 2-D PIR sensor array overlaid on the grayscale image from the vision sensor . DETAILED DESCRIPTION

The application discloses a system and method for detecting occupancy based on exploiting the centre point's distribution for head-top circles. In an exemplary embodiment, occupancy is detected based solely on Gradient Back Propagation Ap- proach. This application expands upon the scope of the prior art, disclosing additional embodiments and methods of imple^¬ menting occupant detection.

Recently significant research interest has been generated in the area of energy efficient control systems using occupancy detection techniques. Occupancy detection has traditionally been accomplished using either vision-based (e.g., using mast/ceiling mounted cameras or non vision-based techniques such as PIR, Ultrasonic, acoustic, etc. Vision-based occu^¬ pancy detectors have been known to be cost effective and power efficient for both indoor/outdoor scenarios, where a camera is mounted on a mast with a clear view of entry/exit points or locations with movement of people. For indoor sce^¬ narios, these sensors can be used to not only detect human occupation but they can also be useful for people counting, detection of unusual activity or possible fall of a person (e.g., in assisted living scenarios).

The term 'region' is used throughout the description refers to both a region as well as various sub-divisions of the re^¬ gion. For instance, in the exemplary embodiment shown in figures, the term 'region' refers to both the area in general as well as to the individual sub-regions or zones e.g. rooms. Therefore, generating occupancy detection for the region would include generating occupancy estimates for each of the individual zones. In addition, the term 'occupancy detection' is used throughout the description and refers generally to output related to occupancy. Therefore, an occupancy detection for a region may include data such as a number of occupants within the region, a probability associated with all possible occupancy levels associated with the region changes in occupancy, detecting something (human) ; catching sight of human, data indicative of the reliability of confidence associated with occupancy (e.g., covariance) , as well as other similarly useful data related to occupancy.

The disclosed novel occupancy detection algorithm using ceil^¬ ing mounted cameras in a system as shown in fig. 1. In such a system, human detection implies detection of human head-tops since the only recognizable feature of human body available in all views is the head-top. Due to near-spherical shape of a human head, the projection of a head on a 2-D plane (e.g. the camera plane) is always near-circular in shape. The char^¬ acteristic of the human head-top to accomplish occupancy de^¬ tection in a circle detection based framework is exploited. To this end, a comparative analysis of two techniques is pre^¬ sented for circle detection in images namely, Hough circles and gradient back propagation. On analysis, it may be seen that the gradient back propagation algorithm yields 40% re^¬ duction in error-rate compared to Hough circles-based ap- proach for human detection problem.

It has been observed that in top-view videos there are no rich features available for human detection. The conventional features like the skin color (hue) , the face of the person, the shape and the movement of the limbs, etc., cannot be used in this view i.e. head-top is the only feature that is visi^¬ ble consistently in all poses for human detection. Hence the problem of human detection boils down to human head-top detection in top-view. A human head resembles solid sphere in shape and the projection of a sphere is always circular at all angles. Head-top detection thus implies finding circles (disks) in top-view videos.

In the gradient back propagation approach the direction of the gradient vector is used, along with its magnitude, in circle detection framework. First the original RGB image is converted to grayscale and then filters are applied to find gradient vector at each point on the grayscale image. The gradient vector can be represented as (gr, Q) in the polar coordinate system, wherein gr is the magnitude of the gradi^¬ ent and Θ is the direction. For a dark colored circular ob^¬ ject in the image, the gradient vectors at the object bound- ary appear to emerge out of the object along radial lines as shown in fig. 3 (a) .

For such an object, if the gradient vectors are propagated backwards using a fixed distance, they all converge at the centre of the object if the distance value selected is equal to the radius of the circle. This is depicted in fig. 4. This is the main concept behind circle detection using gradient back propagation, the algorithm for which is explained below:

1. Choose a range of radius values (rl , r2) to detect cir- cles with radii lying in this range in the image. Here rl and r2 are integer values representing the minimum and maximum radius values respectively in terms of number of pixels.

Hence the possible radius values for the chosen range are in- tegers between (rl, r2) . Let rn be the total no. of radius values .

2. Initialize a 3-D accumulator array (accm) , of size (pn, qn, rn) with zeros, where pn and qn are the width and the height of the grayscale image respectively.

3. Convert the original RGB image to grayscale and apply two dimensional gradient filters (sobel, canny etc.) to find gra^¬ dient vectors at each point on the image. Let (gr_xy, &_xy) be the gradient value at pixel location (im_x, im_y) in the image.

4. Choose a value rv € [rl, r2] and create a vector (rv, $_xy

+ n) . This vector represents propagation of the gradient vec^¬ tor (gr_xy, Q_xy) with distance rv in the reverse direction i.e Back Propagation Vector (BPV) of the gradient.

5. Find projections of the BPV along the image axes (rv_x, rv_y) and round them to integral pixel values. These can be calculated as: rv_x = rv [cos (¾_y + π }] = -rv cos ¾,^■

rv_y =rv [sin(#_xy +π)} = -rvsin ϋ_χγ · (1)

6. Add this BPV to the corresponding pixel coordinates i.e. the coordinates of the point for which the BPV was calcu^¬ lated. The resultant coordinates (acx, acy) can be found us^¬ ing eqn. 1 as: oc_x = im_x * rv_x = im_x- rv cos &_xy - ocy = im_y + rv_y = imy rv sin i¾_y · (2)

7. Add the value of gradient magnitude ( gr_xy) at pixel loca^¬ tion (im_x, im_y) in the grayscale image to the existing value at location ( ac_x, ac_y, rv) in the accumulator array (accm) . Repeat the above steps for all pixels in the grayscale image (pn , qn) and for all radius values (rn) . 8. The accumulator array (accm) gets populated with the gra^¬ dient values of the original grayscale image according to eqn. 3. accm(x,y,r) =∑gr(p,q)

s

S = {(p, q); p+ rcos(Q(p, q) + n) = x,

q+rsin(0(p,q}+T.) =y} (3) It may be noted form equation 3 that the accumulation value is a function not only of the gradient magnitude (gr), but also of its direction (θ) . A suitable threshold can be ap^¬ plied to each 2-D (x, y) plane in accumulator array to detect corresponding circles of radii between (rl, r2) . The choice of threshold depends on the circularity of the object to be detected. For near perfect circles in the image, a high value of threshold is preferable, whereas, if the circles in the original image are imperfect or incomplete, it is better to choose a low value for this threshold.

In the discussion above, a circle detection algorithm for grayscale images utilizing the gradient direction alongwith the gradient magnitude is disclosed. The algorithm also pre^¬ sents a fixed computational complexity for a given image resolution.

For head-top circles, the gradient back propagation approach does not yield a unique centre candidate. Instead, it returns a region around the actual circle centre with all the points in the region satisfying the circle detection threshold. For perfect (near-perfect) circles, clear local maxima in the ac^¬ cumulator array at the centre of the circle is sought. Hence the other centre candidates in the region can be suppressed by finding local maxima around the centre in the accumulator array. However, for imperfect circles, (e.g. head-top cir^¬ cles) , the region around the centre in the accumulator array typically consists of a distribution of points with high ac^¬ cumulation values. Hence, a simple approach to find local maxima may not fetch us the exact centre coordinates. Similar distribution can be observed for different values of radius close to the actual radius value. This distribution of accu^¬ mulation values along the image axes (x, y) for different ra^¬ dius values for the original image in fig. 4 has been de- picted in fig. 5.

As illustrated in fig.5, for imperfect circles candidate cen^¬ tre points are distributed in a 3-dimensional space consti^¬ tuting the 2 image axes (x, y) and a third radial axis (r) . The distribution of centre candidates (for the original image in fig. 4) in 3-D accumulator array (accm) has been plotted in fig. 6. It is worth noting that the distribution of centre point's in the accm array for the two head-tops in the image resembles a cylinder in shape. For a 2-dimensional distribu- tion of centre points, the centre candidates can be concen^¬ trated using a mexican hat-shaped 2-D filter. The same principle is applied in 3-D space (x, y, r) to find centroid of the distribution of the centre candidates. As shown in fig. 6, a cylinder has been selected as the 3-D volume for integrating centre candidates in the accumulator array. This is consistent with the choice of a circular fil^¬ ter along the 2-D image axes (x, y) . The cylindrical shaped 3-D filter is applied to accumulate the potential centre can^¬ didates in the 3-D accumulator array (accm) . The accumulation values at all points within this cylinder in the neighbourhood of a point in the accumulator array are integrated ac- cording to eqn. 4.

S = {(u,v,w) : J(x - u)² + (y - v)² ≤ thxy

I r-w I < th_r } {4}

Where cpa is the 3-D accumulation array obtained after 3-D filtering of accm, th_xy and 2th, are the radius and height of the cylinder, respectively. In our experiments, th_xy=l and th_r=13 were shown to give the best detection results. A threshold can be applied to the integrated values in each 2D (x, y) plane of the 3-D cpa array to detect circles of corre^¬ sponding radii between (rl, r2) . This technique yields -7% improvement in the detection accuracy for head-top circles.

The head-top circles, as can be observed in the snippets in fig. 2, are homogenous in nature i.e. the pixels in the head- top part of the image have similar intensity or grayscale values. The said property of head-top circles is applied to improve the detection performance of our algorithm by apply^¬ ing a homogeneity constraint on the detected circles. This helps in eliminating false positive detection of circles formed with boundaries of heterogeneous objects or due to ex- cessive clutter. Various criteria exist for checking the ho^¬ mogeneity of an object in an image namely, variance/standard deviation of pixel values within the object, number of edge points within the object in the edge-map of the image, sum of , _

gradient values within the object etc. We selected sum of gradients as the homogeneity criteria as we have already com^¬ puted gradients on the grayscale image for earlier steps. The gradient sums have been calculated upfront by applying circu- lar 2- D filters on calculated gradient array for the gray^¬ scale image. This approach yields gradient sums for all the points in the image. We can then find the gradient sum values at circle centres detected via earlier steps and apply threshold only on those selected values. This process is de- picted in fig. 7.

It can be observed from fig. 7 that homogenous regions of the grayscale image have lower values of gradient sums (dark re^¬ gions in the filtered image) compared to heterogene^¬ ous/cluttered regions. This has been used to distinguish ho^¬ mogenous head-top circles from heterogeneous circles. This is accomplished by selecting only those circles from all de^¬ tected circle candidates which have a gradient sum value less than a predefined threshold. The value of the threshold for homogeneity depends on the circularity of the object to be detected. For perfect (in shape) and homogenous circles

(disks) , it is preferable to set a low value for the homoge^¬ neity threshold. For head-top circles, which are imperfect (but homogenous) circles, a higher value should be set as threshold. The gain observed in the detection accuracy with this technique was 3%.

The human head-tops are not perfect circles. In fact the shape of the head varies a lot from person to person. Hence, it is difficult to fit a single model to the head-top. Con^¬ sidering all the variations in the shape and the orientation of the head, a circular shape seems to best model the human head-top. With the help of this model, one can successfully able to detect humans in varying poses such as walking, read^¬ ing, standing, talking etc. More importantly, the model also works for the case when the person is sitting, where the size of the circle is considerably smaller compared to the stand- ing pose. It should be noted that the algorithm is also able to detect a person when he stays stationary for some time while standing or sitting. For the lab scenario, the algo^¬ rithm is able to successfully detect the person in the midst of heavy clutter. Also for the case with two subjects, it successfully detects both the subjects, a feature useful for the people counting scenario. It is worth noting that these results have been achieved with the vision-based algorithm only . In the present novel head-top detection algorithm utilizing gradient back propagation in grayscale images for use in con^¬ junction with ceiling mounted cameras for occupancy detection applications provide 79% improvement in the error-rate per^¬ formance of the overall system. In addition, a database of top-view videos can be generated by installing a ceiling mounted camera. The database captures variability in poses, head-shapes, ceiling height, and illumination conditions. The developed algorithm has been tested extensively using the generated database and a detection accuracy of upto 96% has been achieved. The proposed algorithm is of significance in the design of energy efficient control systems for lighting, heating, air-conditioning, etc.

As per another exemplary embodiment a fusion of infrared and vision-based technologies is used for more accurate detection rates. A Passive Infra Red (PIR) sensor can detect movement of any object in its Field-of-View (FOV) which is not in thermal equilibrium with its surroundings e.g. human beings. Use of heat signatures (obtained via PIR sensors) for human detection helps avoid head like inanimate objects (e.g. hel^¬ met, black bags, dark disk-like objects), from being detected falsely by the vision-based algorithm. This technique of dual mode sensing, in principle, lead to significant improvement in detection accuracy.

As earlier disclosed the circle detection algorithm for grayscale images utilizing the gradient direction along with the gradient magnitude, detects human head-tops by locating cir- cles in grayscale images. The circles are detected by back- propagating the gradients in the image and applying thresholds on accumulated points thereby detecting circle centres. The points are accumulated in a 3-D accumulator array of size (pl,ql,r3), where pi & ql are the width and the height of the image respectively and r3 is the number of radius values. The value of r3 depends upon the range of radius values used for circle detection. After accumulation, a suitable threshold can be applied to each 2-D (x,y) plane in accumulator array to detect circles of corresponding radii. The algorithm per- formance is further improved by utilizing data from two sen^¬ sors - a vision sensor and at least one PIR sensor (Dual-Mode Occupancy Sensing) .

The algorithm performance is further aided by the use of cues about the heat signatures of objects in the scene. An inex^¬ pensive infra-red sensor can be used to get such heat signa^¬ tures. A Passive Infra-Red (PIR) sensor can detect movement of any object in its Field-of-View (FOV) which is not in thermal equilibrium with its surroundings e.g. human beings. Such sensors can, therefore, detect human presence through their movements. A fusion of infra-red and vision based technologies, to boost the detection rates significantly is presented herewith. PIR sensors can detect motion of human beings with sufficient ac^¬ curacy. A 2-D array of infra-red sensors can be used along with a camera in a dual-mode occupancy sensor. Fig. 8 depicts a graphical representation of the working concept of sug^¬ gested dual-mode sensor. The 2-D grid in the left picture is a virtual grid displaying the FOV of each individual sensor in the 2-D PIR sensor array overlaid on the grayscale image from the vision sensor. Optics can be used to limit the FOV of each sensor in the PIR sensor array to a unique block in the grid. The number of partitions of the overall FOV thus depends on the size of the PIR sensor array. A MxN sensor ar^¬ ray, have an MxN grid of blocks in the overall FOV. Heat sig- natures in these blocks are monitored individually by a dedi^¬ cated PIR sensor. Upon detection of occupancy in one or more blocks in the grid through the PIR sensor array, vision algo^¬ rithm is used to detect if there is human presence in the specified block (s). The technique discussed above involves use of vision-based sensing post detection in the PIR domain.

Another mode of fusion, involving use of PIR sensing post vision-based sensing can also be adopted. Visionbased algorithm discussed above can be applied first to detect all head-like objects in the scene. Heat signatures of all detected objects from the vision based approach can then be obtained from the PIR sensor array. Human head-tops can be distinguished from all detected objects by means of their heat signatures. The use of heat signatures for human detection helps avoid head- like inanimate objects (e.g. helmet, black bags, dark disk^¬ like objects), from being detected falsely by the vision- based algorithm. This technique of dual mode sensing can, in principle, lead to significant improvement in detection accu^¬ racy .

The novel fusion of data from PIR and vision-based sensors with specific application to head-top detection is disclosed. Two approaches to the fusion of PIR sensing a pre-vision based detection and a post-vision based detection is proposed. These have been suggested as extensions to the dis^¬ closed head-top detection algorithm utilizing gradient back- propagation. The proposed technique detects head-tops in grayscale images for use in conjunction with ceiling mounted cameras for occupancy detection applications. The algorithm performs person detection in top-view videos captured through ceiling mounted cameras having fish-eye lens camera system. This view has certain advantages like occlusions are minimum and the coverage is good i.e. the number of cameras needed to monitor a certain area is least in this view. But this view also offers some very serious challenges due to the absence of conventional features for person detection such as color (hue) of the skin, facial features and shape/movements of the limbs. Various challenges to person detection in top-view videos have been discussed and a novel fusion technique - based on a) circle-detection utilizing gradient back- propagation & b) PIR based heat signatures is disclosed- has been proposed. The suggested approach, in principle, yield significant, improvement in the detection accuracy when used in conjunction with the system described earlier. The accuracy of the overall system (without PIR sensors) is currently 90%.

Examples

The proposed algorithm was tested extensively using top- view videos captured in two different scenarios: a lab set-up and a discussion room. A ceiling mounted camera was installed in the two locations and 16 test-videos of various subjects were taken in different poses such as walking, sitting, reading, standing etc. The ceiling height in these locations is between 3m- 5m and the lighting is a mix of natural and arti^¬ ficial lights. The proposed algorithm yields upto 96% accu racy (At equal error-rate point i.e. Recall rate and Preci^¬ sion rate are both equal to 96%) when applied to the captured top-view videos.

The invention is further extended to use the occupancy detection by the said system in lighting/HVAC control. This could be achieved by the controlling the power switches by said controller based on human occupancy obtained from said algorithm in addition with PIR related data.

Although the foregoing description of the present invention has been shown and described with reference to particular em^¬ bodiments and applications thereof, it has been presented for purposes of illustration and description and is not intended to be exhaustive or to limit the invention to the particular embodiments and applications disclosed. It will be apparent to those having ordinary skill in the art that a number of changes, modifications, variations, or alterations to the in- vention as described herein may be made, none of which depart from the spirit or scope of the present invention. The par^¬ ticular embodiments and applications were chosen and de^¬ scribed to provide the best illustration of the principles of the invention and its practical application to thereby enable one of ordinary skill in the art to utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated. All such changes, modifications, variations, and alterations should therefore be seen as being within the scope of the present invention as determined by the appended claims when interpreted in accor^¬ dance with the breadth to which they are fairly, legally, and equitably entitled.

Claims

Patent claims

1. A dual mode occupancy detection system for occupancy de tection comprising:

at least one ceiling mounted camera (1) operably connected to a controller (2) at least one Passive Infra Red (PIR) sensor (3) op erably connected to the said controller (2) and is configured to detect movement of any object in its Field-of-View (FOV) which is not in thermal equi^¬ librium with its surroundings; and an output operably connected to the controller and PIR sensor configured to communicate the occupancy detection and predicted movement of the occupants, wherein

the said controller is configured for executing a vision based occupancy detection algorithm which uses gradient back propagation approach in conjunc tion with center point accumulation and/or homogeneity constraint to detect head top circles in the video captured by ceiling mounted camera and util^¬ izes the head shape for occupancy detection; and wherein said PIR sensors obtain heat signatures from objects detected by circle based detection in their respective field of view (FOV) .

2. A dual mode occupancy detection system as claimed in claim 1, wherein the ceiling mounted camera has a fish- eye lens for providing greater field of view.

A dual mode occupancy detection system as claimed in claim 1, wherein vision-based occupancy detection algo- rithm using gradient back propagation approach comprise the steps of : selecting a range of radius values (rl , r2) to de^¬ tect circles with radii lying in this range in the image, wherein rl and r2 are integer values repre^¬ senting the minimum and maximum radius values re^¬ spectively in terms of number of pixels;

initializing a 3-D accumulator array (accm) , of size (pn, qn , rn) with zeros, where pn and qn are the width and the height of the grayscale image re^¬ spectively.

converting the original RGB image to grayscale and applying two dimensional gradient filters (sobel, canny etc.) to find gradient vectors at each point on the image; selecting a value rv € [rl, r2] and create a vector (rv, $_Xy + n) which represents propagation of the gradient vector (gr_xy, Q_xy) with distance rv in the reverse direction i.e Back Propagation Vector (BPV) of the gradient; calculating projections of the BPV along the image axes (rv_x, r_vy) and round them to integral pixel values ; adding the projections of BPV to the corresponding pixel coordinates i.e. the coordinates of the point for which the BPV was calculated; adding the value of gradient magnitude (gr_xy) at pixel location (im_x, im_y) in the grayscale image to the existing value at location ( ac_x, ac_y, rv) in the accumulator array (accm) /and repeating the steps for all pixels in the grayscale image (pn , qn) and for all radius values (rn) .

A dual mode occupancy detection system as claimed in claim 1, wherein a 2-D array of infra-red sensors are used along with vision based head-top circle detection

A dual mode occupancy detection system as claimed in claim 1, wherein the vision-based occupancy detection algorithm using gradient back propagation approach is used in conjunction with Centre Points Accumulation technique for detection of imperfect circles with spe cific application to head-top detection.

A dual mode occupancy detection system as claimed in claim 1, wherein the vision-based occupancy detection algorithm using gradient back propagation approach is used in conjunction with homogeneity constraint for de tection of imperfect homogenous circles with specific application to head-top detection.

A dual mode occupancy detection system as claimed in claim 1, wherein the vision-based occupancy detection algorithm using gradient back propagation approach is used in conjunction with Centre Points Accumulation technique and homogeneity criterion.

A method for occupancy detection comprising the steps of: modelling using vision based occupancy detection algorithm using ceiling mounted cameras; calculating model-based predictions of occupancy within the region by detecting human head-tops in videos captured by ceiling mounted camera wherein the circles are detected by back propagating the gradients in the image and applying thresholds on accumulated points thereby detecting circle cen^¬ tres .

Obtaining heat signatures of detected objects with PIR sensors

A method for occupancy detection as claimed in claim 8, wherein the near-spherical shape of a human head top is used to accomplish occupancy detection in a circle de^¬ tection-based framework.

A method for occupancy detection as claimed in claim 8, wherein the image captured by the ceiling camera is con^¬ verted to grayscale and then filters are applied to find gradient vector at each point on the grayscale image.

A method for occupancy detection as claimed in claim 8, wherein the gradient back propagation approach includes the direction of the gradient vector, along with its magnitude, in circle detection framework.

A method for occupancy detection as claimed in claim 8, wherein the gradient vector is represented as (gr, Θ) in the polar coordinate system, wherein gr is the magnitude of the gradient and Θ is the direction.

A method for occupancy detection as claimed in claim 8, wherein for an object, the gradient vectors emerging out of the object boundary along radial lines are propagated backwards using a fixed distance, converging at the cen^¬ tre of the object if the distance value selected is equal to the radius of the circle.

A method as claimed in claim 8, wherein vision-based oc^¬ cupancy detection algorithm using gradient back propagation approach comprises the steps of :

selecting a range of radius values (rl , r2) to de^¬ tect circles with radii lying in the said range in the image, wherein rl and r2 are integer values representing the minimum and maximum radius values respectively in terms of number of pixels; initializing a 3-D accumulator array (accm) , of size (pn, qn , rn) with zeros, where pn and qn are the width and the height of the grayscale image re^¬ spectively. converting the original RGB image to grayscale and applying two dimensional gradient filters to find gradient vectors at each point on the image; selecting a value rv € [rl, r2] and create a vector (rv, $_xy + n) which represents propagation of the gradient vector (gr_xy, &_xy) with distance rv in the reverse direction i.e Back Propagation Vector (BPV) of the gradient; calculating projections of the BPV along the image axes (rv_x, rv_y) and round them to integral pixel values ; adding the projections of BPV to the corresponding pixel coordinates i.e. the coordinates of the point for which the BPV was calculated; adding the value of gradient magnitude ( gr_xy) at pixel location (im_x, im_y) in the grayscale image to the existing value at location ( ac_x, ac_y, rv) in the accumulator array (accm) /and - repeating the steps for all pixels in the grayscale image (pn , qn) and for all radius values (rn) .

15. A method for occupancy detection as claimed in claim 14, wherein the accumulator array (accm) gets populated with the gradient values of the original grayscale image.

16. A method for occupancy detection as claimed in claim 14, wherein the accumulator array (accm) is given by equation:

=accm(x,y,r) =∑_s gr(p, q)

wherein 5 = {(p, q): p+ rcos(9(p, q) + π) = x

q+rsin(6(p,q)+n) =y}

17. A method for occupancy detection as claimed in claim 14, wherein a suitable threshold is applied to each 2-D (x, y) plane in accumulator array to detect corresponding circles of radii between (rl, r2) and the choice of threshold depends on the circularity of the object to be detected .

18. A method for occupancy detection as claimed in claim 14, wherein a cylindrical shaped 3-D filter is applied to accumulate the potential centre candidates in the 3-D accumulator array (accm) .

19. A method for occupancy detection as claimed in claim 14, wherein the accumulation values at all points within the said cylinder in the neighbourhood of a point in the ac^¬ cumulator array are integrated and is given by the equa^¬ tion : accm (u,v,w)

cpafx; y; r) =¾

(.x-u)² +(y~-u)²+(r-w)²

S = {(u,v,w) : sj(x - u)² + (y— v)² < th_xy

i r-wl < th_r } (4)

20. A method for occupancy detection as claimed in claim 14, wherein the vision-based occupancy detection algorithm using gradient back propagation approach is used in con- junction with Centre Points Accumulation technique for detection of imperfect circles with specific application to head-top detection.

21. A method for occupancy detection as claimed in claim 14, wherein the vision-based occupancy detection algorithm using gradient back propagation approach is used in conjunction with homogeneity criterion for detection of imperfect homogenous circles with specific application to head-top detection.

22. A method for occupancy detection as claimed in claim 14, wherein the vision-based occupancy detection algorithm using gradient back propagation approach is used in conjunction with Centre Points Accumulation technique and homogeneity criterion.

23. A method for occupancy detection as claimed in claim 8, wherein the vision-based sensing is followed by detec^¬ tion in the Passive Infra Red (PIR) sensing.

24. A method for occupancy detection as claimed in claim 23, wherein the Passive Infra Red (PIR) sensing is followed by vision-based sensing.

25. A computer readable storage medium encoded with a ma^¬ chine-readable computer program code for generating occupancy detection for a region, the computer readable storage medium including instructions for causing a controller to implement the method as claimed in claims 8- 24.

26. The computer program product for occupancy detection exploiting the centre point's distribution for head top circles in a region, which includes instructions for calculating model-based detections of occupancy within the region based on state equations that model occupancy of the region;

detecting human head-tops by locating circles in grayscale images by back-propagating the gradients in the image and applying thresholds on accumulated points thereby detecting circle centres wherein the points of the potential centre candidates are accumulated in a 3-D accumulator and after accumulation, a suitable threshold is applied to each 2-D (x,y) plane in accumulator array to detect circles of corresponding radii.

27. Use of method according to any of the claims 8-24, for modelling of cost effective and power efficient in^¬ door/outdoor scenarios .