WO2011129739A1

WO2011129739A1 - Method for video surveillance of an area with behavioural analysis and computer system for carrying out the method

Info

Publication number: WO2011129739A1
Application number: PCT/SE2011/000068
Authority: WO
Inventors: Niclas WADSTRÖRMER; Jörgen AHLBERG; Bengt Gustafsson; Amritpal Singh
Original assignee: Totalförsvarets Forskningsinstitut; Images Systems Ab
Priority date: 2010-04-16
Filing date: 2011-04-14
Publication date: 2011-10-20
Also published as: SE534706C2; SE1000386A1

Abstract

This invention relates to a method and a computer system for video surveillance of an area with behavioural analysis of individuals and groups of individuals and for the generation of a situational picture of the whole area. The analysis is performed in three steps. Firstly, individuals are each analysed separately. One important feature is that this individual analysis is performed in two parts. Firstly, features required to allow an association with an individual known from a previous image to be made are extracted and the attempt at association is carried out. If this is not successful, the individual is identified as a new individual. Then, further relevant features to allow for the analysis of possible membership of a group, etc. are extracted. When the individual analysis is complete, groups of individuals are analysed on the basis of the result of the individual analysis. Finally, the whole area is analysed, resulting in a situational picture based on the result of the individual and group analysis and on features extracted to characterise the situation in the whole area.

Description

Method for video surveillance of an area with behavioural analysis and computer system for carrying out the method

This invention relates to a method for video surveillance of an area with behavioural analysis of individuals and groups of individuals and for the generation of a situational picture of the whole area. The term "individuals" as used in this patent refers not only to people, but also to other moving objects, such as, e.g. vehicles. The invention also includes a computer system for carrying out the method. The surveillance of geographical areas in which people move around for the purposes of detecting unlawful actions and behaviour is traditionally performed by security guards patrolling the area and intervening in the event of unlawful actions and behaviour. In order to extend the area that can be monitored by a security guard, it is known to install video cameras and other sensors capturing the whole area or parts of the area. Data from the sensors is displayed to an operator in a control centre. The operator can then monitor a larger area than that possible when patrolling. In the event of unlawful actions and behaviour or indications that activity of this kind may occur, a security guard is sent to the relevant spot to intervene. A large area with many cameras means that the operator has many video data streams to monitor. This means that it is difficult for the operator to maintain concentration for a long period of time. A human operator is much more capable of sometimes reading people's intentions and guessing what is going to happen.

However, human ability is limited when it comes to identifying vague but significant patterns in human movement behaviour over a long period of time or when many people are moving around in the area. In the event of an incident, it may be difficult for the operator to get an overview of the course of events affecting an area covered by several cameras. In this invention, a computer system is used to carry out the method according to the invention. The method gives a description of the situation in the area with respect to the behaviour and movement patterns of individuals and groups of individuals. The method detects and reports deviations from the normal situation. The method can also detect and report predefined changes in the situation. The method can be used, e.g. to monitor security in public places and then describe the situation with respect to illegal and disorderly behaviour such that a security guard should intervene. E.g. a pickpocket can be detected if the individual stays for a long time in the vicinity of crowds without doing what is normally done in the area, e.g. shopping in a shopping centre or departing from an airport. Pickpockets often operate in pairs or in threes and the method can detect that three people are consistently staying in the vicinity of one another without being so close that they seem to be together. The method can also detect relevant events indirectly, e.g. if nobody wishes to go into the vicinity of a certain person in situations where people would normally not avoid others.

The invention solves this problem of the video surveillance of an area, analysing events in this area and providing various types of alarm and assistance for an operator in that it is designed in the manner that will be clear from the following independent claims.

The invention will now be described with reference to the accompanying drawings, in which: Fig. 1 is a block diagram for the method, and

Fig. 2 is a block diagram for a behavioural analysis filter of Figure 1.

The system of the invention uses one or more video cameras supplying a sequence of images to a computer system. The computer system analyses the video sequences automatically image by image, preferably in a time-synchronised manner if several cameras are used. In addition to video cameras, different individuals can be located by means of other known sensors and the system can also use this information in its analysis. The analysis is improved if there is information about the individuals' membership of a group and definitions of classes of behaviour input into the system by an operator.

The result of the analysis is a situational picture of an area. It consists of a continuously updated, formalised description of the individuals captured by one or more video cameras and their movement behaviour. The result can be displayed as a descriptive text with characteristics of individuals, groups and the whole area and may moreover include possible class membership and possible anomalies. The result can be displayed to the operator or relayed to another system for further use.

It is an essential feature of the invention that the analysis is performed in three steps. Firstly, individuals are each analysed separately. One important feature is that this individual analysis is performed in two parts. Firstly, an attempt is made to associate each observation of an individual with an individual known from a previous image or, if this is not successful, the individual is identified as a new individual. Values for the features used to determine the state variables required to allow the association to be made are extracted and used to this end.

Further features for the various individuals are then extracted depending on the relevant state variables to allow for the analysis of possible membership of a group and the development of the situation in the whole area. When a new observation has been associated with a previously known individual, the state variables characterising the individual can be updated, this including various types of information about the previous history of the individual stored in the previous image. The result of the analysis is a description of the individuals moving around in the area. When the individual analysis is complete, groups of individuals are analysed on the basis of the result of the individual analysis. The analysis of groups is performed in the same manner as that for individuals with the difference that groups can either be detected automatically by way of some common feature in the form of a certain value for one or more state variables or specified by enumerating the individuals belonging to the group. The enumeration is done by an operator or by another part of the system. The classification of individuals in groups does not have to be mutually exclusive and one individual may belong to several groups. The result of the analysis is a description of the groups moving around in the area. Finally, the whole area is analysed, resulting in a situational picture based on the result of the individual and group analysis and on features extracted to characterise the situation in the whole area.

The said state variables are usually numerical, but may take the form of a descrip- tion in words when displayed. The same thing applies to events that can be described as numerical changes in one or more state variables, but can be described in words when displayed.

The value for each state variable is described by a probability distribution over the value range. It is thus possible to express different degrees of certainty when assessing which value applies. It is also possible to designate a certain value as most likely, which is a way of saying that a specific value applies.

The person skilled in the art knows how to select state variables on the basis of the situation to describe relevant features of the moving objects. Further information in this connection is given by A. Yilmaz, O. Javed and M. Shah, "Object tracking: a survey," ACM Computing Surveys, Vol. 38, No. 4, Article 13, Dec 2006, hereby incorporated by reference. There are several common examples of state variables for an individual: position, appearance, posture, bodily movement patterns and spatial movement, speed and direction. There are several common examples of state variables for a group:

individuals in the group, features and values defining the group, position and spatial movement, speed and direction. There are several common examples of state variables for the whole area: number of individuals, number of groups, spatial movement patterns and individual analysis.

Individual analysis (see individual analysis in Figure 1) consists in identifying, tracking and characterising the individuals. In this analysis, there is no reference to the individuals' possible belonging to any group. The detection, extraction and tracking of moving objects are well known to the person skilled in the art. Further information in this connection is e.g. given by A. Yilmaz et al. hereinabove and W. Hu, T. Tan, L. Wang and S. Maybank, "A survey on visual surveillance of object motion and behaviours," IEEE Trans, on systems, man and cybernetics, Vol. 34, No. 3, August 2004, hereby incorporated by reference.

The various parts of the individual analysis step in Figure 1 will now be described. The analysis is performed image by image and in two main parts. The first part consists of detection, extraction and tracking and results in further observations being associated with existing tracks. Tracks may moreover have been added and have disappeared. The second part of the analysis consists in extracting further features and characterising the state and events to update the characterisation of the individuals. Input data, designated 1 in Figure 1 , consists of a sequence of images processed one at a time. Output data, designated 2 in Figure 1 , consists of tracks and the characterisation of the individuals moving around in the scene.

Detection is performed in step D₀

Moving objects can be detected, e.g. by being separated from the background. Background detection of this kind is well known to the person skilled in the art.

Further information in this connection is given by K. Kim, T.H. Chalidabhongse, D. Harwood and L. Davis, "Real-time foreground-background segmentation using codebook model", Real-time imaging, Vol. 11 , No. 3, pp. 172-185, 2005, and C. Stauffer and W.E.L. Grimson, "Learning patterns of activity using real-time tracking", IEEE Trans, on pattern analysis and machine intelligence, Vol. 22, No. 8, August 2000, both hereby incorporated by reference.

Each observation of an object (or part of an object or several objects) is a connected area in the image indicated by means of a silhouette or in some other manner.

Detection is performed for each sensor separately and several different algorithms can be used for each sensor. The detection module produces a number of hypothe- ses for each moving object that can be found in the current image.

Extraction of features is performed in step X₀

Extraction of features required for tracking, such as, e.g. position, appearance and shape. The extraction may vary depending on which detection algorithm is used. The result must be compared with results from other extraction modules in the tracking module.

Tracking is performed in step T₀

Each observation of a moving object is associated with an individual (object track). In many cases, there is a one-to-one correspondence between observations and known individuals, but it can happen both that one individual is described by several observations and that several individuals are described by one observation. Individuals can be created and can disappear. An individual who has not been observed for a relatively long period of time or whom it is known will not return is removed. An observation that cannot be associated with an existing individual leads to the creation of a new individual. The individual is in a sense temporary until several observations have been made.

Extraction of further features is performed in step X,

Further features can be extracted when the detection is associated with a known individual. Changes over time are extracted if the individual includes data about previous observations. If the individual is classified, features relating to the class can be extracted, e.g. number of wheels in the case of a vehicle or height in the case of people. In the case of people, e.g. features describing bodily movement are extracted.

Individual behavioural analysis is performed in the IBF step

In the IBF (individual behavioural filter), the features of the individual are updated with another observation. Classification and anomaly detection are effected to detect undesirable and unusual events.

The result of the individual analysis is that the object moving around in the area is detected and tracked so that it is possible to see how the object is moving around in the scene. The behaviour of the object is characterised continuously so that an alarm can be given in the case of predefined events and in the event of anomalies.

Group analysis consists in identifying, tracking and characterising groups of individuals. A group is defined by enumerating individuals or by way of some common value for some feature. The features forming the basis for the classification in groups can be defined partly by the operator and partly by the analysis of the individuals.

The different parts of the group analysis step in Figure 1 will now be described. The analysis is performed in a similar manner to the analysis of individuals. Input data in this step consists of data from the individual analysis and the result is that further observations can be associated with tracks of the groups. Tracks of groups can be added and can disappear. Input data may also consist of user-defined groups, which in such a case must be input into the system together with the analysis of images. Detection of groups is performed in step D_g

Groups can be specified in that an operator or some other system enumerates the individuals belonging to each group. In this case, the detection of individuals belonging to the group is trivial. Groups can also be specified by way of some common feature provided by an operator or by some other system or the detection module can automatically identify groups of individuals having similar features.

Groups do not have to be mutually exclusive, and an individual can belong to several groups. Extraction of features of the group is performed in step X_g

When a group is identified, the features of the group required to associate the group with one of the known groups are extracted.

Tracking is performed in step T_g

Each observation of a group is associated with a known group or, alternatively, a new group is created.

Extraction of further features is performed in step X_h

When there is another observation with respect to the group, the relevant group features required to characterise the group and its behaviour are extracted.

Group behavioural analysis is performed in the GBF step

In the GBF (group behavioural filter), the features of the group are updated with another observation. Undesirable and unusual behaviour of the group is detected.

The result of the group analysis is that groups in the area are tracked and their behaviour is characterised continuously so that an alarm can be given in the case of predefined events and in the event of anomalies. The method also analyses the individuals moving around in the scene to identify any groups that may be individuals having a common aim, individuals displaying similar behaviour or individuals having a similar appearance, etc..

In addition to individual and group analysis and further information about the whole area, analysis of the whole area is performed. The various parts of the scene analy- sis step (see Figure 1) will now be described. Input data consists of output data from the individual and group analysis. This step is analogous to the other part of the individual and group analysis step. Firstly, features describing the whole area (the scene) are extracted and then the state variables describing the scene are updated. Extraction of features for the whole area X_s

When the state variables for individuals and groups have been updated, relevant features for the whole area required to characterise the situation in the area are extracted. Behavioural analysis of the whole area is performed in the SBF step

In the SBF (scene behavioural filter), features for the whole area are updated.

The result of the analysis of the whole area is a situational picture created on the basis of the individuals moving around in the area and the characterisation of their behaviour and the analysis of possible groups and their behaviour. It should be possible to tell from the situational picture, e.g. if there are several groups behaving in a menacing manner towards others or towards one another or if there are only separate individuals behaving suspiciously. The three filters (for individuals, groups and the whole area) used in behavioural analysis have the same design with four components each (see Figure 2). The contents of the three filters differ. Each filter has a method which calculates the current state on the basis of new observations and the current time and a method which predicts the current state at a time after the last observation. There is a method which classifies the current state either in predefined classes or in classes provided by current and previous data and a method which detects anomalies in the current state by keeping statistics about how often various states occur. The first part can be effected, e.g. by means of a Kalman filter, which must include the prediction of a subsequent measured value, the updating of the state and the calculation of the interval between two states.

The various parts of a behavioural filter will now be described. The filter contains two databases, one with a description of feature classes, designated 5 in Figure 2, which can be estimated from data or specified by the user. The other database, designated 6 in Figure 2, contains a description of normal values to allow for the detection of deviations from normal values of this kind. The description of normal values is estimated continuously from current data.

Prediction is performed in step P

The current state is a function of the time and of all or part of the incoming sensor data. In many cases, the state changes with time and if it is desired to have the current value at a time after the last observation, the current value has to be predicted on the basis of previous data and the current time. Updating of the state is performed in step T

The other part of the filter consists in characterising the current state with respect to dynamically or deterministically predefined classes. The classes may be formed in accordance with directly measurable features, but an explanatory description may also be provided by the definition of the class.

Classification is performed in step C

The current state is classified in step C in Figure 2, i.e. for each state vector, a calculation is made to determine to which class the state vector belongs. The definition of the classes is stored in the database 5 in Figure 2. The definition of the classes is either specified by another part of the system and input into the database on that basis (user-defined classes) or the classes are created in step K in Figure 2. The class definition can be updated for each new state vector that arrives and is then input into the database. The classes can be named to facilitate the interpretation of the current situation. If the classification is updated dynamically, any name provided must be checked when the classes are changed.

Data is classified to facilitate the interpretation of the current situation and to assist the operator with identifying connections between different individuals and events. The method can see connections that are difficult for an operator to detect. The values for the state variables provide more detailed information, but this can be too extensive for an operator to be able to interpret it. Classification is effected by any method known to the person skilled in the art, such as, e.g. K-means. An overview of this method is given in R.O. Duda, P.E. Hart and D.G. Stork, "Pattern Classification", 2^nd Edition, Wiley, 2000, hereby incorporated by reference. Anomaly detection is performed in step AD

The method detects states predefined as abnormal or states deviating from the norm in current sensor data. Normal behaviour can be established, e.g. by means of statistics relating to sensor data. E.g. it is possible to assume that a parameter is distributed normally and to estimate the parameters of the distribution from data. The person skilled in the art is familiar with anomaly detection of this kind. Examples of how anomalies in movements can be detected are given in O. Boiman and M. Irani, "Detecting irregularities in images and in video", hereby incorporated by reference.

The result in the form of the analysis of the invention is a situational picture of the area and can be displayed by a combination of charts, sensor data and textual descriptions. A chart or drawing can show the area being monitored with symbols for basic settings and the current state. The operator is advantageously allowed to select part of all of the information available. The chart can show, e.g. geographical boundaries and the position and range of the sensors. Video sequences can be shown in real time, as well as data from other sensors. Symbols for the current situation and the result of the analysis can be inserted into the data streams. An event log can show a list of events ordered chronologically or sorted in some other manner. Events may, e.g. take different forms in the event log on the basis of their priority to facilitate the work of the operator. It is also possible to allow the operator to freeze the image at a certain point in time and to give him the opportunity to go back in time to view the situation at an earlier point in time.

Claims

Claims:

1. Method for video surveillance of an area with behavioural analysis of individuals and groups of individuals and for the generation of a situational picture of the whole area, comprising,

for the said individuals, groups and area, determining the relevant state variables to describe the said individuals, groups and area and their behaviour and also how the said state variables are dependent on features that can be extracted from an image,

assigning a set of state variables of this kind to each of the said individuals, groups and area,

for each image in an image sequence, detecting the individuals appearing in the image and, by extracting values for relevant features, determining and storing the state variables required to allow an individual to be associated with a corresponding individual in any previous image in the image sequence, and, upon the analysis of an image in the image sequence, attempting to carry out the association for each individual in the image, which is either successful, in which case it is established that an individual is the same as an individual in a previous image, or unsuccessful, in which case the individual is identified as a new individual,

c h a r a c t e r i s e d i n t h a t

values for the features required to determine further relevant state variables for each said individual are extracted,

the said further state variables are determined and stored on the basis of these features and, if each said individual is associated with an individual in a previous image, on the basis of previous values for state variables for each individual,

individual behavioural analysis of each said individual is carried out, then the individual is classified and anomaly detection is performed,

the results of the behavioural analysis for each said individual are used as

starting values for group analysis,

for each image in an image sequence, the groups appearing in the image are detected and, by extracting values for relevant features, the state variables required to allow a group to be associated with a corresponding group in any previous image in the image sequence are determined and stored, upon the analysis of an image in the image sequence, an attempt is made to carry out the association for each group in the image, which is either successful, in which case it is established that a group is the same as a group in a previous image, or unsuccessful, in which case the group is identified as a new group,

values for the features required to determine further relevant state variables for the said groups are extracted,

the said further state variables for the said groups are determined and stored on the basis of these features and, for those groups associated with groups in a previous image, on the basis of previous values for state variables for these groups,

group behavioural analysis of each said group is carried out, then the group is classified and anomaly detection is performed,

the results of the behavioural analysis for each said individual and the results of the behavioural analysis for each said group are used as starting values for behavioural analysis of the whole area,

values for the features required to determine state variables required in addition to the state variables for individuals and groups to update the situational picture of the area are extracted, and

the state variables describing the situational picture are updated on the basis of the state variables for individuals and groups and on the basis of the state variables determined especially to update the situational picture and this situational picture is presented.

2. Computer system for video surveillance of an area with behavioural analysis of individuals and groups of individuals and for the generation of a situational picture of the whole area, comprising that,

for the said individuals, groups and area, the computer system determines the state variables identified as relevant to describe the said individuals, groups and area and their behaviour and also how the said state variables are dependent on features that can be extracted from an image,

the computer system assigns a set of state variables of this kind to each of the said individuals, groups and area,

for each image in an image sequence, the computer system detects the individu- als appearing in the image and, by extracting values for relevant features, determines and stores the state variables required to allow an individual to be associated with a corresponding individual in any previous image in the image sequence, and,

upon the analysis of an image in the image sequence, the computer system attempts to carry out the association for each individual in the image, which is either successful, in which case it is established that an individual is the same as an individual in a previous image, or unsuccessful, in which case the individual is identified as a new individual,

c h a r a c t e r i s e d i n t h a t

the computer system extracts values for the features required to determine further state variables identified as relevant for each said individual, the computer system determines and stores the said further state variables on the basis of these features and, if each said individual is associated with an individual in a previous image, on the basis of previous values for state variables for this individual,

the computer system carries out individual behavioural analysis of each said individual, then classifies the individual and performs anomaly detection, the computer system uses the results of the behavioural analysis for each said individual as starting values for group analysis,

for each image in an image sequence, the computer system detects the groups appearing in the image and, by extracting values for relevant features, determines and stores the state variables required to allow a group to be associated with a corresponding group in any previous image in the image sequence, upon the analysis of an image in the image sequence, the computer system attempts to carry out the association for each group in the image, which is either successful, in which case it is established that a group is the same as a group in a previous image, or unsuccessful, in which case the group is identified as a new group,

the computer system extracts values for the features required to determine further relevant state variables identified as relevant for the said groups, the computer system determines and stores the said further state variables for the said groups on the basis of these features and, for those groups associated with groups in a previous image, on the basis of previous values for state variables for these groups,

the computer system carries out group behavioural analysis of each said group, then classifies the group and performs anomaly detection, the computer system uses the results of the behavioural analysis for each said individual and the results of the behavioural analysis for each said group as starting values for behavioural analysis of the whole area,

the computer system extracts values for the features required to determine state variables required in addition to the state variables for individuals and groups to update the situational picture of the area, and

the computer system updates the state variables describing the situational picture on the basis of the state variables for individuals and groups and the state variables determined especially to update the situational picture and presents this situational picture.