EP2174260A2

EP2174260A2 - Device for identifying and/or classifying movement patterns in an image sequence of a surveillance scene, method and computer program

Info

Publication number: EP2174260A2
Application number: EP08760696A
Authority: EP
Inventors: Julia Ebling; Hartmut Loos; Matthias Koenig
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2007-07-05
Filing date: 2008-06-06
Publication date: 2010-04-14
Also published as: WO2009003793A3; WO2009003793A2; DE102007031302A1; US8761436B2; US20100177936A1

Abstract

The invention relates to video surveillance systems that are used, for example, for surveying public places, stations, streets, industrial estates, buildings or similar. Said video surveillance systems comprise one or more surveillance cameras that are oriented towards surveillance scenes and transfer image data streams, in the form of image sequences, to an evaluation centre. The invention also relates to a device (1) for identifying and/or classifying a movement pattern in an image sequence of a surveillance scene comprising a plurality of moving objects, an interface (2) for recording the image sequence, a calculation module (5) for determining an optical flow field (10) in the surveillance scene by evaluating the image sequence and an identification module (6) that is designed in terms of programming and/or circuitry such that the optical field and/or partial areas thereof are compared to one or more patterns in order to identify the movement pattern in the image sequence.

Description

description

title

Device for detecting and / or classifying movement patterns in one

Image sequence of a surveillance scene, process and computer program

State of the art

The invention relates to a device for detecting and / or classifying movement patterns in an image sequence from a surveillance scene, wherein the

Device has an interface for recording the image sequence and a calculation module for determining an optical flow field in the surveillance scene by evaluating the image sequence. Furthermore, the invention relates to a corresponding method and a computer program.

Video surveillance systems are used, for example, to monitor public places, railway stations, roads, industrial plants, buildings or the like. The video surveillance systems usually comprise one or more surveillance cameras, which are aligned to surveillance scenes and transmit image data streams in the form of image sequences to an evaluation center. While it was common in the past, the

In the meantime, automated image evaluations based on digital image processing are increasingly being carried out by trained monitoring personnel. The advantages of such automated evaluations are in particular that on the one hand the personnel costs are significantly reduced and on the other hand, the monitoring quality is always consistent.

An often practiced approach to detecting and / or classifying motion patterns in surveillance scenes is to separate moving objects from the (essentially static) scene background, track over time, and raise alarms for relevant movements. The methods used in the The first step for the so-called object segmentation is typically the image differences between the current camera image and a so-called scene reference image that models the static scene background to detect moving objects.

By contrast, another approach analyzes the so-called optical flow in the surveillance scene by evaluating the image sequence. In the calculation of the optical flow, the translational movements of pixels or image regions are evaluated from an image to a subsequent image of the image sequence and a vector field is created on the basis of these translational movements, which respectively provides a translational direction and a translation speed for the examined pixels or regions. In this approach, an object segmentation is performed by classifying objects that correspond in terms of the translation vectors, that is to say with regard to the optical flow, to a common object.

Disclosure of the invention

According to the invention, a device for detecting and / or classifying movement patterns in a sequence of images from a surveillance scene with the features of claim 1, a corresponding method with the features of

Claim 12 and a computer program with the features of claim 14 proposed. Preferred or advantageous embodiments of the invention will become apparent from the dependent claims, the following description and / or the accompanying figures.

In contrast to the known prior art, the device according to the invention is characterized in particular in that the movement patterns are not necessarily recognized and classified via the intermediate path of object segmentation and subsequent object tracking, but rather that the optical flow field itself is based on characteristic movement patterns by comparison with predetermined ones and / or predeterminable patterns, in particular flow masks. While a single object approach is thus pursued in the prior art, the invention proposes a multiobject approach which simultaneously analyzes all moving objects in the surveillance scene and / or in a subarea of the surveillance scene. The device is preferably designed as a video surveillance system, which is implemented, for example, as a data processing device, computer, DSP, microcontroller, FPGA, ASIC or the like and / or has one or more interfaces for recording image sequences of one or more surveillance scenes. The interfaces are with a plurality of cameras,

Connected and / or connectable to recorders and / or other storage media on which the image sequences are provided or can be provided. The inventive device is particularly suitable due to a comparatively low computing power requirement for real-time use, but can also analyze already stored image sequences in particular "off-line".

The surveillance scene can - as already shown in the introduction - be designed as a street scene, intersection, public building, station, etc. Preferably, the surveillance scene shows a plurality of moving objects, particularly human, e.g. Passers-by, but can also be designed as vehicles or the like. The image sequence comprises a plurality of in particular temporally equidistantly recorded images of the surveillance scene, wherein the images may have any specification, such as color, grayscale, black and white, infrared and / or UV image are realized.

The device according to the invention has a calculation module, which is formed by program technology and / or circuitry for determining an optical flow field in the surveillance scene by evaluating the image sequence. The optical flow field - also called optical flow - can be represented as a vector field having the preferably two-dimensional direction of movement and / or movement speed for each or every pixel selected, for each or each selected pixel, and / or for each or each selected one The calculation of the optical flow field can be done by differential methods, but also by any other known method.

According to the invention, it is proposed that the device has a recognition module, which is designed in terms of programming and / or circuitry, to compare the optical flow field and / or subregions thereof with one or more patterns in order to recognize a movement pattern in the image sequence. In particular, the recognition of the movement pattern takes place without the intermediate steps of object segmentation and tracking, so that on the one hand inaccuracies in the analysis can be avoided and, on the other, computer time can be saved. The proposed invention thus uses the information obtained by calculating the

Movement can be obtained from small image areas, where an image area can also consist of a single pixel. For the calculation of these optical flow fields, it is also not necessary to have a scene reference image since only the current camera images are used. Preferably, the evolution of the flow fields over time is also considered.

Another useful advantage of the invention is that the underlying method is applicable, even if there are so many moving objects in the surveillance scene that they overlap, merge and / or that one another object segmentation, e.g. via a scene reference image is not or no longer possible with the necessary reliability. Thus, the invention allows the detection of motion patterns in the image sequences even in overcrowded surveillance scenes. In particular, the invention is suitable for detecting movement patterns that are based on mass psychological and / or group dynamic behavior, that is to say in particular on the joint behavior of large ones

Crowds. Examples of such patterns of movement are confluence, flowing apart, flowing around a center, etc.

In a preferred embodiment of the invention, the movement pattern is formed as a global movement pattern which reflects the movement throughout

Surveillance scene describes. This embodiment is particularly suitable when the surveillance scene shows a crowd and a common behavior of the people in the crowd is to be classified through the analysis of the movement pattern.

In another alternative embodiment, the movement pattern is designed as a local movement pattern describing the movement in a subarea of the surveillance scene. The subarea may be set by a user in one possible alternative embodiment, such that, for example, one escalator is selected as a subarea or another Alternatively, alternative embodiments can be determined automatically, for example by selecting subregions in which more object movements are detected than other subregions of the surveillance scene. However, it is preferred that the sub-region of the surveillance scenes comprise a plurality of moving objects in order to best exploit the advantages of the invention.

In a further development of the invention, the device has a classification module which, taking into account the detected movement pattern or patterns of the surveillance scenes, assigns a monitoring situation. In particular, the device according to the invention undergoes an application-specific expression in that the

Monitoring scene is assigned as a real surveillance scene a real monitoring situation. In general, such real monitoring situations can be configured as desired, for example as a human snake, a regularly traveled road, a purpose-used roundabout, etc .. Thus, the invention is suitable real monitoring situations, preferably dangerous situations such as fallen persons, escape behavior, panic and situations which attract onlookers, especially in overcrowded surveillance scenes automatically detect and trigger a reaction, such as the transmission of an alarm call. Preferably, however, it is provided that the classification module has an arbitrary selection of the following assignments:

A movement pattern with flux lines directed at a common point, that is to say a convergence, is assigned to the monitoring situation "gathering." Such a monitoring situation arises, for example, when approaching moving objects, in particular people or onlookers, to a common center.

A movement pattern with flux lines directed away from a common point, ie a divergence, is assigned to the monitoring situation "scattering." Such a monitoring situation is created when moving objects flow apart, for example when a person escapes from a central point.

A movement pattern with flux lines running around a common point is assigned to the "avoidance" monitoring situation and is an example of one Monitoring situation, wherein the surveillance scene has an obstacle, which is bypassed by the moving objects.

In the event that the calculation module can detect with the existing patterns no or only an insufficient match in the pattern comparison, this is

Calculation module designed to determine statistical characteristics of the optical flow field. This embodiment is particularly useful when no structure or motion model is detectable in an optical flow field. The determination of statistical parameters checks whether the surveillance scene can be assigned to the chaos or panic monitoring situation. Even with recognized movement patterns, the device is preferably designed so that the detected movement patterns are confirmed or verizifϊziert with statistical or other characteristics.

In a preferred implementation of the device, the pattern is designed as a two- or three-dimensional data field. It may be provided that the size of the pattern corresponds to the size of an image of the image sequence. Alternatively, it may be provided that the size of the pattern is limited to the size of a subarea, in particular of the subarea of the image to be examined, and thus to the relevant surveillance scene. In one possible alternative, the pattern is as

Window formed with the optical flow field, for example sliding, is traversed.

In a preferred embodiment of the invention, the movement pattern is detected by applying a classification method. In such a

In the classification method, the optical flow field is compared with a first pattern in a first classification stage, with a second pattern in a second classification stage and with an nth pattern in an nth classification stage. A movement pattern is thus recognized by a plurality of positively classified individual patterns. For example, to save computation time, it may be provided that once a pattern of a classification stage can not be found in the optical flow field, this pattern is discarded. In further embodiments, the classification method is designed as a classification tree, wherein the branches of the tree are formed by classification stages.

After applying multiple patterns to the optical flow field as

As a result of an evaluation of the coincidence of the plurality of patterns with the optical flow field, it is preferable that the movement pattern is recognized on the basis of a plurality of found patterns. The assignment of the multiple patterns to a movement pattern can be implemented, for example, automatically by a learning process.

In one implementation, the patterns are formed as linear, shift-invariant patterns and / or the pattern matching is implemented via clifford convolution. The particular advantages of a Clifford convolution and the application to vector fields arise for example from the dissertation of the inventor Julia

Ebling, entitled "Visualization and Analysis of Flowfields based on Clifford Convolution", University of Leipzig, Diss.,.

Another object of the invention relates to a method for detecting and / or classification of a movement pattern or in an image sequence of one or the surveillance scene with the features of claim 12, which is preferably carried out on the device just described or on the device according to the preceding claims becomes.

In the method, in a first step, the optical flow field in the image sequence of the surveillance scene is calculated and in a second step the optical flow field is compared with one or more patterns in order to detect one or the movement pattern in the image sequence or in the surveillance scene. Another optional step is to assign a monitoring situation to the detected movement pattern.

A further subject relates to a computer program with program code means having the features of claim 15. Brief description of the drawings

Further features, advantages and effects of the invention will become apparent from the following description of a preferred embodiment and the accompanying figures. Showing:

Figure 1 is a block diagram of a video surveillance system as a first embodiment of the invention;

Figure 2 is a schematic representation as a first example of an optical

Flow field with a first movement pattern;

Figure 3 is a schematic representation as a second example of an optical

Flow field with a second movement pattern;

Figure 4 is a schematic representation as a third example of an optical

Flow field with a third movement pattern;

Figure 5 is a schematic representation as a fourth example of an optical flow field with a fourth movement pattern.

Embodiment (s) of the invention

FIG. 1 shows a schematic block diagram of a video surveillance system

1, which can be used for monitoring squares, buildings, intersections, traffic junctions and the like, and is designed, for example, as a computer system.

The video surveillance system 1 is connected via an interface 2 with one or more

Security camera (s) 3 and one or more VCR (s) 4 connected. The surveillance cameras 3 are positioned to monitor the said surveillance scenes and direct the image data stream either first for caching in the video recorder or 4 or directly in the video surveillance system. 1 Starting from the interface 2, the image data streams, which correspond to image sequences of the surveillance scene, are first transferred to a calculation module 5. The calculation module 5 is program-technically and / or circuit-wise designed to generate an optical flow field or a time-varying optical flow field from an image sequence of a surveillance scene. The optical

For example, the flow field is created for the translational movements of pixels or image regions in the images of the image sequence over at least two and preferably more than ten images. It can also be provided that the optical flow field is generated in a manner that is virtually sliding for the image sequence, in that in each case the oldest image from the calculation for the optical flow field is calculated for the calculation of the current optical flow field

Flow field is removed and replaced by the latest image of the image sequence (FiFo).

The generated optical flow field is passed to a recognition module 6, which is designed to compare the optical flow field with patterns or pattern masks from a pattern memory 7, wherein the patterns in the pattern memory 7 are designed such that movement patterns in the image sequence or in the surveillance scene can be recognized. For this purpose, the patterns in the pattern memory 7 can be designed as two- or three-dimensional data fields and have a size that is similar in size to an image of the image sequence or as a subregion of an image of the image sequence. For detecting a movement pattern in the optical flow field, a single pattern from the pattern memory 7 may be applied to the optical flow field, alternatively, multiple patterns from the pattern memory 7 are applied to the optical flow field and the motion pattern is recognized by classification procedures, decision trees, etc.

After a movement pattern has been detected, it is transferred to a classification module 8, wherein the classification module 8 is designed to assign the detected movement pattern to a specific real monitoring situation, such as flight, confluence, chaos, panic, etc. As soon as the assignment of the movement pattern to the monitoring situation has been completed, a message can optionally be sent to a monitoring personnel via a notification module 9.

The process running on the video surveillance system 1 thus comprises the steps of accepting images or image sequences of a surveillance scene, which Calculation of the optical flow field for the current image of the image sequence, optionally temporal filtering of the optical flow field to suppress interference and the determination of monitoring situations on the comparison of the optical flow fields or with predetermined flow masks or patterns that describe typical movement patterns, an analysis of calculated similarities, optionally a statistical analysis of the directions of movement for further information and to secure the results and finally the classification of the monitoring situation, where appropriate for different image areas.

FIG. 2 shows in a highly schematized representation a first example of an optical flow field 10 with a movement pattern such as occurs during the influx of persons, such as onlookers. As can be seen from the figure, all or most of the motion vectors 11 are directed to a common center 12, so that this movement pattern can be easily recognized automatically by the recognition module 6. For example, the movement pattern may be referred to as a convergent movement pattern. The movement pattern recognized by the recognition module 6 is then reported to the classification module 8, which assigns to this convergent movement pattern the monitoring situation "oncoming persons." After the influx of persons to a common center 12 is usually triggered by a not in the common center 12 has happened, such as an accident or the like, the classification of the converging movement pattern leads to the triggering of an alarm in the notification module 9.

FIG. 3 shows a second exemplary embodiment of a schematic optical system

Flow field 10, which has a movement pattern that corresponds to the flow of people, such as happens when escaping. All or most of the flow vectors 11 are star-shaped away from the common center 12 in this embodiment. The recognition module 6 will recognize this movement pattern, for example, as a divergent movement pattern and to the

Pass the classification module 8. The classification module 8 will assign the monitoring situation "escape" to this divergent movement pattern and instruct the notification module 9 to trigger an alarm. Figure 4 shows a third example of an optical flow field 10, this third example showing the movement pattern of persons circulating an obstacle. Here, two flows are produced for the flow vectors 11, wherein the flow arranged on the left side of FIG. 4 runs from top to bottom and the flow arranged on the right side runs from bottom to top. The common center 12 is bypassed by both streams. After the classification module 8 has received this movement pattern, it will assign the monitoring scene the monitoring situation of an obstacle that has occurred and also trigger an alarm in the notification module 9.

Finally, FIG. 5 shows a last exemplary embodiment of an optical flow field 10, wherein the flow vectors 11 have no uniform direction and no common center can be detected. In the transfer of such a movement pattern, which may be denoted as a chaotic movement pattern, or the message that no movement pattern was found, the classification module 8 will assume a panic situation as a monitoring situation and also activate the notification module 9.

In all of the movement patterns shown in FIGS. 2 to 5, it is possible to improve the quality of the results and the classification of the monitoring situation by additionally performing statistical evaluations of the flow vectors 11 and checking whether the monitoring situation to be classified is based, for example, on the movement speeds the flow vectors is logically compatible. Statistical hedging thus represents an additional review of the results.

Overall, a video-based surveillance system is presented with the invention, which can be used for the observation of complete scenarios, and it is particularly suitable for the observation of dangerous situations in crowded scenes, ie in scenes in which move many moving objects.

Claims

claims

A device (1) for detecting and / or classifying a movement pattern in an image sequence from a surveillance scene with a plurality of moving objects

with an interface (2) for recording the image sequence and

with a calculation module (5) for determining an optical flow field (10) in the surveillance scene by evaluating the image sequence,

marked by

a recognition module (6), which is designed in terms of programming technology and / or circuit technology, to compare the optical flow field and / or subregions thereof with one or more patterns in order to obtain the

Recognize movement patterns in the image sequence.

2. Device (1) according to claim 1, characterized in that the movement pattern is formed as a global movement pattern that describes the movements in the entire surveillance scene.

3. Device (1) according to claim 1 or 2, characterized in that the movement pattern is formed as a local movement pattern, which describes the movement in a partial area of the surveillance scene, wherein the partial area has and / or may comprise a plurality of the moving objects.

4. Device (1) according to one of the preceding claims, characterized by a classification module (7), which is designed to assign the monitoring scene of a particular real monitoring situation based on the detected movement pattern.

5. Device (1) according to claim 4, characterized in that the classification module (7) has an arbitrary selection of the following assignments:

a movement pattern with flux lines (11) directed to a common point (12) is assigned to the "gathering" monitoring situation;

a movement pattern with flux lines (11) directed away from a common point (11) is assigned to the monitoring situation "scattering";

- A movement pattern with around a common point (12) running flux lines (11) is assigned to the monitoring situation "Dodge".

6. Device (1) according to one of the preceding claims, characterized in that the calculation module (5) is designed to determine statistical characteristics of the flow field in the case of a missing or insufficient match in the pattern comparison.

7. Device (1) according to one of the preceding claims, characterized in that the pattern is designed as a two- or three-dimensional data field.

8. Device (1) according to one of the preceding claims, characterized in that the detection module (6) is designed to detect the movement pattern by applying a classification method.

9. Device (1) according to one of the preceding claims, characterized in that the detection module (6) is designed to detect the movement pattern based on several found patterns.

10. Device (1) according to one of the preceding claims, characterized in that the one or more patterns are formed as linear displacement invariant pattern.

11. Device (1) according to one of the preceding claims, characterized in that the detection module (6) is designed to implement the pattern comparison via a Cliffordfaltung.

12. Method (1) for recognizing and classifying a movement pattern or patterns, preferably using the device according to one of the preceding claims, comprising the steps:

- calculating one or the optical flow field (10) in one or the image sequence of one or the moving object surveillance scene;

- Comparison of the optical flow field (10) with one or more patterns to detect the movement pattern in the image sequence.

13. The method according to claim 12, characterized by the step: - Assignment of the detected movement pattern to a particular real

Monitoring situation.

A computer program comprising program code means for performing all the steps of the method of any of claims 12 or 13 when the program is on a computer and / or device of each

Any one of claims 1 to 11 is executed.