EP1897032A1

EP1897032A1 - Method and image evaluation unit for scene analysis

Info

Publication number: EP1897032A1
Application number: EP06741041A
Authority: EP
Inventors: Martin Litzenberger; Bernhard Kohn; Peter Schön; Michael HOFSTÄTTER; Nikolaus Donath; Christoph Posch; Nenad Milosevic
Original assignee: Austrian Research Centers GmbH ARC
Current assignee: AIT Austrian Institute of Technology GmbH
Priority date: 2005-06-15
Filing date: 2006-06-14
Publication date: 2008-03-12
Also published as: CN101258512A; KR20080036016A; WO2006133474A1; AT502551A1; AT502551B1; CA2610965A1; US20080144961A1; JP2008547071A

Abstract

The invention relates to a method for scene analysis, whereby the scene or the objects in the scene and an optical sensor perform a relative movement and the scene information obtained is evaluated. According to the invention, the visual information of the scene is detected by the individual pixels of the optical sensor; the pixel co-ordinates of the established variations in intensity are determined; the temporisation of the established variations in intensity is determined; local amassment of the variations in intensity of the pixels is determined by means of statistical methods; the local amassments are evaluated in terms of the number and/or position thereof by means of statistical methods and data area clearing methods; the determined values are used as parameters of a detected scene region; at least one of the parameters is compared with a pre-determined parameter considered characteristic of an object; and when the pre-determined comparison criteria are fulfilled, the evaluated local amassment associated with the respective scene region is seen as an image of said object.

Description

Method and image evaluation unit for scene analysis

The invention relates to a method according to the characterizing part of claim 1 and to an image evaluation unit according to the preamble of patent claim 7. The invention relates to the processing of information which is recorded by means of optical sensors. The subject of the invention is a method based on a special optical semiconductor sensor with asynchronous, digital data transmission to a processing unit, in which special algorithms for scene analysis are implemented. The method provides selected scene content information which is evaluated and, for example, to control machines or installations, etc., can be used.

The sensors used asynchronously continue or release the preprocessed scene information in the form of signals, and this only if the scene undergoes changes or individual image elements of the sensors detect certain features in the scene. This principle considerably reduces the amount of data required compared with an image display, and at the same time increases the information content of the data by already extracting properties of the scene.

The scene capture with conventional digital image processing relies on the evaluation of image information provided by an image sensor. In this case, usually the image, pixel by pixel sequentially, in a predetermined clock (synchronously), read many times per second from the image sensor and evaluated the information contained in the data about the scene. Due to the large amounts of data and complex evaluation methods, this principle encounters the following difficulties, even when using correspondingly powerful processor systems:

1.) The data rate of digital transmission channels is limited and insufficient for some high-performance image processing tasks.

2.) Powerful processors have too high an energy consumption for many, especially mobile, applications. 3.) Powerful processors require active cooling. Systems that use such processors can therefore not be built compact enough for many applications. 4.) Powerful processors are too expensive for many applications.

According to the invention, these disadvantages are solved by the features cited in the characterizing part of claim 1. An image evaluation unit according to the invention is characterized by the features of the characterizing part of claim 7. With the procedure according to the invention, a rapid Processing of the signals and a correspondingly rapid detection of significant information in the scene under consideration. The statistical methods used provide an exact evaluation with regard to scene parameters of interest or recognition of objects. In the following the invention will be explained in more detail with reference to the drawings, for example. Fig. 1 shows schematically differences between the usual procedure and the procedure of the invention. 2 shows a diagram of an image evaluation unit according to the invention. FIGS. 3 a and 3 b, as in FIGS. 4 and 5, show the procedure according to the invention schematically on the basis of recorded images. In Fig. 1, the difference between the prior art and the procedure according to the invention is explained in detail. So far, the information or data supplied by an image sensor has been forwarded synchronously and, after a digital image preprocessing and scene analysis, the result has been passed on via the interface of the device (FIG. 1 a). According to the invention, the processing of the image signals of the optical sensor takes place in a specific manner, namely such that in the pixels of the optical sensor the brightness information recorded by a photosensor is preprocessed by means of an analog, electronic circuit. In general, it is noted that in one pixel the processing of the signals of several neighboring photosensors can be summarized. The output signals of the picture elements are transmitted via an interface of the sensor asynchronously to a digital data evaluation unit, in which a scene analysis is performed and the result of the evaluation is provided at an interface of the device (FIG. 1 b). Based on Fig. 2, the procedure of the invention is schematically illustrated. In this case, a scene is imaged on the image plane of the optical sensor 1 via an optical recording arrangement, not shown. The visual information is captured by the picture elements of the sensor and continuously processed in electronic circuits in the picture elements. This processing recognizes certain features in the scene content in real time. Features to be detected in the image content may include static edges, local intensity changes, optical flow, etc.

The detection of a feature is hereafter referred to as an "event." Each time an event occurs, the pixel generates in real time a digital output on the asynchronous data bus containing the address of the pixel and thus the coordinates in the frame where the feature is detected This date will be referred to as the "Address Event" (AE). It can In addition, further properties of the feature, in particular the time of occurrence, are encoded in the data. The sensor 1 sends this information to the processing unit CPU as relevant data via the asynchronous data channel. A bus controller 2 prevents data collisions on the transmission channel. In some cases, it may be advantageous to use a buffer memory 3, eg a FIFO, between sensor and processing unit to compensate for irregular data rates according to the asynchronous transfer protocol (FIG. 2).

The procedure according to the invention is based on the combination of the specially designed sensor, the data transmission and the statistical mathematical methods for data processing provided. The intended sensor detects changes in light intensity and therefore responds, e.g. on moving edges or light-dark boundary lines in a scene. The sensor tracks the changes in the photocurrent of a photosensor in each pixel. These changes are summed for each pixel in an integrator. If the sum of the changes exceeds a threshold, the pixel immediately sends this event asynchronously over the data bus to the processing unit. After each event, the value of the integrator is cleared. Positive and negative changes of the photocurrent are processed separately and generate events of different polarity (so-called "on" and "off" events). The sensor used does not generate images in the conventional sense. in the

However, for better understanding, two-dimensional representations of events are used below. For this, the events for each pixel within a time interval are counted. Picture elements without events are assigned a white pixel. Image elements with "on" or "off" events are displayed with gray or black pixels.

For the following explanations, terms are introduced to avoid confusion with terms from digital image processing:

AE frame is defined as the AE's stored in a buffer which have been generated within a defined period of time. AE-Bid is the representation of an AE frame in an image in which polarity and

Frequency of events can be assigned to colors or gray scale values.

Fig. 3 shows (a) a video image of a scene and (b) an AE image of the same scene produced by a sensor responsive to changes in light intensity. In the data processing unit CPU, the features from the scene are examined by means of static mathematical methods, and higher-quality, abstract information on the scene content is obtained. Such information can be eg the number of persons in a scene or the speed and distance of vehicles on a road.

It's easy to see that the amount of data is significantly less than in the original image. The processing of events requires less processing power and memory than with digital image processing and can therefore be much more efficient.

A room counter can be realized by having the image sensor e.g. is mounted on the ceiling in the middle of a room. The individual events are assigned by the processing unit corresponding square areas in the image field, which are approximately the size of a person. Simple statistical methods and a correction mechanism allow easy estimation of the area covered by moving objects. This is proportional to the number of people in the field of view of the sensor. The calculation effort for the number of people is low, so that this system can be implemented with simple and inexpensive microprocessors. If no people or objects move in the sensor's field of view, no events are generated and the microprocessor can switch to a power-saving mode, which significantly reduces system power consumption. This is not possible in state-of-the-art image processing systems because the sensor image has to be processed and searched for people at all times.

For a door people counter, the image sensor is mounted above the door or other entrance or exit of a room. The persons are not distorted in perspective and the AE's are projected onto axes (eg: vertical axes) when the persons pass through the observation area and are summed up in a histogram (FIG. 4). If a person moves under the sensor through the door, one or more maxima 1 running in the direction of movement can be detected in the histogram. By means of statistical weighting, the calculation of the maximum and the direction of movement can be made robust against disturbances. For each AE frame, the index of the histogram containing the largest number of events is determined and compared to the index of the last AE frame. As the index shifts, it is an indicator that the person is moving, and the likelihood of movement is increased. The probability increases until a threshold is reached. In this case the person is counted and both probabilities are reset to defined values. In this way it is possible for the system to distinguish incoming and outgoing persons and to increment or decrement a counter when persons enter or leave the room. Resetting both probabilities has proven to be beneficial in making the algorithm more robust when high activity is in the field of view. With the choice of negative Values introduce an artificial time constant to prevent double counting of people. Several people walking in parallel can be recognized by splitting the projection areas into different "tracks" along the direction of movement

Warn pedestrians. These warning lights flash around the clock and are often ignored by drivers, as they usually show no real danger. Intelligent sensors that only trigger a warning signal when a pedestrian crosses the street or approaches the protective route can help to improve traffic safety by paying more attention to warning lights. For automatic activation of warning lights on protection routes, an image sensor and a digital processor are used, which are able to monitor protective paths and their immediate surroundings and to detect objects (people, cyclists, ...) crossing the road. The proposed system of image sensor and simple digital

Processing unit is capable of segmenting and tracking the AE's of people and vehicles near and on the protection path in the data stream (Figure 5). The system recognizes the size and speed of the objects and allows them to be categorized into pedestrian and vehicle categories. 5 shows a scene taken by the sensor at two points in time, the corresponding AE images and the result of the mathematical-statistical evaluation which recognizes the individual objects and determines their direction of movement. After a certain period of observation, it is possible for the system to recognize the location and orientation of roads, footpaths and paths through the use of learning methods based on static concepts. As a result, it is then possible to be warned of any pedestrian who is moving towards the protective path or moving along the protective path. Pedestrians, e.g. Moving on footpaths parallel to the roadway does not trigger a warning due to their detected direction of movement.

Systems with simple sensors (e.g., infrared motion sensors) are only able to detect the presence of persons in the vicinity of the protection path, but not to detect their direction of movement and thus specifically warn pedestrians to move directly to the protection path.

Claims

Godfather:

1. A method for scene analysis, in which scene information is recorded with an optical sensor, wherein the scene or the objects in the scene and the optical sensor perform a relative movement and the scene information obtained is evaluated, characterized

- That the visual information of the scene is detected by the individual pixels of the optical sensor, wherein the individual pixels on detection of a predetermined threshold exceeding absolute intensity change or relative intensity change (contrast) of the received light output an output that is relevant for a relative movement between the recorded scene point and the optical sensor and / or for a scene content change,

that the locations or picture element coordinates of the detected intensity changes are determined or recorded,

- that the time dependence of the observed changes in intensity, in particular time and sequence, are determined or recorded,

- that with statistical methods, in particular averaging, histograms, focus, document or order-forming methods, filtering over time od. Like., Local clusters of intensity changes of the picture elements are determined

- that the local clusters with statistical methods, e.g. Weighting, threshold value with respect to number and / or position, data range correction methods or the like, in particular with regard to the temporal change of the clustering density and / or change of the local distribution, that the determined values are used as parameters, e.g. Size, speed,

Movement direction, shape od. Like., Are considered a detected scene area, - that at least one, preferably a number of, parameters (n) with at least one, preferably a number of, predetermined parameter (s) is compared

(to be), which is (are) regarded as characteristic of an object (and)

if the predetermined comparison criteria are met, the local cluster (s) associated with the respective scene area is / are considered to be a representation of this object.

2. The method according to claim 1, characterized in that the local clusters are examined with respect to line-related coherent on the recorded scene locomotive movement intensity changes and considered as coherently assessed or exceeding a predetermined amount exceeding intensity changes as a trajectory moving relative to the optical sensor object become.

3. The method of claim 1 or 2, characterized in that the change in the size of a local accumulation is interpreted as approaching the sensor or removing an object from the sensor.

4. The method according to any one of claims 1 to 3, characterized in that a temporal and / or spatial change in the structure of the clusters is regarded as characteristic of a particular feature of a scene area.

5. The method according to any one of claims 1 to 4, characterized in that monitors in each pixel, in particular due to changes in brightness entering change in the photocurrent and generated or integrated, that when a threshold value is exceeded, the pixel immediately a signal asynchronously to the processing unit and that in particular after each signaling a new start of the summation or integration takes place.

6. The method according to any one of claims 1 to 5, characterized in that positive and negative changes of the photocurrent detected separately or detected and processed or evaluated.

7. image evaluation unit for recording scene information, wherein the scene or the objects in the scene and the optical sensor perform a relative movement to each other and with an evaluation unit for the obtained scene information, wherein the optical sensor comprises picture elements that capture the visual information of the scene, in particular for carrying out the method according to any one of claims 1 to 6, characterized in that the individual pixels or pixels emit an output signal, which is relevant for a relative movement taken upon detection of a predetermined threshold exceeding absolute intensity change or relative intensity change (contrast) of the recorded light between the recorded scene point and the optical sensor and / or for a scene content change, - That a unit for determining the locations or picture element coordinates of the detected intensity changes and for determining the time dependence of the detected intensity changes, in particular time and sequence, is provided, - that a computing unit is provided or assigned to this unit, in the statistical methods, in particular Averaging, histograms, focus formation, document or order formation methods, filtering over time or the like, local accumulations of the intensity changes of the picture elements are determined, - that a valuation unit is provided with which the local accumulations with statistical methods, eg weighting, threshold value with respect Number and / or location, Datenbereichsbereinigungmethoden od. Like., Are evaluated, in particular with regard to the temporal change of the accumulation density and / or change of the local distribution, wherein the determined values are the parameters, eg size, speed indent, direction of movement, shape or the like, of a detected scene area,

that a comparison unit is provided or assigned to the evaluation unit that compares at least one of, preferably a number of, parameters with at least one, preferably a number of, predetermined parameters that are considered to be characteristic of an object (become) and

A computer program product having program code means stored on a computer readable medium for carrying out the method of any one of claims 1 to 6 when the program product is executed on a computer.