GB2282294A

GB2282294A - Method of detecting moving objects

Info

Publication number: GB2282294A
Application number: GB9417967A
Authority: GB
Inventors: Rudolf Mester; Michael Hoetter
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 1993-09-25
Filing date: 1994-09-07
Publication date: 1995-03-29
Anticipated expiration: 2014-09-07
Also published as: DE4332753C2; GB2282294B; GB9417967D0; DE4332753A1; FR2710433B1; FR2710433A1

Description

2282294 1 METHOD OF DETECTING MOVING OBJECTS

Prior art

The invention is based on a method of detecting moving objects of the type indicated in the main claim. A method of detecting moving objects in chronologically successive images in which however the image signal is compared with a predetermined reference image signal in order to detect a moving object is already known from J. Eliss et al "A knowledge-based approach to automatic alarm interpretation using computer vision, on image sequences", ICCST, 1989, page 215 ff. In addition detected displacement vectors are compared with predetermined displacement vectors for detecting a moving object.

Advantages of the invention In contrast to the above the method according to the invention with the features of the main claim has the advantage that the reference image signal, used for comparison, is adaptively updated for each region of the image in dependence on the evaluation result of an automatic image analysis. As a result thereof it is unnecessary to give different comparison thresholds for the image signal or different comparison movement vectors for different scenarios to be monitored. By means of the method according to the invention the comparison thresholds are automatically adapted to the different image regions owing to the image analysis used. This leads to greater reliability in the detection of moving objects irrespective of the area to be observed and of the meteorological influences.

Advantageous further developments and improvements to the method given in the main claim are possible as a result of the measures given in the subclaims. It is particularly advantageous to compare the image region of the actual image with the corresponding image region of the reference image in 2 relation to texture features, thereby increasing the reliability of the detection of a moving object.

In order to determine texture features it is particularly advantageous to use two adjacent pixels. A sufficient amount of data is thereby obtained and simultaneously the amount of data to be processed is small and can be processed rapidly.

The method has the advantage that in object detection the measured variables: image signal variation, displacement and texture, detected for a pair of images (reference image and actual image) are not compared with a fixed threshold which applies for the entire image but these thresholds are determined from a measurement of the chronological statistical distribution of these measured variables - described by the mean value and variance - performed separately for each image region.

The rapidity and reliability of the detection of moving objects is optimized in that firstly the image signal variation is used for detecting a moving object and, if a moving object is detected, the texture features are used for detecting the moving object and, if this comparison also enables a moving object to be detected, finally the displacement vector is considered for detecting the moving object and only when this comparison also produces a moving object is a moving object considered to have been detected with certainty and a signal is emitted for indicating the object alarm. As a result thereof a processing time which is as short as possible is attained in spite of the high degree of reliability in the detection of moving objects.

The comparison of the measured variables: image signal variation, displacement vector and texture feature, is performed simply and rapidly if the actual measured value of the measured variable of an image region minus the chronological mean value of this measured value is compared c i 1 3 with the product of the chronological standard deviation_ of the measured variable assessed with a weighting constant. A simple and rapid comparison of the measured variables for the object detection process is thus attained.

The assessment of the mean value and of the variance of the measured variables: image signal, displacement vector and texture features, are advantageously assessed with a recursive low-pass filter of the first order, a so-called infinite impulse respond low-pass filter. As a result thereof the mean value and variance are assessed simply.

Drawings The drawings show an embodiment of the invention which will be described in further detail in the following description. Figure 1 shows an arrangement for detecting a moving object, Figure 2 shows an image unit with clicks, Figure 3 shows a distribution density function for evaluating a measured variable, and Figure 4 shows a schematic programme sequence of the method.

Description of the embodiment

Figure 1 shows an arrangement for detecting a moving object in chronologically successive images. An image recording unit 1 is connected via a data line 7 to a computing unit 2. The computing unit 2 is in turn connected via a data line 7 to an input unit 3, via a further data line 7 to a memory 4, via an additional data line 7 to a screen 6 and via a further data line 7 to a signal display 5.

Figure 2 shows an image unit 31 which consists of 5x5 pixels 32 of an image 30 for example. So-called click features are used in the method described here in order to determine texture features. Thus a click comprises an arrangement of two locally adjacent pixels. The total of the squared differences between 4 r the image signals of the pixel pairs, of an image unit, determined by the click is used as the measured variable for the texture features. Three different types of clicks are used as clicks: a horizontal click 33, a diagonal click 34 and a vertical click 35.

Figure 3 shows a distribution density function P(vi) 40 of the measured variable vi for a typical scenario. The chronological mean value mvi of the measured variable vi is shown. The "typical" range of the measured variable lies in the value range of the chronological mean value mvi (cvi X Svi) ' Cvi designating a predetermined weighting constant and svi designating the chronological standard deviation of the measured variable vi. The value range 41 in which there is an "untypical" measured variable is shown by the hatched areas in Figure 4. An untypical measured variable is an indication of a moving object. The value range 42 in which there is a typical measured variable is shown by the unhatched area. A distribution density function for each measured variable (image signal variation, texture features and displacement vector) is stored for each image region in the memory 4.

Figure 4 shows a schematic sequence of the method for detecting a moving object. At programme point 11 the image recording unit I records an image and relays it to the computing unit 2 whereupon at programme point 12 the computing unit 2 stores an image signal for each pixel. Thus the image 30 is divided into image units 31. In this embodiment an image unit consists of an image area comprising 5xS pixels 32.

At the following programme point 13 the computing unit 2 compares the image signal variation between the recorded image and a reference image stored in the memory 4. The image signal variation is compared with a distribution function, of the image signal variation, valid for this image unit, which image signal variation is described by the mean value and variance and which reproduces a typical scenario. The image units whose C F image signal variations are untypical are marked by the computing unit 2 in a f irst binary mask and stored in the memory 4. A binary mask consists of a storage f ield containing a storage location for each image unit. This storage location is occupied by 0 or 1 in order to qualify it.

At programme point 14 the computing unit 2 determines texture features for each image unit according to a predetermined method stored in the memory 4 and stores these texture features in the memory 4. At programme point 20 the computing unit 2 determines whether the texture features determined for the image units are untypical texture features. The image units whose texture features are untypical are marked by the computing unit 2 in a second binary mask and the mask is stored in the memory 4.

Subsequently at programme point 15 the computing unit 2 determines for each image unit the displacement vector of the image unit from the comparison of the actual image with the reference image stored in the memory 4 and stores it in the memory 4. In a third binary mask the computing unit marks the image units displaying an untypical displacement factor and stores the third mask in the memory 4. At programme point 21 the enquiry is made as to whether the displacement vectors of the image differ from a typical displacement vector of the corresponding image unit.

If this is the case, programme point 22 then follows. At programme point 22 the computing unit 2 detects a moving object in the image for the image unit whose image signal variation, texture and displacement vectors are untypical by comparing the binary masks and displays it via the signal display 5. The system then branches to programme point 26.

At programme point 26 the reference image is reworked. If a moving object is detected, the computing unit 2 replaces the image signal of each image unit of the reference image by the 6 image signal of the image unit of the actual image. The system then branches to programme point 11, the next image is recorded and the programme sequence is run through again.

If the enquiry at programme point 21 produces the response that the displacement vectors of all the image units are typical, a new chronological mean value and a new chronological variance of the measured variables are determined according to formulae (3) and (4) for the actual description of the distribution function of the measured variables for each image unit of the image in programme point 18 for the displacement vectors, in programme point 17 for the texture features and in programme point 1G for the image signal variations in each case using the computing unit 2 with a low-pass filter of the f irst order according to formula (2) taking account of the actual measured variables: image signal variation, texture and displacement. In addition at programme point 16 the computing unit 2 replaces the image signal of the image units of the reference image by the image signal of the image units of the actual image whose displacement vector is equal to zero. The image signal of the image units whose displacement vector is not zero is unchanged.

At programme point 19 the mean value and the variance of the measured variables: image signal variation, texture and displacement vector, are stored in the memory 4 for each image unit and the mean values and variances stored hitherto for the image signal variation, texture and displacement vector are deleted.

In the case of this particular method the measured variables: image signal variation, texture and displacement vector, are determined for each pixel of an image unit. The computing time is optimized in that the measured variables are determined for one image unit in each case.

The image is processed particularly rapidly if, after programme point 13, at programme point 14, the computing unit 2 only 1 Q 7 determines texture features for the image units whose image L. signal variation is untypical. And at programme point 15 the computing unit 2 only determines displacement vectors for the image units whose image signal variation and texture features are untypical.

In the following a particular embodiment will be described with reference to the schematic programme sequence of Figure 4 and with the inclusion of Figures 1, 2 and 3. At programme point 11 the image recording unit 11, which in this particular case is in the form of a CCD camera, records an image in the form of an analogue video signal. A typical resolution of an image is 756x288 pixels, one image signal being recorded for each pixel. The analogue video signal is firstly low-pass filtered in the image. recording unit 1, digitized and relayed to the computing unit 2. At programme point 12 the computing unit 2 stores the actual image in the memory 4. The stored image is then undersampled by the computing unit 2, the horizontal undersampling factor being four and the vertical undersampling f actor being two in this particular embodiment. Thus 2x4 pixels are combined to form one pixel. The computing unit stores the undersampled image in the memory 4. The entire object detection process is performed on the undersampled image. This procedure considerably reduces the expenditure on evaluation and on the other hand only has a slight influence on detectability. If a moving object is detected, the image is immediately available with full resolution and can be evaluated for recognizing a culprit or checking an alarm.

In the event of operation in the dark or twilight, the lighting time is virtually extended to more than a half image interval of 1/50 of a second by a local accumulation of the pixels in the scanning raster of the video signal and by an accumulation of a plurality of chronologically successive images in order to increase the sensitivity of the camera. For this purpose actual image signal statistics are measured and contrast and average brightness of the camera image are determined from the 8 image signal statistics. If the lighting conditions are insufficient, the contrast is increased by local and chronological accumulation of the pixels of the video signal and the average brightness is followed up in an appropriate manner. In unfavourable lighting conditions these measures lead to an improvement of the image signal and thus to reliable object detection even in darkness or twilight.

The computing unit 2 then determines for each image signal of each pixel of the undersampled image the difference from the image signals of the pixels of a reference image stored in the memory 4. The image signals of the reference image are occupied by pre-determined initial values which the input unit 3 gives to the computing unit 2. After a few images the image signals of the reference image are occupied by values which correspond to a typical scenario. In this embodiment the squared pixel differences between the undersampled actual image and the reference image are accumulated in order to acquire chronological image signal variations for each image unit. This sum of squares is used as a measure for describing the chronological image signal variation of an image unit.

During object detection, i.e. when a moving object is detected, it is examined whether the measured variables (image difference signal, texture feature, displacement vector) actually determined describe a typical or atypical scenario. This evaluation is based on a statistical analysis of the measured variables, a chronological statistical distribution of the measured variables being determined for each image unit. When a measured variable is evaluated, it is checked whether or not the actual measured value of the distribution of a typical measured variable is sufficient. Since the measurement of the distribution function is expensive, the mean value and variance are assessed in order to describe the distribution. This assessment is always performed when no object alarm has been detected and thus the image unit corresponds to a typical scenario. In this way the mean value and variance describe the 9 r, distribution of the measured variables in typical scenarios. If an actual measured value of a measured variable differs greatly from this description, this event is indicative of an atypical scenario, i.e. of a moving object.

The decision as to whether a measured variable differs greatly (significantly) from its statistical distribution is made by means of a threshold value decision. The following formula is used in this respect:

(V,(Xl y) _ Mv,(X' y))2/(CV, X Sv,(X' y))2:, 1(1) where Cvi vj (X, Y) M,i (X, Y) svi (X, Y) weighting constant [0<cvj<50 in dependence on the measured variables examined] actual measured value of the measured variable vi at the local position (x, y) chronological mean value of the measured variable vi at the local position (x, y) chronological standard deviation of the measured variable vi at the local position (X, Y) - The weighting constant c,i thus depends on the measured variable examined in each case. It is a measure as to the number of multiples of the standard deviation by which the actual measured value may differ from the assessed mean value of the measured variable. The greater this weighting constant, the more greatly the actual measured value must differ from the assessed mean value in order to trigger an object alarm.

Figure 3 shows the inequation (1) in graph form. The distribution density function P(vi) of a measured variable vi at a local position (x, y) of the camera image is described by the two characteristic quantities mvi (mean value) and svi (standard deviation). If the actual measured variable vi lies within the interval [_t Vil +tlil (eg. vi(')), then the probability density p(vi(l) for the measured value vi(l) is greater than a required threshold Pmin. This threshold Pmin is the probability density of the measured value vi for which the following condition applies:

(V, _ Mv,) 2 = (C, X S,) 2.

If the actual measured value vi lies outside the interval [t,i, +t,il (eg. vil (2)), the object alarm is triggered. The residual error probability, i.e. the probability that an "atypical" scenario has erroneously been indicated, although the actual measured value of the distribution is sufficient, is the integral of all the possible measured values outside the interval E-tvi, +t,il and is shown by the hatched areas in Figure 3.

In order to be able to make a threshold value decision, the necessary chronological statistical characteristic quantities, mean value and variance, are assessed separately for all the image units and measured variables.

This assessment is with an IIR (Infinite Impulse Response) low-pass filter of the first order according to the following formulae:

mvi (x, y, t): rvi (x, y) + (1-r) mvi (x, y, t-1) (2), pvi (x, y, t): rV,2 (X ' y) + (1 - r) p,i (x, y, t - 1) (3) and Svi 2 (X 1 y 1 t): pv, (_X 0 y, t) _ MV,2 (X, y 1 t) where:

Mvi (X, Y, t) ' Mvi (X, Y, t - 1):

P,i (X, Y, t) ' P,i (X, y, t - 1) (4) chronological mean value of the measured variable vi at the moment t or tl; chronological capacity of the measured variable vi at the moment t or t-l; r, 11 11 2 i (X, Y, t) vj (X, Y) chronological variance of the measured variable vi at the moment t; value of the measured variable vi at the moment t at the local position (X, Y); r:recursive factor of the IIR low-pass filter.

This filter is distinguished in that the assessment of the respective characteristic quantity per detector cell only requires a few computer steps (operations) and can be performed with only one additional memory in each case per measured variable and characteristic quantity.

The mean response time of the IIR low-pass filter (number of steps of a unit jump at the filter input until 0.9 times the amplitude of the input is at the filter output) varies in dependence on the recursive factor r set. This response time is a measure as to the minimum number of images which have to be processed until the statistical characteristic quantities have been determined sufficiently accurately. In principle, the lower the recursive factor, the longer the response time. The recursive factor determines whether the characteristic quantities determined acquire the long-term statistics of the measured variable (recursive factor very small) or whether medium or short-term variations of the measured variables have also been taken into account in the statistics (recursive factor as great as possible). In addition the description of the distribution of the measured variables is adapted to varying scenarios by the recursive chronologically continuous assessment of the characteristic quantities.

At programme point 13 chronological image signal variations between the reference image and the actual image are detected by a variation detection process. The image regions displaying atypical image signals are marked in the first binary mask.

12 r, Small image signal variations corresponding to a typical scenario, i. e. which are brought about by the background noise of the video sensor for example, are suppressed for object detection. In the embodiment described here the decision about a chronologically preceding image or image region is taken into account in addition to the image signal difference. According to whether an image region is detected as having varied or not during the preceding variation detection process, two different thresholds are used in the evaluation f or the actual image signal variation. This method is described in the unpublished patent application bearing the file number P 43 11 972.7.

The computing unit 2 checks whether the squared pixel differences of the image unit between the reference image and the actual image differ more from the mean value mi of the squared pixel differences of the corresponding image unit than the chronological variance s,i of the squared pixel differences multiplied by the predetermined weighting constant cvi stored in the memory 4, i.e. it is checked whether the inequation shown in formula (1) is satisfied. The computing unit marks the image units satisfying formula (1) as untypical in the first binary mask.

In programme point 14 the computing unit 2 examines whether the untypical image signal variations are caused by a variation of the image texture. In this respect a texture is described by click features. In the embodiment described here two directly locally diagonally adjacent pixels are used as clicks, as Figure 2 shows. By measuring diagonally adjacent pixels account is taken of local correlations in the diagonal direction.

By analyzing the texture the following scenarios can be detected and the object alarm thus suppressed:

diffused movements (eg fluttering of leaves, rain, snow), overall changes in brightness (day, night, shadows cast over large areas) and small camera movements (wobbling, shaking).

-1 13 r- The computing unit 2 determines the texture for each image unit. A click thus describes the local arrangement of two adjacent pixels. In this embodiment clicks in the form of diagonal clicks 34 are used. The sum of the squared differences of the pixel pairs, of an image unit, determined by the click are used as the measured variable for the texture features. This measured variable depends on the type of click used and on the local correlation of the image signal within an image unit. The computing unit 2 stores the texture features determined in the memory 4. Further clicks are horizontal clicks 33 and vertical clicks 35 which can be used for determining the texture.

At programme point 20 the computing unit 2 then examines the measured variables of the texture features according to formula (1). The image units whose texture features are untypical, i.e. which satisfy formula (1), are marked in the second binary mask.

At programme point 15 the computing unit 2 determines a displacement vector in order to describe movement data for each image unit. When assessing the displacement vector, it is checked whether an image unit of the actual image can be found again in its local environment in the preceding image. For this search suitable degrees of similarity are defined, for example the absolute or squared sum of the pixel differences of the image region determined by the image unit.

The local position of the image unit in the preceding image with the greatest similarity, i.e. with the lowest sum of the squares of the pixel differences for example, is then associated with the actual image unit by means of a displacement vector. The displacement vector determined is stored in the memory 4. The difference of the local position of an image unit is thus described in two chronologically successive images. A three step blockmatching algorithm which is described by T. Kogy, K. Linuma, A. Hirano, T. Iijiama and 14 T. Ishiguro in "Motion- compensated interframe coding for video conferencing", IEEE, National Telecommunications Conference, New Orleans, pp 65.3.1.-65.3.5., December 1981, is used to assess the displacement vectors in the method described here. The three step blockmatching algorithm is distinguished by low complexity and moderate computer expenditure. The system then branches to programme point 21.

At programme point 21 the enquiry is made for each image unit of the actual image as to whether the displacement vector corresponds to an untypical scenario, i.e. whether the measured variable for the displacement vector satisfies the inequation (1). If the measured variable of the displacement vector of all the image units does not satisfy the inequation (1), displacement vectors for a typical scenario exist and the system branches to programme point 18.

At programme point 18 the computing unit 2 determines a new chronological mean value, a new chronological capacity and a new chronological variance of the displacement vectors of the image units for the displacement vectors of the image units using formulae (2), (3) and (4). Subsequently at programme point 17 the computing unit 2 uses the texture features determined for each image unit in order to determine a new chronological mean value, a new chronological capacity and a new chronological variance of the texture features for each image unit according to formulae (2), (3) and (4).

At programme point 16 the computing unit determines a new chronological mean value, a new chronological capacity and a new chronological variance of the squared pixel differences for each image unit using the squared pixel differences determined and according to formulae (2), (3) and (4). The computing unit 2 then replaces the image signals of the image units of the reference image whose displacement vectors are zero by the image signals of the corresponding image units of the actual image.

is At programme point 19 the computing unit 2 replaces the previous chronological mean values, the chronological capacity and the chronological variance of the displacement vectors, the texture features and the pixel differences by the new chronological mean values, the new chronological capacity and the new chronological variance which the computing unit calculates according to formulae (2), (3) and (4).

At programme point 21 if a displacement vector of an image unit of the actual image satisfies the inequation (1), i.e. - on the basis of the described procedure in the acquisition of measured variables - if all the measured variables (image signal variation, texture and displacement vector) of the image unit in question describe an untypical scenario, the system branches to programme point 22. At programme point 22 the computing unit 2 detects a moving object and emits a signal by means of the signal display 5. At the same time the actual image is shown in full resolution on the screen 6. At the following programme point 26 the computing unit 2 replaces the reference image by the actual image.

It is thus achieved that the actual scene with the moving object describes the typical scenarios subsequently used. If the typical scenario changes, the reference image is adapted to the actual scenario by the object alarm. This adaptation by re- initialization is repeated until the object comes to a standstill or leaves the image.

The system then branches to programme point 11 and the programme is run through again.

The variation of the image signals of the reference image is decided on in dependence on the value of the displacement vectors. In this respect the following cases are differentiated between:

16 The displacement vector is zero, i. e. there is no directed movement in the image. Chronological image signal variations cannot be described by movement in the image region observed. Causes: many different movements within an image region, local variations in brightness.

The displacement vector is not zero: the object alarm is only triggered in the event of significant movement. In order also to be able to detect moving objects moving slowly and in a directed manner, the statisticalsignificance test of the movement description is also used in addition to object detection for re-working the reference image. In all image regions in which there is a displacement vector which is not zero but which corresponds to a typical situation the reference image is not ' re-worked, i.e. the reference image is unchanged in these image regions.

This procedure leads to the f act that small and directed movements accumulate chronologically during a movement assessment between the actual camera image and the reference image. In this way it is possible to distinguish between an intruder moving slowly through the image and a tree moving to and f ro in the wind since in the f irst case the movement accumulates and thus increases quantitatively and chronologically whilst in the second case the movement has no mean value since the tree is securely anchored in the ground.

A.

17 1% i.

Claims

1. Method of detecting moving objects in chronologically successive images of a sequence, an image being divided into image regions and f or each image region a comparison being made by a comparison of the image signal with a reference image and/or a displacement vector being determined from the comparison and the variation of the image signal and/or the value of the displacement vector being compared with a threshold and a moving object being detected from the comparison and a signal being emitted in the event of a moving object being detected, characterized in that, if a moving object is detected, the image signal is stored as the new image signal for the reference image and is used as the reference image for th( next comparison.

2. Method according to Claim 1, characterized in that the image region of the actual image is compared with the corresponding image region of the reference image in relation to texture features.

3. Method according to Claims 1 and 2, characterized in that the statistical distribution of the measured variables described by the mean value and variance - is used in the threshold value test of the measured variables.

4. Method according to Claim 3, characterized in that the mean value and the variance are determined separately for each image region and are used for the comparison.

S. Method according to Claims 3 and 4, characterized in that the difference between two locally adjacent pixels is used for determining texture features.

6. Method according to any one of Claims 3 to 5, characterized in that the image signal of the image unit is compared with the corresponding reference image signal; and in 18 t% I.

iL- that if the image signal variation lies a predetermined_ amount above or below a predetermined image signal threshold, the texture variation of the image unit of the actual image in comparison with the corresponding image unit of the reference image is compared with a predetermined threshold; and in that, if the texture variation differs by more than a predetermined value from the predetermined threshold, the displacement vector of the image region is determined in relation to the corresponding image region of the reference image; and in that finally a moving object is detected and a signal is emitted if the displacement vector is greater than a predetermined threshold.

7. Method according to any one of Claims 3 to 6, characterized in that the following formula is used as the threshold comparison for the measured variables: image signal variation and/or displacement vector and/or texture:

(V, (X, y) - rrL,, (X 1 y)) 2 / (CV, Sv, (X, y)) 2:. 1 # c,i designating a weighting constant, vi(x, y) designating the actual measured value of the measured variable vi at the local position (x, y), mvi (x, y) designating the chronological mean value of the measured variable vi at the local position (x, y), and svi (x, y) designating the chronological standard deviation of the measured variable vi at the local position (x, y).

8. Method according to Claim 7, characterized in that the mean value and the variance of the measured variables: image signal, displacement vector and texture, are assessed for each image region by means of a low-pass filter of the first order according to the following formulae:

M,i (X, Y, t) = rvi (x, y) + (1-r)mi(x, y, t-1), P,i (X, Y, t) = rV,2(X. y) + (1-r)pvi(x, y, t-1) and Svi 2 (X, y ' t) = P,i (X, Y, t) - mvi 2 (X, Y, t) mvi (x, y, t), mi(x, y, t-1) designating the chronological mean value of the measured variable vi at the moment t or t-1, pvi (x, y, t), pi(x, y, t-1) designating the chronological capacity of the measured variable vi at the moment t or t-1, k 19 S 2V, (x, y, t) designating the chronological variance - of the measured variable vi at the moment t, vi(x, y) designating the value of the measured variable vi at the moment t at the local position (x, y), and r designating a predetermined recursive factor of the IIR low-pass filter.

9. A method of detecting moving objects substantially as herein described with reference to the accompanying drawings.