WO2005098751A1

WO2005098751A1 - Crowd detection

Info

Publication number: WO2005098751A1
Application number: PCT/IL2005/000382
Authority: WO
Inventors: Pinchas Reisman; Ofer Mano; Shmuel Avidan; Amnon Shashua
Original assignee: Mobileye Technologies Limited
Priority date: 2004-04-08
Filing date: 2005-04-07
Publication date: 2005-10-20

Abstract

A method of determining the presence of an ensemble of moving objects in an environment comprising: acquiring a plurality of images of a scene in the environment; processing the images to determine optic flow of features in the scene; and determining whether an ensemble of moving objects is present in the environment responsive to the optic flow.

Description

CROWD DETECTION RELATED APPLICATIONS The present application claims benefit under 35 U.S.C. 1 19(e) of US Patent application 60/560,048 filed on April 8, 2004, the disclosure of which is incorporated herein by reference. FIELD OF THE INVENTION The present invention relates to methods of detecting an ensemble of moving objects, such as for example, a crowd of moving people. BACKGROUND OF THE INVENTION Automotive accidents are a major cause of loss of life and dissipation of resources in substantially all societies in which automotive transportation is common. It is estimated that over 10,000,000 people are injured in traffic accidents annually worldwide and that of this number, about 3,000,000 people are severely injured and about 400,000 are killed. A report "The Economic Cost of Motor Vehicle Crashes 1994" by Lawrence J. Blincoe, published by the United States National Highway Traffic Safety Administration, estimates that motor vehicle crashes in the U.S. in 1994 caused about 5.2 million nonfatal injuries, 40,000 fatal injuries and generated a total economic cost of about $150 billion. The damage and costs of vehicular accidents have generated substantial interest in collision warning/avoidance systems (CWAS) that detect potential accident situations in the environment of a driver's vehicle and alert the driver to such situations with sufficient warning to allow him or her to avoid them or to reduce the severity of their realization. In relatively dense population environments typical of urban environments, it is advantageous for a CWAS system to be capable of detecting and alerting a driver to the presence of a person or persons in the path of a vehicle. Systems that may be used to detect the presence of one or a few pedestrians in the path of a vehicle exist. An article, "Pedestrian Detection Using Wavelet Templates"; Oren et al Computer Vision and Pattern Recognition (CVPR) June 1997 describes a global shape-based detection system for detecting presence of a person. The system uses Haar wavelets to represent patterns in images of a scene and a support vector machine classifier to process the Haar wavelets to classify a pattern as representing a person. A component based detection system for detecting a person is described in "Example Based Object Detection in Images by Components"; A. Mohan et al; IEEE Transactions on Pattern Analysis and Machine Intelligence; Vol 23, No. 4; April 2001. The disclosures of the above noted references are incorporated herein by reference. However, as the density of people in the path of a vehicle increases and the people become a crowd, such as for example, as often occurs at a zebra crossing of a busy street corner, cues useable to determine presence of an individual often become masked and obscured by the commotion of the individuals in the crowd. As a result, system for detecting one or a few human forms exhibit a tendency to become "confused" by the presence of a crowd in the path of a vehicle and for such situations, their ability to provide reliable detection of pedestrians can be compromised. SUMMARY OF THE INVENTION An aspect of some embodiments of the present invention relates to providing a method for determining whether an ensemble of moving objects is present in an environment. An aspect of some embodiments of the invention relates to determining the presence of the ensemble responsive to optical data acquired for the environment. In accordance with an embodiment of the invention, the optical data comprises patterns of optic flow generated responsive to optical data comprised in a sequence of camera images of a scene in the environment. The inventors have determined that in a sequence of camera images of a scene, an ensemble of moving objects in the scene will often generate patterns of optic flow having characteristics associated with the presence of the ensemble in the scene. These characteristics are usually sufficiently correlated with the ensemble so that a degree to which the characteristics are expressed in the sequence of images may be used to indicate the presence or absence of the ensemble in the scene. According to an aspect of some embodiments of the invention, the sequence of camera images is acquired by a camera moving towards the environment and a degree to which the images exhibit inward optic flow is used to determine whether or not the ensemble is present. In an embodiment of the invention, optical data in the sequence of camera images is represented as a function of coordinates in a space-time (ST) volume defined by the images.

An ST volume is an optionally rectangular volume defined by arraying the images parallel to each other and aligned one behind the other in the order in which they were acquired. A location of a given pixel in the images is determined by a time coordinate and two spatial "image" coordinates. The time coordinate is measured along a t-axis perpendicular to the planes of the camera images. The two spatial image coordinates are measured along spatial axes parallel to the planes of the camera images, which are conventionally x and y orthogonal image axes. Planes parallel to the xt-plane of an ST volume are referred to as epipolar or EPI planes. The x and y image coordinates of a pixel in a camera image acquired at a given time t, as measured along the t-axis, correspond to "real world" x and y-coordinates of a feature in the scene imaged on the pixel at the time t. Pixels in the ST volume that image a same feature in the scene at different times t, trace out a line, hereinafter referred to as an "image trajectory", in the ST volume. Image trajectories of features in a scene at a constant distance from a moving camera are located in an EPI plane. Hereinafter, to distinguish camera image coordinates from real world coordinates, real world coordinates are represented by capitalized letters, while camera image coordinates are represented by small letters. Optionally, the image x-axis and y-axis are defined to correspond respectively to real world X and Y-axes so that for a displacement of the camera along the positive world X-axis or Y-axis, a feature in a camera image corresponding to a stationary feature in the real world displaces along the positive image x-axis or positive image y-axis respectively. Optionally, the origin of world coordinates is assumed to be at the optical center of the camera. For images of a scene acquired by a land based camera, conventionally, the world X-axis is a horizontal axis parallel to the ground, the world Y-axis a vertical axis perpendicular to the ground and the world Z-axis coincides with the optic axis of the camera. If the camera is moving in the direction of its optic axis, it is moving along the positive Z-axis. As the camera moves towards the environment, stationary features in the environment that are imaged in the sequence of camera images are characterized by "outward" optic flow, away from the optic axis of the camera. In the ST volume, their corresponding image trajectories move outward, away from a line, hereinafter the "ST axis", in the ST volume corresponding to the camera optic axis. However, an ensemble of objects moving in different directions in the environment generally provides a plurality of features in the sequence of images that exhibit "inward^'' optic flow towards the optic axis. The corresponding image trajectories of the features in the ST volume move inward, towards the ST axis. In accordance with an embodiment of the invention, a measure of a degree to which image trajectories in the ST volume exhibit inward optic flow is used to indicate presence of the ensemble in the environment. The inventors have noted that image trajectories associated with an ensemble of moving object in an environment often exhibit a relatively high incidence of intersecting image trajectories. For example, a crowd of people crossing a street in both directions at a zebra crossing will produce many image trajectories that cross each other as people pass one another and randomly occlude one another in the sequence of images. In accordance with an embodiment of the invention, a measure of the frequency with which image trajectories in the ST volume intersect is used to indicate presence of the ensemble in the environment. In some embodiments of the invention, the environment is an automotive environment and the ensemble is a crowd of people. Optionally, methods in accordance with an embodiment of the invention are used in a CWAS comprised in a moving vehicle to determine whether a crowd is present in the path of the vehicle. There is therefore provided in accordance with an embodiment of the present invention, a method of determining the presence of an ensemble of moving objects in an environment comprising: acquiring a plurality of images of a scene in the environment; processing the images to determine optic flow of features in the scene; and determining whether an ensemble of moving objects is present in the environment responsive to the optic flow. Optionally, the method comprises determining a degree to which the optic flow exhibits inward optic flow. Optionally, the method comprises determining whether the ensemble is present responsive to the degree of inward optic flow. In some embodiments of the invention, determining optic flow comprises determining image trajectories of features in the scene and using the image trajectories to determine optic flow. Optionally, determining image trajectories comprises determining image trajectories that lie in at least one EPI plane of a space time volume defined by the images. Optionally, the at least one EPI plane comprises a plurality of planes. In some embodiments of the invention the method comprises determining a degree to which the image trajectories intersect. Optionally, the method comprises determining whether the ensemble is present responsive to the degree to which the image trajectories intersect. In some embodiments of the invention, the images are acquired under conditions for which stationary features in the environment exhibit outward optic flow. In some embodiments of the invention, the images are acquired by a camera mounted to a vehicle. Optionally, the vehicle is an automotive vehicle. In some embodiments of the invention, the ensemble of moving objects is a crowd of people. There is further provided in accordance with an embodiment of the invention apparatus for detecting presence of a crowd of people in an environment comprising: a camera that acquires images of a scene in the environment; and a processor that processes the images to determine presence of a crowd of people in accordance with an embodiment of the invention. Optionally, the Apparatus according is adapted to be mounted in a vehicle. Optionally, the vehicle is an automotive vehicle. BRIEF DESCRIPTION OF FIGURES Non-limiting examples of embodiments of the present invention are described below with reference to figures attached hereto, which are listed following this paragraph. In the figures, identical structures, elements or parts that appear in more than one figure are generally labeled with a same numeral in all the figures in which they appear. Dimensions of components and features shown in the figures are chosen for convenience and clarity of presentation and are not necessarily shown to scale. Fig. 1A schematically shows a vehicle comprising a CWAS moving along a road in an urban environment and image trajectories of features in the environment in an ST volume defined by a sequence of images acquired by the CWAS in accordance with an embodiment of the invention. Fig. IB shows a plan view of the environment and vehicle shown in Fig. 1A; Fig. 1C schematically shows a plan view of an epipolar (EPI) plane through the ST volume and image trajectories that lie in the plane, in accordance with an embodiment of the invention; Fig. 2A schematically shows the vehicle and urban environment shown in Fig. 1A with the addition of a crowd of people present in the path of the vehicle and image trajectories associated with the crowd in an ST volume defined by a sequence of images acquired by the CWAS, in accordance with an embodiment of the invention; Fig. 2B shows a plan view of the environment and vehicle shown in Fig. 2A; Fig. 2C schematically shows a plan view of an EPI plane through the ST volume and image trajectories associated with the crowd that lie in the plane, in accordance with an embodiment of the invention; and Fig. 3 shows a flow diagram of an algorithm used to determine presence of a crowd, in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS Figs. 1A and IB schematically show perspective and plan views of an urban environment 24 in which a vehicle 20 comprising a CWAS 30, in accordance with an embodiment of the invention, is moving along a road 22. The vehicle is moving towards an intersection 26 that has zebra crosswalks 27 and 28. CWAS 30 optionally comprises a single camera 31 that acquires images of the environment through which vehicle 20 moves and a processor (not shown) for processing the images. Camera 31 has an optic axis 32 and a field of view schematically delineated by lines 34. For convenience of presentation, CWAS 30 and camera 31 are shown greatly enlarged relative to vehicle 20 and mounted on the roof of the vehicle. Generally a CWAS is mounted in a substantially less obtrusive location, inside a vehicle, such as under the hood and comprises appropriate optics to enable a camera in the CWAS to image the vehicle's environment. Features of urban environment 24 are optionally located relative to a coordinate system 40 having an origin located at the optic center of camera 31, a Z-axis coincident with optic axis 32, a horizontal X-axis and a vertical Y-axis. Vehicle 20 is shown at a first location Pj at a time t\ and at a second location jsj at a time tjvf as it moves along road 22 towards intersection 26. A sequence of images 50 acquired by camera 31 in the time it takes vehicle 20 to move from position Pj to position P^ are schematically shown in Fig. 1A in an inset 60 aligned one behind the other to define an ST volume 52. A first image in the sequence acquired at time t] when vehicle 20 is located at P\ is labeled IMj and an N-th, image in the sequence, acquired at a time tj_sj when vehicle 20 is located at position Pxj, is labeled IMχj. Pixels in images 50 comprised in ST volume 52 are located by coordinates measured relative to a coordinate system 41 having a time axis perpendicular to the planes of the images and x and y spatial axes that are parallel to the planes of the images. The x and y-axes correspond respectively to X and Y-axes of coordinate system 40. ST volume 52 has an ST "optic" axis 42 corresponding to optic axis 32 of camera 31. ST optic axis 42 passes through pixels in images 50 that image features in environment 24 lying along camera optic axis 32. As a matter of convenience, the t-axis of ST volume 52 is chosen to coincide with ST axis 42. A given pixel in ST volume 52 is located by a time coordinate along the t-axis, which designates a particular image 50 in which the pixel lies by a time at which the particular image is acquired, and x and y-coordinates, which designate where in the particular image 50 the pixel is located. Pixels in images 50 that image a same feature in urban environment 24 lie along a same line, i.e. an image trajectory, in ST volume 52. For stationary features in environment 24, as vehicle 20 approaches intersection 26, the Z-coordinates (relative to coordinate system 40) of the features decrease, while their respective X and Y-coordinates remain the same. As a result, all stationary features in the environment that do not lie along the Z-axis, i.e. optic axis 32, exhibit outward optic flow. The location of pixels in an image 50 of the sequence of images 50 that image a given stationary feature in environment 24 that does not lie on the Z-axis, "drifts" away from the center and towards the perimeter of the image as the acquisition time of the image increases. As a result, image trajectories in ST volume 52 of stationary features in environment 24, except for features that might lie along optic axis 32, are "outward" moving trajectories that veer away from ST axis 42 of the ST volume, i.e. the t-axis of coordinate system 40. To illustrate characteristics of image trajectories of features in environment 24 that are imaged by camera 31 six features in the environment are indicated by reference labels F₀, Fi , ¥2, F3, F4 and F5. Image trajectories in ST volume 52 corresponding to features F₀, Fj, ¥2, F3, F4 and F5 are schematically shown in the ST volume and are indicated by reference labels T₀, Ti , T2, T3, T4 and T5 respectively. All features F₀-F5 are, by way of example, stationary features in environment 24 and are assumed for convenience of presentation to have a same Y-coordinate equal to zero. Feature F₀ lies along the Z-axis, features ¥\ and 2 have negative X-coordinates and lie to the right of the driver of vehicle 20 and features F3, F4 and F5 have positive X-coordinates and lie to the left of the driver. As vehicle 20 approaches intersection 26, the Z-coordinates (relative to coordinate system 40) of the features decrease while their respective X and Y-coordinates remain the same. As a result, all features except feature F₀, which lies along the Z-axis, exhibit outward optic flow and their respective image trajectories T\ - T5 in ST volume 52, move outward and veer away from the t-axis as t increases. Image trajectory T₀ corresponding to feature F₀ is a straight line lying along the ST axis 42. Because features F₀-F5 have a same Y-coordinate equal to zero, image trajectories T₀-T5 are coplanar and lie in an EPI plane 54 of ST volume 52 that is coincident with the xt-plane of the ST volume. It is noted, that in general, a given feature in environment 24 does not have an image trajectory that is coplanar with an EPI plane of ST volume 52. However, many objects in an environment such as environment 24, especially if they have sufficiently large vertical extents and optical properties that change relatively slowly with change in Y-coordinate, generate traces on EPI planes that are projections of image trajectories associated with the regions. The projections share characteristics of the image trajectories and in general may be used as approximations of the image trajectories. Hereinafter, projections of image trajectories on EPI planes of an ST volume are assumed to be image trajectories and are not distinguished from actual image trajectories. Fig.lC schematically shows an enlarged plan view of a portion of EPI plane 54 and details of image trajectories T₀, Tι , T2, T3, T4 and T5. The image trajectories have been generated, by way of example, assuming that at position Pj of vehicle 20, the X and Z-coordinates of features F₀, ¥\, F2, F3, F4, and F5 are respectively (0, 80), (-3.5, 32.6), (-3.5, 43), (8.5, 32.6), (8.5, 43), (10.7, 48). In each set of parenthesis, the first and second number give the X-coordinate and Y-coordinate in meters of its corresponding feature. Between position Pj at which image IMj is acquired, and position P vj at which image

is acquired, it is assumed that vehicle 20 moves twenty two meters. Camera 31 is assumed to have a focal length of 10 mm and values along the x-axis in the image of EPI plane 54 shown in Fig. 1C are given in mm. Units along the t-axis, which passes through x = 0 and coincides with ST axis 42 (also shown in Fig. 1A) and image trajectory T₀, are arbitrary. To prevent clutter, tj, tN and equidistance time intervals between time t\ and time txj are indicated by witness lines along an axis 51 parallel to the t-axis at the bottom of the image of EPI plane 54. The small icons along a given trajectory T₀-T5 indicate an x-value for the trajectory at a time indicated by a witness line directly below the icon. For convenience of presentation, distance vehicle 20 has traveled from position Pj at each of the times indicated by a witness line along axis 51 is indicated in meters. It is noted that in addition to each of image trajectories T₀-T5 veering away from the

ST axis 42 (i.e. the t-axis in Fig. 1C), none of the image trajectories intersect. The lack of intersection points is due to a fortuitous choice of the locations of features F₀-F5- For example, were a feature Fg, which is shown in Fig. IB, added to the group of features, F₀-F5, its image trajectory would intersect with image trajectory Tj of feature Fjat a time between t\ and ^. However, in general, for a region of an environment in which none of the features is moving, intersections between image trajectories occur less frequently than for a region of an environment comprising an ensemble of moving objects. In addition, since there are no inward moving image trajectories in an environment in which there are no moving features, there are no intersection points between an inward and an outward moving trajectory. All intersection points are only between image trajectories moving in a same, outward direction relative to ST axis 42. A configuration of image trajectories different from that shown in Figs. 1A and 1C arises for an environment comprising an ensemble of non-stationary features that move in different directions. For example, Figs. 2A and 2B schematically show perspective and plan views of vehicle 20 and urban environment 24 having, in addition to the features shown in Figs. 1A and IB, a crowd 70 of people that are crossing road 22 at zebra crosswalk 27 from both sides of the road. By way of example, the crowd comprises six people, three of whom are moving from left to right and three of whom are moving from right to left as seen by the driver (and camera 31) of vehicle 20. The motion of the people in crowd 70 generates features in a sequence of images of the crowd acquired by camera 31 that exhibit inward optic flow and corresponding image trajectories in an ST volume defined by the images that move inward in spite of the motion of vehicle 20. Insets 61 and 62 in Figs. 2A and 2B respectively show enlarged schematic views of crowd 70 at time t], when vehicle 20 is located at position

Pi - Fig. 2A schematically shows in an inset 63, a sequence of images 80 acquired by camera 31 as the vehicle moves from location Pj to location Pfsj and an ST volume 82 defined by the images. ST volume 82 shows image trajectories TPj, TP2, TP3, TP4, TP5 and TPg for six features, each of which is associated with a different person in crowd 70, that are imaged by camera 31 in images 80. A feature associated with a given person in crowd 70 may, for example, be a region of the person's body or clothing or something the person is carrying. Image trajectories having an odd subscript are associated with persons in crowd 70 moving from right to left and trajectories having an even subscript are associated with persons in the crowd moving from left to right. Each of the features is assumed, for convenience of presentation, to have a Y-coordinate equal to zero. Image trajectories TPj - TPg are therefore coplanar and lie in an EPI plane 84 that lies in the xt-plane of ST volume 52. Fig. 2C shows an enlarged plan view of a region of EPI plane 84 and details of image trajectories TPj-TPg. Image trajectories TPI, TP2, TP3, TP4, TP5, and TP6 are generated assuming that the persons they are associated with move at constant velocities equal respectively to -2.8, 1, -1.3, 2, -1 and 1.4 m/s between times t] and jsj and that at time tj their respective X-coordinates are, in meters, 1.75, -0.5, 1.2, -2, 0.5, -1.5. Because of the motions of the persons associated with image trajectories TP1-TP6, each of the trajectories initially moves inward towards the t-axis. In addition, the trajectories exhibit a relatively high concentration of intersection points, which tend to be clustered close to the t axis (the ST axis corresponding to optic axis 32 (Fig. 2A) of camera 31) and intersection points between trajectories moving in opposite directions as well as trajectories moving in same directions. In accordance with an embodiment of the invention, CWAS 30 determines whether crowd 70 is present in environment 24 responsive to characteristics of image trajectories in ST volume 82. Optionally, CWAS 30 determines whether crowd 70 is present or not present responsive to a degree to which ST volume 82 exhibits inward moving image trajectories. In some embodiments of the invention, CWAS 30 determines whether or not crowd 70 is present responsive to a frequency with which image trajectories in the ST volume intersect. Optionally, CWAS 30 determines a degree to which ST volume 82 exhibits inward moving and/or intersecting trajectories responsive to a degree to which at least one EPI plane in the ST volume exhibits such trajectories. The y-coordinate of the at least one EPI plane in ST volume 82 is determined so that it corresponds to Y and Z-coordinates at which features of people in a crowd in the path of vehicle 20 are expected to be located and which will therefore generate image trajectories in the at least one EPI plane responsive to human motion. In an embodiment of the invention, the at least one EPI plane comprise a plurality of EPI planes located at appropriate image y-coordinates. By way of example, assume that five EPI planes are used to detect crowds, that camera 31 (Figs. 1A and 2 A) has a focal length of about 10 mm and that optic axis 32 of the camera is located about a meter from the ground (unlike in Figs. 1A and 2A where for convenience of presentation camera 31 is shown located on the roof). Optionally, the y-coordinates of the planes are determined so that at a distance of about 30 meters from vehicle 20 features in a person's body at locations within a range from about knee height to about shoulder height of the person generate image trajectories in the EPI planes. Optionally, the planes are evenly spaced. Assume that knee height to head height extends from about 0.25 to about 1.75 meters above ground. Then a central EPI plane of the five EPI planes would have a y-coordinate equal to zero (it would lie in the xt- planes of ST volumes 52 or 82) and the planes would be spaced apart by about 0.5 mm. For each EPI plane, to determine presence of inward moving and intersecting image trajectories in the EPI plane, in accordance with an embodiment of the invention, CWAS 30 generates a "rolling", sample image of the EPI plane. At any given time t, the rolling sample image comprises a line of pixels parallel to the x-axis of ST volume 82 at the y-coordinate of the EPI plane from a last image 80 acquired by camera 31 and from each of a plurality of "M-l" images 80 preceding the last image. The sample image therefore comprises pixel lines from a total of M images. Each time a new last image is acquired by camera 31, the line of pixels from the earliest acquired image of the M images is, optionally, discarded and a line, hereinafter a "last pixel-line" of pixels from the new last image is added to the sample image. In some embodiments of the invention, M is equal to about 20. CWAS 30 identifies image trajectories in the rolling sample image optionally using a Hough transform that maps pixels in the rolling sample image to a Hough accumulator space. For each pixel in the rolling sample image, a gradient of intensity is determined. For pixels having a relatively well defined intensity gradient, CWAS 30 defines a straight line that passes through the pixel and has a slope that is perpendicular to the gradient. A pixel is optionally assumed to have a well defined gradient if its gray level differs from that of it neighbors by a sufficient amount. For example, in some embodiments of the invention, for pixels that have gray levels in a range from 0-255 a gray level difference is required to be greater than about 16 for a pixel to have a well defined gradient. The straight line defined for a given pixel is assumed to approximate an image trajectory along which the pixel lies. Whereas, an image trajectory along which a pixel lies is, as is shown for example in Figs. 1A, 1C, 2A and 2C, generally not a straight line, for relatively short distances, the trajectory can usually be approximated by a straight line. Portions of image trajectories that are comprised in rolling sample images that do not have a large dimension along the t-axis are generally sufficiently short so that they may be reasonably well approximated by straight lines. For example, for a rolling image for which M is about 20 and camera images are acquired at a rate of about 10 images per second, the inventors have found that image trajectories in a rolling sample may advantageously be approximated by straight lines. The straight line image trajectory for a pixel is described by an equation optionally of the form, x = s(t-t_n) + x₀, 1) where t_n is a time at which the last image 80 is acquired, s is the slope that characterizes the straight line and x₀ is the x-coordinate of the straight-line trajectory at time t_n. The parameter, x₀ is the x intercept of the pixel's straight-line trajectory with the last pixel line added to the rolling sample image. CWAS 30 determines values for s and x₀ for the pixel and maps the pixel into a two dimensional Hough accumulator space having discrete accumulator bins, each of which defines a range of values for _Q and a range of values for s. Mapping the pixel comprises increasing by one a count in the Hough space bin that brackets the values of x₀ and s determined for the pixel. For example, for a j-th pixel having coordinates (tj,xj) and a relatively well defined intensity gradient, the slope s is determined to have a value s; that, optionally, corresponds to a direction perpendicular to the intensity gradient at the coordinates (tj,xj). The x₀ intercept is determined to have a value x₀ , optionally in accordance with an expression ^xoJ ^{= x}j - ^sj(^tj'^tn)- 2)

CWAS 30 maps the j-th pixel into the Hough accumulator space by increasing a count in the bin that brackets the values (sj, x₀ ) by one. It is noted that in accordance with equation 1) given above, a trajectory in the sample image is a right or left moving trajectory relative to the driver of vehicle 20 if its slope s is positive or negative respectively. A trajectory in the sample image is an outward moving trajectory if its slope s and x₀ have opposite signs and is an inward moving trajectory if they have same signs. Relatively high counts for both positive slope Hough bins and negative slope Hough bins that are associated with a same value of x₀ indicate that x₀ is an intersection point of an outward and an inward moving image trajectory. In accordance with an embodiment of the invention, CWAS 30 uses values accumulated in the Hough accumulator space for a rolling sample image to define probability functions that are used to determine a degree to which inward moving and/or intersecting image trajectories are found in the sample image. The probability functions are used to determine presence of a crowd, such as crowd 70, in the path of vehicle 20. Let the accumulated count in a Hough space bin having central values s and x₀ be represented by H(s,x₀) and let a probability of a value for x₀ lying on a trajectory having a positive or negative slope be represented by P(s+, x₀) and P(s_, x₀) respectively. Then, in accordance with an embodiment of the invention, P(s+, x₀) = [∑H(s,x_o)/ ∑H(s,x_o) ] for s > K_± 3) S S, X o P(s_, x₀) = [∑H(s,x_o)/ ∑H(s,x_o) ] for s < K_±, 4) S S, X o where K± is a predetermined threshold, which is used to increase signal to noise. By way of example, in some embodiments of the invention, K-j- has a value equal to about 0.25. Optionally, the rolling sample image is determined to exhibit inward moving trajectories to a degree that indicates presence of crowd 70 if, ∑P(s₊, x_o) > K_INW 5) x > 0 o or ∑ P(S_, X_Q) > K_INW, 6) x < 0 o where Krj^ is a predetermined threshold controllable to control sensitivity of the inward motion determination. It is noted that the above procedure assumes that trajectories in a rolling sample image are readily detected. The inventors have found that fast moving features, which generate image trajectories having relatively large slopes and leave a sample region quickly, have a tendency to generate artifacts in the rolling sample image and escape detection. To reduce severity of the artifacts and improve efficiency for detecting trajectories, optionally, each rolling sample images is processed to generate a Gaussian pyramid of images, using methods known in the art. Image trajectories having large slopes are generally more easily detected in a higher level image of a Gaussian pyramid than in a lower one and in accordance with an embodiment of the invention, higher level Gaussian pyramid images of rolling sample images are used to determine to what extent they exhibit inward motion and intersecting trajectories. In accordance with an embodiment of the invention, a highest level image in the Gaussian pyramid of a rolling sample image is first processed optionally in accordance with equations 3) - 6) to determine if it exhibits inward moving trajectories. If "inward motion" is not found in the highest level, it is looked for in a next lower level. The process is continued to determine if at some level of the pyramid, including possibly the lowest level of the pyramid (i.e. the original rolling sample image for the EPI plane), inward motion is found. The process of looking for inward motion is stopped at a highest level of the pyramid at which such motion is found. Depending on whether or not inward motion is found in some level of the pyramid, the original rolling sample image and its corresponding EPI plane are determined respectively to exhibit or not exhibit inward motion. In accordance with an embodiment of the invention, following testing of each EPI plane for inward motion, the results from all the EPI planes are combined to determine if the images acquired by camera 31 exhibit a degree of inward motion indicative of presence of a crowd. Optionally, to determine whether the images exhibit inward motion a weighted sum of the results from each of the EPI planes is determined and if the weighted sum is greater than an appropriate threshold the images are determined to exhibit inward motion to a degree indicating presence of a crowd. Following determination as to whether the camera images exhibit inward motion, the camera images are processed to determine, in accordance with an embodiment of the invention, whether or not they exhibit a degree of intersecting trajectories sufficient to indicate presence of a crowd. Optionally, the rolling sample region for each EPI plane is processed to determine if it exhibits intersecting image trajectories. Optionally, processing a given rolling sample region for intersections is performed in a same level of the Gaussian pyramid generated for the sample region for which inward motion is found. If inward motion was not found, processing for intersections is performed on the original rolling sample region. In accordance with an embodiment of the invention, each value x₀ that labels a Hough space bin is vetted to determine if it is an intersection point of image trajectories. Optionally, x₀ is determined to be an intersection point for at least two image trajectories if the product of the probabilities determined in equations 3) and 4) above satisfy an equation of the form P(s+, x₀)P(s_, x₀) > K_C, 7) where KQ is a predetermined "sensitivity" threshold. In accordance with an embodiment of the invention, if NQ represents the number of points x₀ that satisfy equation 7), the rolling sample region and its associated EPI plane are determined to exhibit a degree of intersecting image trajectories indicative of a crowd if

N_C > K_NC, 8) where is an appropriate sensitivity threshold. In accordance with an embodiment of the invention, following testing of the rolling sample regions of the EPI planes for trajectory intersections, the results from all the EPI planes are combined to determine if the images acquired by camera 31 exhibit a number of intersecting trajectories to indicate presence of a crowd. Optionally, the determination is made responsive to whether a weighted sum of the "intersection results" from all the EPI planes is greater than a predetermined threshold. The results from testing the rolling sample regions of the EPI planes at a given time t for inward motion and multiplicity of intersections are processed by CWAS 30 to provide an assessment as to whether at the given time t a crowd is present in front of vehicle 20.

Optionally, CWAS 30 determines whether a crowd such as crowd 70 (Figs. 2A and 2B) is present in accordance with an algorithm 200 similar to that shown in a flow diagram in Fig. 3. After processing images provided by camera 20 as described above to determine at time t whether or not images provided by camera 31 exhibit inward motion and multiple intersecting trajectories, in a step 201 CWAS 30 optionally proceeds to a junction 202. At 202, if the images exhibit multiple intersections, CWAS 31 proceeds to a junction 203 and if not it proceeds to a junction 204. At junction 203 the CWAS optionally determines whether any of the following three conditions prevail: 1) there is substantial clustering of intersections close to the t-axis (as shown for example in Figs. 2A and 2C for crowd 70); 2) in an immediately preceding decision, CWAS 30 determined there was a crowd present; or 3) a pedestrian detection system comprised in the CWAS determined that many individuals are present in front of vehicle 20. Any of various pedestrian detection systems known in the art may be used in the practice of the present invention to provide an indication if many individuals are present in front of vehicle 20. Optionally, the pedestrian detection system is a component based detection system such as described in a PCT patent application entitled "Pedestrian Detection" filed on even date with the present application, the disclosure of which is incorporated herein by reference. If at least one of the three conditions exists, CWAS 30 proceeds to a decision block 205 and determines that a crowd is present. If on the other hand none of the conditions are extant CWAS proceeds to junction 204. At junction 204 if the camera images provided by camera 31 have been determined to exhibit inward motion, CWAS 30 proceeds to decision junction 206. At junction 206, the CWAS optionally, determines if either of the following two conditions prevail: 1) in an immediately preceding decision, CWAS 30 determined there was a crowd present; or 2) a pedestrian detection system comprised in the CWAS determined that many individuals are present in front of vehicle 20. If at least one of the conditions prevails, the CWAS proceeds to a decision block 207 and determines that a crowd is present. If neither of the two conditions are present, CWAS 30 proceeds to a junction 208 and determines if vehicle 20 is or is not stationary. If the vehicle is not stationary, CWAS proceeds to block 209 and determines that a crowd is not present. Any of various methods and devices known in the art may be used to determine if vehicle 20 is moving or not. For example, CWAS 30 optionally determines whether vehicle 20 is moving from an accelerometer it comprises or from signals that it receives from a speedometer system in the vehicle. If at junction 208 the vehicle is stationary, CWAS 30 proceeds to a junction 210. At 210 if in an immediately preceding decision CWAS 31 determined that a crowd was not present, the CWAS proceeds to a decision block 21 1 and determines that a crowd is not currently present. If on the other hand, at junction 210 the preceding decision was that a crowd was present, CWAS proceeds to a junction 212. At 212 if the camera images did not exhibit outward optic flow, optionally determined using methods similar to those used to determine inward flow, CWAS 30 determines in a decision block 213 that a crowd is currently also not present. If the images did exhibit outward flow, CWAS 30 proceeds to a decision block 214 and determines that a crowd is currently present. In using outward flow as a criterion for deciding whether a crowd is present, it is noted that if vehicle 20 is stationary, outward flow can be generated in images acquired by camera 31 only if moving objects are imaged in the images. In the description and claims of the present application, each of the verbs, "comprise" "include" and "have", and conjugates thereof, are used to indicate that the object or objects of the verb are not necessarily a complete listing of members, components, elements or parts of the subject or subjects of the verb. The present invention has been described using detailed descriptions of embodiments thereof that are provided by way of example and are not intended to limit the scope of the invention. The described embodiments comprise different features, not all of which are required in all embodiments of the invention. Some embodiments of the present invention utilize only some of the features or possible combinations of the features. Variations of embodiments of the present invention that are described and embodiments of the present invention comprising different combinations of features noted in the described embodiments will occur to persons of the art. The scope of the invention is limited only by the following claims.

Claims

1. A method of determining the presence of an ensemble of moving objects in an environment comprising: acquiring a plurality of images of a scene in the environment; processing the images to determine optic flow of features in the scene; and determining whether an ensemble of moving objects is present in the environment responsive to the optic flow.

2. A method according to claim 1 and comprising determining a degree to which the optic flow exhibits inward optic flow.

3. A method according to claim 2 and determining whether the ensemble is present responsive to the degree of inward optic flow.

4. A method according to any of claims 1- 3 wherein determining optic flow comprises determining image trajectories of features in the scene and using the image trajectories to determine optic flow.

5. A method according to claim 4 wherein determining image trajectories comprises determining image trajectories that lie in at least one EPI plane of a space time volume defined by the images.

6. A method according to claim 5 wherein the at least one EPI plane comprises a plurality of planes.

7. A method according to any of claims 4-6 and comprising determining a degree to which the image trajectories intersect.

8. A method according to claim 7 and comprising determining whether the ensemble is present responsive to the degree to which the image trajectories intersect.

9. A method according to any of the preceding claims wherein the images are acquired under conditions for which stationary features in the environment exhibit outward optic flow.

10. A method according to any of the preceding claims wherein the images are acquired by a camera mounted to a vehicle.

1 1. A method according to claim 10 wherein the vehicle is an automotive vehicle.

12. A method according to any of the preceding claims wherein the ensemble of moving objects is a crowd of people.

13. Apparatus for detecting presence of a crowd of people in an environment comprising: a camera that acquires images of a scene in the environment; and a processor that processes the images to determine presence of a crowd of people in accordance with any of the claims 1 - 12.

14. Apparatus according to claim 13 adapted to be mounted in a vehicle.

15. Apparatus according to claim 14 wherein the vehicle is an automotive vehicle.