WO2013135969A1

WO2013135969A1 - Method, arrangement, and computer program product for analysing regularities in the behaviour of moving objects

Info

Publication number: WO2013135969A1
Application number: PCT/FI2013/050285
Authority: WO
Inventors: Markus KUUSISTO
Original assignee: Mirasys Oy
Priority date: 2012-03-14
Filing date: 2013-03-13
Publication date: 2013-09-19
Also published as: FI20125279L

Abstract

The analysed behaviour of moving objects is prepared for display. For a num- ber of objects, a number of discrete locations are determined at which the ob- ject was detected to appear In a record space where records represent loca- tions of the area of interest, records that correspond to the determined discrete locations are weighted with values representative of the stops the object was detected to make at each respective discrete location. After selecting a visual appearance for records of said record space, records of said record space are displayed in a form that geometrically corresponds to an image of the area of interest.

Description

Method, arrangement, and computer program product for analysing regularities in the behaviour of moving objects

TECHNICAL FIELD The invention concerns in general the technology of analysing the observed movements of a number of objects that move freely or essentially freely within an observed area. In particular the invention concerns the task of providing information about the stops that the objects made, and preparing such information for presentation in a way that is intuitive and illustrative for a human observer.

TECHNICAL BACKGROUND

For an actor like a shopkeeper, trade fair organizer, traffic planner, or other whose interests concern the way in which living beings (meaning humans and animals) move, it is important to obtain experience-based data about how a certain population has been observed to behave under some known circumstances. In order to make the collection of such experience-based data possible in a wide variety of applications and at a reasonable cost, it is advantageous if it can be automatized and performed without highly specialized hardware or instrumentation. A known way to automatically collect experience-based data is shown schematically in fig. 1 . A digital video camera 101 has been configured to view an area of interest 102, where objects of interest (here: walking humans) move. Fig. 2 illustrates a single 2D video frame taken from the footage produced by the camera 101 . It is well known that a video processing algorithm can "learn in" and ignore the stationary background and recognize moving objects. In the video frame illustrated in fig. 2, a known video processing algorithm would recognize the two shapes 201 and 202 as representing objects of interest.

Digital images consist of image elements called pixels (or voxels for 3D images), and it is known to form a "cumulative image" by analysing a sequence of video frames taken by the camera 101 and incrementing a value of a pixel every time when it coincides with a recognized moving object in a frame. If we assume that shape 201 represents a human who walked from right to left, and that shape 202 represents one who walked from top to down, a cumulative image might look like the one illustrated in fig. 3. To facilitate easier comparison to fig. 2, the groups of pixels 301 and 302 have been hatched and outlined to represent the most recent locations of the detected moving objects. The traces 303 and 304 represent the paths they have taken. Their paths have crossed on area 305, which explains why the corresponding pixels have accumulated higher values and are shown shaded in fig. 3. Continuing with the same principle, and assuming that a number of humans or other moving objects move on the area of interest 102, the cumulative image eventually evolves into a kind of activity map in which pixels have highest accumulated values at those areas where the most people have been through.

A drawback of the method shown in figs. 1 to 3 is that it may lose information about the dynamics of the movements of the objects. For example, a highly emphasized pixel or voxel area in the cumulative image does not actually tell, whether a large number of objects just passed through there or whether a possibly much smaller number of objects happened to stop there. One person stopping (however, still recognized as an object and not as a part of the background) at one location for ten seconds is shown similarly on the cumulative image as ten persons passing through, each spending only one second at the observed location.

SUMMARY OF THE INVENTION

According to an aspect of the invention there are provided a method, an arrangement and a computer program product that are capable of converting the observed movements of objects of interest into a data set that is applicable for visualisation in a highly intuitive and practical manner. According to another aspect of the invention, especially regularities like stops of the objects at specific locations within the area of interest are revealed.

Objectives of the invention are preferably achieved by detecting stops of objects of interest, and weighting records - representative of locations on the ar- ea of interest - according to detected stops, or according to how long objects were observed to stop at each corresponding location. If the weighted records are given a particular visual appearance, they can be displayed as a kind of a heat map that geometrically corresponds to an image of the area of interest. A human observer can easily conceive those spots of the displayed "heat map" as "hottest" where of the stops or most significant stops were observed.

A method according to the invention is characterised by the features recited in the characterising part of the independent claim directed to a method. The invention concerns also an arrangement that is characterised by the features recited in the characterising part of the independent claim directed to an arrangement.

Additionally the invention concerns a computer program product that is characterised by the features recited in the characterising part of the independent claim directed to a computer program product.

The novel features which are considered as characteristic of the invention are set forth in particular in the appended claims. The invention itself, however, both as to its construction and its method of operation, together with additional objects and advantages thereof, will be best understood from the following de- scription of specific embodiments when read in connection with the accompanying drawings.

The exemplary embodiments of the invention presented in this patent application are not to be interpreted to pose limitations to the applicability of the appended claims. The verb "to comprise" is used in this patent application as an open limitation that does not exclude the existence of also unrecited features. The features recited in depending claims are mutually freely combinable unless otherwise explicitly stated.

BRIEF DESCRIPTION OF DRAWINGS

Fig. 1 illustrates a known case of monitoring an area of interest,

fig. 2 illustrates an exemplary video frame take by the camera of fig. 1 , fig. 3 illustrates a known method of tracking the movements of objects, fig. 4 illustrates an example of the detected stops and movements of six different objects,

fig. 5 illustrates the detection results of fig. 4 displayed in a single image, fig. 6 illustrates a heat map type visualization of detected stops of objects, fig. 7 illustrates an example of technical means for detecting the movements and stops of objects, fig. 8 illustrates another example of technical means for detecting the movements and stops of objects,

fig. 9 illustrates laser hits obtained in the arrangement of fig. 8,

fig. 10 illustrates a principle called the gravity-based method,

fig. 1 1 illustrates a coordinate-based environment,

fig. 12 illustrates the concept of revisiting an environment,

fig. 13 illustrates an arrangement and a computer program product according to an embodiment of the invention, and

fig. 14 illustrates an example of realizing one part of fig. 13. DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

All further subheadings until the claims belong under the heading given immediately above.

As a prerequisite we will assume that technical means exist for observing and documenting the movements of objects within an area of interest. We will pro- vide some examples of such technical means later in this description. The objects may be for example humans, and their movements may be observed for example in a shop, a mall, a trade fair or other public space. As objects, one may also consider vehicles like cars, motorcycles, bicycles and/or the like moving within a trafficable area. Also animals may be considered as objects, and in some cases the object may be only a part of a larger entity, like the hand of a human or the snout of an animal. Of the movements of the objects we assume that of particular interest are their stops and/or slower than usual movement at particular locations. In many cases a stopping or slow moving object may be interpreted as paying specific attention to some localized target, so that gain- ing knowledge about such "hot spots" helps to understand what a typical population of objects would consider interesting or attractive. In other cases objects stop or slow down as a sign of uncertainty or experienced difficulty of getting forward (or deciding which way to go), so that gaining knowledge about common stopping locations helps in making the area of interest more easily navi- gable.

MOVEMENTS vs. APPEARANCE AT DISCRETE LOCATIONS

Whether the movements or (apparently) static appearance at discrete locations of objects are discussed, is in some cases subject to interpretation. A digital data processing system always involves a certain degree of discretization by nature. Therefore we may assume that using technical means to observe and document the movements of an object within an area of interest results in determining a number of discrete locations at which the object was detected to appear within said area at consecutive moments of time. As an example, if a digital video camera points at the area of interest, it produces a sequence of video frames, each representing a snapshot at a particular moment of time. An object moving on the area of interest appears at a different location in each video frame. Using a suitable detection and video processing algorithm, it is possible to identify where the object appears in each video frame, and to represent the more or less irregular pixel group that is the image of the object with something more specific. Typically the location of the object in the video frame is expressed as the location of a representative point, like the geometric center point or the so-called center of mass of said pixel group, as a pixel count from two perpendicular edges of the video frame. A digital video camera produces the video frames at regular intervals, or at least at known moments of time. Thus documenting the discrete locations at which the image of the object was detected to appear within the pixel matrix that constitutes the video frame is analogous to determining the discrete real- life locations at which the object itself was detected to appear within the area that is enclosed in the field of view of the camera. The video frames come as an ordered sequence, so - especially if the frame rate is sufficiently high - a good approximation of the observed movements of the object is the assumption that it moved essentially linearly between two consecutively detected locations during the period of time that elapsed between taking the corresponding two consecutive video frames.

Of particular interest to embodiments of the present invention are stops that the objects make, which as a concept should be kept apart from mere loitering within an area. It is known to set up an observation plane, for example by arranging an invisible optical grid in front of a shelving. When a shopper reaches towards an item on a shelf, his or her hand cuts the optical grid, and a signal is received about an object appearing within the observation plane. When the shopper draws back his or her hand, another signal is received, and the "loitering time" (during which the hand of the shopper remained within the observation plane; i.e. during which at least one ray of the optical grid was cut) can be calculated. However, a typical shopper's behaviour involves local movement, for example moving a hand between various alternative products on the shelf or walking past racks filled with different products. Systems that measure loitering time are not capable of discriminating between stops and continuous movement, which makes them inaccurate in following actual object behaviour. Technically it is far from obvious to define what constitutes a stop. Above it was pointed out that for example in each video frame, an object appears as "stopped" at a particular location. However, a vast majority of these "stops" are just snapshots taken from what actually is continuous movement. On the other hand, for example a living human is never stationary in terms of exact physics, because muscles, nerves, and the sensory system constantly interact to execute a complicated series of delicate correction movements and balancing. An aspect of the present invention involves a method that can be employed to identify locations where objects remained essentially stationary, so that for all practical purposes they can be considered to have stopped for a certain dura- tion of time.

MAPPING THE AREA OF INTEREST INTO RECORD SPACE

In this description, an area of interest means a concrete real-life location where actual objects can move. For processing in a computer, the real-life area of interest may be represented in a number of ways. We assume that in general, there is a record space where computer-readable records represent locations of the area of interest.

As an example, still considering a digital video camera as the technical means for making observations, we may assume that there are as many records in the record space as there are pixels (or blocks, which are agglomerations of adjacent pixels) in the image produced by the camera. It is not necessary to cover the whole field of view of the camera with records, for example if there are obstacles within the area of interest that preclude any objects from appearing at the corresponding locations. However, obstructed areas may also be represented with dummy records that exist but are not used for the purposes of the present invention. It is also possible to have fewer records than pixels and to apply some kind of integration that deliberately degrades resolution, i.e. combines several adjacent pixels into a common record. Yet another possibility is to use a number of digital video cameras as the technical means for making observations, and to make records of the record space correspond to pixels or pixel groups of a composite image combined from the outputs of the video cameras. This way it is possible to employ methods according to embodiments of the present invention on a wider area than what could be included in the field of view of just a single camera. When an object appears at a particular location within the area of interest, its appearance may be documented by marking the record in the record space that - according to the employed mapping scheme - corresponds to that particular location.

In order to facilitate intuitive displaying of data that has been processed in the record space, it is advantageous to maintain geometric correspondence between records in the record space and real-life locations. This follows quite naturally for instance in the example mentioned above, in which each record corresponds to a pixel or block of pixels in a video frame or a composite image combined from video frames provided by several cameras. In such a case, if there is a selected visual appearance for each record, displaying the visual appearances of the records in the same geometric order in which the corresponding pixels or blocs appear in the video frame reveals the information that the selected visual appearances indicate as if just another, specifically processed video frame was displayed. EXAMPLE OF PRODUCING A VISUAL APPEARANCE

Fig. 4 illustrates the observed movements and stops of six different objects on the same area of interest. That is, an area of interest was monitored, and six different objects were seen to pass through it, each following its own trajectory and each stopping at its own stopping points. Each large rectangle in fig. 4 il- lustrates the observed trajectory of a particular object with a line, and the observed stopping points of that object with hatched circles. The small rectangle next to each stopping point provides a visual indication about the duration of the respective observed stop. We may assume that the lines have been drawn by making a technical apparatus determine a number of discrete locations at which the respective object was detected to appear within the an area of interest at consecutive moments of time. The stopping points illustrate those of said discrete locations where the respective object was detected to remain stationary or essentially stationary for a duration of time that exceeds a predetermined minimum duration of a stop. Fig. 5 illustrates the observed trajectories and stopping points of all six objects drawn in a single image. In this case the stops of different objects seem to concentrate on particular locations. According to an aspect of the present invention, the existence and location of such apparently attractive areas should be visualized in a highly intuitive and practical manner, preferably so that a human user could use the visualization to easily convey not only their existence and location but also their relative importance. The last-mentioned could be interpreted so that a visualized attractive area is the more important, the more there were objects stopping at or near it and/or the longer the objects were observed to spend at or near it.

Above it was assumed that a record space exists where records represent locations of the area of interest. The illustration of the trajectories and stopping points in fig. 5 can be considered to consist of small, adjacent pixels, so that for each pixel or group of immediately adjacent pixels there is a record in a record space. Records - that correspond to the determined stopping point locations - may be weighted with values representative of stops each object was detected to make at each respective discrete location. As more and more objects are observed and the as the weighting continues, those records begin to accumulate distinctively more weight that correspond to the most frequented stopping points.

For example a so-called "heat map" style visual appearance may be selected for the records of the record space. This means selecting for each record a color that appears to be the "hotter" the more weight the record has accumulated. This way the selected visual appearance becomes indicative of the ac- cumulated weight of the respective record. In order to enable more intuitive illustration, it may be advantageous to make some kind of averaging between records close to each other, so that the selected visual appearance becomes indicative of the accumulated weight more in a vicinity of the respective record than at the exact record itself. Above it was mentioned already that a predetermined minimum duration of a stop may be set, so that an object is only considered to have "stopped" if it remained (essentially) stationary for a time that was longer than the minimum duration. Additionally or alternatively other filtering strategies may be applied, at the step of weighting the records and/or at the step of selecting the visual ap- pearance. Changing the value of the predetermined minimum duration is an example of filtering of the former kind, because it affects the way in which the records will be weighted; a longer predetermined minimum duration results in weighting the corresponding records only if relatively long stops were observed. Filtering of the latter kind may be used to select various visual appear- ances for the same (weighted) records of the record space. One may for example select a neutral visual appearance for all but the most highly weighted records, so that the result emphasizes exclusively the most popular stopping points.

One example of filtering is that one only observes stops, regardless of their du- ration, and weights the records according to where the stops were observed to take place. Another example is to produce different heatmaps according to observed stop lengths, so that one of them illustrates the selected visual appearance of records that were weighted according to observed short stops (like 3 to 7 seconds), another one illustrates the selected visual appearance of records that were weighted according to observed medium length stops (like 7 to 15 seconds), and yet another one illustrates the selected visual appearance of records that were weighted according to observed long stops (like over 15 seconds, or 15 to 60 seconds).

Filtering according to the observed stop length has specific importance to for example certain consumer brands, which aim at catching the attention of a potential customer for a particular duration of time.

Fig. 6 illustrates how the weighted records of a record space corresponding to the area of interest can be displayed as a simplified "heat map". Here the selected visual appearance is not a color but a hatch, the intensity of which rep- resents the corresponding "temperature". The selected visual appearances are used, and the records of said record space are displayed in a form that geometrically corresponds to an image or ground plan of the area of interest. The "hottest" spot is the one referred to with number 601 , which provides a graphical illustration of the fact that essentially all objects were observed to stop there within a relatively small area, and also that the stops of the objects there were relatively long.

AUTOMATIZING THE OBSERVATION OF OBJECTS

Using automatically operating technical means to determine the discrete locations at which the object was detected to appear within the area of interest in- volves great advantages. The observation of objects may take place continuously or regularly without any such limitations that the limited alertness or persistence of human observers would pose, and once installed, the automatically operating technical means can in many cases be left operating for significant periods of time at little or no significant accumulating cost. Automatically operating technical means can also be made quite immune to potentially biasing factors such as the age, sex, size, outer appearance, or subjective attractiveness of the objects to be observed. The accuracy and repeatability of location- related detection can be made much better with automatically operating tech- nical means than e.g. with human observers. In many cases the automatically operating technical means can also be built so small and integrated into other structures so well that humans or other objects to be observed do not notice them or at least do not consider them annoying or intruding. It goes without saying that local laws and regulations must be obeyed; in many countries they place strict limits to the application of technical observation means to monitor the behaviour of people at public locations.

Fig. 7 illustrates the use of a digital camera as a part of automatically operating technical means. In the arrangement of fig. 7 a camera 701 has been installed above an area of interest 702, which in this case is the aisle between two shelves in a shop. If the room is sufficiently high and/or the camera has a wide enough imaging angle, the camera may cover even a much wider area, such as the whole shop, the environment of a large fair stand, or the like. The camera 701 may be for example a digital video camera or a digital still image camera configured for repeated exposures at relatively small intervals.

Fig. 8 illustrates another example of automatically operating technical means. In the arrangement of fig. 8 an electromagnetic beam scanner 801 has been installed so that the electromagnetic beam it emits may sweep across an area of interest, which in this case is the imaginary plane immediately in front of goods that stand on a shelf 802. As examples, the beam is shown with dashed lines at ten different locations during the sweep. An example of an electromagnetic beam scanner of this kind is a laser scanner that is configured to emit one or more laser beam(s) on a wavelength that is invisible and harmless to the human eye, to rotate the beam emitter head, and to measure the angle and distance at which a reflection of the laser beam is observed. The laser scanner gives the rotation angle and distance (and in some cases time) at which a reflection was observed as output information. Assuming that the re- flection was caused by an object such as a hand 803 that appeared in the plane defined by the sweeping beam(s), the rotation angle and distance already as such constitute coordinates of a detected location of the object. If necessary, a coordinate transformation of a known kind can be applied to transform the angular coordinates into corresponding other coordinates, like Cartesian coordinates.

Fig. 9 illustrates schematically a number of reflections, also called laser hits, that could have been obtained with one or more sweeps of an electromagnetic beam scanner. Much like the images of fixed structures in a video frame, the output given by an electromagnetic beam scanner may comprise a number of reflections or "image elements" that come from stationary objects and surroundings and thus remain essentially constant in every sweep even for longer durations of time. For example in fig. 9 we may assume that the essentially regular rectangle of reflections come from the floor, walls, ceiling, or other limit- ing surfaces of the space in which the electromagnetic beam scanner was installed. Additionally there may be certain essentially constant reflections from shelf edges and/or goods on sale or the like; in fig. 9 these appear in essentially straight lines here and there inside the regular rectangle.

At location 901 a group of reflections appear that do not reappear constantly from sweep to sweep over a long period of time, so a processing algorithm could recognize them as representing an object of interest, like the hand of a shopper who is about to pick something from the shelf. A processing algorithm that receives a list of reflections that - when plotted in accordance with their associated angle and distance values - would look like that in fig. 9 as input can recognize the group of reflections at 901 much like the group of pixels representing an object in an video frame, calculate its geometric center point or other representative point, and use its location in terms of the same kind of coordinates as those used for the reflections to correlate it with a record in the record space. As a further example of automatically operating technical means, a three- dimensional reflection analyser can be considered. This technology is particularly well known on the technical field of video games, where it is used for example in the Kinect® sensor of the Xbox® 360 gaming console. Kinect, Xbox, and Xbox 360 are registered trademarks of Microsoft Corporation. The three- dimensional reflection analyser projects a very large number of invisible infra- red laser dots on all visible surfaces of a three-dimensional space. By measuring the reflections of the dots and observing their movement, the analyser is able to detect the locations and movements of objects within the observed area. A drawback of three-dimensional reflection analysers is that they typically require a calibration step in which the location of the sensor in relation to the dimensions of the space to be observed is measured and fed in manually.

DETERMINING THE STOPPING POINTS

It is advantageous to employ an effective algorithm to decide, which points will be identified as the stopping points of an object and how should one calculate the dwelling time, i.e. the length of time for which the object is considered to have stayed at the stopping point. Basically it would be possible to consider each detected location of an object as a stopping point for a duration of time that is equal to the time between two consecutive detections. However, in many cases it is more advantageous to preprocess the detection data so that only actual stopping points and the corresponding stop durations are singled out, as was illustrated earlier in association with figs. 5 and 6. As a result of such preprocessing, only those records are weighted that correspond to the determined discrete locations at which the object was detected to stay longer than a predetermined limit. We assume that corresponding to each detected momentary location of the object, a location record is created that contains coordinates and time information. Location records should not be confused with the records of the record space discussed above. In order to find stopping points there is calculated, for at least a number of the location records, a location-record-specific descriptor value according to a "gravity-based" rule. The rule being "gravity-based" means that the rule makes the descriptor value the more significant the more densely there are other location records around the location record for which the descriptor value is currently calculated. Fig. 10 illustrates schematically the process of calculating the descriptor value for a location record illustrated as 1001 . Each vector represents a pulling force that a unit mass located at 1001 would experience because of another unit mass located at a nearby detected location. Only the most significant vectors are shown in order to preserve graphical clarity. The location-record-specific descriptor value calculated for the location 1001 is then the scalar sum of all lengths of the vectors. Mathematically we could say that the rule defines the calculation of said location-record-specific descriptor value as

where di means the location-record-specific descriptor value of an

z^':th location record,

i and j are indices that identify location records,

Ni means an environment covering all location records around the z^':th location record that are to be taken into account in calculating the location-record-specific descriptor value of an z^':th location record,

G is a scalar constant,

D (ij^') is the distance between the z^':th and y^':th location records,

a_min is a constant, and

p is a positive exponent. Defining the environment Ni is a practical way of limiting the amount of processing that needs to be done in calculating the location-record-specific descriptor values. It is of course possible to have N cover all location records for all z, but since the significance of far-lying other location records may become very small (depending on the magnitude of the exponent p), the required pro- cessing can be significantly limited without losing very much accuracy by defining N to cover for example only such other location records that according to their coordinates are closer than a predetermined limit to the z^':th location record.

The scalar constant G does not have much significance, because it will appear similarly in all calculated location-record-specific descriptor values. Mainly it reminds of the gravity analogy; in order to fully resemble the calculation of gravitational interaction, the formula should include the gravity constant. The value of G could well be 1 .

Calculating the distance between the z^':th and y^':th location records is made for example according to the formula D (i,j if (x;, ;)

and { j. yj) are the two-dimensional Cartesian coordinates of the z^':th and y^':th location records respectively. The general notation D (i,j) covers all ways of expressing the (scalar) distance between the locations represented by the z^':th and y^':th location record.

The constant a_min is a useful way of ensuring that for no value of j the corre- sponding term in the sum could go to infinity (which would happen if D (i,j) = 0; i.e. if at the available accuracy the two detected locations would be the same). It introduces a certain minimum distance between the two locations. The value of a_min is advantageously of the same order of magnitude as the estimated typical error in detecting the locations. If digital imaging is used for de- tection, the value of a_min could be a small number of pixels. If a laser scanner is used for detection, the value of a_min could be in the order of some millimetres or centimetres.

The magnitude of the exponent p essentially determines how quickly the effect of another location record should die away as a function of increasing dis- tance. For best analogy with gravity, one should select p = 2, but the value of p could also be e.g. one or smaller than one, or between one and two, or larger than two. The larger value p has, the more the calculation of the descriptor value emphasizes only very close other location records. Concerning the detected movements of the object, a small value of p allows also less significant slow-downs to be detected as stops while a large value of p tends to limit the consideration to longer and more clearly outstanding stops.

The rule illustrated by the mathematical formula above gives the largest descriptor values as the most significant ones. It is naturally possible to present an alternative formula that would give the descriptor values so that the small- est of them is the most significant one, for example by summing only the powers of location differences + a_min)^p instead of their inverses. By knowing what is the exact rule applied to calculating the descriptor values, it can be said which of the calculated descriptor values are the most significant ones. Selecting a location record that has the most significant descriptor value in its surroundings gives most likely stopping point of the object. For example in fig. 10, the location illustrated by the left-hand one of the two topmost circles would most probably be selected as the stopping point.

Saying that the location-record-specific descriptor value is calculated at least for a number of all location records means that some selectivity may be ap- plied for example when the number of location records is very large and they are located very densely along the detected trajectory of the object. In such a case it is in many cases perfectly acceptable to sacrifice some accuracy by only calculating the location-record-specific descriptor value for e.g. every second, every third, or every tenth location record. After having identified a stopping point, a representative time value should be found for telling, how long the object remained stationary or essentially stationary at or around the identified stopping point. Fig. 1 1 illustrates an exemplary way of doing this. From a coordinate-based environment of the location record that was selected (illustrated as 1 101 ) there are found those other loca- tion records that according to their time information are the latest and earliest in said environment.

The coordinate-based environment is illustrated as the circle 1 102 in fig. 1 1 . Being coordinate-based means that whether or not another location record belongs to that environment may be decided by investigating the coordinates of the other location records. If the coordinate-based environment of an z^':th location record is a circle like in fig. 1 1 , the y^':th location record belongs to that environment if the distance D (i,j) is smaller than the radius of the circle. In an analogous example, if the coordinate-based environment would be e.g. a rectangle centered on the detected stopping point, a relatively straightforward comparison of coordinate values would reveal, whether some other location record belongs to that environment.

After all those location records have been found that belong to the coordinate- based environment 1 102, the latest and earliest location record in that environment can be found by examining the time information contained in the loca- tion records. From the time information of the latest and earliest location record in the environment, a stopping time can be calculated. The most straightforward alternative is to just calculate the time difference between time stamps contained in the latest and earliest location records in the environment. If the time information has been expressed in some indirect way like as an ordinal number of a location record or as a reference to the number of a video frame from which the location detection was made, slightly more advanced calculations may be needed. Additionally it is possible to process the time difference somehow, for example by only taking some representative fraction of the time difference as the stopping time, in order to account for the fact that the envi- ronment covered also points where the object was still moving or again moving.

According to an analysis result that can be output and stored, the object stopped at the coordinates of the selected location record for the stopping time that was calculated from the time information of said latest and earliest location records. How many other stopping points will be announced as analysis results, depends on what is required of the analysis. In some cases it is sufficient to only find the overall most significant descriptor value and use it to derive the results. In other cases it may be that for example all stops longer than a limit, say three seconds, should be revealed.

In the last-mentioned case one possible way to proceed is to go to the location-record-specific descriptor values, find the second most significant descriptor value (because the most significant one was considered already) and use the corresponding location record as a starting point for again placing a coordinate-based environment, finding the latest and earliest location record therein, and so forth. However, certain problems may arise because in many cases the location record with the second most significant descriptor value is very close to the first one, and actually belongs to the same stop of the object.

As an example, we may assume that the location record associated with the detected location 1 101 in fig. 1 1 had the overall most significant descriptor value among all location records. Since the emergence of that descriptor value was based on the close proximity of other detected locations, it is very much possible that the overall second most significant descriptor value will emerge for one of the detected locations immediately to the right or left from the de- tected location 1 101 , which also have very close neighbours.

In order to focus the search of another stopping point to somewhere else along the trajectory of the object after having selected the location record that has the most significant descriptor value in its surroundings, according to an aspect of the invention the method comprises excluding from further selection location records that are closer than a limit value to the selected location record. Setting the limit value may follow the way in which the environment 1 102 was defined, but this is not a requirement of the invention. However, it is a relatively logical choice, because as was noted above, the duration of time of the stop at detected location 1 101 could be announced as equal to the time difference be- tween the latest and earliest location record within the environment 1 102. In other words, all location records within the environment 1 102 could be made to logically belong to the stop at the detected location 1 101 , which would advocate excluding all of them from further selection.

Among the location records that remain not excluded, there can be repeated the steps of:

- selecting a (remaining) location record that has the most significant descriptor value in its surroundings,

- from a coordinate-based environment of the selected location record, finding those other location records that according to their time information are the lat- est and earliest in said environment, and

- outputting and storing (a further) analysis result, according to which said object stopped at the coordinates of the selected location record for a stopping time calculated from the time information of said latest and earliest location records. The output and stored further analysis result thus indicates that the object also stopped at the coordinates of the location record that was selected in this repeated execution of the selection step.

Some further considerations may apply if the object proceeds along a trajectory that revisits the environment of a stopping point. Fig. 12 illustrates a case in which the object first moved along what is illustrated as line 1201 , and later moved along what is illustrated as line 1202. At detected location 1 101 the object made a stop. In order to find out the time duration of the stop, a coordinate-based environment 1 102 was considered. However, of all location records that happen to fall within the environment 1 102, only those are eligible for finding the latest 1 104 and earliest 1 103 location record concerning the stop at 1 101 for which a continuous sequence of temporally consecutive location records exists - within environment 1 102 - that connects them with the selected location record (the one associated with detected location 1 101 ). In other words, even if all location records along the line 1202 are temporally later than the one associated with detected location 1 104, none of them is eligible as the latest location record.

Since none of the location records along the line 1202 thus belongs to the stop that will be associated with detected location 1 101 , it is logical that when location records are then excluded from further selection, said excluding does not apply to location records within environment 1 102 if time information of such location records reveals them to be either earlier than the earliest location record 1 103 or later than the latest location record 1 104. It is possible that a further significant stop of the object will be found at some of the detected locations along 1202, even if some of them are within the coordinate-based envi- ronment 1 102 that applied to the stop at location 1 101 .

Whether or not the location records belonging to line 1202 are taken into account in calculating the descriptor values for the location records belonging to line 1201 , is not limited by the present invention. Both alternatives are possible. Allowing them to be taken into account results in more straightforward cal- culation of the descriptor values, because location records need not then be examined and discriminated on the basis of their time information. Also, if there was some reason for the object to later come back very close to a previous location, it may be only advantageous (for the purpose of finding the most relevant factors that affect the movements of the object) if the repeated visit makes the associated descriptor values more significant and increase the possibility of a stopping point being identified. However, in some cases allowing repeated visits to be taken into account in calculating the descriptor values may also result in a situation where a location will be identified as a stopping point even if the object actually never stopped there but only passed by a number of times. If discrimination on the basis of time information is to be applied, it can be realized in many ways. For example, when calculating a descriptor value for a particular location record, one can simply take a time dimension into the definition of the environment N_;. That is, one may omit a y^':th location record from the calculation of the descriptor value for the z^':th location record if the time infor- mation of these two location records shows them to be more distant in time than a predetermined limit. Another possibility is to apply a similar "continuous sequence of temporally consecutive location records" type rule as above, including a y^':th location record into the environment Ni only if - in addition to the sufficiently short mutual distance in location - a continuous sequence of loca- tion records connects the two without going prohibitively far from the z^':th location record at any point.

In the embodiments described so far the question "how densely there are other location records around the location record for which the descriptor value is calculated" is answered by summing together contributions from other location records, weighted with their inverse spatial distance from that location record for which the descriptor value is calculated. An alternative way is to take each location record at a time, to define a coordinate-based environment for it, and to look for the earliest and latest other location record within said environment. From the time information associated with said earliest and latest other loca- tion record within the environment, an indicative stopping time length can be calculated for the location record that is concerned. In this embodiment there is thus no separate step for calculating a descriptor value; for each location record, a stopping time is directly calculated. Whether or not such other location records (those along line 502 in fig. 5) should be ignored that represent "revis- iting" the environment, is a matter of design choice.

DESCRIPTOR VALUES vs. WEIGHTING OF RECORDS

Basically it would be possible to use location-record-specific descriptor values of the kind described above directly as the weights of the records of the record space. However, that could easily lead to a situation where a small number of very highly weighted records dominated the generation of the heatmap or other visual representation. Additionally it could make the filtering mentioned earlier unnecessarily difficult, because the descriptor values do not as such contain any information about the length of a stop at a particular location.

It has been considered more advantageous to refine the available raw data so that the weights associated with location records are (or represent) the length of time an object was detected to stay at the corresponding location during an observed stop. Even then there exists the possibility that some very long observed stops dominate too much over a larger number of relatively short stops. Such distortion could be avoided to some extent by taking the stop lengths through a logarithmic or otherwise length-balancing function before using them as weights. Another possibility, which has been mentioned above in association with filtering, is to use the same basic material (the weighted records) to produce a number of heatmaps, each of which only takes into account stops the lengths of which fit a particular range of lengths. Yet another possibility is to weight each observed stop equally, irrespective of its length. In order to maintain conformity with the language used elsewhere in this description, the last-mentioned case could be characterised so that the records that correspond to the determined discrete locations are weighted with values representative of the fact that a length of time exists for which the object was de- tected to stay at each respective discrete location. ARRANGEMENT- AND COMPUTER PROGRAM PRODUCT EMBODIMENTS

Fig. 13 is a schematic illustration of an embodiment of the invention that can represent both an arrangement and a computer program product. The hardware parts of an arrangement according to an embodiment of the invention can be implemented in a single apparatus or they can be distributed in a number of physically separate devices, like separate computers that are linked together through one or more communications networks.

Block 1301 represents the technical means for detecting the trajectories and stopping points of objects on an area of interest. Examples for its implementa- tion are described in more detail later in association with fig. 14. For now it suffices to assume that technical means exist that are capable of detecting the location and timing of stops that objects make within the area of interest, and of creating and storing information of the detected stops in a machine-readable form that allows correlating the detected locations of the stops with records that represent locations within the area of interest. A storage of such information is separately shown in fig. 13 as the object trajectory and stopping point database 1302. Together, blocks 1301 and 1302 can be said to implement a stopping point locatore that is configured to determine discrete locations at which objects were detected to appear within the area of interest. In the schematic illustration of fig. 13, block 1303 is a stopping point selector and weight calculator, the task of which is to select those detected stops of the objects that are meaningful for an analysis to be performed, and to use the information about such selected stops to weight the corresponding records. Block 13 could also be described as a record manager configured to receive information about how objects were detected to make stops at discrete locations of said area of interest and to weight records that correspond to the determined discrete locations with values representative of said information. What is meaningful for each kind of analysis can be given as input information to the record manager, for which purpose a control input is shown in fig. 14. A storage of the records and their associated weights is separately shown in fig. 14 as block 1304. Conceptually the record space weights maintenance functionality of block 1304 could quite as well be considered as a part of the record manager, because in many cases a data structures and the routines used to update and maintain it constitute an integrated entity. A visual rendering manager 1305 is configured to select a visual appearance for records of said record space, said visual appearance being indicative of the accumulated weight at or in a vicinity of the respective record. For commands that control the visualization process, a control input to block 1305 is shown. The output from block 1305 is the "heat map" type information that can be displayed; for this purpose the arrangement may comprise a display that is configured to use the selected visual appearances for displaying records of said record space in a form that geometrically corresponds to an image of the area of interest. An additional visual and/or geometric data section 1306 is shown to illustrate the fact that during the visualization process, some information is needed about how the displayed form should be composed so that a human user can intuitively understand the correspondence between displayed "hot spots" and actual physical locations within the area of interest. For example, the additional visual and/or geometric data section 1306 may provide a map or a photograph of the area of interest to be used as a background, on top of which the heatmap may be displayed. A background data input to block 1306 is shown, through which such background data can be introduced into the arrangement.

Fig. 14 illustrates schematically an exemplary arrangement 1401 for imple- menting the functionalities of a stopping point locator. The blocks 1402, 1403, and 1404 illustrated as location data acquisition, detection of object location, and location record formation respectively may implement, individually and/or in various combinations, a location record storage that is configured to store location records, each location record comprising coordinates of a detected or assumed location of an object, and time information. The location data acquisition block 1402 is illustrated as implementing an input to the arrangement, through which information in various forms may enter the arrangement. As an example, the location data acquisition block 1402 may comprise a video subsystem that includes one or more video cameras, and the detection of object location block 1403 may comprise a location calculator configured to calculate coordinates of where the object is located in a coordinate system defined in respect of a video image produced through said one or more video cameras. Such coordinates then constitute the input of forming the actual location records in block 1404. It may derive time information to be included in the location records from the way in which the location data is acquired, for example by noting, from which video frame the coordinates of a detected object were de- rived. Alternatively or additionally the arrangement may comprise a dedicated time data acquisition block 1405, from which comes the time data that will end up in the location records.

The arrangement of fig. 14 comprises also a descriptor value calculator 1406. It is configured to calculate, for a number of location records, a location-record- specific descriptor value according to a rule. The rule makes said descriptor value the more significant the more densely there are other location records around the location record for which the descriptor value is calculated, i.e. the more there are other location records that according to their coordinates are close to the location record under consideration.

The arrangement of fig. 14 comprises also a selector 1407 configured to select a location record that has the most significant descriptor value within a number of location records and to find, from a coordinate-based environment of the selected location record, those other location records that according to their time information are the latest and earliest in said environment. Further comprised in the arrangement is a result storage 1408 that is configured to store an analysis result, according to which said object stopped at the coordinates of the selected location record for a stopping time calculated from the time information of said latest and earliest location record. For calculating the stopping times there is shown a particular stopping time calculator 1409. Conceptually it may also assist the selector in block 1407 in excluding from further selection location records that are closer than a limit value to location record that was selected on the basis of its most significant descriptor value.

Many additions and modifications to the schematic principles shown in figs. 13 and 14 are possible without departing from the scope of protection defined by the appended claims. For example, in addition or as an alternative to the video subsystem, the arrangement may comprise an electromagnetic beam scanner that is configured store coordinates of where in a coordinate system defined in respect of the electromagnetic beam scanner the object is located when an electromagnetic beam emitted by the electromagnetic beam scanner is found to hit the object. Other means for detecting the momentary location of an object include but are not limited to the use of radio waves and/or acoustic waves. For example, if the object is carrying a radio transmitter such as a blue- tooth or wlan transmitter, a mobile phone, or other wireless terminal of a cellu- lar radio system, it is possible to receive the transmitted signals at multiple lo- cations and use the differently tinned receptions to calculate the position of the transmitter. If the object is emitting acoustic noise, acoustic microphones can be used for similar reception time based positioning calculations.

The functions that have been shown as separate blocks in figs. 13 and 14 can be combined into common blocks and/or distributed into different kinds of functional entities.

Saying that some limit or threshold value or other reference is "predetermined" does not require that the limit, threshold value or other reference should remain fixed: for example, it is possible to use an adaptive algorithm that chang- es the limit, threshold value or other reference adaptively in relation to some observed conditions. Such a changing value is "predetermined" in the sense that the algorithm will react in a deterministic way and perform the adaptation in a way that follows the intentions of the programmer.

Claims

1 . A method for preparing analysed behaviour of moving objects for display, comprising:

- for a number of objects, - determining a number of discrete locations at which the object was detected to appear within an area of interest, and

- in a record space where records represent locations of the area of interest, weighting records that correspond to the determined discrete locations with values representative of stops the object was detected to make at each respective discrete location;

- selecting a visual appearance for records of said record space, said visual appearance being indicative of the accumulated weight at or in a vicinity of the respective record, and

- using the selected visual appearances, displaying records of said record space in a form that geometrically corresponds to an image of the area of interest.

2. A method according to claim 1 , wherein only those records are weighted that correspond to the determined discrete locations at which the object was detected to stay longer than a predetermined limit.

3. A method according to claim 1 or 2, wherein records are weighted with values representative of the length of time the object was detected to remain stationary at each respective discrete location.

4. A method according to claim 2 or 3, comprising, for determining the length of time the object was detected to stay at one of said discrete locations: a) for a number of location records representative of locations at which the object was detected to appear, calculating a location-record-specific descriptor value according to a rule, wherein said rule makes said descriptor value the more significant the more densely there are other locations around at which the object was detected to appear, b) selecting a location record that has the most significant descriptor value within a number of location records, c) from an environment of the location corresponding to the selected location record, finding those other locations at which the object was detected to ap- pear latest and earliest in said environment, and d) determining the length of time the object was detected to stay at the location corresponding to the selected location record to be equal to a time difference of detecting the object to appear at said latest and earliest locations in said environment.

5. A method according to claim 4, comprising:

- after having selected the location record that has the most significant descriptor value in its surroundings, excluding from further selection location records corresponding to locations that are closer than a limit value to the location corresponding to the selected location record, and - among the location records representative of locations at which the object was detected to appear that remain not excluded, repeating steps b), c) and d) in order to determine the length of time the object was detected to stay at at least one more of said discrete locations.

6. A method according to any of the preceding claims, wherein said deter- mining of a number of discrete locations at which the object was detected to appear within the area of interest comprises at least one of the following:

- videoing the movements of said object and calculating coordinates of where the object is located in a coordinate system defined in respect of a video image produced through said videoing, - scanning a monitored space with an electromagnetic beam scanner and storing coordinates of where in a coordinate system defined in respect of the electromagnetic beam scanner the object is located when an electromagnetic beam emitted by the electromagnetic beam scanner is found to hit the object.

7. An arrangement for preparing analysed behaviour of moving objects for display, comprising: - a stopping point locator configured to determine discrete locations at which objects were detected to appear within an area of interest,

- a record manager configured to receive information about how objects were detected to make stops at discrete locations of said area of interest and to weight records that correspond to the determined discrete locations with values representative of said information, and

- a visual rendering manager configured to select a visual appearance for records of said record space, said visual appearance being indicative of the accumulated weight at or in a vicinity of the respective record.

8. An arrangement according to claim 7, comprising a display configured to use the selected visual appearances for displaying records of said record space in a form that geometrically corresponds to an image of the area of interest.

9. An arrangement according to claim 7 or 8, wherein said stopping point locator comprises:

- a location record storage configured to store location records, each location record comprising coordinates of a detected or assumed location of an object and time information,

- a descriptor value calculator configured to calculate, for a number of location records, a location-record-specific descriptor value according to a rule, wherein said rule makes said descriptor value the more significant the more densely there are other location records around the location record for which the descriptor value is calculated,

- a selector configured to select a location record that has the most significant descriptor value within a number of location records and to find, from a coordinate-based environment of the selected location record, those other location records that according to their time information are the latest and earliest in said environment, and

- a result storage configured to provide said record manager with said infor- mation, according to which said object stopped at the coordinates of the selected location record for a stopping time calculated from the time information of said latest and earliest location record.

10. An arrangement according to any of claims 7 to 9, comprising a video subsystem that includes one or more video cameras, and a location calculator configured to calculate coordinates of where the object is located in a coordinate system defined in respect of a video image produced through said one or more video cameras.

1 1 . An arrangement according to any of claims 7 to 10, comprising an electromagnetic beam scanner configured store coordinates of where in a coordinate system defined in respect of the electromagnetic beam scanner the object is located when an electromagnetic beam emitted by the electromagnetic beam scanner is found to hit the object.

12. A computer program product comprising machine-executable instructions that, when executed by a programmable device, cause the implementation of:

- for a number of objects,

- determining a number of discrete locations at which the object was de- tected to appear within an area of interest, and

- in a record space where records represent locations of the area of interest, weighting records that correspond to the determined discrete locations with values representative of stops the object was detected to make at each respective discrete location; - selecting a visual appearance for records of said record space, said visual appearance being indicative of the accumulated weight at or in a vicinity of the respective record, and

- using the selected visual appearances, displaying records of said record space in a form that geometrically corresponds to an image of the area of in- terest.