EP2754089A1 - Système et procédé de surveillance permettant de détecter le comportement de groupes d'acteurs - Google Patents

Système et procédé de surveillance permettant de détecter le comportement de groupes d'acteurs

Info

Publication number
EP2754089A1
EP2754089A1 EP12758666.7A EP12758666A EP2754089A1 EP 2754089 A1 EP2754089 A1 EP 2754089A1 EP 12758666 A EP12758666 A EP 12758666A EP 2754089 A1 EP2754089 A1 EP 2754089A1
Authority
EP
European Patent Office
Prior art keywords
behavior
rules
detected
actor
motion
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
EP12758666.7A
Other languages
German (de)
English (en)
Inventor
Wannes Van Der Mark
Arvid Hendrik Roald HALMA
Ernst Gerard Pieter Bovenkamp
Pieter Thijs EENDEBAK
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nederlandse Organisatie voor Toegepast Natuurwetenschappelijk Onderzoek TNO
Original Assignee
Nederlandse Organisatie voor Toegepast Natuurwetenschappelijk Onderzoek TNO
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nederlandse Organisatie voor Toegepast Natuurwetenschappelijk Onderzoek TNO filed Critical Nederlandse Organisatie voor Toegepast Natuurwetenschappelijk Onderzoek TNO
Priority to EP12758666.7A priority Critical patent/EP2754089A1/fr
Publication of EP2754089A1 publication Critical patent/EP2754089A1/fr
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects

Definitions

  • the invention relates to a surveillance system and a method of detecting behavior of actors or groups of actors from surveillance data.
  • Each state graph models defines a number of states and possible transitions between states.
  • the states define possible positions, velocities and postures that are consistent with the states. Examples of simple states are "on sidewalk", “on road”, “on crosswalk”, “has speed”, "is sitting” etc. More complex states can be built from combinations of such states.
  • the detected tracks and postures make it possible to determine whether or not detected persons in individual video frames satisfy the conditions for being in specific states.
  • Gonzalez' system searches for state graph models for a person that are consistent with the states in individual video frames, i.e. wherein the temporal transitions between different states correspond to possible transitions of the model.
  • Kota Yamaguchi et al describe a method of tracking persons in an article titled "Who are you and where are you going?", published in The 2011 IEEE conference on computer vision and pattern recognition pages 1345-1352 (EPO reference XP031672590). An energy function is used to compute predicted motion.
  • Pellerini et al. describe a form of tracking of persons in video images in a first article titled "You'll never walk alone: Modeling of social behavior for multi-target tracking", published in the IEEE 12 th international conference on computer vision, pages 261-268 (EPO reference XP031672590).
  • Pellerini proposes the use of a model that predicts movements in terms of movement towards a goal, keeping the same speed, and avoiding collisions with objects and other persons. Goal points are labeled and the desired direction for each subject is set towards the closest goal.
  • the model defines an energy function and predicted motion is determined by searching for
  • None of the documents discloses matching motion with individual multi- actor rules, and in particular not with rules that depend on the field of view of the modeled actor, or use of a measured indication of the field of view for rule matching.
  • An electronic image based surveillance method comprises
  • each of said rules of behavior defining at least a predicted aspect of motion of a modeled actor, at least part of the rules of behavior being specified dependent on position and/or motion of the modeled actor relative to a further actor;
  • “actors” cover human persons, but also animals or robots, or any other entity that is capable of acting dependent on its perceived environment in the area under surveillance. When applied to robots, the method can be used to detect features of the programs of the robots or reverse engineer or classify the programs. The position and/or motion of actors may be detected for example from images of at least parts of the area under
  • surveillance such as camera images, radar images etc, and/or from signals transmitted by tags carried the actors.
  • Detection of the position and/or motion of a plurality of actors makes it possible account for interaction between actors when it is detected that the motion shows a match with an individual rule of behavior that involves several actors.
  • Use of matching with individual rules makes it possible to detect a combination of actors that may need attention, rather than just an individual actor.
  • behavior can be detected wherein a detected actor follows another detected actor, or an alarm due to an unexpected movement of an actor can be suppressed if the movement can be explained by a rule according to which the actor avoids another actor.
  • both the detected actor and the other actor can be identified, for example in an image displayed to a human observer, to facihtate surveillance.
  • other rules may be used that involve only a single actor and its location and/or motion.
  • An embodiment of the method comprises determining an indication of a field of view of the first one of the actors, for example from image data.
  • determining whether the rule of behavior is satisfied comprises testing whether the position of the second one of the actors is in the field of view of the first one of the actors.
  • the indication of a field of view may be determined by determining an indication of a direction of view of the first one of the actors from an image.
  • At least one of the rules of behavior is specified dependent on perception of a sound, a touch or a smell by the modeled actor.
  • the rule may define that it applies only if the modeled actor is able to hear a sound with properties (e.g. loudness or loudness within one or more spectral bands) in a specified range within a predetermined time interval before a predicted motion. If so, output from a sound sensor in the area under surveillance may used, and the rule may be found to apply only after testing that the sound sensor has detected a sound that has properties in the specified range within the specified time interval.
  • touching for example of a surface like a control button, or the hand of another actor may be used, or a condition on a smell may be tested by measuring a concentration of a predetermined chemical substance that leads to this smell.
  • position dependent measurements of sound, touch or concentration of chemicals may be measured, by using a plurality of sensors at mutually different positions in the area under
  • Rules that account for directions and or speed on a (quasi-) continuous scale such as a "Goto" rule that predicts that a detected actor will move along a shortest possible path to a predetermined destination or a "Follow” rule that predict that an object will move to follow another object, makes it possible to detect specific forms of behavior.
  • additional rules such as a rule that predicts motions to avoid collision with other users can be used to make the method more robust against deviations due to external influences.
  • the method may be used for example to generate an alert, activate a sign (e.g. a warning sign or a direction sign) in the area under surveillance and/or to control a gate of the area under surveillance, in response to detection that a predetermined condition in terms of satisfied rules is met.
  • a human guard can be alerted or an automated response can be activated in specifically selected conditions.
  • an image may be displayed with a mark up to identify the first and second one of the actors and the other of the detected in a displayed image that have been detected to satisfy one or more of the rules of behavior, when their position and/or motion is used for the rule controlled object and the further object in the rule of behavior.
  • the markup may identify a person that has been found to satisfy a rule of behavior as well as other persons that are involved in that rule, such as another person that is being followed.
  • An electronic surveillance system that comprises
  • processing system comprising an actor detection module configured to detect positions and motion of a plurality of detected actors in the area under surveillance from the data representing the sensor data;
  • - a memory storing information representing a set of rules of behavior, each of said rules of behavior defining at least a predicted aspect of motion of a modeled actor, at least part of the rules of behavior being specified dependent on position and/or motion of the modeled actor relative to a further actor; - the processing system being configured to test for each of the rules of behavior whether said aspect of motion and the position and/motion of a first one of the detected actors relative to a second one of the detected actors, when used as the position and/motion of the modeled actor satisfies the rule of behavior.
  • Figure 1 shows an electronic surveillance system
  • FIG. 3 shows flow-chart of operation of a surveillance system
  • Figure 4 illustrates determination of a field of view
  • Figure 5 illustrates how a person's track can be classified
  • Figure 6 illustrates a rule in terms of location, moving direction, speed and visibility
  • FIG. 7 illustrates use of observation of new parts of an environment
  • Figure 1 shows an electronic surveillance system, comprising a plurality of cameras 10, directed at an area under surveillance 11, a processing system 12 and a display screen 14.
  • Processing system 12 is coupled to cameras 10 and display screen 14.
  • a controllable information sign 16 and a controllable gate 18 in the area under surveillance 11 are also shown, processing system 12 having outputs coupled to control inputs of information sign 16 and gate 18.
  • Cameras 10 may be located above area under surveillance 11 looking down on area under surveillance 11 and/or adjacent to area under surveillance 11 looking sideways at area under surveillance 11.
  • each camera 10 captures a temporal series of images of the area under surveillance 11 and supplies data representing the images to processing system 12.
  • Processing system 12 detects image areas in the images that show persons (or more generally actors like humans, animals or even robots that are able to act dependent on their perceived environment), or moving objects in general and determines tracks of motions of these persons or objects.
  • Processing system 12 may translate the position of the image areas into world coordinates, e.g. 3D coordinates or 2D coordinates on the ground in the area under surveillance, for use in the tracks, for example from
  • predetermined coordinates of a surface e.g. the ground
  • a surface e.g. the ground
  • Processing system 12 tests the resulting information by means of a predetermined set of rules of behavior.
  • Each rule of behavior defines a direction of motion of a modeled person as a function of the location of the modeled person and features of an environment of that location.
  • predetermined set of rules of behavior may include a "Goto” rule, a “Follow” rule, an “Avoid” rule, a “Meet rule”.
  • the "Goto" rule for a modeled person has a destination location as parameter and it defines a direction of motion of the modeled person as the direction from the current location of the modeled person along a path from the current location to the destination location.
  • the path may be defined using a path planning algorithm, designed to determine a shortest path between locations without collisions with obstacles.
  • Static and or dynamic obstacles may be used.
  • each detected person other than the modeled person may be included as an obstacle.
  • detected moving objects which may be cars etc.
  • the obstacles may be included in a time dependent way using prediction of their future motion.
  • a plurality of "Goto" rules may be provided, each for a respective different destination location.
  • predetermined set of destination locations may be used to define different "Goto" rules, including for example the locations of exits (e.g. gates), a location of a reception desk, a location of an office, a predetermined waiting area, a predetermined meeting point etc.
  • exits e.g. gates
  • a location of a reception desk e.g. a location of an office
  • a predetermined waiting area e.g. a predetermined meeting point etc.
  • the "Follow” rule requires the indication of a further person detected in the images.
  • This rule defines the direction of motion of a modeled person as a direction of a path from the current location of the modeled person to the current location of the indicated further person. This path may be determined in the same way as for the "Goto” rule, or by a simpler algorithm that merely uses a straight line between the locations as the path.
  • the "Follow” rule may depend on whether the indicated further person is in the field of view of the modeled person.
  • Figure 2a illustrates geometric relations between the modeled person 20, the indicated further person 22, the field of view 24 and the direction of movement 26 defined by the "Follow” rule.
  • the "Follow” rule may define the direction of motion of the modeled person 20 as the direction of movement of the indicated further person 22 (parallel motion) which applies if the distance between the locations of the modeled person 20 and the indicated further person 22 is less than a predetermined threshold and optionally under condition that the indicated further person is in the field of view 24 of the modeled person 20.
  • the rule may define switching to this direction from a path based direction, or mixing these directions, dependent on a distance between the modeled person and the indicated further person.
  • the "Avoid” rule requires the indication of a further person detected in the images.
  • Figure 2b illustrates geometric relations involved in an example of the "Avoid” rule.
  • This example of the "Avoid” rule defines the direction of movement 26 of a modeled person 20 as a direction of opposite to the direction from the modeled person toward the indicated further person 22, as defined for the "Follow” rule.
  • the "Avoid” rule may depend on whether the indicated further person 22 is in the field of view 24 of the modeled person 20.
  • the "Avoid” rule may depend on distance from the modeled person 20 to the indicated further person 22.
  • Another “Avoid” rule may represent a behavior to deviate from behavior defined by other rules in order to avoid collision with other persons.
  • the "Avoid_2" rule requires the indication of a further person detected in the images.
  • Figure 2c illustrates geometric relations involved in an example of the "Avoid_2” rule. This example of the "Avoid_2” rule defines the direction of movement 26 of a modeled person 20 in a negative way as any direction not toward the indicated further person 22, as defined for the
  • the "Avoid_2" rule defines the direction of movement 26 of a modeled person 20 as a standstill or motion in a direction perpendicular to the direction towards the indicated further person 22, as defined for the "Follow” rule.
  • the "Avoid_2" rule may depend on whether the indicated further person 22 is in the field of view 24 of the modeled person 20.
  • the "Avoid_2" rule may depend on distance from the modeled person 20 to the indicated further person 22. For example, the rule may define that it does not apply of the distance is above a predetermined threshold value.
  • the "Meet rule” may require the indication of a group of at least one a further person detected in the images. This rule defines absence of motion when the modeled person is within a predetermined distance from the persons in the group. In a further embodiment, the "Meet” rule may depend on whether the modeled person and the indicated further person(s) are in each others field of view.
  • each rule defines a direction of motion, or an absence of motion
  • rules may be used that define a speed of motion, and/or a range of possible directions and speeds of motion, or a scale factor size for the accuracy of the direction and/or speed.
  • a "Goto" rule may define a range of directions from a current location along paths with travel times to the destination that deviate less than a predetermined absolute or relative amount from the travel time along the shortest path.
  • a "Follow” rule may define a range of directions. The rule may define that size of such a range may depend on other parameters, such as the distance to a destination or indicated further person.
  • FIG. 3 shows a flow chart of operation of processing system 12.
  • processing system 12 initializes information about the area under surveillance. It sets a set of detected persons to an empty set.
  • processing system 12 receives images for a current time point from different cameras 10, or an image from one camera 10.
  • processing system 12 detects image areas in the image(s) that show persons.
  • third step 33 is part of loop wherein the set of detected persons may be expanded or reduced dependent on the images.
  • Each person may be represented by a respective index in an array of data fields, or a label.
  • processing system 12 will use the data field for the person to store data representing a track of locations for the person as a function of time and optionally data representing the person's field of view as a function of time.
  • processing system 12 may use the stored data of persons in the set of detected persons to detect and/or identify image areas that show persons in the set.
  • Motion detection may be used, e.g. by means of a measure of difference between image content in a first patch in an image for a current time point and a second patch in an image for a previous time point, for different displacement vectors between the first and second patch, and a selection of one of the displacement vectors with the least measure of difference.
  • recognition techniques such as histogram
  • processing system 12 may simply us the image positions where image content changes in or adjacent to image areas for the persons in the image for a previous time point.
  • processing system 12 may detect image areas that show persons that are not yet in the set of detected persons.
  • the image areas may be detected on the basis of motion detection, image content change in the image areas etc, optionally limited to image areas within a predetermined distance from the borders of the image(s).
  • processing system 12 updates the data representing tracks of locations for respective ones of the persons for which image areas have that have been detected in third step 33. Furthermore, processing system 12 may add persons to the set, if persons have been detected that cannot be identified with existing persons in the set.
  • the update may comprise integration of data obtained from image areas in different images for the same time point that are associated with the same person and a
  • processing system 12 determines or updates an indication of the fields of view of the detected persons.
  • this comprises a determination of an estimate of viewing direction 40 of each of the detected persons 42 from the images.
  • a field of view 44 of a person is defined by means of a range of directions including the estimated viewing direction 40 (e.g. directions at less than a predetermined angle relative to the viewing direction 40).
  • the field of view 44 may be a set of locations located along these viewing directions.
  • this set may also be limited to locations at no more than a predetermined distance (R) to the location of the detected person 42.
  • locations along a direction in the range of directions may be excluded from the determined or updated field of view if they are found to lie at a greater distance from the location of the detected person than the location of an obstacle 46.
  • Predetermined information about the location of obstacles 46 such as walls in the area under surveillance may be used for this purpose.
  • the location of detected persons 42 and/or other objects may be used to define the locations of obstacles.
  • the field of view may be determined in two or three dimensions. A two dimensional field of view may be defined by selecting only directions relative to the direction of view in a predetermined plane, for example a plane parallel to the ground surface of the area under surveillance. In a three dimensional field of view directions outside this plane may also be used. Use of a two dimensional field of requires less information to be determined from the images.
  • Image processing methods for determining of an estimate of viewing direction of a detected person from images are known per se.
  • a camera system could be used to determine the field of view of an entity such as a human or animal.
  • facial recognition software is used to determine the location and orientation of the face in the camera image.
  • Such a recognition image processing algorithm will detect salient facial features such as the eyes, mouth or chin. From this information the direction can be derived in which the entity is looking. The field of view can be determined based on this viewing direction. In case no observation direction is available; it is assumed that the entity is looking in the direction that it was last moving.
  • processing system 12 may detect image areas in the images that show faces and/or other sides of a head at the top of or above the image areas where persons have been detected. In another example processing system 12 may determine the image locations where the eyes of detected person are visible. From the face areas and adjacent other head area(s) a nominal direction of view may be estimated, be it with the knowledge that the estimate may have an error due to inaccuracy. When the head of the same person is visible in a plurality of images, a more accurate direction of view may be determined. Similarly, the detected eye positions may be used to determine the nominal direction of view, optionally using detected eye positions of the same person from a plurality of camera angles.
  • the direction of motion along detected track of the person may be used to determine the direction of view of the person.
  • the direction of motion may be used as the direction of view for example, or the direction of view may be determined from a combination of the direction of motion and the direction of view that is estimated from the images only.
  • the direction of view for a current time point is determined by updating the direction of view determined for an earlier time point, for example by changing the latter over an angle along an arc from the direction of view determined for the earlier time point to that determined from the images for the current time point, the angle being taken as a
  • processing system 12 it is not necessary that processing system 12 explicitly determines the set of locations in the field of view in fifth step 35. It may suffice that in fifth step 35processing system 12 records detected information that allows a determination whether a location is in the field of view when needed. The detected direction of view may be recorded for example.
  • processing system 12 computes match scores for each of the rules of behavior in its predetermined set of rules of behavior applied to each of the persons in the set of detected persons. That is, each detected person in the set of detected person is substituted as the modeled person in each of the rules of behavior. Moreover, this may be done, using different ones of the other detected persons as the indicated further person in the rules in different instances, for example each of the detected persons within a predetermined detected distance from the person used as modeled person. Each rule is then used to determine a direction of motion D(i) of the person (i) according to the rule, optionally after first determining whether the rule applies to the person.
  • a rule may define that it does not apply for example when a distance from the modeled person to the indicated further person, or to a destination location, is above a predetermined threshold.
  • processing system 12 may use the field of view from fifth step 35 in the rules.
  • the field of view may be used to determine whether a rule applies for a user.
  • the "Follow”, “Meet” and “Avoid” rules may define that they apply only if the detected location of the indicated further person is in the field of view of the modeled person (for example located in a direction from the detected location from the modeled person that is in the field of view of the modeled person).
  • moving obstacles may be used in the path planning algorithm plan as obstacles to be avoided, dependent on whether they are located in the detected field of view of the person.
  • computation of the match score for this rule may comprise using the direction D(i) defined by the rule to compute a function f(D(i), D0(i)) that varies with the difference with the direction D0(i) found in the track of the person, as determined from the images.
  • the computation of the function may comprise taking a scalar product of the directions D(i)*D0(i) for example. A different function may be used when the direction vector is zero. Alternatively a difference D(i)-D0(i) between the directions may be used. In embodiment, the function may depend on the scale factor size for the accuracy.
  • a value D(i)*D0(i)/ ⁇ I v(i)-v0(i) I +1 ⁇ is used, wherein v(i) and v0(i) are velocities along the directions D(i) and D0(i) defined by the rule and the observed track.
  • the result of the function may be used as an instant match score. If it is determined that the rule does not apply to a person, this may be recorded or a predetermined instant match score value may be given that is lower than the maximum possible match score.
  • a match score may be used that may have any value in a quasi continuous range of values. Instead a binary score may be used for example if a rule defines a range of directions within which detected directions D0(i) may be considered to match. Thus binary score may be set to indicate "no-match" for a detected person when it has been determined that the rule does not apply to that person.
  • time integrated match scores for the rules of behavior may be computed using the detected tracks of the persons up to a current time point rather than the direction at the current time point.
  • a time integrated match score may be computed for example by summing instant match scores computed from the directions for different time points, optionally weighted with decreasing weight with increasing distance to the current time point.
  • "Goto" rules a measure of difference between a planned path to a destination and the observed track may be used to determine the time integrated match score.
  • the instant and/or time integrated match score may comprise use of the direction of motion D(i) of the person (i) according to the rules dependent on the outcome of application of the rules to other detected persons.
  • the match score for a rule exceeds a threshold
  • the direction of motion or future path of the modeled person according to that rule may be used to define the location of an obstacle formed by that person in the future, for use in a "Goto" rule or as a future location of an indicated further person involved in rules applied to other persons.
  • the match score for a "Goto" rule for a person exceeds a threshold
  • the path of that person to the destination used in that rule may be used to define the future location of that person for use in the rules applied to other persons.
  • processing system 12 detects whether the match scores and/or time integrated match scores meet a predefined condition for generating an alert. If so, processing system 12 executes an eight step 38, wherein it generates an alert, for example by causing one of the images for the current time point to be displayed on display screen with an indication of the image area(s) that show(s) the persons that contributed to meeting meet the predefined condition. After eight step 38, or after seventh step 37 if the predefined condition is not met, processing system 12 repeats the process from second step 32.
  • processing system 12 may effect automatic precautionary actions in eight step 38.
  • Processing system 12 may activate a sign in the area under surveillance for example (e.g. switch on a light or cause a selected image to be displayed), or it may cause exits or entrances to the areas under surveillance or between different parts of the area under surveillance (herein all termed "gates of the area under surveillance", which includes doors or fences etc) to be closed or opened, according to a
  • a condition is that one of the rules is satisfied for more than a predetermined number of detected persons.
  • a rule is said to be satisfied for a detected person if all conditions for a applying the rule, such as the presence of the indicated further person in the field of view are satisfied and/or its match score exceeds a predetermined value. For example, if an alert is desired when too many persons are heading to the same destination, the predefined condition may be that the "Goto" rule for that destination must be satisfied for more than a predetermined number of detected persons. In an embodiment, a "Goto" rule for a destination is satisfied for a detected person only if its match score is larger than for the "Goto" rule for other destinations for the same detected person.
  • seventh step 37 comprises testing whether this is so for a "Goto" rule when applied to a detected person.
  • the predefined condition may be that the match score for the "Follow” rule for the same indicated further person is satisfied for at least a predetermined number of the detected persons.
  • the predefined condition may be that no predetermined rule from a predetermined set of rules is satisfied for a detected person, i.e. that the person has ceased to behave as allowed or expected.
  • predefined condition may be set so that eight step 38 is executed only when the none of these rules is satisfied during at any time in a time interval of predetermined length.
  • the predefined condition may be deactivated if an "Avoid_2" rule is found to be satisfied. Thus false alerts may be avoided if unexpected deviations from allowed behavior can be accounted for by avoidance.
  • the predefined condition a predetermined rule is not satisfied for a detected person and that no "Avoid_2" rule is found to be satisfied for the detected person in relation to any indicated further person. Such a condition may be used for example to detect whether a detected person continues to be going to a specified
  • rules may be provided for objects that may be persons or vehicles for example.
  • different sets of rules or alert conditions may be used for different types of objects, for example for persons and vehicles, the set of rules that is applied to a detected person depending on a detected type of object.
  • different sets of rules may be used for different types of persons, for example for law enforcement officers and others, or between fans of different sports teams, depending on a detected type of person.
  • a color detected color of clothing in the image areas that show persons may be used to select between different types of person.
  • rules may provide that they are applicable only if the location of the modeled person is in a specified area, or that different sub-rules apply for locations in different areas. In this case the location of the modeled person may be used to select the applicable sub-rule by comparing it with definitions of the different areas.
  • rules may be based on use of other physically limited senses that are applicable for individuals. For individuals such as humans and animals this may be hearing, smell, tactile (touch) or temperature perception.
  • knowledge about the reaction of humans on an alarm can be used to set up rules about human starting an evacuation in case of hearing an audio alarm.
  • Another embodiment would be burglars feeling a scene after setting off an burglar alarm.
  • Other senses such as smell could be used to set up rules why people avoiding foul smelling garbage bins on the streets.
  • a plurality of cameras 10 may suffice to use a single camera. However, use of a plurality of cameras directed at a same location from different angles makes it possible to obtain a more reliable direction of view. Furthermore a plurality of cameras 10 may be used to extend the area under surveillance.
  • processing system 12 may compute use the input from microphones at a set of locations in the area under surveillance to determine a synthetic sound signal at the location of a detected person (and optionally the direction dependence of this sound signal) and to one or more sound features of this synthetic sound signal, such as loudness, or loudness in a predetermined frequency band.
  • at least one of the rules may be defined to apply only if the computed sound feature(s) meet(s) a predetermined condition.
  • the "Follow” rule may depend on whether a sound feature meets a condition representing that the modeled person could have heard a signal.
  • the "Meet” rule might depend on whether a sound feature meets a condition representing that the modeled person could have heard speech at the meeting.
  • additional sensors such as microphones or chemical detectors may be provided that detect phenomena that may result in sensory stimuli for persons.
  • the determination whether rules apply could be performed at least partly dependent on the output of such additional sensors.
  • Another instance of the "Meet" rule may depend on detection of the occurrence of a tactile (touch) event such as the shaking of hands.
  • the cameras may be sensitive to different wavelengths in the electro-optical spectrum; infra-red, visual, ultraviolet or multi-spectral.
  • active sensors could used such as sonar, radar or lidar, which ar based on the transmission, reflection and reception of sound, radio waves and light respectively.
  • Other types of imaging sensors such as millimeter wave scanner or backscatter X-ray could also be applied.
  • the persons themselves can be equipped with devices that can aid their observation by sensors. This can be done with active radio beacons, using the global position system (GPS) or Radio-frequency identification (RFID) devices.
  • GPS global position system
  • RFID Radio-frequency identification
  • One example would be to install such a device in a visitor's badge or in the license plate of an vehicle.
  • Processing system 12 may be implemented as a single computer provided with a program designed to make it execute the process as described.
  • processing system 12 may comprise a plurality of processors, including for example image signal processors for respective ones of cameras 10 and/or parallel processors that perform the described process in a distributed way.
  • different computers each programmed to perform parts of the process may be used to compute match scores for different persons or different rules.
  • an electronic circuit may be used that is hardwired to perform part or all of the process.
  • a processing system, module etc will be said to be configured to perform an operation if it is programmable and has a program that will make it perform the operation and/or if it is an electronic circuit that is hardwired to do so.
  • the processing system may have one or more image processing modules for detecting image areas that show persons, determining the location and motion of these persons and to determine the directions of view of the detected persons.
  • the processing system may have one or more processing modules for testing whether the rules are satisfied.
  • the rules may be stored implicitly as part of the program of the processing system that performs determinations whether the rules are satisfied or as input data representing the rules for use by a program that determines whether the rules are satisfied.
  • the detection of a plurality of persons in the image(s) makes it possible to account for interaction between persons when it is detected whether the image(s) show a match with a rule of behavior.
  • behavior can be detected wherein a detected person follows another detected person, movements can be discounted in an evaluation of the images if an "Avoid" rule applies or deviations from a path to a destination can be discounted if other persons affect the path as obstacles.
  • the determination of fields of view of the detected persons makes it possible to account even more accurately for the interaction between persons, when it is combined with rules of behavior that have conditions dependent on persons in the detected field of view of a modeled person.
  • rule applies to behavior if the modeled person cannot have seen a cause for the behavior.
  • rules that account for directions and or speed on a (quasi-) continuous scale such as a "Goto" or "Follow” rule, makes it possible to detect specific forms of behavior.
  • additional rules such as an "Avoid" rule can be used to make such methods more robust against deviations due to external influences.
  • the viewing angle of an person is set to that of a normal human being; in the range from
  • the system is able to model behavior based processes sensory input and different underlying rules.
  • Behaviors such as collision avoidance, individual/related behavior and joining/avoid behavior discussed below, are predicted in our system using rules.
  • Each specific behavior is a rule and it only predicts the actions of a single entity. However, it takes as input the observations of the entities. For example, the positions of the other persons that are within the field of view. Based on the observations, the rule will predict an action of the person. This can be change in the person's movement direction, speed and/or location.
  • Group behavior is modelled to be a product of all the interactions between the persons that apply their rules individually. Using this approach, there is no need for pattern recognition or clustering approaches that try and find "common” or “similar” behaviors. Because of the application of rule based modeling our system can also predict person behavior instead of only classifying observed behavior.
  • That model has as input the current location of the person, a potential goal, and other persons in its field of view.
  • Figure 5 demonstrates how we can classify a person's track as being
  • Such a basic behavior reduces the number of wrongly classified behavior as being "unusual" by explicitly modeling all persons' observations. This reduces the number of false alarms of such unusual behavior when the proposed system is used in security applications, for instance.
  • Figure 6 illustrates a second example wherein we look at another behavior models based on a person's observations.
  • a person is moving similarly (in same direction, with similar speed) as a nearby agent, both persons are likely to act as a small group (i.e. they are related). In a real world case this could be friends heading for the same goal, or a host and his guest.
  • Figure 6 explains the rule in terms of simple observable relations: location, moving direction, speed and visibility.
  • Figure 7 illustrates an example.
  • This rule allows one to determine whether a person belongs to a particular group or not, based on the active behaviors. For example, for the join behavior the person does not see the other persons due to the visual occlusion (e.g. wall). In this state, it is assumed that the person is an
  • Example summary The examples show that more accurate models, considering an entity's perception will lead to better classification (collision avoidance example) and allows to determine its relation to other entities and its intentions (other two examples).

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Image Analysis (AREA)
  • Alarm Systems (AREA)

Abstract

Une surveillance électronique consiste à détecter des positions et des mouvements d'une pluralité d'acteurs détectés dans une zone surveillée. On utilise un ensemble de règles de comportement définissant chacune au moins un aspect prédit du mouvement d'un acteur modélisé. Au moins une partie des règles de comportement est spécifiée en fonction de la position et/ou du mouvement de l'acteur modélisé par rapport à un autre acteur. Pour chacune des règles de comportement, il est déterminé si ledit aspect du mouvement et de la position et du mouvement d'un premier des acteurs détectés par rapport à un second des acteurs détectés, lorsqu'il est utilisé en tant que position et que mouvement de l'acteur modélisé, respecte la règle de comportement. Il résulte de la détection que les acteurs dont il a été détecté qu'ils respectaient une règle pouvaient être signalés sur un écran d'affichage représentant une image de la zone surveillée, et que des actions telles que l'ouverture de portes pouvaient être contrôlées dans la zone où une alerte peut être émise s'il apparaît qu'une règle est respectée.
EP12758666.7A 2011-09-09 2012-09-10 Système et procédé de surveillance permettant de détecter le comportement de groupes d'acteurs Ceased EP2754089A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP12758666.7A EP2754089A1 (fr) 2011-09-09 2012-09-10 Système et procédé de surveillance permettant de détecter le comportement de groupes d'acteurs

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
EP11180807A EP2568414A1 (fr) 2011-09-09 2011-09-09 Système de surveillance et procédé pour détecter le comportement de groupes d'acteurs
PCT/NL2012/050635 WO2013036129A1 (fr) 2011-09-09 2012-09-10 Système et procédé de surveillance permettant de détecter le comportement de groupes d'acteurs
EP12758666.7A EP2754089A1 (fr) 2011-09-09 2012-09-10 Système et procédé de surveillance permettant de détecter le comportement de groupes d'acteurs

Publications (1)

Publication Number Publication Date
EP2754089A1 true EP2754089A1 (fr) 2014-07-16

Family

ID=46832574

Family Applications (2)

Application Number Title Priority Date Filing Date
EP11180807A Withdrawn EP2568414A1 (fr) 2011-09-09 2011-09-09 Système de surveillance et procédé pour détecter le comportement de groupes d'acteurs
EP12758666.7A Ceased EP2754089A1 (fr) 2011-09-09 2012-09-10 Système et procédé de surveillance permettant de détecter le comportement de groupes d'acteurs

Family Applications Before (1)

Application Number Title Priority Date Filing Date
EP11180807A Withdrawn EP2568414A1 (fr) 2011-09-09 2011-09-09 Système de surveillance et procédé pour détecter le comportement de groupes d'acteurs

Country Status (2)

Country Link
EP (2) EP2568414A1 (fr)
WO (1) WO2013036129A1 (fr)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3336746A1 (fr) 2016-12-15 2018-06-20 Nederlandse Organisatie voor toegepast- natuurwetenschappelijk onderzoek TNO Système et procédé de filtrage de contenu vidéo
CN110046535B (zh) * 2018-01-16 2023-06-23 国民技术股份有限公司 基于机器学习的智能出行时间预测系统、方法及存储介质

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9892606B2 (en) * 2001-11-15 2018-02-13 Avigilon Fortress Corporation Video surveillance system employing video primitives

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
CRISTANI M ET AL: "Socially intelligent surveillance and monitoring: Analysing social dimensions of physical space", COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW), 2010 IEEE COMPUTER SOCIETY CONFERENCE ON, IEEE, PISCATAWAY, NJ, USA, 13 June 2010 (2010-06-13), pages 51 - 58, XP031728367, ISBN: 978-1-4244-7029-7 *
KAMIJO S ET AL: "An incident detection system based on semantic hierarchy", INTELLIGENT TRANSPORTATION SYSTEMS, 2004. PROCEEDINGS. THE 7TH INTERNA TIONAL IEEE CONFERENCE ON WASHINGTON, DC, USA OCT. 3-6, 2004, PISCATAWAY, NJ, USA,IEEE, 3 October 2004 (2004-10-03), pages 853 - 858, XP010772224, ISBN: 978-0-7803-8500-9, DOI: 10.1109/ITSC.2004.1399015 *
LUIS M FUENTES ET AL: "Tracking-based event detection for CCTV systems", PATTERN ANALYSIS AND APPLICATIONS, SPRINGER-VERLAG, LO, vol. 7, no. 4, 1 December 2004 (2004-12-01), pages 356 - 364, XP019381470, ISSN: 1433-755X, DOI: 10.1007/S10044-004-0236-Z *
See also references of WO2013036129A1 *

Also Published As

Publication number Publication date
EP2568414A1 (fr) 2013-03-13
WO2013036129A1 (fr) 2013-03-14

Similar Documents

Publication Publication Date Title
Sathyamoorthy et al. Frozone: Freezing-free, pedestrian-friendly navigation in human crowds
Rudenko et al. Human motion trajectory prediction: A survey
Kooij et al. Context-based pedestrian path prediction
Garcia et al. Sensor fusion methodology for vehicle detection
US10186123B2 (en) Complex event recognition in a sensor network
US8655020B2 (en) Method of tracking an object captured by a camera system
Fernández-Caballero et al. Human activity monitoring by local and global finite state machines
Zhang et al. Modeling pedestrians’ near-accident events at signalized intersections using gated recurrent unit (GRU)
Gidel et al. Pedestrian detection and tracking in an urban environment using a multilayer laser scanner
Morris et al. A survey of vision-based trajectory learning and analysis for surveillance
Schulz et al. A controlled interactive multiple model filter for combined pedestrian intention recognition and path prediction
Zhang et al. Prediction of pedestrian-vehicle conflicts at signalized intersections based on long short-term memory neural network
EP2345999A1 (fr) Procédé de détection automatique et de suivi de plusieurs objets
Nasernejad et al. Modeling pedestrian behavior in pedestrian-vehicle near misses: A continuous Gaussian Process Inverse Reinforcement Learning (GP-IRL) approach
Ferryman et al. Robust abandoned object detection integrating wide area visual surveillance and social context
Castro et al. An expert fuzzy system for predicting object collisions. Its application for avoiding pedestrian accidents
US9977970B2 (en) Method and system for detecting the occurrence of an interaction event via trajectory-based analysis
US9965687B2 (en) System and method for detecting potential mugging event via trajectory-based analysis
Liu et al. People detection and tracking using RGB-D cameras for mobile robots
Chang et al. A hybrid CNN and LSTM-based deep learning model for abnormal behavior detection
Rouse et al. Feature-aided multiple hypothesis tracking using topological and statistical behavior classifiers
Guo et al. Pedestrian tracking based on camshift with kalman prediction for autonomous vehicles
Otto Fusion of data from heterogeneous sensors with distributed fields of view and situation evaluation for advanced driver assistance systems
Patino et al. Abnormal behaviour detection on queue analysis from stereo cameras
Kemmotsu et al. Recognizing human behaviors with vision sensors in a Network Robot System

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20140328

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: NEDERLANDSE ORGANISATIE VOOR TOEGEPAST- NATUURWETE

17Q First examination report despatched

Effective date: 20150625

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

APBK Appeal reference recorded

Free format text: ORIGINAL CODE: EPIDOSNREFNE

APBN Date of receipt of notice of appeal recorded

Free format text: ORIGINAL CODE: EPIDOSNNOA2E

APBR Date of receipt of statement of grounds of appeal recorded

Free format text: ORIGINAL CODE: EPIDOSNNOA3E

APAF Appeal reference modified

Free format text: ORIGINAL CODE: EPIDOSCREFNE

RIN1 Information on inventor provided before grant (corrected)

Inventor name: BOVENKAMP, ERNST GERARD PIETER

Inventor name: VAN REST, JEROEN HENDRICUS CORNELIS

Inventor name: VAN DER MARK, WANNES

Inventor name: HALMA, ARVID HENDRIK ROALD

Inventor name: EENDEBAK, PIETER THIJS

APAF Appeal reference modified

Free format text: ORIGINAL CODE: EPIDOSCREFNE

REG Reference to a national code

Ref country code: DE

Ref legal event code: R003

APBT Appeal procedure closed

Free format text: ORIGINAL CODE: EPIDOSNNOA9E

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED

18R Application refused

Effective date: 20210623