JP2008211781A - System for modeling movement of objects in certain environment and method executed by computer - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and system for tracking models and predicting object movement in an environment. <P>SOLUTION: Sequences of temporally and spatially adjacent events sensed by a set of sensors are linked to form a set of tracklets. Each tracklet has an associated starting and terminating location. The tracklets are used to construct a directed graph including starting nodes, terminating nodes, and intermediate nodes connected by edges. The intermediate nodes can be split nodes where the tracklets diverge onto different tracks, and join nodes where multiple tracklets converge onto a single path. Probabilities are assigned to the edges to model and predict movement of the objects in the environment. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

[Related applications]
This application is a part of US patent application Ser. No. 11 / 565,264, “Monitoring System and Method for Tracking and Identifying Objects in the Environment,” filed November 30, 2006, by Ivanov et al. This is a continuous continuation application.

  The present invention relates generally to surveillance systems and object tracking methods, and more particularly to modeling object motion from surveillance data.

  Video cameras and relatively simple sensors make it possible to construct a mixed-type surveillance system for large environments. The sensor cannot identify the object, but can detect the object in a relatively small area. When the image is available, identification can be made from the video (video) image obtained by the camera.

  Storing or accumulating video obtained by such a system can exceed multi-terabytes of data. Clearly, it is practically impossible to retrieve stored data collected over months for a particular object. Therefore, it is desirable to find an object by a discrete event of motion detected by a simple sensor. In addition, it is desirable to model the movement of an object from a motion event to predict the movement of the object.

  In conventional surveillance systems, tracking of objects such as humans, animals, cars, etc. is usually done by image and video processing. The disadvantage of such a surveillance system is that when a specific object needs to be tracked and identified, it is necessary to observe the object with a camera. However, many surveillance environments require many video cameras to provide the complete surveillance range required for accurate operation. A large amount of video stream increases the computational load on the surveillance system to operate correctly.

  It is an object of the present invention to provide a system and method for tracking and identifying moving objects (eg, people) by using a mixed network of various sensors, cameras and surveillance databases. is there.

  A small number of PTZ cameras are placed in the environment to be under surveillance. Even if the number of cameras is relatively small, the amount of video (video) data can exceed the multi-terabyte storage capacity.

  A video camera can only acquire an image of a part of its environment. For this reason, it is difficult to track and identify an object using only a camera. Even if the monitoring range of the camera is complete, the time for searching for video data becomes impractical.

  Thus, the environment also includes a dense arrangement of sensors, which essentially cover all public areas. An event has an associated sensor identification and time. This makes the total amount of sensor data much smaller and simplifies processing. Sensor activation events are correlated spatially and temporally to a video image to track a particular individual, even if an individual person is not continuously viewed by the camera.

  It is possible to create a directed graph that represents the movement of an object in the environment. These graphs have a start node, an end node, and an intermediate node. Intermediate nodes lead to ambiguities. Probabilities are assigned to the edges connecting the nodes so that the objects show the likelihood of starting and ending their movement at a particular location. The movement (movement) of an object can be predicted from the probabilities even if there is an ambiguity in the movement and the graph representing it.

  Embodiments of the present invention provide a mixed mode monitoring system. The system includes many relatively simple sensors and a relatively small number of movable cameras. This reduces cost, complexity, network bandwidth, storage capacity, and processing time compared to conventional monitoring systems.

  Objects in an environment are tracked by the camera using the available front and back (context) information from the sensors. In order to instantly determine the track (track) of a particular object, the previous and subsequent (context) information collected over many months can be retrieved. The image of the corresponding object can then be used to identify the object. Such a thing is not actually possible with a conventional monitoring system that requires retrieval of a huge amount of video data.

  Embodiments of the present invention also provide a method and system for modeling and predicting object movement by determining the probability for a track (track). As a salient feature, the present invention does not require a complete track (track) prior to analysis. Most conventional surveillance tracking systems require reliable pre-processing steps. This implies high fidelity sensors and computationally complex methods to resolve all ambiguities.

  In contrast, the present invention can also work with relatively simple (binary) motion sensors and imperfect tracks with ambiguities. The present invention resolves ambiguities by assigning probabilities to edges in a graph representing a track.

Monitoring System As shown in FIG. 1, a monitoring system in which a tracking module is implemented in accordance with an embodiment of the present invention includes a relatively large set of wireless network sensors (dots) 101, a relatively A small set of pan-tilt-zoom (PTZ) cameras (triangles) 102. The ratio of sensor to camera is very large, for example 30: 1 or higher.

Sensor The sensor may be a motion (motion) sensor, or may be a door sensor, an elevator sensor, a thermal sensor, a pressure sensor, or an acoustic sensor. A motion sensor such as an infrared sensor can detect the movement of an object in the vicinity of the sensor. The door sensor can typically detect a door opening / closing event indicating a person passing through the doorway. Elevator sensors can similarly indicate the arrival or departure of people in an environment. For example, an acoustic sensor such as a transducer or a microphone can detect an activity in a certain region. The sensor can be mounted on an optical switch in the environment or a power switch of an office device. The mat (rug) pressure sensor can also indicate traffic passing through. Security sensors such as badge readers in the entrance passage to the environment can also be incorporated.

  Each sensor is relatively small, for example 3 × 5 × 6 cm for a motion sensor. In a preferred embodiment, these sensors are closely spaced in a public area, spaced from each other by a distance of about 10 meters or less, and mounted on a ceiling, wall, or floor. It should be noted that the spatial arrangement and density of sensors can be adapted to suit a particular environment and the traffic flow in that environment. For example, areas with high traffic (traffic) have a higher population density than areas with low traffic.

  In one embodiment of the invention, the set of sensors communicates with the processor 110 (see FIG. 1) using industry standard IEEE 802.15.4 radio signals. This is the physical layer typically used by Zigbee type devices. Each battery-powered sensor consumes about 50 μA in detection mode and 46 mA during communication. The communication interval by activation is about 16 milliseconds. It should also be noted that the sensor may be embedded in hardware or may use other communication technologies.

  When an event is detected by any of the sensors 101, a sensor identification symbol (SID) and a time stamp (TS) corresponding to the event are broadcast simultaneously or otherwise sent to the processor 110. The processor stores the sensor data in the memory as a monitoring database. The identification symbol originally indicates the location of the sensor, and thus the location of the event that caused the activation. Only a few bytes are required to record an event. Therefore, compared with video data, the total amount of sensor data collected over a long period is essentially trivial.

  The set of cameras is used to acquire video (video) data (image series). The image has a unique camera identification symbol (CID or location) of the camera and a frame number (FN). As used herein, frame number is synonymous with time. That is, the time can be calculated directly from the frame number. In addition, every time is associated with a set of pan-tilt-zoom parameters for each camera so that the visible part of the scene near the sensor at any time can be calculated during the query from the database. .

  Cameras are typically mounted on the ceiling in strategic locations (locations), such as locations where all traffic in the environment must pass sometime to provide maximum coverage. . The PTZ camera 102 can be pointed and focused in any general direction. Although not necessary, any nearby video camera can be directed to a scene near the sensor by detecting an event in order to acquire a video image. The associated sensor ID and TS can later be used to retrieve a series of images or video clips associated with the event. It should also be noted that if no event is detected in the vicinity of a sensor near a particular camera, image acquisition can be interrupted to reduce the required storage capacity.

  Examining video data acquired over many months of operation to identify specific objects, such as finding specific events or finding tracks of specific objects, is a challenge.

Tracklet and Tracklet Graph As shown in FIG. 2, in one embodiment of the present invention, a set of tracklets (track pieces) 210 is used. A corresponding tracklet graph 200 is collected from a set of tracklets 210. A tracklet is formed by linking a series of temporally adjacent discrete events (events) with a series of spatially adjacent sensors 101. A tracklet is a basic building block of the tracklet graph 210. The term “discrete event” is used to indicate that motion near the sensor (movement) can be transmitted in a single timestamp bit. This is different from a camera that plays a continuous signal in the form of video.

  We call the process to find the previous predecessor or successor event linked to the current event. To improve system performance, tracklet linking and storage (storage) can be performed periodically. For example, linking and saving can be done at the end of one business day or every hour. As described above, when it is necessary to perform a search, it is possible to easily use a tracklet stored in advance.

  In the assembled tracklet graph 200, tracklets are directed edges connected by nodes of the graph. The nodes of the graph encode the relationship of each tracklet with the immediately following or immediately preceding tracklet. A node can have one of four types: That is, a track start (start) 201, a track join (join) 202, a track split (branch) 203, and a track end (end) 204.

Track start (start)
The track start node represents the first (first) event in the tracklet, and no preceding event can be associated with that sensor within a predetermined time interval. As used herein, “predecessor” means a previous event at an adjacent sensor. The “time interval” can be substantially limited to the time required for the pedestrian to move from one sensor to the next adjacent sensor.

Track join
A track join node represents an event in the tracklet graph such that there are multiple preceding events that can be associated with that sensor within a predetermined time interval. That is, the tracklet join node represents that a plurality of preceding tracklets converge to a single successor tracklet. A single valid predecessor tracklet cannot exist because it is already linked to the current tracklet.

Track split (branch)
A track split node represents an event in a tracklet such that there are multiple successor tracklets that can be associated with that sensor within a predetermined time interval. That is, the tracklet split node indicates that a single preceding tracklet branches to a plurality of successor tracklets. A single valid successor tracklet cannot exist because it is already linked to the current tracklet.

Track end (end)
The track end node represents the last event in the tracklet that is not associated with any subsequent event within a predetermined time interval. All tracklets form a set of graphs, each of which represents an inherent ambiguity with respect to the actual track (wake) that the object has traveled.

  A tracklet graph is a set of tracklets related to events that can be imposed by a user or collected by temporal and spatial constraints that can be “learned” over time.

  In the tracklet graph of FIG. 2, there are two starting tracklets, which then converge to a single track. The converged tracklet is branched twice to become four end points. The tracklet graph is a central representation of the events we use for object tracking.

Extended Tracklet Graph For the purpose of extended tracking when objects disappear from view of the sensor network, it is possible to collect two spatially adjacent and temporally adjacent tracklet graphs. it can. This situation frequently occurs in an environment where people being tracked exit a public area such as a corridor and enter an area such as an office. An event entering the office terminates the preceding tracklet at the tracklet end node when a person is no longer sensed or observed. When leaving the office, the person can be tracked again in the successor graph. When a person enters the office, he may have to leave, for example after a long period of time. In this case, spatial constraints can be strictly enforced, but temporal constraints can be relaxed.

  The graph is collected under the condition that one of the track end nodes of the tracklet in the preceding graph has a time stamp that is smaller than the time stamp of at least one tracklet start node of the tracklet in the successor graph.

Determining sensor visibility One goal of the present invention is to determine from which of a plurality of cameras the region in the vicinity of a sensor can be viewed. This minimizes the amount of irrelevant images presented to the user.

  To achieve this goal, all cameras in the system are calibrated to the sensor location. In our system, each sensor is associated with various pan, tilt and zoom parameters for each camera that make the event that caused the sensor activation visible from that camera. If the PTZ parameters of each camera are stored in the monitoring database each time the camera direction is changed, the “visible” area will be displayed when the tracklet is retrieved from the database every time the sensor is activated. It can be compared with the PTZ parameter. Once the camera's PTZ parameters are in the visible region of the sensor, the sensor activation (event) is then considered visible and a series of images from the corresponding camera are retrieved as video evidence. As will be discussed below, this evidence is subsequently displayed to the user during the tracklet selection process using the user interface.

Human-guided tracking We can show in simple scenarios the tasks related to human-induced tracking and searching that we solve in our system.

  A laptop computer was reported stolen from the office between 1:00 pm and 2:00 pm. There was no direct camera surveillance area available for the office. The user needs to find all the people who were able to pass the office at that time, and if possible, identify them and gather evidence that links the individual to the event. In such a situation, the operator wants to identify the individual by identifying all trucks (wakes) starting at the office door and examining all available video evidence. Will.

General principles of object tracking in mixed-mode sensor networks The track start and track end nodes are the clear beginning and end of a complete track. By the way, automatic resolution of the ambiguity between the track split (wake branch) and the track join (wake combination) is not possible by using only the sensed event. The ambiguity between splits and joins is due to the perceptual limitations of the sensor network to any feature other than events at or near the sensor.

  In such a situation, an event where two people cross a passage in the hallway (meet) will cause the system to generate at least four tracklets containing events for each person before and after a possible intersection. Without further information, there is an inherent ambiguity in the interpretation of this set of tracklets. For example, the two people can cross each other or meet and turn back the way they came. It is impossible to map the identity (identity) of these tracks (tracks) and maintain their continuity with absolute certainty only from those events.

  From the point of view of these ambiguities, we conduct observations with the following simplification.

  The user need not remove the ambiguity of the entire graph. The user may remove the ambiguity of the track join node that starts the selected tracklet or the ambiguity of the track split node that ends the tracklet for forward or backward graph traversal. Each is only needed.

  Resolving track join and track split ambiguity can be simplified by taking into account the video clips associated with each candidate track.

  The first observation significantly reduces the amount of tracklets that need to be considered as possible candidates to be collected on the track. In one embodiment, the user tracks only one person at a time. Thus, the system need only resolve (analyze) the person's movement while effectively ignoring other events. For the example of two humans crossing in the aisle, we see that one tracklet is selected before the intersection and only two tracklets are considered to be a possible sequence rather than all four Assume that there is a need. With this iteratively focused approach to tracking and track ambiguity, we can potentially reduce the complexity of the problem from exponential to linear (linear function).

  The second observation is that when split and join ambiguity arises, the system correlates the time and placement of the tracklet with the video from the most recent camera, and To determine whether a tracklet is a likely sequence, it implies that the corresponding video clip can be displayed to the user.

  By using only a network of sensors, it may be possible to develop an automatic tracking procedure that attempts to infer the motion dynamics of an object. However, it is inevitable that any such processing procedure will make a mistake. In surveillance applications, commitments (involvement, commitment, response) to the results of tracking processes that are only slightly inaccurate can be quite expensive.

  Thus, our tracking scheme uses human-guided techniques with tracklet graphs as basic contextual information representing tracking data. It should be noted that the sensor data on which tracking and retrieval is based is very small and can therefore proceed quickly, especially when compared to conventional retrieval of video data.

  The main focus of our system is to retrieve a large amount of video data efficiently in a very short time by using events. For this reason, we are primarily interested in reducing the false negative rate (false negative rate), but reducing the false positive rate (false positive rate) is a distant secondary goal. In order to achieve these goals, we employ a track assembly mechanism as described below.

Tracklet Set Processing The human-guided tracking process of our system includes a subset (subset) of one or more sensors that we expect a track to start with, and optionally a time interval. Start with a choice. For example, in our system, if the sensor is placed in a public area outside the office, the user can use a floor plan to show that the person is probably activated when leaving a particular office A subset of sensors can be selected.

  By performing a fast search in the event database, we can identify every instance (example) of a tracklet that starts with one of the selected sensors. Here, the user can select a single instance (example) of the tracklet for further exploration. By identifying the approximate point in time when the track starts, the above search can be expedited.

  When the first tracklet is selected, a corresponding tracklet graph is constructed. The collected track graph includes a plurality of tracklets associated with a series of temporally and spatially adjacent events. The selected tracklet is drawn on the floor plan to the point where the end (end), split (branch) or node (join) is, as shown in FIG. When the end point (end point) is reached, the track 300 is completed. The location of the person along the track 300 in the floor plan is visually shown on the user interface by a thick line 301 on the track 300.

  If the end of a tracklet has a split or join node, the track will not be terminated and multiple tracklet graphs will be created to collect candidate tracklets into a consistent track. The tracklet aggregation process is repeated using the above. During this process, at each ambiguous (unclear) point (split node or join node) in the graph, the user selects a subgraph for further traversal. In order to identify people and select the correct successor tracklet, available video images from cameras directed to any of the sensor activations belonging to the corresponding tracklet can be displayed. Also, automated techniques such as object and face authentication can be used for the identification described above.

  The process is illustrated in FIG. 4 using a selection graph. In the selection graph, the video image 401 represents a plurality of video clips available from a plurality of cameras directed to a plurality of sensors included in a corresponding plurality of tracklets. The diamond 410 shows an ambiguous point (point) and possible competing tracklets following the ambiguous point. The edges in the graph indicate the presence of tracklets.

  Note that the tracklet selection graph of FIG. 4 relates to the tracklet graph of FIG. 2, but is not the same. In fact, the graph of FIG. 4 represents a general selection graph, which can be used for traversing the tracklet graph forward (as shown) or backward in time. In the former case, the start (start) and end (end) nodes of the selection graph in FIG. 4 have the same meaning as those in the tracklet graph, but only the diamond represents a split (branch). . Track joins are independent of the forward selection process because they do not offer forward selection alternatives. In contrast, if the selection graph is used for backward traversal, the start and end nodes of the selection graph have the opposite meaning of that of the tracklet graph, and only the diamonds are joined. Represents.

  In any case, the tracklet selection graph represents a set of tracks according to the tracklet graph, starting with the first selected tracklet and traversing the available camera frame 401 indicated by the start node 201. . Since the ambiguous points (points) are known, at each such point, the system can present an ambiguous set of tracklets to the user for resolution.

  For example, in the first step, the ambiguous point 410 represents a three-way split (three-way branch) from the current node. The leftmost tracklet leads to two camera views 431. The middle tracklet ends without camera view. The third tracklet has one camera view and then leads to a two-way split (bidirectional branch). Each of these tracklets can be drawn on the floor plan. After the selection is made, rejected tracklets are removed from the floor plan. Processing continues until an end track 204 is encountered.

  When the end of the track is encountered, the track set processing process can be terminated. However, if the user has reason to believe that the actual track will continue from the end point, a tracklet graph expansion mechanism as described above is used. The system searches the database within a predetermined time interval to find a new tracklet that begins at the position of the terminated track. If such a tracklet is found, the corresponding video clip is identified and displayed to the user in the tracklet selection control panel, as described below. When the user selects the first track for an extended line of tracks, a tracklet is added at the end of the collected track and a new tracklet graph is created starting with the selected tracklet. The selection process is then continued repeatedly as before to further extend the complete track of the object. In a complete track, all join and split nodes are removed and the track only contains a single start tracklet and a single end tracklet.

User Interface As shown in FIG. 5, in one embodiment, the user interface has five main panels: floor layout diagram 501, timeline (time series) 502, video clip bin (storage device) 503, track. A let selector 504 and a camera view panel 505 are included.

  The floor plan is as shown in FIG. The position of the person along the track 300 in the floor plan is indicated by a “bulge” 301 (broken line) in the track 300. For each sensor, timeline 502 shows the event. Each column in the timeline corresponds to one sensor, and time progresses from left to right. The vertical line 510 indicates the “current” playback time. A menu and icon 520 can be used to set the current time. A “knob” 521 can be used to adjust the speed of playback. You can move the timeline forward and backward by dragging the line with the mouse. The short line segment 200 represents the tracklet, and the line 300 represents the resolved track, see FIG.

  The video clip bin shows an image of a clip (image series) selected for object identification. In essence, the collected series of images associated with a track in a video clip bin is video evidence associated with the track and the object.

  The tracklet selection control shows the current state of the decision graph of FIG.

  An image corresponding to the current time and the selected position is shown on the camera view panel 505. The image can be selected by the user or automatically selected by a camera scheduling procedure. To form the video clip bin 503, a scheduling procedure can be invoked during clip playback.

Tracking Method In an embodiment of the present invention, the tracking process includes two phases. That is, a phase for recording monitoring data and a phase for searching for tracking an object.

  The recording phase is shown in FIG. FIG. 6 shows a method for storing sensor data in the monitoring database 611. The monitoring database stores events 103 acquired by a set of sensors 101. A series of events that are temporally and spatially adjacent to a selected subset (subset) of sensors are linked 630 to form a set of tracklets 631. Each tracklet has a tracklet start node and a tracklet end node. The tracklet is stored in the monitoring database.

  Simultaneously with sensor activation, the sequence of images 104 acquired by the set of cameras 102 is recorded in the computer storage device 612. Each event and image is associated with one camera (position) and time. As described above, the PTZ parameter of the camera can also be determined.

  The tracking phase is shown in FIG. This phase selects 620 a subset of the sensors that a track is expected to start 620, finds multiple tracklets that can be used as the start of multiple tracks 625, and sets the first as the start of that track. Including selecting one tracklet 640 and performing track set processing 680.

  The track set process begins by configuring 650 a tracklet graph 651 for the selected tracklet. The tracklet graph 651 shows possible tracklet join nodes where multiple preceding tracklets merge into a single successor tracklet and possible tracks where a single preceding tracklet branches into multiple tracklets. And let split node.

  The tracklet graph 651 is traversed repeatedly, starting with the first selected tracklet. Following the graph, the next ambiguous node is identified, and images that are temporally and spatially correlated with sensor activations (events) included in the candidate tracklet are retrieved from computer storage 612 and displayed 660. And the next tracklet 670 to be combined with the collected track 661 is selected 670.

  The process ends when the collected track 661 is terminated with a tracklet having a track end node as its end point, and all join and split nodes are removed from the graph.

Object Track Modeling and Object Movement Prediction So far, we have described a method for determining where an object (eg, a human) is located according to discrete motion events. Next, we will try to identify ambiguous movements (movements) of objects by predicting where the objects may go from monitoring data acquired in an environment. Specifically, we will try to predict movements that may behave abnormally or suspiciously.

  We describe multiple methods for modeling the movement of objects in the environment, even when high quality (video) tracking is not possible or not feasible. These methods are effective for any object of interest (human, car, etc.) moving in a complex environment. The term “complex environment” means that we are likely to have hundreds or thousands of sensors and a similar number of large office buildings, manufacturing plants or parks with moving objects such as workers or cars. It means a parking lot. In complex environments, the likelihood of objects associated with each other is relatively high, and the identity (identity) of the objects may be unknown.

  First of all, we compare high quality and low quality by tracking. High quality tracking is the act of processing a continuous stream of sensor data from a collection of sensors, such as cameras and motion sensors, to generate a model of a particular individual's movement. The model can be used to answer questions about the person's start, middle and end positions along the track the object (person) has moved. In general, tracking is considered to generate a deterministic track model (ie, the best guess about the person's track through the environment).

  The ambiguity of the model is thought to be errors, tracking failures, unusual or unusual movements. High quality tracking systems can become very complex in an attempt to resolve all ambiguities. If there are multiple individuals in the environment, ambiguity will inevitably arise.

  In a surveillance system with the goal of generating a deterministic motion (movement) model for each object, those ambiguities are the only true for hypotheses about the true composition of the object and for future events. Until the hypothesis remains, it is represented as a set of environments that must be propagated, tested, and possibly pruned.

  Since ambiguities arise from the interaction of objects or individuals, the number of hypotheses that must be considered can increase exponentially compared to the number of individuals in the environment.

  Low quality tracking is when it is impractical to resolve the ambiguity or it is practically impossible to allow ambiguity.

  A high quality tracking system generally requires a high quality sensor because the event must contain enough special information to resolve the ambiguity. People tracking systems may resolve such ambiguities by recognizing faces, matching clothing colors, or relying on key cards and biometric scans. unknown.

  In situations where these types of high-quality sensors are not available, simple sensors that can only transmit discrete events (for example, the sensor turns off when there is no movement and the sensor turns on when there is movement) In an environment with, it may still be useful to derive inferences about patterns of object movement (movement). Rather than relying on the full tracking output, we rely on these inferences from a set of incomplete tracking models obtained by low quality motion sensors that can only send discrete motion events. How to obtain will be described.

Tracklets We use the concept of tracklets, embedding their representation in a probabilistic framework. The basic concept of one tracklet is that it collects discrete events that are clearly related to each other. The concept of the tracklet, as described above, allows the system to efficiently interact with the user of the system in order to refine the imperfect tracking model in the tracklet graph into a high quality track model. It was developed as a method to express ambiguity in object tracking for application to surveillance.

  Now we embed tracklets and tracklet graphs in a probabilistic model to allow inferences from a set of probabilistic graphs. In this way, the present invention understands the pattern of movement (movement) of a group of individuals or other objects over a long period, rather than a specific movement of a single or small group of individuals over a short period of time. Make it possible to do.

  As shown in an extremely simplified manner in FIG. 8, a set of events 801 are arranged in a line through space and time. These events can be, for example, motion events detected by a simple low quality motion sensor, as described above.

  In the following, these events are synonymous with their location in the environment. One sensor can only detect movement in a relatively small area of the environment, for example, movement of an object in an area of about 5-10 meters. Thus, multiple sensors detect moving objects in a relatively small area, and multiple (detected) events can be directly related to multiple sensor locations. If those sensors are placed in the corridor and assigned to a particular individual adjacent to the office, a particular individual's traffic (traffic) pattern can be predicted with a relatively high probability.

  If multiple events are spatially close to each other, arranged in time, and isolated from other events, a simple model of object movement in that environment It allows us to collect the event on a single track 802, as shown by the solid arrows passing through. The arrow is an abstraction of a series of discrete events into one tracklet, which tracks the fact that all events in the set were generated by the same individual or a small group of individuals. This is a model with very high probability.

  Furthermore, if those series of events are not followed by any event that is near the final event in space and time, we say the tracklet has ended. This is represented by box node 803. Similarly, if a tracklet is not preceded by an event that appears to be connected to other events in space and time, the tracklet will begin in that space and time, and this is indicated by triangle node 804. Subsequently, we can only drop the high-level tracklet abstraction 805 from the start node A to the end node Z and drop those events themselves.

A tracklet graph is shown. We call the simple graph 805 of FIG. 8 a graph γ ₀ . Graph γ ₀ shows that the probability P that the end event detected at node Z is generated by the same individual who generated the event at start node A is high.

で表される確率８００は、ノードＺで検出されたイベントが、ノードＡのイベントに続いて起こるという確率が明らかに（曖昧ではなく）１であることを意味する。ここで、

は、後に続くということを示している。 If there is no loss of generality, considering the graph γ ₀ ,

The probability 800 represented by means that the probability that the event detected at node Z will follow the event at node A is clearly 1 (not ambiguous). here,

Indicates that it will follow.

  In general, the probability P is 0 if the object is not in a known position, the probability P is 1 if the object is in a known position, and in other cases the probability P is 0 <P <. 1.

Tracklet graph The ambiguity in the environment produces a more complex graph. Our graph includes nodes connected by directed edges or tracklets. All events that share a possible common cause are tied to the same connected graph. The nodes in the graph can include a start node, an intermediate node, and an end node. For purposes of this description, the graph includes at least one intermediate node, such as a split node or join node. Otherwise, there will be no ambiguity, and the problem of tracking objects becomes deterministic and trivial. Intermediate nodes can include split nodes and join nodes.

  Probabilities are assigned to edges (tracklets) to indicate the likelihood that an object will start at a particular node and end at another node. In other words, probability is the likelihood that an event at the end node follows the event at the start node, and that these events are caused by the same object. The track extends from the start node to the end node and includes intermediate nodes to create ambiguities.

  FIG. 9 shows a more complex graph representing two or more individuals crossing a track in the environment.

  Angle portions 901 and 902 represent a join j and a split s, respectively. That is, several events occur following the split s within a sufficiently small temporal and spatial neighborhood, creating an ambiguity point. In the split case, there are several individuals who are moving together or very close to each other so that they are considered to be co-actors in one tracklet. After the split, each individual's track that crossed the tracklet from j to s is ambiguous. That is, it is ambiguous whether each individual has moved from the split s to the end node Z or moved from the split s to the end node Y along the tracklet.

ここで、我々は、両方の結果が等しく起こりうるように、ｐ＝ｑと決めてもよいし、或いは、我々の予測を一方或いは他方に偏向させるように、移動パターンを学習しても良い。そのパターンは、システムのユーザにより手動的に学習されることもできるし、或いは時間の経過とともに自動的に学習されることもできる。ノードＡの個人または物体は、どこかに行かなければならないので、グラフにおけるその確率は、合計すると１、すなわちｐ＋ｑ＝１にならねばならない。 Considering this graph we call γ ₁ , we don't know the personal track starting at node A. The probability of this ambiguous point is expressed by the following equation (1).

Here we may decide p = q so that both results can occur equally, or we can learn the movement pattern to bias our prediction to one or the other. The pattern can be learned manually by the user of the system or can be learned automatically over time. Since the individual or object at node A has to go somewhere, its probabilities in the graph should sum to 1, ie p + q = 1.

The graph can be arbitrarily complex. The graph γ _{2 in} FIG. 10 is much more complex with several start nodes A, B and C, several splits s and joins j, and end nodes W, X, Y and Z. It is a simple graph.

It is possible to connect the start node A to any end node {W, X, Y, Z} by tracing the directed edge in the graph. For example, the probability is as follows:

我々は、それらの全ての結果が等しく起こりそうであると決めてもよいし、或いはまた、個人の習慣的な動き等のような、以前に学習された情報に基づいて、それらの確率を偏倚させてもよい。 Now, so that an individual starting at node A can end up at any of those positions with non-zero probabilities, we determine the probabilities below (2) through (5) Shown in

We may decide that all those outcomes are likely to occur equally, or we bias their probabilities based on previously learned information, such as personal habits etc. You may let them.

で表される確率が成立するので、ノードＣで開始しノードＷまたはノードＸで終了する個人に対する妥当な説明は無い。そのため、次式（６）〜次式（９）が成立する。

A more interesting case is that individuals starting at node C do not have as much ambiguity at their final destination. Since the edge is directed from s1 to j2, that is,

Therefore, there is no valid explanation for an individual who starts at node C and ends at node W or node X. Therefore, the following expressions (6) to (9) are established.

It is worth noting that the graph is complete. That is, if there is a series of possible events associated with an object starting at node C and ending with one end node, that end node is in the graph. However, if node V is not an end node in the graph, the probability is expressed by the following equation.

Similarly, for a start node that cannot be found in a particular graph, there are no possible connections in the graph between the start node and the end node for the time spanned by that graph. For example, the probability is expressed by the following equation.

A set of graphs. At some moment, several objects or individuals who move around in the environment so that they are not visually linked accidentally cross the track, creating an ambiguity, resulting in a specific tracklet graph. Instance (example) becomes a cohort (group) of individuals who generated γ. The above analysis makes some weak assertions (assertions, assertions) on those individual tracks.

  We now consider a set of graphs Γ that illustrate the movement of a group of objects over a longer period of time.

ここで、Ｎは、そのセットΓにおけるグラフの数である。ノードＡを開始ノードとして含むセットΓにおけるグラフの数はＭであり、それは、正規化因子（係数）として使用できる。１つの接続に対する証拠、すなわち次式

で表される確率は、トラックレットグラフを作図して、次にグラフの数Ｍと共に、各センサ対に対する証拠を蓄積することによって、イベントが受信されるにつれて蓄積され得る。 We can ask questions in the form of "what this set of graphs tells us about the probability that an individual moves from one place to another in a building". We search all graphs that contain, for example, node A and start with node A, and find the number of graphs that contain possible connections between nodes A and Z, possibly weighted by individual probabilities. Counting can answer this question. This is expressed by the following equation.

Here, N is the number of graphs in the set Γ. The number of graphs in the set Γ including node A as a start node is M, which can be used as a normalization factor (coefficient). Evidence for one connection, ie

Can be accumulated as events are received by plotting tracklet graphs and then accumulating evidence for each sensor pair along with the number M of graphs.

  It is possible to accumulate evidence for repetitive movements (motions) even when there is no obvious (unambiguous) tracklet showing the movement of the object as a whole. If the traffic starting at node A always ends at node Z, then all graphs in the set Γ including node A as the starting node will contain valid tracks from node A to node Z. This is a significant amount of evidence.

は、平均証拠確率

である。その確率は、その動きのあらゆる例が完全に明白（曖昧さの無い）であるときよりも小さく、そして、確率

は、同様に１である。 probability

Mean probability of evidence

It is. The probability is less than when every instance of the movement is completely obvious (unambiguous), and the probability

Is 1 as well.

  Conversely, accidental or extremely rare movements (movements) that are never repeated collect little evidence in the numerator, but the denominator is the number of graphs that contain the accidental start node and result The value is very small for a set of graphs.

ここで、Θは終了ノードのセットであり、また、正規化因子Ｍは、全てのトラフィックが通過しなければならない環境における隘路等の、開始ノードとして対象となる特定のノードＡを含むセットΓにおけるグラフの数である。これは、我々がその環境における異なる領域の間の「接続性」を測定することを可能にする。 By gathering more evidence as probabilities across the population of sensors and across the graph, the traffic pattern in an environment can be quantified as:

Where Θ is the set of end nodes and the normalization factor M is in the set Γ including the particular node A that is the target as the start node, such as a bottleneck in an environment where all traffic must pass. The number of graphs. This allows us to measure “connectivity” between different areas in the environment.

ここで、Φは開始ノードのセットである。セットΦ及びΘは、不連続の場合があり、その環境に亘って分散配置させてもよい個人のグループ間の関係を測定する。 A more general solution is to collect probabilities for the start and end nodes across multiple graphs, as shown in the following equation.

Where Φ is a set of start nodes. The sets Φ and Θ may be discontinuous and measure the relationship between groups of individuals that may be distributed across the environment.

で表される確率は、グラフのセットΦにおけるあるノードで開始するトラックが、グラフのセットΘにおけるあるノードで終了する相対的な尤度を表している。 In either case, the following formula

Is represented by the relative likelihood that a track starting at a node in the graph set Φ ends at a node in the graph set Θ.

で表される確率は、その環境におけるある場所から他の場所への、繰り返される人々の流れがある確率の推定値を表している。オフィスビル等の環境で、個々の人々は、典型的には、特定のオフィスに関連している。このため、通常、ある個人を特定のオフィスの外で開始するトラックに関連づけることが可能である。同様に、あるトラックがあるオフィスで終了するならば、その個人を比較的高い確率で再び認識できる。また、通常、少数の共用資源が、例えば、トイレ、台所、コピー機、及びプリンタ等の既知の場所にある。そして、ノードがそれらの位置に対応する確率は、個々人と場所との間の潜在的な接続性の間接的な測定を与える。 Connectivity graph The following expression between nodes A and Z of all start and end pairs in the environment

Is an estimate of the probability that there is a repeated flow of people from one place to another in the environment. In an environment such as an office building, individual people are typically associated with a particular office. Thus, it is usually possible to associate an individual with a track that starts outside a particular office. Similarly, if a track ends in an office, the individual can be recognized again with a relatively high probability. Also, typically a small number of shared resources are in known locations such as toilets, kitchens, copiers, and printers. And the probability that nodes correspond to their location gives an indirect measure of potential connectivity between individuals and locations.

  These probabilities can be grouped into two or more classes to produce a deterministic graph of connections and connection quality between individuals.

  In the prior art, these types of social “networks” are measured by using sensors worn on the human body, such as RFID tags and badges, which are easily lost or damaged. It may be uncomfortable or inconvenient to wear and detect. Inferring a network from indirect, simple sensors placed in the environment is an important advantage of the present invention.

Prediction If a tracklet starts with sensor (node) A, the set or subset of all end probabilities for that node is given by the probability:

Here, it is possible to predict the end node with the highest accuracy for the tracklet as follows.

  Also weights for stochastic decision systems such as elevators, lighting, heating systems, or other systems that predict demand for limited environmental resources by ordering all end nodes by rank. It is also possible to use relative (conditional) probabilities as

In addition, this technique generalizes the probability to include a subset (subset) of nodes such as “A, R, S, Z” by creating a more limited graph.

  Nodes A and R need not be sequential (sequential, continuous). Those nodes need only be part of the likely (reasonable) part of the graph. That is, there may be an intermediate node between the node A and the node R and between the node R and the node Z. This is useful, for example, when there are many equivalent tracks between node A and node R, which represents a particularly useful node in the graph.

Such a format can be extended to include a subset (subset) that includes one or more intermediate nodes. Next, the prediction takes a probabilistic form as follows:

  FIG. 11 shows each step of the general method. An event 1111 caused by a moving object is detected 1110 by a set of sensors 101 located at a known location in an environment. A series of temporally and spatially adjacent events are linked (connected) 1120 to form a set of tracklets 1121. A set of directed graphs 1131 are assembled 1130 from the set of tracklets. The graphs include at least one start node, at least one end node, and one or more intermediate nodes to which a plurality of tracklets connect while generating an ambiguity with respect to the movement (movement) of the object. Probabilities 1141 are assigned 1140 to the edge to indicate the likelihood that the object was at a particular location in order to model the movement of the object in the environment. The probabilities 1141 can be refined based on information such as a movement (movement) pattern 1146 learned 1145 over time. And the graphs annotated with those probabilities can be used to predict the motion of the object.

  Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other modifications and changes can be made within the spirit and scope of the invention. Accordingly, the purpose of the appended claims is to cover all modifications and variations as fall within the true spirit and scope of the invention.

It is explanatory drawing which shows the environment where a tracking system is implemented according to embodiment of this invention. It is explanatory drawing which shows the tracklet (small wake) graph by embodiment of this invention. FIG. 2 is a block diagram illustrating a track of a tracked object in the environment of FIG. 1 according to an embodiment of the present invention. It is explanatory drawing which shows the decision graph by embodiment of this invention. It is an image which shows the user interface by embodiment of this invention. 6 is a flowchart illustrating a method for recording monitoring data according to an embodiment of the present invention. 6 is a flowchart illustrating a method for retrieving monitoring data for tracking an object according to an embodiment of the present invention. It is explanatory drawing which shows the discrete event (event) detected by the motion (motion) sensor and the tracklet. It is explanatory drawing which shows the tracklet (small wake) graph by embodiment of this invention. It is explanatory drawing which shows the tracklet (small wake) graph by embodiment of this invention. 3 is a flowchart illustrating a method for performing modeling and prediction.

Claims (17)

Detecting a discrete event caused by an object moving in an environment with a set of sensors disposed at known locations in the environment;
Linking a series of temporally and spatially adjacent discrete events to form a set of tracklets;
A set comprising at least one start node, at least one end node, and one or more intermediate nodes to which the plurality of tracklets connect while generating an ambiguity in the movement of the object from the set of tracklets Creating a directed graph of
Assigning a probability to the node indicating the likelihood that the object was at a particular known position to model movement of the object in the environment;
A computer-implemented method for modeling the movement of an object in an environment comprising:   The computer-implemented method for modeling object movement in an environment as recited in claim 1, wherein the sensor is a motion sensor.   The computer-implemented method for modeling object movement in an environment as recited in claim 1, wherein the sensor uses a wireless transmitter to transmit the event.   The computer-implemented method for modeling object movement in an environment as recited in claim 1, wherein the linking is performed according to temporal and spatial constraints.   The computer-implemented method for modeling object movement in an environment as recited in claim 4, wherein the temporal and spatial constraints are learned over time.   The computer-implemented method for modeling object movement in an environment as recited in claim 1, wherein a particular probability is associated with a particular object. When the start node is node A, the end node is node Z, and the graph γ ₀ models the movement of a specific object from node A to node Z, the specific object starting at node A is always said If we end with node Z in graph γ _0, the probability is

The computer-implemented method for modeling the movement of an object in an environment according to claim 1, characterized in that   The intermediate node includes a split node from which a plurality of tracklets diverge and a join node from which a plurality of tracklets converge to model the movement of an object in an environment according to claim 1. A computer-implemented method. When the start node is node A, the end node is node Z, and Γ is a set of graphs γi, the probability is

The computer-implemented method for modeling the movement of an object in an environment according to claim 1, characterized in that If a set of end nodes is a set Θ, the probability is given by

A computer-implemented method for modeling the movement of an object in an environment as claimed in claim 9. If a set of start nodes is a set Φ, the probability is given by

A computer-implemented method for modeling the movement of an object in an environment according to claim 10 characterized by:   The computer-implemented method for modeling object movement in an environment as recited in claim 11, wherein the set Φ and set Θ can be discontinuous.   The computer-implemented method for modeling object movement in an environment as recited in claim 1, further comprising predicting movement of the object from the set of graphs with an assigned probability. Method.   The computer-implemented method for modeling object movement in an environment as recited in claim 1, comprising ordering the probabilities of the end nodes by rank.   The computer-implemented method for modeling object movement in an environment as recited in claim 1, wherein the environment is complex.   If the particular object is not in a particular known position, the probability P is 0, if the particular object is in the known position, the probability P is 1, and otherwise The computer-implemented method for modeling object movement in an environment according to claim 1, wherein the probability P is 0 <P <1. A set of sensors disposed at known locations in an environment and configured to detect discrete events caused by objects moving in the environment;
Means for linking together a series of temporally and spatially adjacent discrete events to form a set of tracklets;
A set comprising at least one start node, at least one end node, and one or more intermediate nodes to which a plurality of tracklets connect while generating an ambiguity in the movement of the object from the set of tracklets A means of creating a directed graph of
Means for assigning to the node a probability indicating the likelihood that the object was in a particular known position to model movement of the object in the environment;
A system for modeling the movement of an object in an environment characterized by comprising:

Images