JP2010057105A - Three-dimensional object tracking method and system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object tracking method with which complicated occlusion can be handled. <P>SOLUTION: In order to track an object in a three-dimensional (3D) manner, a video image of a 3D camera observing a predetermined environment from an obliquely upper side is analyzed to form a depth map for a background of the environment and a depth map for a current scene. The created depth maps are compared to form a depth map of a foreground, and on the basis of the foreground depth map, 3D coordinates at respective points are computed. The 3D coordinates are transformed into world coordinates, and for each of the point of the foreground, only zw values of the world coordinates are extracted and mapped to respective pixels of an image plane, thereby forming a world-Z map. A point, with a great change amount of the zw values, between neighboring pixels on the world-Z map is detected to detect a boundary of one or more objects included in the foreground. On the basis of the detected boundary, the objects are segmented, thereby creating a 3D locus. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a method and system capable of detecting one or a plurality of moving objects (hereinafter referred to as objects) from a series of moving images (video sequences) in real time and tracking them in three dimensions.

  In order to track and detect an object, such as a person, in a video sequence, it is necessary to detect the person or persons appearing in the monitoring area of the video image, as long as the person is in this area. It is necessary to assign a unique ID to each person and to reproduce the track of each person tracked in three dimensions. In order to construct an ideal system, it is necessary to satisfy the following requirements.

  First, it is necessary to be able to track even complex scenes with frequent occlusions (overlapping objects), for example, crowded scenes. Occlusion includes (1) partial occlusion where only a portion of the subject is visible to the sensor, (2) short occlusion where the subject is completely occluded for a short time, and (3) subject. There are three types of extended occlusion that leave the field of view for a long period of time, and these occlusions need to be identified and tracked.

  Second, robustness is required for changes in lighting and background complexity in the scene. The tracking system must operate in an indoor or outdoor environment and with a complex and dynamic background. Third, it is necessary to accurately reproduce the trajectory of each person in a three-dimensional space or floor plane, and to distinguish and accurately track a person with a similar appearance, and fourthly, processing time is fast, That is, it is required to operate in real time to analyze a person's behavior and detect an abnormality, and fifthly, the system installation cost is low and setup and maintenance are easy. In order to satisfy the fifth requirement, it is desirable that the parameters for system adjustment are minimized.

  Several tracking systems have already been proposed. Non-Patent Document 3 proposes a single monocular camera-based system. This system introduces the Bayesian-multi-blob concept for object tracking. Once an object to be tracked is detected (blobed) with a single monocular camera, this tracking is performed based on the appearance of each blob. They proposed applying a multi-blob likelihood function and Bayesian filtering for multi-object tracking when the number of objects in the scene is unknown and changes over time.

  However, this system cannot effectively identify occlusion, fails to distinguish between persons with similar appearances, is sensitive to changes in lighting conditions, and cannot reproduce the 3D trajectory of the tracking object.

  Non-Patent Document 1 proposes a plurality of monocular camera-based systems. This system uses a plurality of monocular cameras set around the monitor scene. Perform color-based background subtraction to extract the foreground image. The intersection of cones corresponding to the foreground on the overhead plane produces a set of candidates to track. A new trajectory of the candidate to be tracked is generated by performing an association between the current candidate and the person detected in the previous frame based on motion, color and appearance.

  This system can process occlusion and reproduce 3D trajectories as long as the person is not occluded in all fields of view. Since a patch of 100 frames is processed at each time, it can be said to be a relatively fast process with a delay of 2 seconds.

  Further, since each person is recognized based on the color and appearance, it is considered that the ID of each person can be stored. However, since this system operates based on the intersection of cones, it cannot handle a case where a plurality of persons merge. It is also impossible to identify a person with a similar color and a similar appearance. Furthermore, since background subtraction and tracking are performed with color-based data, it is not robust to changes in illumination.

  The systems of Non-Patent Document 2 and Patent Document 1 use a 3D camera that is fixed at a high position and observes a scene from an oblique direction. This 3D camera provides a depth map of the scene. The operation of this system is as follows.

  1) Start by subtracting the background (background) recorded with no objects in the scene from the foreground (foreground).

  2) Compute 3D points from the foreground depth and render a virtual overhead plane. Only the highest point is displayed on the overhead plane.

  3) By post-processing the blob by some morphological operation, each composite blob will represent each individual in the scene.

  4) The detected person is associated based on tracking that provides a temporal trajectory of the height (height) pattern of each person.

  Although this system can handle occlusion, it cannot handle well when people are close to each other and have similar appearances, especially similar heights. On the other hand, since 3D data is used instead of color, it is robust against changes in illumination. Also, although the 3D trajectory is reproduced, the processing speed is fast, and the cost is low, when the person leaves the scene and reenters, the person's ID is not kept. In addition, because the height pattern is used for tracking, confusion arises when a person sits down, jumps or raises his hand.

U.S. Patent No. 7,003,136 F. Fleuret, J. Berclaz, R. Lengagne and P. Fua, “Multi-Camera People Tracking with a Probabilistic Occupancy Map”, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), February (30) 2, pp. 267 -282, Feb.2008. M. Harville. “Stereo person tracking with adaptive plan-view templates of height and occupancy statistics” Journal of Image and Vision Computing (22), No. 2, pp. 127-142, 2004. M. Isard and J. MacCormick. Bramble: “A bayesian multiple-blob tracker”. In Proceedings of the International Conference on Computer Vision, ICCV’01, pages 34-41, 2001. Arun Hampapur, Lisa M. Brown, Jonathan Connell, Max Lu, Hans Merkl, S. Pankanti, Multi-scale Tracking for Smart Video Surveillance, IEEE Transactions on Signal Processing, Vol. 22, No. 2, March 2005. IBM smart surveillance system: http://reaserch.ibm.com/peoplevision/

  The present invention has been made for the purpose of solving the above-mentioned problems of the prior art, and is intended to provide an occlusion of a moving object or person having a similar appearance and robustness in a more complex and dynamic background in indoor and outdoor environments. It is an object of the present invention to provide a three-dimensional tracking method and system for an object that can be effectively processed.

In order to solve the above-described problem, the method of the present invention analyzes a video of a 3D camera that observes a predetermined environment obliquely from above, and obtains a depth map of the background of the environment and a depth map of the current scene of the environment. Forming and comparing the depth map of the background and the current scene to form a depth map of the foreground, calculating three-dimensional coordinates of each point of the foreground based on the depth map of the foreground,
Converting the three-dimensional coordinates into world coordinates, extracting only the zw value of the world coordinates for each point of the foreground, and mapping it to each pixel of the image plane to form a world-Z map, and the world-Z map Detecting a point with a large amount of change in zw value between adjacent pixels on the top, detecting a boundary of one or more objects included in the foreground,
Each step includes segmenting the object based on the detected boundary and creating a three-dimensional trajectory for the segmented object.

  In order to solve the above-described problem, the system of the present invention includes a 3D camera that observes a predetermined environment obliquely from above, and a processor that analyzes an image of the 3D camera, and the processor uses the image to the environment A depth map of each of the background and the current scene is formed, the depth maps are compared to calculate a foreground depth map, and three-dimensional coordinates at each point of the foreground are calculated based on the depth map of the foreground. Converting the calculated three-dimensional coordinates into world coordinates, extracting only the zw value of the world coordinates for each point of the foreground and mapping it to each pixel of the image plane to form a world-Z map, One or more objects included in the foreground are detected by detecting a point with a large change in zw value between adjacent pixels on the world-Z map. Detecting the door of the boundary, and segmenting the object based on the boundary and the detecting, to create a three-dimensional trajectory to the segmented object, it performs each procedure.

  In the above method or system, the depth map of the background is formed based on an image captured when no object exists in the environment, and the depth map of the foreground is generated from the depth map of the current scene. You may make it form by subtracting a depth map.

  Furthermore, in the above method or system, the foreground three-dimensional coordinate data may be filtered by an incident angle with respect to the 3D camera to remove noise.

  The video obtained from the 3D camera is analyzed, a depth map of the foreground of the region (ROI, region of interest) to be observed is formed, and the three-dimensional coordinates of each point of the foreground are calculated based on this depth map. Next, the calculated three-dimensional coordinates are converted into world coordinates. At this time, the world coordinate system is created with the XY plane of the world coordinate system parallel to the floor surface of the ROI and the world coordinate of the center of the 3D camera as (00h) (h is an arbitrary value). From each foreground point converted to world coordinates, only the coordinate zw value on the Z axis is extracted and mapped on the image plane to form a foreground world-Z map. In the world-Z map formed in this way, the zw values of points belonging to the same object have a characteristic that changes continuously and uniformly in the Y-axis direction of the image plane.

  By using this characteristic, it is possible to detect boundaries between a plurality of objects included in the foreground. That is, when the world-Z map is scanned, if there is a point where the zw value does not decrease (or increase) uniformly between adjacent pixels but changes rapidly, that point is a boundary between objects. it is conceivable that. Therefore, if a plurality of objects included in the foreground are detected based on a large change in zw value that suggests a boundary between objects, and these are separately segmented, a plurality of objects are mixed, so-called occlusion occurs frequently. Even in such a case, it is possible to distinguish and segment a plurality of objects with high accuracy.

  When individual objects are segmented, it is possible to correlate how the objects have moved between time t-1 and time t, for example by using a linear probability model. As a result, the three-dimensional trajectory of each object can be detected. Object association is realized by performing probability calculation based on feature quantities such as 3D coordinates, average color, and facial features of the object.

  In the method and system of the present invention, the detection of an object is basically performed based on a depth map, so that it is robust against a change in illumination of the environment to be detected and the complexity of the background. Furthermore, even in an environment where the occurrence frequency of occlusion is high, a plurality of objects can be effectively segmented. Also, when the method and system of the present invention are implemented, the only important parameter for system setup is the installation angle of the 3D camera, and therefore the setup cost and maintenance cost of the system can be lowered.

  Various embodiments of the present invention will be described below with reference to the drawings. Note that, in the following drawings, the same reference numerals indicate the same or similar components, and thus redundant description will not be given.

  FIG. 1 is a flowchart showing a three-dimensional tracking method of a moving object (hereinafter, object) according to an embodiment of the present invention, and FIG. 2 is a diagram showing a system configuration for carrying out the method of FIG. FIGS. 3A to 3F are image diagrams for explaining operations in several steps of the flowchart of FIG.

  As shown in FIG. 2, the object three-dimensional tracking system according to the present embodiment is basically a single 3D camera (for example, a stereoscopic camera) 2 fixed obliquely above a space 1 in which the movement of the object is to be tracked. And a processing device (including a processor) 3 for processing information obtained from the camera 2. The processing device 3 may be a normal information processing device such as a computer, or may be a camera system configured integrally with the camera 2. The 3D camera 2 observes the space 1 from an inclined angle, and the processing device 3 creates a depth map (depth map) of the space 1 from the observation image.

  As pre-processing for object tracking, a background model is first created. Steps S1 and S2 in FIG. 1 are steps for creating a background model. First, in step S1, a background image of the space 1 is acquired by the camera 2. A background image is an image of a space when there is no object to be tracked. In step S2, a depth map, that is, a background model is created from the acquired background image. FIG. 3C shows the background model created in this way. It should be noted that once the background model is created, it may be used for each scene image processing, and therefore it is not necessary to repeat Step S1 and Step S2 for shooting a scene every time t.

  In step S <b> 3 shown in FIG. 1, an actual scene in the space 1 is imaged by the 3D camera 2. The actual scene contains objects to be tracked. FIG. 3A shows an example of a scene image. Note that scene shooting is performed at regular time intervals t, and therefore steps S3 and subsequent steps are repeatedly performed at each time interval t. In step S4, a depth map of the captured scene is created. FIG. 3B shows a depth map created from the scene image of FIG. In order to identify an object and track its movement, it is necessary to cut out only the object from the image depth map (see FIG. 3B). Accordingly, in step S5, the depth map of the entire object, that is, the depth map of the foreground (foreground) image is created by subtracting the depth map of the background image from the depth map of the scene image.

  In the next step S6, 3D data is created for each point of the foreground image based on the depth map of the foreground image created in step S5. In step S7, a process for removing noise included in the 3D data is performed. The noise removal processing will be described in detail with reference to FIG. 4 in the section [Noise removal processing in 3D data].

  In step S8, the 3D data after noise removal is converted into world coordinates. Note that 3D data is based on camera coordinates, and when this is converted into world coordinates, the floor surface of the scene is made parallel to the XY plane of the world coordinates. Also, the world coordinate system is set so that the center coordinates of the camera are (00h) (h: arbitrary value) of the world coordinate system. The conversion from the camera coordinate system to the world coordinate system will be described in detail with reference to FIG.

  In step S9, for each foreground point, a value on the Zw axis of the world coordinates, that is, a zw value is taken out and mapped onto each pixel of the image plane to create a world-Z map. FIG. 3D shows a world-Z map created in this way.

  Note that the maps shown in (b), (c), and (d) of FIG. 3 are actually sets of numerical data, but the functions of the map are shown as an image for explanation. Therefore, in the world-Z map of FIG. 4D, the pixels having a large zw value are shown bright and the small pixels are shown dark. Parts other than the foreground are processed with a zw value of 0.

  As a result of the above processing, when a world-Z map having only the zw value in the world coordinates is created for each point in the foreground, in step S10, this map is scanned to detect each object in the foreground image. . The map is usually scanned from the top left edge to the top right edge of the map, similarly from top to bottom of the map, but any scanning algorithm may be used based on the system configuration.

  In the world-Z map, the zw value of the world coordinate, that is, the height from the floor surface of the object is mapped in each pixel. Therefore, in the same object, the zw value is one in the Yw axis direction of the world coordinate. Changes. In step S11, the boundary between objects is detected in the world-Z map using this characteristic. A specific procedure for detecting the boundary between objects will be described in detail in the section of [World-Z map characteristics] with reference to FIGS.

  When an object boundary is detected, each detected object is segmented in step S12. FIG. 3E shows a state in which two objects are identified from the world-Z map of FIG. The segmentation in step S12, that is, the segmentation will be described in detail in the [Segmentation] section with reference to FIG.

  In step S13, for example, an association between the object segmented at time t-1 and the object segmented at current time t is performed using a linear probability model. In step S14, a 3D trajectory of each object is created based on the association in step S13. By repeating the above procedure, that is, step S3 to step S14 at regular intervals, 3D tracking of the object can be performed.

  Next, details of the main steps of FIG. 1 will be described.

[Noise removal processing in 3D data]
In order to remove noise specific to the depth information by the 3D camera, noise removal processing is performed. Such noise is likely to occur mainly at the boundary of an object, and the presence thereof degrades the accuracy in object segmentation. The inventors have noted that such noise has a larger incident angle (with respect to the camera) than a point cloud (point cloud). By using this characteristic, it is possible to determine a filter threshold value for removing noise.

  That is, on the image plane, an incident angle map in which each pixel records the incident angle of one point cloud is formed. The map is then filtered to preserve only those points that have an angle of incidence that is less than a predetermined threshold. The map thus formed is used as a mask for the world-Z map.

FIG. 4 shows noise generated in stereo processing. As shown in this figure, the incident angle β of the noise 40 to the camera 20 tends to be larger than the incident angle γ of the point 30 to the camera 20. In FIG. 4, 32 indicates incident vectors of the noise 40 and the point 30, and 34 indicates a normal vector of the noise 40. The cosine of the incident angle of a given point p is the unit normal vector of point p
And a unit vector connecting the center of the camera and the point p
Is shown as the product of The normal vector can be calculated using effective points p1 and p2 that are adjacent to the point p but are not aligned with the point p. That is, normal vector
Is
When
Is given by the outer product of.

As a result, the incident map entry I (p) for point p is
Given in.

[Create World-Z Map]
The world-Z map is a record of the world-Z coordinates of each three-dimensional point of the scene on each pixel of the image plane. FIG. 5 shows the relationship between camera coordinates and world coordinates. In FIG. 5, Xc, Yc, and Zc indicate coordinate axes of camera coordinates, and Xw, Yw, and Zw indicate coordinate axes of world coordinates. The camera 2 is disposed at a height h from the floor surface so as to observe the floor surface at an angle α. Further, it is assumed that the floor surface is parallel to the XY plane, and the world coordinate of the center of the camera 2 is (00h). In this case, the world coordinates (xw, yw, zw) of the point P whose camera coordinates are (xc, yc, zc) are
xw = zcsin (α)
yw = xc
zw = h−zccos (α) (1)
Is calculated as The world-Z map is created by taking only the world Z coordinate value calculated based on equation (1) and mapping it to each pixel on the image plane.

[Characteristics of World-Z Map]
6 and 7 are diagrams showing the characteristics of the world-Z map. FIG. 6 shows a case where a scene including objects A and B located close to each other on the floor surface 10 is observed by the camera 2, and FIG. 7 shows a world-Z map of the scene of FIG. In FIG. 7, 12 is an image plane. When the scene of FIG. 6 is observed by the camera 2, the distance d1 between the camera 2 and the object A is smaller than the distance d2 between the camera 2 and the object B. Therefore, on the image plane 12 of the world-Z map, the object A is Appears below object B.

  Therefore, when the object A and the object B have the same height (height) as in the case of a person, the point at the highest position (Y coordinate) on the world-Z map is farthest from the camera 2. You can think of it as belonging to the object at the position. Specifically, the point P3 in FIG. 7 is considered to be a point belonging to the object B far from the object A. Further, as apparent from FIG. 6, the point P1 of the object A and the point P2 of the object B appear at the same position on the image plane 12. However, the object A is closer to the camera than the object B, and therefore the world-Z coordinate value Z1 of the point P1 is larger than the world-Z coordinate value Z2 of the point P2. That is, Z1> Z2.

  In the same object, the value of the world-Z coordinate decreases uniformly as the Y-axis coordinate of the image plane increases. That is, in the object B of FIG. 7, the zw value of the point decreases uniformly as the Y coordinate increases from the point P3. Similarly, in the object A, the zw value of the point P1 is the largest, and the zw value decreases uniformly as the Y coordinate increases.

  In summary, the World-Z map has the following characteristics.

1) Since the World-Z map includes all 3D points calculated from the depth map, there is no loss of information.
2) The zw value in one object decreases uniformly as the Y-axis value increases.
3) An object near the camera has a larger zw value on the world-Z map than an object far from the camera. This means that when the object has almost the same height (height) as in the case of a person, the point at the highest position (Y coordinate) on the world-Z map is the object farthest from the camera. Means belonging to.

[Object segmentation]
By using the above characteristics 2) and 3) of the world-Z map, a plurality of objects partially overlapping on the image plane can be identified and segmented. That is, when the world-Z map shown in FIG. 7 is scanned from right to left and from top to bottom, for example, the zw value may change greatly between adjacent pixels without being uniformly reduced. In such a case, it can be considered that there is a boundary between a plurality of objects. For example, in the world-Z map, the zw value in the object B decreases uniformly from the point P3 toward the point P2, but when it reaches the point P2, it corresponds to the point P1 of the object A. The zw value increases rapidly. This is because Z1> Z2 as shown in FIG. In the object A, the zw value Z1 of the point P1 is maximized and thereafter decreases uniformly.

  As described above, the boundary between objects can be detected by scanning the world-Z map and detecting a point where the change in the zw value is sharp between adjacent pixels. The method uses this property to identify multiple objects from the world-Z map.

[segmentation]
FIG. 8 is a diagram for explaining a procedure for segmenting the objects A and B from the world-Z map including the two objects A and B in the monitor area. FIG. 8A shows a world-Z map created based on an image obtained from a 3D camera. The map starts scanning from the top left end and scans the screen from right to left and further from top to bottom to detect the boundary 13 between the object B and another object. In order to detect the boundary, a user-specified threshold ε may be provided for a change in the zw value between adjacent pixels. Next, based on the detected boundary 13, the object B is segmented as shown in FIG.

  Next, segment B ′ is removed from the world-Z map and the remaining map is scanned to determine object A (FIG. (C)) and segment it as shown in FIG. (D) ( Segment A ′). As a result, the objects A and B are segmented as a segment A ′ and a segment B ′ in the world-Z map.

[Segment tracking]
FIG. 9 shows an overview of 3D object tracking based on a linear probability model. FIG. 9A shows the positions of the objects X1, X2 and X3 detected at time t-1, and FIG. 9B shows the positions of the two segments Y1 and Y2 detected at time t. . Tracking is to find the correspondence between the segments Y1, Y2 detected at time t and the objects X1, X2 and X3 detected at time t-1. Each variable Xi (i = 1, 2, 3) encodes certain features of the object Xi related to tracking. These features include 3D position, placement, appearance and movement.

  For example, if (x1, y1, z1) is the 3D position of the object, (h1, w1) is the height and width of the object, and (r1, g1, b1) is the average color of the object, It can be assumed that X1 = (x1, y1, z1, w1, r1, g1, b1). The same assumption is made for Yj (j = 1, 2). In order to build this correspondence, it is assumed that the number of objects in the scene may change over time. For example, one person can leave the scene and another person can enter the scene. The system automatically handles this situation.

In general, the set of objects detected at time t−1 is
X = (X1, X2,..., Xn), n is the number of objects,
The segment set detected at time t is
Y = (Y1, Y2,..., Yn), m is the number of segments,
Can be shown as Note that m ≠ n.

  For tracking purposes, consider a matrix P having m + 1 rows and n columns. Each column corresponds to an object detected in the previous frame, and each row corresponds to a segment detected in frame t. Add additional rows to this matrix. This line corresponds to the probability that the person Xi detected at time t−1 disappears at time t. When i is a row number and j is a column number, each entry P (i, j) in the matrix has a probability that Yi corresponds to Xj.

  FIG. 9D shows an example of such a matrix P. Once the matrix P is determined, the next step is to find the optimal assignment. FIG. 9C shows an example of such allocation. That is, in FIG. 3C, Y1 is assigned to X3, Y2 is assigned to X2, and X1 is considered to disappear at time t (assigned to φ). The probability of this assignment is P (1,3) × P (2,2) × P (3,1) = 0.343, which is the highest value among other possible assignments.

  If m> n, it indicates that a new object has entered the scene. This algorithm finds no assignments for objects that have entered the scene. In this case, they are interpreted as newly coming objects and are added to the list of persons detected at time t. The value of the matrix P is calculated based on the degree of similarity between the objects X1, X2, X3 and the segments Y1, Y2. As this similarity, it is also possible to use several properties between objects. For example, 3D distance between objects, color similarity, face similarity, or a combination thereof can be used.

The flowchart which shows the procedure for implementing the object tracking method which concerns on one Embodiment of this invention. 1 is a schematic diagram showing a configuration of an object tracking system according to an embodiment of the present invention. The image figure for demonstrating the method of FIG. The figure which shows the relationship between a point cloud and noise. The figure which shows the relationship between a camera coordinate and a world coordinate. The figure which shows the relationship between an object and a camera. The figure which shows a world-Z map. The figure which shows the procedure of object segmentation. The figure which shows the procedure of 3D tracking of an object.

Explanation of symbols

1 Space 2 3D Camera 3 Processing Device (Computer)

Claims (6)

Analyzing an image of a 3D camera that observes a predetermined environment from obliquely above, and forming a depth map of the background of the environment and a depth map of the current scene of the environment,
Comparing the background and current scene depth maps to form a foreground depth map;
Calculating the three-dimensional coordinates of each point of the foreground based on the depth map of the foreground;
Converting the three-dimensional coordinates into world coordinates;
Extracting only the zw value of the world coordinates for each point in the foreground and mapping it to each pixel in the image plane to form a world-Z map;
Detecting a point with a large amount of change in zw value between adjacent pixels on the world-Z map, detecting a boundary of one or more objects included in the foreground;
Segment the object based on the detected boundary;
Creating a three-dimensional trajectory for the segmented object;
An object three-dimensional tracking method comprising each step.   The method of claim 1, wherein the depth map of the background is formed based on a video taken when no object exists in the environment, and the depth map of the foreground is derived from the depth map of the current scene. 3D tracking method of an object formed by subtracting a depth map of   The method according to claim 1, further comprising a step of filtering the foreground three-dimensional coordinate data by an incident angle with respect to the 3D camera to remove noise. A 3D camera for observing a predetermined environment obliquely from above;
A processor for analyzing the video of the 3D camera,
The processor is
Forming a depth map of the background of the environment and the current scene from the video,
Compare the respective depth maps to calculate a foreground depth map;
Calculating three-dimensional coordinates at each point of the foreground based on the depth map of the foreground;
The calculated three-dimensional coordinates are converted into world coordinates,
Extract only the zw value of the world coordinates for each point in the foreground and map to each pixel in the image plane to form a world-Z map;
Detecting a point with a large amount of change in zw value between adjacent pixels on the world-Z map, detecting a boundary of one or more objects included in the foreground;
Segment the object based on the detected boundary;
An object three-dimensional tracking system that executes each procedure that creates a three-dimensional trajectory for the segmented object.   The system of claim 1, wherein the depth map of the background is formed based on a video taken when no object exists in the environment, and the depth map of the foreground is derived from the depth map of the current scene. 3D tracking system for objects formed by subtracting the depth map of.   The system according to claim 4, further comprising a step of filtering the foreground three-dimensional coordinate data by an incident angle with respect to the 3D camera to remove noise.