JP2002218449A - Device for tracking moving object - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a moving object tracking device capable of acquiring the position information of an object and stably identifying the object wherever the object is. SOLUTION: Cameras 2#1 to 2#n respectively independently fetch downward images, and individual observing parts 4#1 to 4#n acquire the position information of a person in response to the matter that a corresponding camera picks up the person directly under the camera, acquire information for identifying his/her face and clothes in response to the matter that another camera picks up the side of the person and perform feature extraction processing. Obtained feature quantity is made to correspond to each target to be tracked based on a predictive observation position sent from a tracking part 8 and subsequently transmitted to the tracking part 8, and the feature quantity that cannot be made to correspond is sent to a finding part 6. The finding part 6 detects a new person by using uncorresponded point information. The detection results of the new person are transmitted to the tracking part 8. The tracking part 8 updates each person information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a moving object tracking apparatus, and more particularly to a moving object tracking apparatus that tracks a moving object with a multi-viewpoint image obtained asynchronously.

[0002]

2. Description of the Related Art Stereo measurement, which obtains depth information in a scene from a plurality of images obtained from different viewpoints, occupies an important position in computer vision research in which restoring three-dimensional information from images is one of the main tasks. ing.

[0003] In stereo measurement, what is particularly important is correspondence between images. In order to reduce erroneous correspondence, a feature amount is appropriately selected (Marr D. And Poggio).
TA theory of human stereo vision. In Roy. Soc. L
ondon, pp. 301-328. 1979), a method of effectively using more viewpoints (Ramakant Nevatia. Depth)
measurement by motion stereo. Computer Graphics an
d Image Processing vol. 5, pp. 203-214, 19
76) has been developed. These are mainly for static scenes, and there have been many reports of errors in the stereo method in this case (Jeffrey J. Rod
riguez and JK Aggarwal.Stochastic Analysis of
stereo quantization error. IEEE Pattern Anal. Mach
ine Intell., Vol. 12, No. 5, pp. 467-470,
5 1992).

In research on scenes with motion,
Robot Navigation Based on Stereo Observation (Larr
y Matthies and Steven A. Shafer.Error Modeling in
stereo navigation.IEEE Rebotics and Automation,
vol. RA-3, No. 3, pp. 239-248, 1987),
Tracking of known shape with one camera (Akio Kos
aka and Goichi Nakazawa.Vision-based motion track
ing of rigid objects using prediction of uncertaini
es.In Proc. of International Conference on Roboti
cs and Automation, pp. 2637-2644, 195
5) Tracking of known shape by stereo measurement (Ge
m-Sun J. Young and Rama Chellappa.3-dmotion estim
ation using a sequence of noisy stereo images: Mod
els, estimation, and Uniqueness results.IEEE Patt
ern Anal. Machine Intell., Vol.12, No.8, pp.7
35-759, 1990), tracking of multiple objects based on stereo observation (Jae-Woong Yi and Jun-Ho Oh. Recursive)
resolving algorithm for multiple stereo and motion
matches.Image and Vision Computing, Vol. 15, pp.
181-196, 1997). These question settings, which track multiple objects while associating image features with the target object model, are called "motion association"
(Motion Correspondence) (Ingem
ar J. Cox.A review of statical data association te
chniques for motion correspondence. International
Journal of Computer Vision, Vol. 10: 1, pp. 53
-66, 1993).

[0005]

By the way, all of the above-mentioned conventional methods are based on the premise that observations from a plurality of viewpoints are simultaneously performed at a fixed interval or a static scene (equivalent to simultaneous observation). If you observe at the same time,
Since the same physical feature is observed, it is easy to associate motion.

[0006] However, even in the conventional method of performing simultaneous observation, "correlation of motion" is still required between different times in order to track an object in time series.

Furthermore, many conventional systems using multi-viewpoint images are based on the premise that observations are performed simultaneously between viewpoints in order to acquire position information during tracking. When processing an image, the overall processing speed is limited by the slowest process. In addition, such a system requires a separate means for synchronizing between viewpoints in addition to the exchange of image information. These problems of the conventional system on the premise of synchronization become more remarkable as the number of viewpoints used increases.

Therefore, the inventor of the present application has filed Japanese Patent Application No. 11-308.
No. 187 proposed a “moving object tracking device” that performs a feature extraction process on an image captured and obtained by a plurality of cameras. However, a viewpoint (camera position) that is advantageous in acquiring a plurality of types of information on a tracking target from an image in the moving object tracking device is not always advantageous in acquiring other information.

For example, when the person 2 is tracked by the camera 2 installed on the ceiling as shown in FIG. 25, both the position information and the information unique to the object such as the color of the clothes and the height of the clothes are detected. However, it was not possible to cope with a camera arrangement in which only one of them could be obtained, and the tracking stability was sometimes lowered.

[0010] That is, in FIG. 25, the arrow a indicates that the position can be detected with high accuracy when the person MAN is present directly below the camera 2, and that the two can be detected separately even if two persons MAN exist. It indicates that the accuracy is high. The arrow b indicates that the more the person MAN is away from the camera 2, the more the information unique to the person MAN can be detected, and the higher the accuracy of identifying the person.

SUMMARY OF THE INVENTION It is, therefore, a primary object of the present invention to provide a moving object tracking apparatus capable of stably acquiring position information and identifying an object regardless of the position of the object.

[0012]

SUMMARY OF THE INVENTION The present invention relates to a moving object tracking apparatus for tracking a moving object moving in a scene from multiple viewpoints, comprising a plurality of moving objects having a fixed height and a fixed interval in the scene. A plurality of photographing means for photographing the moving object in the downward direction from the viewpoint of the camera; and a plurality of photographing means provided in correspondence with each of the plurality of photographing means. Acquisition of the position information of, according to the other imaging means image the side of the moving object,
Obtains information for identifying the moving object, extracts a feature point from the obtained result and outputs it as observation information, and according to a request from the observation means that extracted the feature point among the plurality of observation means. And tracking means for predicting the state of the moving object by integrating observation information from a plurality of observation means.

The plurality of observing means are associated with each tracking target based on the predicted observation position sent from the tracking means and transmitted to the tracking means, and are not associated with the plurality of observing means. Detecting means for detecting a new moving object in response to the feature amount and sending the detected moving object to the tracking means, wherein the tracking means updates each moving object information based on the new moving object detected by the detecting means. I do.

[0014] Each of the plurality of observation means includes an area dividing means for dividing an image received from the corresponding photographing means into a background image and a moving object area, and a center of gravity of the moving object area extracted by the area dividing means. Means for calculating and outputting information of the center of gravity as the feature point.

Further, the tracking means is constituted by a computer, and each of the plurality of observation means is constituted by a computer, and each of the plurality of observation means communicates with the tracking means.

[0016]

FIG. 1 is a layout diagram of a camera according to an embodiment of the present invention. In FIG. 1, a plurality of cameras 2
# 1, # 2, and # 2 are embedded at regular intervals in the ceiling of a room at a fixed height so that the shooting direction is downward. Cameras 2 # 1, 2 # embedded in these ceilings
When a person is tracked by 2, 2 # 3, position information and information unique to the object such as clothes color and height are used as information necessary for tracking.

In order to obtain position information, it is advantageous to use a camera directly above a person in order to avoid hiding from other objects, since a plurality of persons can be separately observed. For example,
At the position below the camera 2 # 2, the position detection (separation) by the camera 2 # 2 is facilitated as described with reference to FIG. 25, but the other cameras 2 # 1 and 2 # 3 observe the side. Therefore, position detection becomes difficult. Arrow a indicates each camera 2 #
This indicates that the position detection accuracy of the object directly below each of 1, 2 # 2 and 2 # 3 is high.

On the other hand, the unique information such as the color of the clothes, the height and the face image is more advantageous because the cameras 2 # 1 and 2 # 3 observe the sides more than the camera 2 # 2. Arrow b indicates each camera 2 #
This indicates that the acquisition of the unique information becomes good when 1, 2, and 2 # 3 observe the sides, respectively.

According to the present invention, as shown in FIG.
1, 2 # 2 and 2 # 3 are placed on the ceiling, and by switching which camera is used to acquire which information according to the distance to the person, the object is less affected by hiding over a wide area. It is possible to perform both identification and position detection stably.

FIG. 2 is a block diagram showing the overall configuration of the moving object tracking device 1 according to one embodiment of the present invention. Referring to FIG. 2, moving object tracking device 1 includes cameras 2 # 1, 2 # 2,..., 2 # n arranged on the ceiling as shown in FIG.
, 4 観 測 n, observation units 4♯1, 4♯2,.
And a tracking unit 8.

Each of the observation sections 4 # 1, 4 # 2,..., 4 # n is provided corresponding to each of the cameras 2 # 1, 2 # 2,. 2. Observation unit 4). And each observation part 4 # 1, 4 #
Each camera 2 # 1, 2 # 2,..., 2 #
n imaging outputs are provided. Observation unit 4♯1, 4♯2
.., 4♯n, the discovery unit 6 and the tracking unit 8 can operate independently of each other. For example, they are composed of different computers, and each computer is connected by a local area network LAN.

In the figure, a symbol A0 represents observation information of a corresponding point (tracking target) transmitted from the observation unit 4 to the tracking unit 8, and a symbol A1 represents an uncorresponding point (the tracking target) transmitted from the observation unit 4 to the discovery unit 6. Represents the observation information of points that cannot be matched with the tracking target), and the symbol A
2 represents predicted position information transmitted from the tracking unit 8 to the observation unit 4, symbol A3 represents position information (initial value) of a new person transmitted from the finding unit 6 to the tracking unit 8, and symbol A4 is the tracking unit. 8 represents the position information (after updating) transmitted to the discovery unit 4.

The observing section 4 acquires the position information of the person in response to the corresponding camera capturing the image of the person directly below, and at least obtains the face information in response to the other camera capturing the side of the person. ， Acquire information to identify clothes,
A feature extraction process is performed from the obtained result.

Note that the camera is not necessarily limited to an image of a person directly below or a person from the side. For example, a state in which a robot runs away or an animal such as a dog or a cat moves is viewed from above and from the side. What is necessary is just to be able to image. Each observation unit 4 operates independently. The feature amount (centroid and distance conversion value) obtained by the observation unit 4 is associated with the tracking target based on the predicted position information A2 transmitted from the tracking unit 8 described later, and then the observation time information Is transmitted to the tracking unit 8. The feature amount that could not be corresponded is transmitted to the discovery unit 6 as observation information A1 of an uncorresponding point.

The discovery unit 6 detects a newly appearing person (new person) in the scene by using the transmitted uncorresponding observation information A1. The detection result of the new person, that is, the position information A3 of the new person,
Sent to. Thereby, a new person is added as a new tracking target. Then, tracking is started in the tracking unit 8.

The tracking section 8 updates the position information of the person using the Kalman filter, using the position information A3 of the new person as an initial value, the observation information A0 as an input value, and predicts the position based on the observation model. The predicted position information A2 is transmitted to the observation unit 4. The location information (after update) A4 is transmitted to the discovery unit 6 as described later.

FIG. 3 is a flowchart for explaining the operation of one embodiment of the present invention. FIG. 4 is a flowchart showing the correspondence between each person area and the tracking model and the position / posture estimation operation in FIG. FIG. 5 is a diagram illustrating an example of a change in a human image based on a distance from a camera installed on a ceiling.

In step SP1 shown in FIG. 3 (abbreviated as SP in FIG. 3), when the corresponding image signal is input from the camera 2, the observation section 4 detects a person area in step SP2. In step SP3, it is determined whether a person has been detected. Step S until a person is detected
P1 to SP3 are repeated. When the presence of the person is detected, at step SP4, the time at which the person was observed is tracked by the tracking unit 8.
Send to As a result, the observation unit 4 receives from the tracking unit 8 the position / posture calculated based on the tracking model and the observation time, and the predicted values of human characteristics such as face images and clothes.

In step SP6, each person area is associated with a tracking model. This processing is performed by executing the processing shown in FIG. That is, in step SP21, each person area is specified, and in step SP22, the position / posture is estimated. This process is performed by the process shown in FIG. In step SP31, position estimation is performed. In step SP32, the distance d between each person region and the camera 2 is calculated. The posture is estimated under the constraint of the distance d, and the reliability (weight) is calculated. The posture is estimated based on silhouette information of a rotation (posture angle) that can be taken by a person at a distance d from the camera as shown in FIG.

Thereafter, step SP2 shown in FIG.
In step 3, human characteristics such as a head region image and clothes color are extracted, and the reliability (weight) is calculated. Then, in step SP24, an association process based on the position / posture information and the person characteristic information is performed. In step SP25, it is determined whether or not the processing has been completed for all the person areas,
If not, the process returns to step SP21 to process the next person area. If the processing has been completed for all the person regions, the process proceeds to step SP7 shown in FIG.

At step SP7, each person area is specified, and at step SP8, it is determined whether or not there is a correspondence with the tracking model. If there is a correspondence with the tracking model, in step SP9, observation information such as the position / posture and the person's characteristic is transmitted to the tracking node, and at that time, a reliability corresponding to the distance from the camera 2 is added. If there is no corresponding relationship with the tracking model, in step SP10, the observation information such as the position / posture and the person characteristic is transmitted to the discovery node,
At that time, a reliability corresponding to the distance from the camera 2 is added. In step SP11, it is determined whether or not the processing has been completed for all the person areas.
Return to P7, and if it is completed, return to step SP1.

Here, a human body model used in one embodiment of the present invention will be described with reference to FIG. FIG.
FIG. 3 is a diagram for describing a human body model used in the embodiment of the present invention. In the figure, X, Y, and Z indicate three axes of the world coordinate system. The person MAN is modeled by an elliptical cylinder h. The center axis Xh of the person model h is set as the rotation axis of the person, and the angle r between the normal axis (the axis in the short axis direction perpendicular to the rotation axis Xh) and the X axis is set as the posture angle of the person. In addition,
It is assumed that the rotation axis Xh of the person MAN is perpendicular to the floor (the plane defined by the X axis and the Y axis). Camera 2
It is arranged perpendicular to the rotation axis Xh of the person, that is, the Z axis.

Next, the observation section 4 shown in FIG. 2 will be described. FIG. 7 is a diagram for describing an outline of the configuration of the observation unit 4. Referring to FIG. 7, the observation unit 4 includes a region dividing circuit 10, a pixel value calculating circuit 12, a centroid point selecting circuit 14,
And a feature point association circuit 16.

The area dividing circuit 10, the pixel value calculating circuit 12, and the center-of-gravity point selecting circuit 14 perform a feature extracting process based on the input human image. The feature point associating circuit 16 associates the extracted feature points with the tracking target (model).

The area dividing circuit 10 divides an input image into a person area and a background area. For example, “Image Segmentation for Person Tracking by Hierarchical Adaptation Based on Continuous Images” Computer Vision and Pattern Recognition (CVPR'9
8), IEEE Computer Society Transactions, p. 911-
916 describes a method of region division.

More specifically, a mask image is generated by binarizing the difference between the image of the current frame photographed by the camera and the images of the preceding and succeeding frames, and superimposing several frames of the difference image (that is, a mask image). A rough moving object region is extracted by a coarse division). When the size of the mask image is within a certain range, the luminance information, the color information, or the pixel information, the texture information and a combination of any of these are used to estimate the distribution of the parameter. For each pixel on the newly obtained image, the probability that the moving object region exists is calculated (that is, precise division is performed). Thus, a person region is cut out from the input image captured by the camera.

Subsequently, the pixel value calculation circuit 12 performs distance conversion on the obtained human image. Specifically, a pixel value indicating the shortest distance from each of the pixels constituting the person region to the boundary of the person region is calculated. The center-of-gravity point selection circuit 14 selects a point having a maximum pixel value in the person area as a center-of-gravity point (that is, a feature point) of the person area. The pixel value at the center of gravity is defined as a distance conversion value. A centroid point and a distance conversion value of the centroid point (centroid point distance conversion value) are used as the feature amount.

More specifically, the area dividing circuit 10 outputs a binarized image as shown in FIG. The pixel value calculation circuit 12 generates a distance conversion image shown in FIG. 8B for the binarized image shown in FIG. FIG.
In (b), a pixel having a larger distance conversion value is represented in black,
Pixels with smaller distance conversion values are represented whiter. As the distance from the contour of the human body increases, the color of the pixel becomes darker. The centroid selection circuit 14 selects a pixel corresponding to the symbol “X” in FIG. 8B as a centroid. The detection of the center of gravity by the distance conversion has a feature that it is hardly affected by a change in the pose of the person.

Next, a description will be given of the feature point associating circuit 16 for associating the extracted center of gravity with a tracking target that has already been discovered. FIG. 9 is a diagram for explaining the feature point associating process. In the figure, symbol 2
0♯1, 20♯2, ..., 20♯k, ..., 20♯m, ...,
Each of 20、２n is a camera 2♯1, 2♯2,.
, 2m,..., 2n (hereinafter, generically referred to as an image plane 20).

At time ta, camera 2 @ k uses time tb
First, it is assumed that observation is performed by the camera 2 m (however, ta <tb). At time ta,
It is assumed that two persons h0 and h1 exist, and the person h2 first appears in the scene at time tb.

The symbol x attached to the image plane 20 indicates the center of gravity detected in each image. In this case, the time t
Two centroid points (feature points) are detected from the image plane 20 @ k at a, and three centroid points (feature points) are detected from the image plane 20 @ m at time tb.

For each of the persons h0 and h1, the predicted position at the time tb is obtained by an inquiry to the tracking unit 8 based on the observation result up to the time ta. As will be described later, in the tracking unit 8, the motion of the person is assumed to be constant speed motion, and the predicted position of the person hj at a certain time t is represented by a two-dimensional Gaussian distribution. The position of the person hj at time t in the world coordinate system is referred to as position X _{hj, t} , the average of the two-dimensional Gaussian distribution is referred to as / X _{hj, t} , and the covariance matrix is referred to as / _{Shj, t} .
The average and covariance matrix are transmitted from the tracking unit 8 as predicted position information A0.

In the expression, / X _{hj, t} is _expressed as a symbol X _{hj, t} with "-" added _thereto, and / S _{hj, t} is expressed as a symbol "S _{hj, t} "-".

Prediction position distribution N (/ X _{hj, t} , / S _{hj, t} )
Is weakly perspective projected on an image 20♯i (i = 1, 2,..., N) to obtain a one-dimensional Gaussian distribution n (/ x _{hj, t, i} , / s _{hj, t, i} is obtained, which indicates the probability of the presence of a person in the image 20♯i.In equation (1), the symbol x is the person position X in the world coordinate system.
Is projected onto the image plane, symbol / x is the projection of the average / X in the world coordinate system onto the image plane, symbol / s
Represents the projection of the covariance matrix / S in the world coordinate system onto the image plane. In the equation,
x is represented as "-" added to the symbol x,
/ S is expressed as "-" added to the symbol s.

[0045]

(Equation 1)

The feature point that maximizes the probability expressed by the equation (1) is set as the observation value corresponding to the person hj at the observation time, and the feature point is labeled hj. The observation information A0 of the labeled feature points is transmitted to the tracking unit 8. However, for feature points that can be associated with multiple people,
It determines that occlusion has occurred at the time of observation, and stops transmission.

After these processes, the feature points not associated with each other are assumed to belong to an unknown person, that is, a new person, and the position and time are transmitted to the discovery unit 6 as unsupported observation information A1. You. In FIG. 9, observation information A1 on the person h2 is transmitted to the discovery unit 6.

Next, the tracking section 8 shown in FIG. 2 will be described. The tracking unit 8 updates the position and direction of the tracking target person based on the observation information A0 sent from each of the observation units 4.

FIG. 10 is a diagram for explaining an outline of the configuration of the tracking unit 8 shown in FIG. Referring to FIG. 10, tracking section 8 includes a position estimation circuit 22 and a direction angle estimation circuit 24. The position estimating circuit 22 estimates a predicted position based on the position of the center of gravity obtained from the observation unit 4, and outputs the estimation result to the direction estimating circuit 24. Direction angle estimation circuit 24
Is based on the estimation result output from the position estimation circuit 22,
The direction angle is estimated based on the center-of-gravity point distance conversion value obtained from the observation unit 4.

First, the position estimating process in the position estimating circuit 22 will be described in detail. The position of the person being tracked is updated using the characteristic information (center-of-gravity point position) associated with each observation unit 4. FIG. 11 is a diagram for describing the position estimation processing. Referring to FIG. 11, the distance between camera 2 # _i and image plane 20 # _{i is referred} to as distance l _i , and the distance between camera 2 # _i and person hj is referred to as distance L _{hj, i} . The angle between the epipolar line and the Y axis is denoted by w _{hj, i} .

In the position estimating process, it is assumed that the person is moving at a constant speed. The state of the person hj at the time t is expressed by the equations (2) and (3) on the world coordinates (X, Y). However, it is assumed that the initial state is determined by the information of the new person (model) transmitted from the discovery unit 6. "'" Added to the matrix indicates transposition.

Let ∧X _{hj, t-1 be} the estimated value of the person position X _hj at time (t-1), and ∧S _{hj, t-1} be the estimated value ∧X
_{Assuming that} the variance matrix is _{hj, t-1} , the state at time t is given by equation (4)
And equation (5). In the formula, ∧X
_hj, a _t-1, symbol X _hj, expressed as marked with "∧" over the _t-1, ∧S _hj, a _t-1, symbol S _hj, on the _t-1 "∧" It is expressed as the one with.

The transition matrix F is represented by the following equation (6) (Δt: t−1 → t). The symbol Q represents a covariance matrix at the transition.

[0054]

(Equation 2)

Here, it is assumed that the first observation is performed by the observation unit 4 # i. Based on the position information sent from the observation unit 4 # i, this observation can be represented by the equations (7) to (9).

H = [1 0 0 0] (7)

[0057]

(Equation 3)

[0058] symbol C _i is, the position of the camera, the symbols R
_{hj, t, i} represents the clockwise rotation of the angle w _{hj, t, i} between the epipolar line and the Y axis. The symbol e represents an observation error, which is assumed to be mean 0 and standard deviation σ _{hj, t, i} . The standard error σ _{hj, t, i} is considered to increase as the distance of the camera increases, and is expressed as in equation (9).

[0059] In this case, the camera position C _i and the person of (X
_hj, the distance L _hj between _{t), t, i} because of unknown, X _hj, predicted position / X _hj of _t, the distance is calculated by _t / L _hj, using _t, a _i as an approximation ( In the formula, / L _{hj, t, i} is represented by the symbol L
_{hj, t, i} are expressed as "-".

In the observation equation (8), the left side represents observation information, and the right side represents the result of projecting the position of the person on the image.

The position estimation circuit 22 forms a Kalman filter based on the above observation model, and updates the state of the person hj.

[0062]

(Equation 4)

Formula (10) and formula (10) are independently set for each camera.
The update process according to (11) is performed, and the state is predicted.
The state prediction of the person hj at the time (t + 1)
/ X _{hj, t + 1}, The covariance matrix / S_{hj, t + 1}Gaussian distribution
Given by The result of the state prediction corresponds to the request of the observation unit 4.
Calculated and transmitted in the same way, and associating feature points as described above
Used for Person who has moved out of range where the camera can detect
The object model is deleted and tracking of the person is stopped.

Next, the processing of the direction angle estimation circuit 24 will be described. In the direction angle estimation circuit 24, the posture angle r of the person
(See FIG. 6) is estimated using the center-of-gravity point distance conversion value that does not cause occlusion. This estimation is performed based on the human observation model (ellipsoid model) shown in FIG. Referring to FIG. 12, the center-of-gravity point distance conversion value is used assuming that the width of the ellipsoid on the image captured by camera 2 is s. Assuming the weak perspective transformation, the angle-of-gravity point distance conversion value s to be observed follows Expression (12), where θ is the angle between the optical axis and the normal to the rotation axis of the ellipsoid model (person). Here, the symbol L in Expression (12) indicates the distance from the rotation axis of the ellipsoid model to the camera, and the symbols A and B are constants. Assuming that the observation involves a Gaussian error, the probability P (s | θ) of observing the center-of-gravity point distance conversion value s is expressed by the equation (1)
It is expressed as 3). The constants A and B and the error variance σ _s are determined in advance based on the learning data.

[0065]

(Equation 5)

The posture of the human body is determined by the Euler angles (a, e, r)
Can be expressed by Here, r is the azimuth angle (posture angle).
, A represents an azimuth angle, and e represents an elevation angle.
Since the azimuth angle a and the elevation angle e are determined by the direction of the rotation axis of the human body, only the rotation angle r around the rotation axis is unknown.

The normal vector N of the human body is expressed by equation (14). Therefore, the optical axis vector C of the camera 2♯i
The angle θ _i (r) formed between the normal vector N of the human body and the normal vector N is expressed by Expression (15).

N = R _Z (a) R _Y (e) R _X (r) e (14) θ _ci (r) = cos ⁻¹ · N ^T · C (15) In the equation (14), , R _Z , R _Y , and R _X denote rotation matrices about the Z, Y, and X axes, respectively, and e _Z denotes Z
The unit vector in the axial direction is shown. Equation (15)
, The symbol ^NT indicates a transposed vector of the normal vector N.

The probability P that the set W (s ₁ , s ₂ ,..., S _n ) of the center-of-gravity point distance conversion values is observed by the _n cameras 2 # ₁ to _{2 #n} is expressed by the following equation (13). It is represented by equation (16). The value r that maximizes the probability P (W | r) in Expression (16) is set as the estimated value of the posture angle.

[0070]

(Equation 6)

The azimuth estimation is updated every time an observation is performed, as in the above-described position estimation. FIG.
The observation model shown in (1) has the state r _{hj, t} represented by Expression (17), using the input as the feature information (the center-of-gravity point distance conversion value s) transmitted from the observation unit 4. The presence or absence of occlusion is determined at the time of associating feature points, and is not used as observation information when occlusion occurs.

Next, the outline of the discovery unit 6 shown in FIG. 2 will be described. The discovery unit 6 detects a person who has newly appeared in the scene (a new person) and adds a corresponding model to the tracking unit 8.

Since the observation information is obtained asynchronously, normal stereo correspondence cannot be applied as it is. Therefore, the following (discovery) method using time series information is used.

First, among the observation information of the uncorresponding points sent from each of the observation units 4, observation information at four different times is selected one by one (denoted by γ). Observation time t1, t
Observation viewpoints are respectively C ₁ , C ₂ , C ₃ , and C ₄ for each of 2, 2, t3, and t4. With observation information as input,
The above-described Kalman filter update processing (formula (8)) is performed. However, it is assumed that initial dispersion / _{Shj, t} = 0.

By this operation, a predicted trajectory in four observations is obtained. Using this predicted trajectory, position estimation is performed using equation (4) (where Δt = t4-ti: i
= 1, 2, 3). The set of position estimation results is [∧
_{X t1, ∧X t2, ∧X t3} , and ∧X _{t 4].}

As described above, in the observation expression of Expression (7),
The left side shows the observation information, and the right side shows the result of projecting the person's position on the image. An error evaluation function f (γ) using the Mahalanobis distance shown in Expression (17) is defined as the difference between the observation information and the result of projecting the person position on the image.

[0077]

(Equation 7)

A combination in which the value of the evaluation function shown in Expression (17) is within a certain threshold value is a feature point set belonging to a new person, and an estimated position at the latest observation time (here, t4) is set as an initial discovery position. Transmit to the tracking unit 8.

Next, the following two simulation experiments were performed to clarify the effectiveness of the present invention. FIG. 13 is a diagram for describing a simulation experiment for confirming the accuracy of position following. Figure 1
In the simulation experiment shown in FIG. 3, two cameras (2
Using {1, 2} 2), a circular orbit indicated by a dotted line has a period of 10 ×
The tracking target object h0 that makes a uniform circular motion at t _f was observed. t _f
Is the shooting interval of the camera. The Kalman filter has
A model of constant velocity linear motion was given. Observation is assumed to be parallel projection. In the experiment, four angles θ (0 °, 30 °, 45 °) formed by the respective optical axes of the two cameras 2 # 1 and 2 # 2 are set.
°, 90 °). Under each condition, the difference Δt in the photographing time between the camera 2 # 1 and the camera 2 # 2 is set to 0.
From t _f / up to 2 is varied, t _f from the observation of the camera 2♯1
Later (in the direction parallel to the shooting plane of camera 2-1)
The position prediction error was recorded.

FIG. 14 is a diagram for explaining the results of experiments on FIG. In FIG. 14, the horizontal axis represents the shift Δt of the photographing time, and the vertical axis represents the prediction error. As shown in FIG. 14, the angle θ between the optical axes is 0 °, 30 °, 45 °
In particular, in the case of, the prediction error decreased with an increase in the photographing interval, and the prediction error took the minimum value at Δt = t _f / 2. Accordingly, when the motion of the object to be tracked deviates from the motion model that is the premise of the Kalman filter, it can be seen that, in particular, higher tracking performance can be obtained by performing asynchronous observation as compared with synchronous observation.

FIG. 15 is a diagram for explaining a simulation experiment for clarifying the ability to estimate the motion of a plurality of persons. In the simulation experiment shown in FIG. 15, five cameras 2 # 1 to 2 # 5 are arranged. Each of the cameras is connected to one computer constituting a corresponding observation unit. Image processing is performed on these computers. The processing speed is about 1-2 frames / sec.

Each computer is connected to the local area network LAN and synchronizes internal clocks with each other. Two computers (not shown) configuring the discovery unit 6 and the tracking unit 8 are respectively connected to the local area network LA.
N.

Each camera has been calibrated in advance, and the calibration information of each observation unit 4 is
It is transmitted to the discovery unit 6 and the tracking unit 8 together with the observation information (observation time, feature point). In the experiment, position tracking was performed on two persons h0 and h1. As shown in FIG. 15, two persons h0 and h1 appear in the scene 55 in order.

FIGS. 16 to 20 are diagrams for explaining the experimental results for FIG. In the simulation experiment shown in FIG. 15, the transition of the feature point obtained by each camera over time is shown. FIG. 16 shows the camera 2-1.
17 corresponds to camera 2 # 2, FIG. 18 corresponds to camera 2 # 3, FIG. 19 corresponds to camera 2 # 4, and FIG. 20 corresponds to camera 2 # 5. 16 to 20, the horizontal axis represents time, and the vertical axis represents the tracking position.

Referring to FIGS. 16 to 20, at the beginning of tracking,
Since a feature point is hardly observed except for the camera 2♯1, no person is found. As the first person approaches the center of the experimental environment (scene 55), feature points are obtained from other viewpoints (cameras 2♯3, 2♯4, 2♯5), and tracking is started (about from the start). 20 seconds have passed). Similarly, tracking of the second person is started about 37 seconds after the movement of the second person. 16 shown in FIG. 16 to FIG.
The × indicates the tracking result of the second person, and × indicates the tracking result of the second person. From the above experimental results, it can be said that the present invention can track a plurality of persons.

In the above-described embodiment, the movement of a person is tracked based on feature points in an image taken asynchronously from a plurality of viewpoints. Next, attribute values unique to the person are extracted.
An embodiment will be described in which the extraction result is output as observation information, the state of the corresponding person is detected based on the extracted attribute value, and the tracking model of the person is switched according to the detected state. For example, it is considered that the possibility of a person moving while seated is extremely low, so the reliability of the position estimation value is increased.

FIG. 21 is a diagram showing an outline of an observation unit and a tracking unit according to the second embodiment of the present invention, and corresponds to FIGS. 7 and 10 of the first embodiment. In FIG. 21, the area dividing circuit 10, the pixel value calculating circuit 12, and the center point selecting circuit 1
4 is the same as that of FIG. 7, but the observation unit 4 of this embodiment is newly provided with a silhouette height detection circuit 17. The silhouette height detection circuit 17 is a region division circuit 10
Detects the height of the person from the binarized image binarized by.

FIG. 22 is a diagram showing a binarized image when a person is standing and when a person is seated. As shown in FIG. 22A, the silhouette height detection circuit 17 calculates the height T1 of the binarized image and the height T of the binarized image shown in FIG.
2 is detected. For example, if a person is standing, T
1 is 180 cm and T2 is 1 if a person is seated.
It is detected as 40 cm. The detection output of the silhouette height detection circuit 17 is provided to the feature point association circuit 16.

As described in the first embodiment, the feature point association circuit 16 associates the centroid point extracted by the centroid point selection circuit 14 with the tracking target already found. Based on the detection output of the silhouette height detection circuit 17, information obtained by measuring the height of the associated person is included in the observation information and output to the tracking unit 8.

On the other hand, the tracking section 8 includes a height information extraction circuit 30, a backrest conversion circuit 32, and a state estimation circuit 34, as shown in FIG. 21, in addition to the configuration shown in FIG. The height information extraction circuit 30 extracts height information from the observation information sent from the observation unit 4 to the tracking unit 8 and supplies the height information to the backrest conversion circuit 32. The back conversion circuit 32 converts the height information extracted from the observation information into a back.

If this backrest changes from, for example, 180 cm to 140 cm, the state estimating circuit 34 estimates that the person has changed from a standing state to a seated state.
Conversely, when the backrest changes from 140 cm to 180 cm, it is estimated that the person has risen from a seated state.

The observation unit 4 and the tracking unit 8 operate in the same manner as in FIG. 10 described above, and the tracking unit 8 estimates a predicted position based on the centroid position included in the observation information obtained from the observation unit 4,
The direction angle is estimated based on the estimation result, and a specific person among the plurality of persons is tracked.

[0093] Of the observation information sent from the observation unit 4 to the tracking unit 8, height information is extracted by a height information extraction circuit 30, converted into a back by a back conversion circuit 32, and a person is estimated by a state estimation circuit 34. Is estimated to be standing or sitting. Tracking unit 8
Then, the person to be tracked is switched according to the estimated state.
That is, if the person being tracked is seated,
Since the person no longer moves, follow other people.

In the above description, the state where the person is standing and the state where the person is seated are estimated. However, the present invention is not limited to this. Behavior information such as a state of stopping from a state of being stopped may be estimated. In that case,
When the speed of the person reaches a certain speed or more, it is estimated that the person is moving, and when the speed becomes zero, it is estimated that the person stops. In this case, the above equations (4) to (6) are converted into the following equation (1).
8) to (20). These equations are at time t
from information up to the observation of the last performed in _a, shows a case where the prediction of the state of the person h ₁ at time t _b.

[0095]

(Equation 8)

In the above equation, Q _x and Δt indicate the fluctuation of the movement between Δt, and it is considered that this magnitude changes depending on the current state of the movement. In the embodiment described above, it is conceivable that this value is changed such that the walking width is large, while the vehicle is stopped, it is medium, and when it is seated, it is small. As described above, by changing the motion model, the reliability of the estimation result can be evaluated according to the motion state of the object.

In the second embodiment shown in FIG. 21, the state estimation is made as to whether a person is standing or sitting and the state whether the person is stopped or has started moving. . However, in associating the feature of each observed image with the tracking model, an attribute value unique to the target object observable from the image can be used instead of the position information. Here, the unique attribute value is not limited to the back of the target object and the action information described above, but may be the color of the clothes.

FIG. 23 is a block diagram showing such an embodiment. The observation section 4 shown in FIG. 23 is provided with a silhouette height and color detection circuit 18 instead of the silhouette height detection circuit 17 shown in FIG.
1 instead of the height extraction circuit 30 and the backrest conversion circuit 32,
A height and color information extraction circuit 36 and a back, and a color conversion circuit 38 are provided. The silhouette height and color detection circuit 18 of the observation unit 4 detects the height of the person as described with reference to FIG. 21 and also detects the chest portion (FIG. 24) in the image divided into regions as shown in FIG. (Shown by the diagonal lines) is also detected and given to the feature point association circuit 16.

On the other hand, the height and color information extraction circuit 3 of the tracking unit 8
Numeral 6 extracts height information and color information from the observation information, the backing and color conversion circuit 38 converts the backing and color from the height information and the color information, and the state estimating circuit 34 estimates the state of the person. . In this case, the color information _Stn obtained in the previous observation and the color information _{Stn + 1} obtained in the current observation are represented by the following equations.

[0100]

(Equation 9)

As described above, by using attribute values unique to the target object, such as backrest and color information, as observation information, there is an advantage that confusion does not occur even when the target object is approaching. is there.

The embodiments disclosed this time are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

[0103]

As described above, according to the present invention, the downward direction is photographed by the photographing means from a plurality of viewpoints at a constant height and at a constant interval in the scene, and the photographing means moves downward. Acquiring the position information of the moving object in response to the imaging of the moving object, and acquiring information for identifying the moving object in response to the other imaging means imaging the side of the moving object. Then, extract the feature points from the obtained results, output them as observation information, and predict the state of the moving object by integrating the observation information from multiple observation means according to the request from the observation means that extracted the feature points. It is possible to obtain position information and identify an object stably regardless of the position of the object, and to follow a moving object with high accuracy.

[Brief description of the drawings]

FIG. 1 is a layout view of a camera according to an embodiment of the present invention.

FIG. 2 is a block diagram showing an overall configuration of a moving object tracking device 1 according to an embodiment of the present invention.

FIG. 3 is a flowchart for explaining the operation of the embodiment of the present invention;

FIG. 4 is a flowchart showing the association between each person region and the tracking model and the position / posture estimation operation of FIG. 3,

FIG. 5 is a diagram illustrating a change example of a human image captured by a camera installed on a ceiling.

FIG. 6 is a diagram for describing a human body model used in one embodiment of the present invention.

FIG. 7 is a diagram for describing an outline of a configuration of an observation unit.

FIG. 8 is a diagram for describing a feature extraction process in an observation unit.

FIG. 9 is a diagram for explaining feature point association processing;

FIG. 10 is a diagram illustrating an outline of a configuration of a tracking unit illustrated in FIG. 2;

FIG. 11 is a diagram for describing position estimation processing.

FIG. 12 is a diagram for explaining processing in a direction angle estimation circuit.

FIG. 13 is a diagram for describing a simulation experiment for confirming the accuracy of position following.

FIG. 14 is a diagram for explaining an experimental result with respect to FIG. 13;

FIG. 15 is a diagram for describing a simulation experiment for clarifying the ability to estimate the motion of a plurality of persons.

FIG. 16 is a diagram for explaining an experimental result for FIG. 15;

17 is a diagram for explaining an experimental result with respect to FIG.

FIG. 18 is a diagram for explaining an experimental result for FIG. 15;

FIG. 19 is a diagram for explaining an experimental result for FIG. 15;

FIG. 20 is a diagram for explaining an experimental result for FIG. 15;

FIG. 21 is a diagram illustrating aspects of an observation unit and a tracking unit according to the second embodiment of the present invention.

FIG. 22 is a diagram illustrating a binarized image when a person is standing and when a person is seated.

FIG. 23 is a diagram illustrating aspects of an observation unit and a tracking unit according to a third embodiment of the present invention.

FIG. 24 is a diagram illustrating a state in which an average color of a chest portion in an area-divided image is detected.

FIG. 25 is a diagram illustrating an example of tracking a person by a camera installed on a ceiling.

[Explanation of symbols]

1 Moving object tracking device, 2♯1-2♯n camera, 4♯
1-4♯n Observation unit, 6 discovery unit, 8 tracking unit, 10
Area division circuit, 12 pixel value calculation circuit, 14 center of gravity selection circuit, 16 feature point association circuit, 17 silhouette height detection circuit, 18 silhouette height and color detection circuit,
22 position estimation circuit, 24 direction angle estimation circuit, 30 height information extraction circuit, 32 back conversion circuit, 34 state estimation circuit, 36 height and color information extraction circuit, 38 back
Color conversion circuit.

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5L096 AA02 BA02 CA05 EA43 FA60 FA67 FA69 GA08 GA38 GA55 HA05 JA11

Claims (4)

[Claims] 1. A moving object tracking apparatus for tracking a moving object moving in a scene from multiple viewpoints, wherein a downward direction is photographed from a plurality of viewpoints at a certain height and at a certain interval in the scene. A plurality of photographing means for performing, and provided in correspondence with each of the plurality of photographing means, and acquires position information of the moving object in response to the corresponding imaging means imaging a moving object in a downward direction. A plurality of observations that acquire information for identifying the moving object in response to the other imaging means imaging the side of the moving object, extract feature points from the obtained result, and output as observation information. Means, and a tracking means for predicting the state of the moving object by integrating observation information of the plurality of observation means in accordance with a request from the observation means that extracted the feature point among the plurality of observation means. , Moving object tracking Trace device. 2. The plurality of observing means, based on the predicted observing position sent from the tracking means, transmit to the tracking means in association with each tracking target, and further correspond to the plurality of observing means. Receiving means for detecting a new moving object in response to the unattached feature amount and sending the detected moving object to the tracking means, wherein the tracking means calculates each moving object information based on the new moving object detected by the finding means; The moving object tracking device according to claim 1, wherein the moving object tracking device is updated. 3. Each of the plurality of observation units includes: a region dividing unit that divides an image received from the corresponding photographing unit into a background image and a moving object region; and a moving object region extracted by the region dividing unit. 2. A moving object tracking apparatus according to claim 1, further comprising: means for obtaining a center of gravity and outputting information on the center of gravity as the feature point. 4. The tracking means is constituted by a computer, each of the plurality of observation means is constituted by a computer, and each of the plurality of observation means communicates with the tracking means. 2. The moving object tracking device according to 1.