JP3520050B2 - Moving object tracking device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving object tracking device, and more particularly to a moving object tracking device for tracking a moving object by multi-view images obtained asynchronously.

[0002]

Stereo measurement, which obtains depth information in a scene from a plurality of images obtained from different viewpoints, occupies an important position in computer vision research whose restoration of three-dimensional information from an image is one of the main subjects. ing.

Correspondence between images is particularly important in stereo measurement, and feature amounts are appropriately selected to reduce erroneous correspondence (Marr D. And Poggio
TA theory of human stereo vision. In Roy. Soc. L
ondon, pp. 301-328.1979), a method for effectively utilizing more viewpoints (Ramakant Nevatia. Depth
measurement by motion stereo. Computer Graphics an
d Image Processing vol. 5, pp.203-214, 19
76) has been developed. These are mainly for static scenes, and there are already many reports about the error of the stereo method in this case (Jeffrey J. Rod
riguez and JK Aggarwal. Stochastic Analysis of
stereo quantization error. IEEE Pattern Anal. Mach
ine Intell., Vol.12, No.5, pp.467-470,
5 1992).

[0004] In the research targeting scenes with motion,
Robot navigation based on stereo observation (Larr
y Matthies and Steven A. Shafer. Error Modeling in
stereo navigation. IEEE Rebotics and Automation,
vol. RA-3, No.3, pp.239-248, 1987),
Tracking known shapes with one camera (Akio Kos
aka and Goichi Nakazawa. Vision-based motion track
ing of rigid objectsusing prediction of uncertaini
es. In Proc. of International Conference on Roboti
cs and Automation, pp.2637-2644,195
5), tracking of known shape by stereo measurement (Ge
m-Sun J. Young and Rama Chellappa. 3-dmotion estim
ation using a sequence of noisy stereo images: Mod
els, estimation, and Uniqueness results.IEEE Patt
ern Anal. Machine Intell., Vol.12, No.8, pp.7
35-759, 1990), Tracking of multiple objects based on stereo observation (Jae-Woong Yi and Jun-Ho Oh. Recursive
resolving algorithm for multiple stereo and motion
matches. Image and Vision Computing, Vol.15, pp.
181-196, 1997) and the like have been proposed. These problem settings, which track multiple objects while associating image features with the object model of interest, are "motion associations."
Known as (Motion Correspondence) (Ingem
ar J. Cox. A review of statical data association te
chniques for motion correspondence. International
Journal of Computer Vision, Vol.10: 1, pp.53
-66, 1993).

[0005]

By the way, all of the above-mentioned conventional methods are based on the premise that observations from a plurality of viewpoints are made simultaneously at fixed intervals, or that they are still scenes (equivalent to simultaneous observations). If the observations are made simultaneously,
Since the same physical characteristics will be observed, it becomes easy to associate the movements.

However, even in the conventional method of performing simultaneous observation, "movement correspondence" between different times is still necessary in order to track an object in time series.

Further, since many conventional systems using multi-view images are based on the premise that observations are simultaneously made between the viewpoints in order to acquire position information at the time of tracking, they are obtained at each viewpoint. When processing an image, there is a problem that the overall processing speed is limited by the slowest processing. Further, in such a system, means for synchronizing the viewpoints in addition to the exchange of image information is required separately. The problems of the conventional system based on these synchronizations become more remarkable as the number of viewpoints used increases.

Therefore, the inventor of the present application filed Japanese Patent Application No. 11-308.
In No. 187, we proposed a "moving object tracking device" that performs feature extraction processing on images obtained by capturing images with a plurality of cameras. However, a viewpoint (camera position) that is advantageous for acquiring a plurality of types of information regarding a tracking target from an image with this moving object tracking device is not always advantageous for acquiring other information.

For example, when the person MAN is tracked by the camera 2 installed on the ceiling as shown in FIG. 25, both the position information and the information peculiar to the object such as the color and the length of the clothes are detected. With regard to (1), it was not possible to deal with the camera arrangement that only one of them could be obtained, and there were cases where the tracking stability became low.

That is, in FIG. 25, the arrow a has high accuracy in detecting the position when the person MAN exists directly under the camera 2, and even if there are two person MAN, they can be detected separately. It shows that the accuracy is high. The arrow b indicates that the farther the person MAN is from the camera 2, the more the information unique to the person MAN can be detected, and the more accurate the person can be identified.

Therefore, a main object of the present invention is to provide a moving object tracking device which can stably acquire position information and identify an object regardless of the position of the object.

[0012]

SUMMARY OF THE INVENTION The present invention is a moving object tracking device for tracking a moving object moving in a scene from multiple points of view, and a plurality of moving object tracking devices having a constant height and a fixed interval in the scene. A plurality of photographing means for photographing a downward direction from the viewpoint and a plurality of observation means provided corresponding to each of the plurality of photographing means and operating independently of each other.
It Each of the plurality of observation means is photographed by the corresponding photographing means
Position information of the moving object from the image signal of the moving object
Position information estimating means for estimating
The distance between the moving image capturing means and the moving object,
Shooting corresponding to the moving object according to the estimated distance
Weight information indicating whether it is closer to the lower side or the side of the means
Information estimating means for estimating information and feature points from image signals
Object identification information that identifies the moving object by performing extraction processing
The object identification information calculation means for calculating
Observations that send constant information and weight information as observation information
And information transmission means. The moving object tracking device is
Position information and object identification information transmitted from the measurement information transmission means
Distribution, and integrates the weighted information, obtain Preparations tracking means for predicting the state of the moving object.

Further, each of the plurality of observation means, based on the predicted observation position that will be transmitted from the tracking unit, corresponding shooting
Correspondence between the moving object and the tracking target means have taken
It includes associating means for performing. Moving object tracking device
And a discovery means for detecting a new moving object. Observation
The information transmitting means transmits the observation information of the associated moving object.
A view of a moving object that was sent to the tracking means and was not associated
The measurement information is transmitted to the finding means. Tracking means is compatible
Based on the new moving object discovered by the discovery means based on the morning it was not observed information, related to each moving object
To update the that information.

Each of the plurality of observing means further includes area dividing means for dividing the image photographed by the corresponding photographing means into a background image and a moving object area . Object identification information calculation
Detecting means includes a determined Mel hand stage the center of gravity point of the moving object region extracted by the region dividing means as a characteristic point.

Further, the tracking means is composed of a computer,
Each of the plurality of observation means is constituted by a computer, and each the tracking means of a plurality of observing means, communicate with each other.

[0016]

1 is a layout view of a camera according to an embodiment of the present invention. In FIG. 1, a plurality of cameras 2
# 1, # 2 and 2 # 3 are embedded in the ceiling of the room having a constant height at regular intervals so that the photographing direction is directed downward. Cameras 2 # 1, 2 # embedded in these ceilings
When the person is tracked by 2, 2 # 3, position information and information peculiar to the object such as the color and the length of the clothes are used as the information necessary for the tracking.

In order to obtain position information, it is advantageous to use a camera directly above a person in order to avoid obscuration with other objects because a plurality of persons can be observed separately. For example,
At the position below the camera 2 # 2, the position detection (separation) by the camera 2 # 2 becomes easy as described above with reference to FIG. 25, but the other cameras 2 # 1 and 2 # 3 perform side observation. Therefore, it becomes difficult to detect the position. Arrow a indicates each camera 2 #
It shows that the position detection accuracy of the objects directly below each of 1, 2 # 2 and 2 # 3 is high.

On the other hand, the unique information such as the color of the clothes, the height and the face image is advantageous because the cameras 2 # 1 and 2 # 3 observe the sides more than the camera 2 # 2. Arrow b indicates each camera 2 #
It is shown that 1,2 # 2 and 2 # 3 each have good acquisition of unique information when observing their sides.

According to the present invention, as shown in FIG.
1, 2 # 2, 2 # 3 are placed on the ceiling, and by switching which camera is used to acquire which information according to the distance to the person, it is possible to prevent the influence of hiding over a wide range. It is possible to stably perform both identification and position detection.

FIG. 2 is a block diagram showing the overall configuration of the moving object tracking device 1 according to the embodiment of the present invention. Referring to FIG. 2, moving object tracking device 1 includes cameras 2 # 1, 2 # 2, ..., 2 # n arranged on the ceiling as shown in FIG.
, Observation section 4 # 1, 4 # 2, ..., 4 # n, and discovery section 6
And a tracking unit 8.

Observation units 4 # 1, 4 # 2, ..., 4 # n are provided corresponding to cameras 2 # 1, 2 # 2, ..., 2 # n, respectively (hereinafter collectively referred to as cameras). 2 and observation section 4, respectively). Then, the observation units 4 # 1 and 4 #
2, ..., 4 # n have respective cameras 2 # 1, 2 # 2, ..., 2 #
n imaging outputs are given. Observation units 4 # 1 and 4 # 2
4 # n, discovery unit 6 and tracking unit 8 can operate independently of each other. For example, these are composed of different computers, and each computer is connected to the local area network LAN.

In the figure, the symbol A0 represents the observation information of the corresponding point (tracking target) transmitted from the observing unit 4 to the tracking unit 8, and the symbol A1 represents the uncorresponding point (transmitting from the observing unit 4 to the finding unit 6 ( It represents the observation information of points that cannot be traced to the tracking target, and the symbol A
Reference numeral 2 represents predicted position information transmitted from the tracking unit 8 to the observation unit 4, symbol A3 represents position information (initial value) of a new person transmitted from the discovery unit 6 to the tracking unit 8, and symbol A4 is the tracking unit. 8 represents the position information (after updating) transmitted from 8 to the discovery unit 4.

The observing section 4 acquires position information of the person directly under the image of the person directly under the corresponding camera, and at least the face according to the fact that the other camera images the side of the person. ， Acquire information to identify clothes,
Feature extraction processing is performed from the obtained result.

It should be noted that the camera is not necessarily limited to the one that directly images the person or person directly below, but for example, a robot may run away or an animal such as a dog or cat may move from above or from the side. It is only necessary to be able to take an image. Each observation unit 4 operates independently. The feature amount (the center of gravity point and the distance conversion value) obtained by the observation unit 4 is associated with the tracking target based on the predicted position information A2 transmitted from the tracking unit 8 described later, and then the observation time information is obtained. It is sent together with the tracking unit 8. The feature amount that is not matched is transmitted to the discovery unit 6 as the observation information A1 of the uncorresponding point.

The finding section 6 detects a person (new person) who appears newly in the scene by using the transmitted observation information A1 of the uncorresponding points. The detection result of the new person, that is, the position information A3 of the new person is acquired by the tracking unit 8
Sent to. As a result, the new person is added as a new tracking target. Then, the tracking unit 8 starts tracking.

The tracking unit 8 updates the position information of the person using the Kalman filter with the position information A3 of the new person as the initial value and the observation information A0 as the input value, and further predicts the position based on the observation model. The predicted position information A2 is transmitted to the observation unit 4. The location information (after updating) A4 is transmitted to the finding unit 6 as described later.

FIG. 3 is a flow chart for explaining the operation of one embodiment of the present invention, and FIG. 4 is a flow chart showing the correspondence between each human area and the tracking model and the position / orientation estimation operation of FIG. FIG. 5 is a diagram showing an example of a change in a person image based on the distance from a camera installed on the ceiling.

When a corresponding image signal is input from the camera 2 in step SP1 (abbreviated as SP in the figure) SP1 shown in FIG. 3, the observation section 4 detects a person area in step SP2. In step SP3, it is determined whether or not a person is detected. Step S until detecting a person
Repeat P1 to SP3. When the presence of a person is detected, the time when the person is observed is tracked by the tracking unit 8 in step SP4.
Send to. As a result, the observation unit 4 receives from the tracking unit 8 the position / orientation calculated from the tracking model and the observation time, and the predicted value of the human feature such as the face image and clothes.

In step SP6, each person area is associated with the tracking model. This process is performed by executing the process shown in FIG. That is, in step SP21, each person area is specified, and in step SP22, the position / posture is estimated. This process is performed by the process shown in FIG. Position estimation is performed in step SP31, and the distance d between each person area and the camera 2 is calculated in step SP32. The posture is estimated under the constraint of the distance d, and the reliability (weight) is calculated. The estimation of the posture is performed based on the rotation (posture angle) silhouette information that can be taken by a person who is a distance d from the camera as shown in FIG.

Then, step SP2 shown in FIG.
In 3, the human feature such as the head region image and the clothes color is extracted and the reliability (weight) is calculated. Then, in step SP24, the associating process based on the position / orientation information and the person characteristic information is performed. In step SP25, it is determined whether or not the processing has been completed for all the person areas,
If not completed, the process returns to step SP21 to process the next person area. If the processing has been completed for all the person regions, the process proceeds to step SP7 shown in FIG.

In step SP7, each person area is specified, and in step SP8 it is determined whether or not there is a correspondence relationship with the tracking model. If there is a correspondence relationship with the tracking model, the observation information such as the position / orientation and the human feature is transmitted to the tracking node in step SP9, and the reliability according to the distance from the camera 2 is added at that time. If there is no correspondence with the tracking model, the observation information such as the position / orientation and the human feature is transmitted to the discovery node in step SP10,
At that time, the reliability according to the distance from the camera 2 is added. In step SP11, it is determined whether or not the processing has been completed for all the person regions, and if not completed, step S
Return to P7, and if completed, return to step SP1.

A human body model used in the embodiment of the present invention will be described with reference to FIG. Figure 6
FIG. 4 is a diagram for explaining a human body model used in the embodiment of the present invention. In the figure, X, Y, and Z indicate the three axes of the world coordinate system. The person MAN is modeled by an elliptic cylinder h. The center axis Xh of the person model h is the rotation axis of the person, and the angle r formed by the normal axis (the axis in the short axis perpendicular to the rotation axis Xh) and the X axis is the posture angle of the person. In addition,
It is assumed that the rotation axis Xh of the person MAN is perpendicular to the floor surface (the plane formed by the X axis and the Y axis). Camera 2
It is arranged perpendicular to the rotation axis Xh of the person, that is, the Z axis.

Next, the observation section 4 shown in FIG. 2 will be described. FIG. 7 is a diagram for explaining the outline of the configuration of the observation unit 4. Referring to FIG. 7, the observation unit 4 includes an area dividing circuit 10, a pixel value calculating circuit 12, a center of gravity point selecting circuit 14,
And a feature point correspondence circuit 16.

The area division circuit 10, the pixel value calculation circuit 12, and the center-of-gravity point selection circuit 14 perform the feature extraction processing based on the inputted person image. The feature point association circuit 16 associates the extracted feature points with a tracking target (model).

The area dividing circuit 10 divides the input image into a person area and a background area. For example, "Image segmentation for person tracking by hierarchical adaptation based on continuous images" Computer Vision and Pattern Recognition (CVPR'9
8), Journal of IEEE Computer Society, p. 911 to
In 916, a method of area division is described.

Specifically, the mask image is generated by binarizing the difference between the image of the current frame captured by the camera and the images of the preceding and following frames, and superimposing some frames of the difference image (that is, Extract a rough body region by coarse division). When the size of the mask image is within a certain range, further estimate the distribution of the parameter by luminance information, color information, or pixel information, texture information and any combination thereof, and further by the estimated parameter distribution, The probability that the moving object region exists for each pixel on the newly obtained image is calculated (that is, precise division is performed). As a result, the person area is cut out from the input image captured by the camera.

Subsequently, the pixel value calculation circuit 12 performs distance conversion on the obtained person image. Specifically, a pixel value indicating the shortest distance from each of the pixels forming the person area to the boundary of the person area is calculated. The center-of-gravity point selection circuit 14 selects a point having the maximum pixel value in the person area as a center of gravity (that is, a feature point) of the person area. The pixel value at the center of gravity is used as the distance conversion value. The center-of-gravity point and the distance conversion value of the center-of-gravity point (the center-of-gravity point distance conversion value) are used as the feature amount.

Specifically, the area dividing circuit 10 outputs a binarized image as shown in FIG. The pixel value calculation circuit 12 generates a distance conversion image shown in FIG. 8B for the binarized image shown in FIG. Figure 8
In (b), a pixel with a larger distance conversion value is represented in black,
Pixels with smaller distance conversion values are shown in white. The color of the pixel becomes darker as the distance from the contour of the human body increases. The center-of-gravity point selection circuit 14 selects the pixel corresponding to the symbol “X” in FIG. 8B as the center of gravity point. The detection of the center of gravity by the distance conversion is characterized by being less susceptible to changes in the pose of a person.

Next, the feature point associating circuit 16 for associating the extracted barycentric point with the already-discovered tracking target will be described. FIG. 9 is a diagram for explaining the feature point association processing. Symbol 2 in the figure
0 # 1, 20 # 2, ..., 20 # k, ..., 20 # m ,.
Each of 20 # n has a camera 2 # 1, 2 # 2, ..., 2
The image planes obtained by #k, ..., 2 # m, ..., 2 # n are respectively represented (hereinafter, generically referred to as image plane 20).

At time ta, camera 2 # k detects time tb.
It is assumed that the observation is performed by the camera 2 # m (note that ta <tb). At time ta,
It is assumed that there are two persons h0 and h1 and the person h2 first appears in the scene at time tb.

The symbol x attached to the image plane 20 indicates the center of gravity detected in each image. In this case, time t
Two barycentric points (feature points) are detected from the image plane 20 # k at a, and three barycentric points (feature points) are detected from the image plane 20 # m at time tb.

For each of the persons h0 and h1, the predicted position at time tb is obtained by inquiry to the tracking unit 8 based on the observation result up to time ta. As will be described later, in the tracking unit 8, the motion of the person is assumed to be a uniform velocity motion, and the predicted position of the person hj at a certain time t is represented by a two-dimensional Gaussian distribution. The position of the person hj in the world coordinate system at time t is _defined as the position X _{hj, t} , the average of the two-dimensional Gaussian distribution is represented as / X _{hj, t} , and the covariance matrix is represented as / S _{hj, t} .
The average and covariance matrix are transmitted from the tracking unit 8 as the predicted position information A0.

In the expression, / X _{hj, t} is _expressed as a symbol X _{hj, t with} a "-" added, and / S _{hj, t} is denoted by a symbol S _{hj, t} on ". It is expressed as "-".

Predicted position distribution N (/ X _{hj, t} , / S _{hj, t} )
Is weakly perspectively projected onto the image 20 # i (i = 1, 2, ..., N), the one-dimensional Gaussian distribution n (/ x _{hj, t, i} , / s _{hj, t, i} is obtained, which indicates the existence probability of a person in the image 20 # _i, where the symbol x is the person position X in the world coordinate system.
Is projected on the image plane, the symbol / x is the average / X in the world coordinate system projected on the image plane, the symbol / s
Represents the projection of the covariance matrix / S in the world coordinate system on the image plane. In the formula, /
x is expressed as a symbol x with "-" added,
/ S is expressed as a symbol s with "-" added.

[0045]

[Equation 1]

The feature point that maximizes the probability expressed by the equation (1) is taken as the observation value corresponding to the person hj at the observation time, and the feature point is labeled hj. The observation information A0 of the labeled feature points is transmitted to the tracking unit 8. However, for the feature points that can be associated with multiple people,
It is determined that occlusion has occurred at the time of observation, and transmission is stopped.

After these processes, the feature points not associated with each other are assumed to belong to an unknown person, that is, a new person, and the position and time are transmitted to the finding unit 6 as uncorresponding observation information A1. It In FIG. 9, the observation information A1 about the person h2 is transmitted to the discovery unit 6.

Next, the tracking unit 8 shown in FIG. 2 will be described. The tracking unit 8 updates the position / direction of the tracking target person based on the observation information A0 sent from each of the observation units 4.

FIG. 10 is a diagram for explaining the outline of the configuration of the tracking unit 8 shown in FIG. Referring to FIG. 10, tracking unit 8 includes a position estimation circuit 22 and a direction angle estimation circuit 24. The position estimation circuit 22 estimates the predicted position based on the position of the center of gravity obtained from the observation unit 4, and outputs the estimation result to the direction estimation circuit 24. Direction angle estimation circuit 24
According to the estimation result output from the position estimation circuit 22,
The direction angle is estimated based on the barycentric point distance conversion value obtained from the observation unit 4.

First, the position estimation processing in the position estimation circuit 22 will be described in detail. The position of the person being tracked is updated by using the feature information (centroid position) associated with each observation unit 4. FIG. 11 is a diagram for explaining the position estimation process. Referring to FIG. 11, the distance between camera 2 # _i and image plane 20 # _{i is referred} to as distance l _i , and the distance between camera 2 # _i and person hj is referred to as distance L _{hj, i} . The angle formed by the epipolar line and the Y axis is denoted by w _{hj, i} .

In the position estimation processing, it is assumed that the person is moving at a constant velocity. The state of the person hj at time t is represented by equations (2) and (3) on world coordinates (X, Y). However, the initial state is determined by the information of the new person (model) transmitted from the discovery unit 6. The “′” attached to the matrix represents transposition.

Let ∧X _{hj, t-1 be} the estimated value of the person position X _hj at time (t-1), and let ∧S _{hj, t-1} be the estimated value ∧X.
_{Assuming that hj, t-1 is} the covariance matrix, the state at time t is given by equation (4).
And equation (5). In the formula, ∧X
_{Express hj, t-1} as a symbol X _{hj, t-1} with "∧" added, and ∧S _{hj, t-1} on the symbol S _{hj, t-1} with "∧" It is expressed as the one with.

The transition matrix F is expressed by the equation (6) (Δt: t-1 → t). The symbol Q represents the covariance matrix at the transition.

[0054]

[Equation 2]

Here, it is assumed that the observation section 4 # i makes the first observation. This observation can be expressed by equations (7) to (9) based on the position information sent from the observation unit 4 # i.

H = [1 0 0 0] (7)

[0057]

[Equation 3]

The symbol C _i is the position of the camera and the symbol R is
_{hj, t, i} represents the clockwise rotation of the angle w _{hj, t, i} formed by the epipolar line and the Y axis. The symbol e represents an observation error, and has an average of 0 and a standard deviation σ _{hj, t, i} . The standard error σ _{hj, t, i} is considered to increase as the distance of the camera increases, and is expressed as in equation (9).

Here, the position C _{i of} the camera and the person (X
_Since the distance L _{hj, t, i} from ( _{hj, t} ) is an unknown number _, the predicted position of X _{hj, t} / the distance calculated by X _{hj, t} / L _{hj, t, i} is used as an approximate value ( In the formula, / L _{hj, t, i} is the symbol L
It is expressed as _{hj, t, i} with "-" added.

In the observation formula (8), the left side represents the observation information and the right side represents the result of projecting the person position on the image.

The position estimation circuit 22 constitutes a Kalman filter by the above observation model and updates the state of the person hj.

[0062]

[Equation 4]

For each camera, equations (10) and
The update process according to (11) is performed to predict the state.
The state prediction of the person hj at time (t + 1) is the average.
/ X _{hj, t + 1}, The covariance matrix / S_{hj, t + 1}Gaussian distribution
Given in. The result of the state prediction meets the request of the observation unit 4.
Correspondence of feature points as described above
Used for. A person who has moved outside the detectable range of the camera
Delete the physical model and stop tracking the person.

Next, the processing of the direction angle estimating circuit 24 will be described. In the direction angle estimation circuit 24, the posture angle r of the person
(See FIG. 6) is estimated using the center-of-gravity point distance conversion value that does not cause occlusion. This estimation is performed based on the human body observation model (ellipsoidal model) shown in FIG. Referring to FIG. 12, the barycentric point distance conversion value is used with s being the width of the ellipsoid on the image captured by camera 2. Assuming a weak perspective transformation, the observed center-of-gravity point distance conversion value s follows Equation (12), where θ is the angle formed by the optical axis and the normal to the rotation axis of the ellipsoidal model (person). Here, the symbol L in the equation (12) indicates the distance from the rotation axis of the ellipsoidal model to the camera, and the symbols A and B are constants. Assuming that the observation is accompanied by Gaussian error, the probability P (s | θ) that the centroid distance conversion value s is observed is given by the equation (1)
3). The constants A and B and the variance σ _{s of the} error are determined in advance based on the learning data.

[0065]

[Equation 5]

The posture of the human body is the Euler angles (a, e, r)
Can be expressed by Where r is the azimuth angle (posture angle)
, A represents an azimuth angle, and e represents an elevation angle.
Since the azimuth angle a and the elevation angle e are determined by the rotation axis direction of the human body, only the rotation angle r around the rotation axis is an unknown number.

The normal vector N of the human body is expressed by equation (14). Therefore, the optical axis vector C of the camera 2 # i
The angle θ _i (r) formed between the normal vector N of the human body and the normal vector N of the human body is expressed by Expression (15).

N = R _Z (a) R _Y (e) R _X (r) e (14) θ _ci (r) = cos ⁻¹ · N ^T · C (15) In the formula (14), , R _Z , R _Y , and R _X represent rotation matrices about the Z, Y, and X axes, respectively, and e _Z is Z.
A unit vector in the axial direction is shown. Also, equation (15)
In, the symbol N ^T indicates a transposed vector of the normal vector N.

The probability P that the set W (s ₁ , s ₂ , ..., Sn) of the barycentric point distance conversion values is observed by the _n cameras 2 # ₁ to ₂ # _n is given by the equation (13). It is expressed by equation (16). A value r that maximizes the probability P (W | r) in Expression (16) is used as the estimated value of the posture angle.

[0070]

[Equation 6]

The azimuth angle estimation is updated every time observation is performed, like the position estimation described above. Note that FIG.
The observation model shown in (3) has the input as the characteristic information (the center-of-gravity point distance conversion value s) transmitted from the observation unit 4 _, and has the state r _{hj, t} represented by the equation (17). The presence / absence of occlusion is determined at the time of associating feature points, and is not used as observation information when occlusion occurs.

Next, the outline of the finding section 6 shown in FIG. 2 will be described. The finding unit 6 detects a person (new person) who newly appears in the scene, and adds a corresponding model to the tracking unit 8.

Since the observation information is acquired asynchronously, ordinary stereo correspondence cannot be applied as it is. Therefore, the following correspondence (discovery) method using time series information is used.

First, of the observation information of uncorresponding points sent from each of the observing units 4, the observation information at different four times is selected one by one (denoted by γ). Observation time t1, t
The observation viewpoints are C ₁ , C ₂ , C ₃ , and C ₄ for 2, 2, and t ₄ , respectively. With observation information as input,
The above-described Kalman filter update processing (equation (8)) is performed. However, the initial variance / S _{hj, t} = 0.

By this operation, the predicted orbits in the four observations can be obtained. Using this predicted trajectory, position estimation is performed using equation (4) (where Δt = t4-ti: i
= 1, 2, or 3). The set of position estimation results is [∧
_{X t1, ∧X t2, ∧X t3} , and ∧X _{t 4].}

As described above, in the observation formula of the formula (7),
The left side shows the observation information, and the right side shows the result of projecting the person position on the image. The error evaluation function f (γ) using the Mahalanobis distance shown in Expression (17) is defined as the difference between the observation information and the result of projecting the person position on the image.

[0077]

[Equation 7]

A combination in which the value of the evaluation function shown in equation (17) is within a certain threshold is a set of feature points belonging to a new person, and the estimated position at the latest observation time (here, t4) is the initial found position. It transmits to the tracking unit 8.

Next, in order to clarify the effectiveness of the present invention, the following two simulation experiments were conducted. FIG. 13 is a diagram for explaining a simulation experiment for confirming the accuracy of position tracking. Figure 1
In the simulation experiment shown in 3, two cameras (2
# 1, # 2) and the circular orbit indicated by the dotted line with a cycle of 10 ×
An object h0 to be tracked that moves in a uniform circular motion at t _f was observed. t _f
Is the shooting interval of the camera. The Kalman filter has
A model of constant velocity linear motion is given. In the observation, parallel projection is assumed, and in the experiment, there are four angles θ formed by the optical axes of the two cameras 2 # 1 and 2 # 2 (0 °, 30 °, 45 °).
, 90 °). Under each condition, the difference Δt in the shooting time between the camera 2 # 1 and the camera 2 # 2 is 0.
To t _f / 2 from the observation of camera 2 # 1 to t _f
Later (for the direction parallel to the shooting surface of camera 2 # 1)
The position prediction error was recorded.

FIG. 14 is a diagram for explaining the experimental results for FIG. In FIG. 14, the horizontal axis represents the deviation Δt of the shooting time, and the vertical axis represents the prediction error. As shown in FIG. 14, the angle θ formed by the optical axes is 0 °, 30 °, 45 °
In particular, the prediction error decreased as the shooting interval increased, and the prediction error took the minimum value at Δt = t _f / 2. From this, it can be seen that when the motion of the tracking target object deviates from the motion model that is the premise of the Kalman filter, a higher followability can be obtained by performing the asynchronous observation, in particular, as compared with the synchronous observation.

FIG. 15 is a diagram for explaining a simulation experiment for clarifying the ability to estimate the movements of a plurality of persons. In the simulation experiment shown in FIG. 15, five cameras 2 # 1 to 2 # 5 are arranged. Each camera is connected to one computer that constitutes the corresponding observation unit. Image processing is performed on these computers. The processing speed is about 1-2 frame / sec.

Each computer is connected to the local area network LAN and has internal clocks synchronized with each other. Two computers (not shown) that respectively configure the finding unit 6 and the tracking unit 8 are local area network LA.
It is connected to N.

Each camera is calibrated in advance, and the calibration information of each observing unit 4 is:
It is transmitted to the discovering unit 6 and the tracking unit 8 together with the observation information (observation time, characteristic points). In the experiment, position tracking of two persons h0 and h1 was performed. As shown in FIG. 15, two persons h0 and h1 appear in the scene 55 in order.

16 to 20 are views for explaining the experimental results for FIG. In the simulation experiment shown in FIG. 15, the transition of the characteristic points obtained by each camera over time is shown. 16 shows the camera 2 # 1
17 corresponds to the camera 2 # 2, FIG. 18 corresponds to the camera 2 # 3, FIG. 19 corresponds to the camera 2 # 4, and FIG. 20 corresponds to the camera 2 # 5. 16 to 20, the horizontal axis represents time and the vertical axis represents tracking position.

Referring to FIGS. 16 to 20, at the beginning of tracking,
Since no characteristic points are observed except for the camera 2 # 1, the person is not found. As the first person approaches the center of the experimental environment (scene 55), feature points are obtained from other viewpoints (cameras 2 # 3, 2 # 4, 2 # 5) and tracking is started (about from the start). 20 seconds have passed). Similarly, tracking of the second person is started about 37 seconds after the movement of the second person. 16 in FIG. 16 to FIG. 20 is 1
The tracking result of the second person is shown, and the cross shows the tracking result of the second person. From the above experimental results, it can be said that the present invention enables tracking of a plurality of persons.

In the above-described embodiment, the movement of the person is tracked based on the feature points in the images asynchronously photographed from a plurality of viewpoints. Next, however, the attribute value peculiar to the person is extracted,
An embodiment will be described in which the extraction result is output as observation information, the state of the corresponding person is detected based on the extracted attribute value, and the tracking model of the person is switched according to the detected state. For example, since it is considered that there is a very low possibility that a person moves while seated, the reliability of the position estimation value is increased.

FIG. 21 is a diagram showing an outline of the observation section and the tracking section of the second embodiment of the present invention, and corresponds to FIGS. 7 and 10 of the first embodiment. In FIG. 21, a region dividing circuit 10, a pixel value calculating circuit 12, and a center point selecting circuit 1
4 is the same as that of FIG. 7, but the observation unit 4 of this embodiment is newly provided with a silhouette height detection circuit 17. The silhouette height detecting circuit 17 is the area dividing circuit 10.
The height of the person is detected from the binarized image binarized by.

FIG. 22 is a diagram showing a binarized image when a person is standing and seated. As shown in FIG. 22 (a), the silhouette height detection circuit 17 detects the height T1 of the binarized image and the height T of the binarized image shown in FIG. 22 (b).
2 is detected. For example, if a person is standing, T
1 is 180 cm and T2 is 1 if a person is seated
It is detected as 40 cm. The detection output of the silhouette height detection circuit 17 is given to the feature point correspondence circuit 16.

As described in the first embodiment, the feature point associating circuit 16 associates the barycentric point extracted by the barycentric point selecting circuit 14 with the already-discovered tracking target. Based on the detection output of the silhouette height detection circuit 17, the information obtained by measuring the height of the corresponding person is included in the observation information and output to the tracking unit 8.

On the other hand, tracking unit 8 includes a height information extraction circuit 30, a back conversion circuit 32, and a state estimation circuit 34, as shown in FIG. 21, in addition to the configuration shown in FIG. The height information extraction circuit 30 extracts height information from the observation information sent from the observation unit 4 to the tracking unit 8 and supplies it to the backrest conversion circuit 32. The backrest conversion circuit 32 converts the height information extracted from the observation information into a backrest.

If the backrest changes from 180 cm to 140 cm, for example, the state estimating circuit 34 estimates that the person is in a standing state from a standing state.
On the contrary, if the backrest changes from 140 cm to 180 cm, it is estimated that the person is standing up.

The observing section 4 and the tracking section 8 operate in the same manner as in FIG. 10, and the tracking section 8 estimates the predicted position based on the position of the center of gravity included in the observation information obtained from the observing section 4,
The direction angle is estimated based on the estimation result, and a specific person among a plurality of persons is tracked.

Among the observation information sent from the observing unit 4 to the tracking unit 8, the height information is extracted by the height information extracting circuit 30, converted into the ignorance by the displacing circuit 32, and the state estimating circuit 34 exemplifies the person. It is estimated whether the person is standing or seated. Tracking unit 8
Then, the person to be tracked is switched according to the estimated state.
That is, if the person being tracked is seated,
The person no longer moves, so he tracks other people.

In the above description, the state in which the person is standing and the state in which the person is seated are estimated, but the present invention is not limited to this. It is also possible to estimate action information such as a state of stopping from a state of being. In that case,
It may be estimated that the person is moving when the speed of the person becomes a certain speed or more, and it is estimated that the person is stopped when the speed becomes zero. In this case, the above equations (4) to (6) are expressed by the following equation (1
8) to (20). These expressions are time t
_It shows a case where the state of the person h ₁ at time t _b is predicted from the information up to the previous observation made in a.

[0095]

[Equation 8]

In the above equation, Q _x and Δt represent the fluctuation of the motion between Δt, and it is considered that the magnitude thereof changes depending on the current motion state. In the above-described embodiment, it is possible to change this value such that the walking width is large, the walking width is medium, and the sitting width is small. In this way, by changing the motion model, the reliability of the estimation result can be evaluated according to the motion state of the object.

In the second embodiment shown in FIG. 21, the state estimation of whether the person is standing or seated and the state of whether the person is stationary or has started moving are estimated. . However, in associating the feature of each observed image with the tracking model, the attribute value peculiar to the target object that can be observed from the image can be used instead of the position information. Here, the unique attribute value is not limited to the backrest of the target object or the action information described above, and the color of clothes may be considered.

FIG. 23 is a block diagram showing such an embodiment. The observation unit 4 shown in FIG. 23 is provided with a silhouette height / color detection circuit 18 in place of the silhouette height detection circuit 17 shown in FIG.
1 instead of the height extraction circuit 30 and the backrest conversion circuit 32,
A height / color information extraction circuit 36 and a color conversion circuit 38 are provided. The silhouette height of the observation unit 4 and the color detection circuit 18 detect the height of the person as described with reference to FIG. 21, and at the same time, as shown in FIG. The average color of the shaded area) is also detected and given to the feature point correspondence circuit 16.

On the other hand, the height / color information extraction circuit 3 of the tracking unit 8
The reference numeral 6 extracts height information and color information from the observation information, the backrest / color conversion circuit 38 converts backrest and color from the height information and color information, and the state estimation circuit 34 estimates the state of the person. . In this case, the color information S _tn obtained in the previous observation and the color information S _{tn + 1} obtained in the current observation are expressed by the following equation.

[0100]

[Equation 9]

As described above, by using the attribute values specific to the target object such as the backrest and the color information as the observation information, there is an advantage that the target object is not confused even when the target object is approaching. is there.

The embodiments disclosed this time are to be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and is intended to include meanings equivalent to the claims and all modifications within the scope.

[0103]

As described above, according to the present invention, the photographing means photographs the downward direction from a plurality of viewpoints at a constant height and at constant intervals in the scene, and the photographing means moves downward. The position information of the moving object is acquired according to the image of the moving object, and the information for identifying the moving object is acquired according to the image of the side of the moving object by another imaging unit. Then, the feature points are extracted from the acquired results and output as observation information, and the state of the moving object is predicted by integrating the observation information of the plurality of observation means according to the request from the observation means that extracted the feature points. The position information can be acquired and the object can be stably identified regardless of the position of the object, and the moving object can be tracked with high accuracy.

[Brief description of drawings]

FIG. 1 is a layout view of a camera of an embodiment of the present invention.

FIG. 2 is a block diagram showing an overall configuration of a moving object tracking device 1 according to an embodiment of the present invention.

FIG. 3 is a flowchart for explaining the operation of the embodiment of the present invention.

FIG. 4 is a flowchart showing the correspondence between each human region and the tracking model in FIG. 3 and the position / orientation estimation operation,

FIG. 5 is a diagram showing a change example of a person image captured by a camera installed on a ceiling.

FIG. 6 is a diagram for explaining a human body model used in the embodiment of the present invention.

FIG. 7 is a diagram for explaining an outline of a configuration of an observation unit.

FIG. 8 is a diagram for explaining a feature extraction process in the observation unit.

FIG. 9 is a diagram for explaining a feature point association process.

FIG. 10 is a diagram for explaining an outline of the configuration of the tracking unit shown in FIG.

FIG. 11 is a diagram for explaining position estimation processing.

FIG. 12 is a diagram for explaining processing in the direction angle estimation circuit.

FIG. 13 is a diagram for explaining a simulation experiment for confirming the accuracy of position tracking.

FIG. 14 is a diagram for explaining an experimental result for FIG.

FIG. 15 is a diagram for explaining a simulation experiment for clarifying the ability to estimate the movements of a plurality of persons.

FIG. 16 is a diagram for explaining an experimental result for FIG. 15.

FIG. 17 is a diagram for explaining an experimental result for FIG. 15.

FIG. 18 is a diagram for explaining an experimental result for FIG. 15.

FIG. 19 is a diagram for explaining an experimental result for FIG. 15.

FIG. 20 is a diagram for explaining an experimental result for FIG. 15.

FIG. 21 is a diagram showing aspects of an observation unit and a tracking unit according to the second embodiment of the present invention.

FIG. 22 is a diagram showing a binarized image when a person is standing and when a person is seated.

FIG. 23 is a diagram showing aspects of an observation unit and a tracking unit according to the third embodiment of the present invention.

FIG. 24 is a diagram showing a state of detecting an average color of a chest portion of an image obtained by region division.

FIG. 25 is a diagram showing an example of tracking a person with a camera installed on the ceiling.

[Explanation of symbols]

1 Moving Object Tracking Device, 2 # 1-2 # n Cameras, 4 #
1-4 # n Observation part, 6 Discovery part, 8 Tracking part, 10
Area dividing circuit, 12 pixel value calculating circuit, 14 barycentric point selecting circuit, 16 feature point associating circuit, 17 silhouette height detecting circuit, 18 silhouette height, color detecting circuit,
22 position estimation circuit, 24 direction angle estimation circuit, 30 height information extraction circuit, 32 back conversion circuit, 34 state estimation circuit, 36 height and color information extraction circuit, 38 back support,
Color conversion circuit.

─────────────────────────────────────────────────── ─── Continuation of the front page (58) Fields surveyed (Int.Cl. ⁷ , DB name) H04N 7/18 G06T 1/00 G06T 7/00

Claims (4)

(57) [Claims] 1. A moving object tracking device for tracking a moving object moving in a scene from multiple viewpoints, wherein a downward image is taken from a plurality of viewpoints at a constant height and at regular intervals in the scene. a plurality of imaging means for, provided corresponding to each of the plurality of imaging means, each other
And a plurality of observing means for operating independently to have, each of the plurality of observing means from the image signal of the moving object corresponding imaging means photographed,
Position information estimating means for estimating the position information of the moving object
When photographing means and the transfer of the corresponding, based on the position information
Estimate the distance to the moving body and adjust the distance according to the estimated distance.
Then, if the moving object is below the corresponding photographing means,
Is a weight for estimating weight information that indicates which side is closer to
Information estimating means and a feature point extraction process from the image signal to obtain the moving object.
Object identification information calculator for calculating object identification information
Step, the position information, the object identification information, and the weight information
Observation information transmitting means for transmitting as the observation information, the moving object tracking device further includes the position information transmitted from the observation information transmitting means,
A moving object tracking device , comprising tracking means for integrating the object identification information and the weight information to predict the state of the moving object. Each wherein said plurality of observation means, based on the predicted observation position that will be transmitted from said tracking means, said corresponding
Includes correlation means for capturing means performs <br/> correspondence between the moving object and the tracking target taken for the moving object tracking apparatus further comprises a discovery means for detecting a new moving object, wherein The observation information transmitting means is used to view the associated moving object.
The measurement information was sent to the tracking means and was not associated
The observation information of the moving object is transmitted to the discovery means, and the tracking means adds the observation information that is not associated with the observation information.
Based on the basis of the new moving object found by the finding means, to update the information on each moving object, the moving object tracking apparatus according to claim 1. 3. Each of the plurality of observing means further includes area dividing means for dividing an image photographed by the corresponding photographing means into a background image and a moving object area, and the object identification information calculating means, calculated Mel including <br/> hand stage, moving object tracking apparatus according to claim 1 as a point wherein the center of gravity of the moving object region extracted by the region dividing means. Wherein said tracking means comprises a computer, each of the plurality of observing means is constituted by a computer, with each said track means of said plurality of observing means can communicate with each other, claim 1. The moving object tracking device according to 1.