JP6628494B2 - Apparatus, program, and method for tracking object using discriminator learning based on real space information - Google Patents

Description

  The present invention relates to an object tracking technique for analyzing a time-series image group acquired from a camera capable of photographing an object and tracking the object.

  2. Description of the Related Art For the purpose of surveillance, marketing, and the like, a technology for tracking a position of a moving object in a real space using a time-series image group captured and generated by a camera has been developed. As the object to be tracked, various objects that can be photographed, such as a person and a vehicle, are targeted.

  In such an object tracking technique, in general, a target image area in an image in which a tracking target object is reflected is tracked, and the image area is converted into a position in a real space, so that the position of the object in the real space is obtained. Achieve tracking. Here, when one point in the two-dimensional image area is projected onto the three-dimensional real space, the height of the point in the real space corresponding to this point, that is, the z coordinate value is fixed to a predetermined value. There is a need to.

  For example, based on the result of tracking the area in the image, the place where it is clear that the object is in contact with the floor or the ground in the image, such as the foot position, is assumed to have a height of zero and is positioned on the floor of the real space. Can be projected. However, in practice, it is not easy to continue to specify the location where the object is in contact with the floor or the ground in the image. In general, in a captured image, a place where an object is in contact with a floor or the ground is often hidden behind another object such as a desk, a table, a person, or a car.

  For such a problem of projection into a real space, for example, Patent Document 1 discloses a case where a person's head is difficult to hide in an image compared to his / her feet, and a person's foot position is unknown in the image. A technique for detecting a head and applying a point indicating the position of the head in an image to a real space by applying an average height value preset as a height is disclosed.

  Further, Patent Literature 2 discloses a technique of photographing an object from a plurality of viewpoints and estimating a position where the object contacts a road surface based on a plurality of images having different viewpoints.

JP 2014-229068 A JP 2014-194361 A

  However, in the related arts described in Patent Literature 1 and Patent Literature 2, there has been a problem that it is difficult to keep track of an object while maintaining high positional accuracy in a real space.

  For example, in the method described in Patent Literature 1 in which the average height of an object is set in advance and used, the deviation between the height of the tracking target object and the average height is large, or the shape of the object is If it changes, the estimated position in the real space will deviate significantly from the correct position. For example, when tracking a child, the difference between the actual height of the child and a preset average height becomes large, and when projecting the head position in the image to the real space, the head position from the original head position is reduced. When viewed, a large shift occurs. Furthermore, setting the height to the average height assumes that the person to be tracked is only upright. As a result, when a shape change such as sitting or bowing occurs in the tracking target person, a large deviation occurs in the estimated position.

  Further, in the method using an image from a plurality of viewpoints as described in Patent Literature 2, it is more likely that a portion in contact with the floor or the ground is shown in any of the images as compared with a monocular camera. However, for example, a situation can easily occur in which the location is not photographed by any camera due to being surrounded by another moving object. That is, there is no guarantee that a portion in contact with the floor or the ground will be reflected on any of the cameras even from a plurality of viewpoints. In addition, since a plurality of cameras must be used, there is also a problem that the introduction / operation cost is higher than that of the monocular camera.

  Furthermore, the techniques described in Patent Literature 1 and Patent Literature 2 perform tracking in consideration of the amount of movement of a region corresponding to an object in an image, and therefore, consider the amount of movement in a real space. Absent. As a result, when an error occurs in the estimated position in the image, even if the amount of movement in the image is small, the amount of movement in the real space is abruptly changed such that it is almost impossible in reality. May be caused.

  Therefore, an object of the present invention is to provide an apparatus, a program, and a method capable of tracking an object while maintaining high positional accuracy in a real space by using an acquired image group.

According to the present invention, an apparatus capable of tracking an object to be tracked using a time-series image group acquired from one or more cameras capable of capturing the object,
As the object motion information which is information including the position information on the position of the object in the real space at one time, at least the change from the previous time in the position of the object in the real space is adopted, and the one time Candidate information calculation means for calculating a plurality of candidate object motion information that is object motion information different in the amount of change from each other,
And image information according to the acquired image, a classifier to learn the data set comprising a the object motion information that is the correct answer, the probability that a change in the position of the real space of the object a variable density For the evaluation function having a term relating to the function and a term for evaluating the closeness of the appearance (appearance) of the image area calculated from the candidate object motion information with respect to the image area relating to the object, Applying the candidate object motion information and the image information related to the image at the one time point, the candidate object motion information that maximizes the score of the evaluation function is calculated based on the position of the object in the real space at the one time point. An object tracking unit that obtains instantaneous position information of the object in real space by a discriminator that outputs the position information as a correct answer ;
Object tracking apparatus having is provided.

Further, as another embodiment of the object tracking device according to the present invention, the candidate information calculating means includes, as the object motion information at the one time point , a change in the position of the object in the real space from the previous time point , adopted and change from the previous point in the height of the object, at least one of the variation in the single point in time is to calculate a plurality of candidate object motion information different,
The classifier of the object tracking means is
(A) a term relating to a probability density function having a change in the position of the object in real space as a variable,
(B) a term relating to the probability density function having a change in the height of the object as a variable;
And (c) a term for evaluating the apparent proximity of the image area calculated from the candidate object motion information to the image area of the object. Applying the image information of the image at one point in time , the candidate object motion information that maximizes the score of the evaluation function is calculated based on the position of the object in the real space and the height of the object at the one point in time . It is also preferable to output the information as correct information.

Furthermore, as still another embodiment of the object tracking device according to the present invention, the candidate information calculating means includes, as the object motion information at the one time point , a change in the position of the object in the real space from the previous time point. If, adopts a change from the previous point in the height of the object, at least one of the variation in the single point in time is to calculate a plurality of candidate object motion information different,
The classifier of the object tracking means is
(A) a term relating to a probability density function having a change in the position of the object in real space as a variable,
(B) a term relating to the probability density function having a change in the height of the object as a variable;
(C) a term that evaluates the degree to which a change due to the motion of the object in the image area related to the object matches a change amount related to the object motion information;
(D) a term for evaluating the apparent proximity of the image area calculated from the candidate object motion information to the image area relating to the object, to the input plurality of candidate object motion information and the Applying the image information of the image at one point in time , the candidate object motion information that maximizes the score of the evaluation function is calculated based on the position of the object in the real space and the height of the object at the one point in time . It is also preferable to output the information as correct information.

Further, in each of the embodiments described above, the candidate information calculation unit further employs a change in the inclination of the object from the previous time as the object motion information at the one time ,
It is also preferable that the discriminator of the object tracking means outputs candidate object motion information that maximizes the score of the evaluation function as information including information on the correct answer related to the tilt of the object at the one point in time .

Further, in each of the embodiments described above, the object is detected based on the acquired image, and the position of the object in the real space is determined based on the lowest position of the detected image area of the object. The ground position of the object is calculated, the position in the real space calculated based on the detected top position of the image area of the object, and the height of the object based on the calculated ground position It is preferable to further include an object detection unit that calculates

  Further, in each of the embodiments described above, the discriminator of the object tracking means determines the weight coefficient of each term of the evaluation function by learning, and uses the evaluation function having the determined weight coefficient to input the weight coefficient. It is also preferable that the image information relating to the image is processed to calculate the object motion information to be output.

Further, the object tracking apparatus according to the present invention, the identifier of the object tracking means, before the time of one time point, the data set generated by using the object motion information outputted as the correct learns, the one It is also preferable that image information relating to the image at the time is input, the image information is processed using the parameters determined by the learning, and the correct object motion information at the one time is output.
Further, in the object tracking device according to the present invention, the discriminator of the object tracking means may include, as an image area of the object, the object from the upper end of the object in the real space to a position lower by a predetermined ratio of the height of the object. It is also preferable to adopt an image area calculated by performing coordinate conversion on a portion.

  Furthermore, in the object tracking device according to the present invention, it is preferable that the discriminator of the object tracking means is constructed by a structured SVM (Structured Support Vector Machine) algorithm.

According to the present invention, a program for causing a computer mounted on a device capable of tracking an object to be tracked by using a time-series image group acquired from one or more cameras capable of capturing the object to be tracked is provided. So,
As the object motion information which is information including the position information on the position of the object in the real space at one time, at least the change from the previous time in the position of the object in the real space is adopted, and the one time Candidate information calculation means for calculating a plurality of candidate object motion information that is object motion information different in the amount of change from each other,
And image information according to the acquired image, a classifier to learn the data set comprising a the object motion information that is the correct answer, the probability that a change in the position of the real space of the object a variable density For the evaluation function having a term relating to the function and a term for evaluating the closeness of the appearance (appearance) of the image area calculated from the candidate object motion information with respect to the image area relating to the object, Applying the candidate object motion information and the image information related to the image at the one time point, the candidate object motion information that maximizes the score of the evaluation function is calculated based on the position of the object in the real space at the one time point. the discriminator to output as the position information of the correct, object tracking program for causing a computer to function as <br/> the object tracking means for acquiring every moment the position information of the real space of the object is provided is It is.

According to the present invention, there is further provided a method of tracking an object to be tracked by a computer using a time-series image group acquired from one or more cameras capable of capturing the object to be tracked,
As the object motion information which is information including the position information on the position of the object in the real space at one time, at least the change from the previous time in the position of the object in the real space is adopted, and the one time Calculating a plurality of candidate object motion information, which are different object motion information from each other,
A probability classifier that learns from a data set including image information of an acquired image and the object motion information that is regarded as a correct answer, and uses a change in position of the object in real space as a variable. , And a term for evaluating the closeness of the appearance (appearance) of the image region calculated from the candidate object motion information with respect to the image region of the object, The candidate object motion information that maximizes the score of the evaluation function is obtained by applying the object motion information and the image information of the image at the one time point to the correct answer based on the position of the object in the real space at the one time point. Repeating the step of determining the position information of the object in the real space at the one point in time by the discriminator that outputs the position information of the object. An object tracking method for acquiring is provided.

  ADVANTAGE OF THE INVENTION According to the object tracking apparatus, program, and method of this invention, an object can be tracked using an acquired image group, maintaining high positional accuracy in real space.

1 is a schematic diagram illustrating an embodiment of an object tracking system including an object tracking device according to the present invention. 5 is a flowchart schematically showing a flow of processing in an embodiment of the object tracking device according to the present invention. FIG. 2 is a functional block diagram showing a functional configuration in an embodiment of the object tracking device according to the present invention. It is a schematic diagram which shows one Embodiment of the method of calculating the height of the object in a height calculation part. It is a schematic diagram which shows roughly the relationship between the acquired time-series image and the discrimination function in the tracking discriminator. FIG. 2 is a schematic diagram for describing an embodiment in which a tracking target object is projected onto an image coordinate system. It is a schematic diagram which shows the relationship between the element which concerns on the amount of change in real space in object motion information, and an object model. FIG. 9 is a schematic diagram illustrating an embodiment of acquiring candidate object motion information by sampling a position in a real space. It is a schematic diagram which shows one Embodiment of acquisition of candidate object motion information by sampling which concerns on the height in real space. FIG. 7 is a schematic diagram illustrating an embodiment of obtaining candidate object motion information by sampling relating to a tilt in an image coordinate system. 9 is a graph showing an example of a probability density function for a position change in an evaluation function. 9 is a graph showing an example of a probability density function for a height change in an evaluation function. It is an image figure showing an example of a difference picture. Is a schematic diagram showing an embodiment of a feature vector of the apparent _yt | image region x ^t.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Object tracking system]
FIG. 1 is a schematic diagram showing an embodiment of an object tracking system including an object tracking device according to the present invention.

The object tracking system of the present embodiment shown in FIG.
(A) one or more cameras 2 capable of photographing an object to be tracked and capable of transmitting information of the photographed image in a time series via a communication network;
(B) an object tracking device 1 that can track the object using a time-series image group acquired from the camera 2 via a communication network.

  Here, the object to be tracked includes various objects that can be photographed, such as a person, an animal, a vehicle, and other movable physical objects. In particular, in the present embodiment, a person, an animal, or the like that can change its overall shape by standing, sitting, or bending may be used. Further, the place where the image is taken is not particularly limited, and may be, for example, an outdoor where spectators, commuters, shoppers, pedestrians, runners, etc. can be seen, but may take a seat or bow. It is also preferable to be indoors, such as inside a company, school, home, or store where such situations are expected.

  In addition, a communication network that is a transmission path of image information can be a wireless LAN (Local Area Network) such as Wi-Fi (registered trademark). Alternatively, the camera 2 and the object tracking device 1 are connected to each other via the Internet via a wireless access network such as LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), or 3G (3rd Generation). It may be.

  Further, the camera 2 and the object tracking device 1 may be communicatively connected via the Internet via a fixed access network such as an optical fiber network or an ADSL (Asymmetric Digital Subscriber Line), or via a private network. As a modification, the camera 2 and the object tracking device 1 may be directly connected by wire. Further, a camera control device (not shown) that can collect image information output from the plurality of cameras 2 and transmit the image information to the object tracking device 1 may be provided.

Similarly, as shown in FIG. 1, the object tracking device 1 is provided for each image of the object tracking target that is acquired moment by moment.
(A1) The tracking discriminator 114a that outputs at least the correct position information in the real space of the object to be tracked by inputting the image information related to the image, and at least the instantaneous position of the object to be tracked in the real space. Object tracking unit 114 for acquiring position information
Having.

Here, the tracking discriminator 114a
(A2) It is constructed and updated by learning with a data set including image information on an acquired image and “correct object movement information” including position information on the position of the tracking target object in the real space. Is done. The "object motion information" is described later in detail, is an amount containing a variation Delta] p _x ^t and Delta] p _y ^t from the previous time at the location in the real space of at least tracked object as an element.

  As described above, the object tracking device 1 performs tracking in consideration of not only image information relating to an acquired image but also “object motion information” including position information of a tracking target object in a real space (observation target space). It is carried out. For example, taking into account not only the change in the image area corresponding to the object in the image but also the change in the object equivalent area in the real space, that is, taking into account the constraints in the real space, It estimates position information every moment. As a result, when an error occurs in the estimated position in the image, even if the amount of movement in the image is small, the amount of movement in the real space is abruptly changed such that it is almost impossible in reality. Can be avoided. That is, by adopting the “object motion information” in consideration of the real space, it is possible to track the object while maintaining high positional accuracy in the real space while using the acquired image group.

Incidentally, in calculating the change in Delta] p _x ^t and Delta] p _y ^t like from the previous time position in the real space of the tracking target object in the "object motion information", the position information of the position information of the image in real space Must be converted to In the object tracking device 1, the position coordinates (u, v) in the image coordinate system uv set in the image are converted into the position coordinates (g in the world coordinate system G _x -G _y -G _{z set} in the real space. _x , g _y , g _z ) is used to calculate position information such as a change in position in the real space from image information of the object to be tracked, using a coordinate conversion operation.

For example, if the position (u, v) at the previous time (t−1) in the image of the tracking target object changes to the position (u ′, v ′) at the current time t, this object is in real space. Moved from the position (g _x , g _y , g _z ) at the previous time (t−1) to the position (g _x ′, g _y ′, g _z ′) at the current time t in (observation target space) Is estimated, and the change in the position in the real space from the previous time (t−1) can be obtained.

  Here, the time to be used is a time set every time the unit time elapses with the unit time set to 1, and the time immediately before the time t is t-1. Further, the coordinate conversion from the image coordinate system to the world coordinate system as described above can be determined by setting external parameters relating to the installation position and the shooting direction of each camera 2 by calibration in advance. is there. Note that, even when images are acquired from each of the plurality of cameras 2, these images can be integrated to construct one image space, and the image coordinate system can be applied to this image space.

  Further, in the object tracking device 1, the tracking discriminator 114a can be made to learn immediately, that is, online, using each time-series image acquired from the camera 2. As a result, it is possible to immediately grasp the position of the tracking target object and perform tracking. Furthermore, since tracking is performed using the tracking discriminator 114a that learns every moment, even if the appearance of the target object changes every moment, it can be recognized as the same object, and, for example, a unique It is easy to continue appropriate tracking while continuing to assign an identifier ID.

  Incidentally, as shown in the above (A1), the image information input / output by the tracking discriminator 114a of the object tracking unit 114 and the position information in real space (“object motion information”) both have a structure inside. This is the data that we have. That is, the tracking discriminator 114a can output information that is a correct answer in the real space based on the structure learning. As described above, the object tracking device 1 performs the object identification based on the structure learning in consideration of the structural relationship between the real space and the camera image, and thereby, for example, as described in detail later, the height and the shape of the tracking target object. Even if a change occurs, for example, it is possible to keep track of an accurate real space position while continuously assigning a unique identifier ID.

[Device Function Overview]
FIG. 2 is a flowchart schematically showing a flow of processing in an embodiment of the object tracking device according to the present invention.

  According to FIG. 2, when the object tracking device 1 of the present embodiment acquires an image to be analyzed from the camera 2, the object tracking device 1 sends a learned tracking identifier 114 a (FIG. 1) corresponding to the tracking target object. The object is tracked by inputting the image and outputting “object motion information” as a correct answer. Here, as many tracking discriminators 114a as the number of tracking target objects are used. At this time, each object is continuously assigned an identifier ID unique to the object.

  Further, the object tracking device 1 uses the correct “object motion information” output from the tracking discriminator 114a and the acquired image as a teacher data set, and causes the tracking discriminator 114a to perform online learning.

  The object tracking device 1 performs a detection process in parallel with the above-described tracking process. Specifically, a new or possibly untracked object is determined by using a learned detection classifier to determine whether or not the object is captured in the acquired image. It is assumed that the object is detected when a region appears on the image.

  When an object is detected, the object tracking device 1 calculates a similarity between the detected object and an object that has been tracked in the past before starting a new tracking, and when the similarity is equal to or greater than a predetermined value. In some cases, it is considered that the object once tracked has returned to the space where the image can be captured. In this case, the same identifier ID as the past object having a high similarity is given to the detected object, that is, the identifier ID is integrated and the tracking is restarted. On the other hand, when the calculated similarity is smaller than the predetermined value, it is considered that a new object has appeared in the space where the image can be captured, and a new identifier ID is assigned to the detected object.

  Next, the object tracking device 1 acquires a new image again in time series and repeats the processing cycle as shown in FIG.

  As described above, the object tracking device 1 performs the tracking process and the detection process using the time-series image group from one or more cameras 2 at the same time. In addition, integrated tracking can be performed.

[Device configuration, object tracking method]
FIG. 3 is a functional block diagram showing a functional configuration in an embodiment of the object tracking device according to the present invention.

  According to FIG. 3, the object tracking device 1 includes a communication interface 101 that can be communicatively connected to one or a plurality of cameras 2, an image storage unit 102, an ID storage unit 103, a tracked object management unit 104, a processor And a memory. Here, the processor memory realizes the object tracking function by executing a program that causes the computer of the object tracking device 1 to function.

  Further, the processor memory includes, as functional components, an object detection unit 111, an ID (identifier) management unit 112, a candidate information calculation unit 113, an object tracking unit 114, an object position / shape estimation unit 115, and a communication unit. And a control unit 121. Here, it is preferable that the object detection unit 111 includes a detection classifier 111a and a height calculation unit 111b. Further, it is preferable that the ID management unit 112 includes an object integration unit 112a and an object registration unit 112b. Furthermore, it is preferable that the object tracking unit 114 includes a tracking identifier 114a and a teacher data set generation unit 114b. The flow of processing shown by connecting the functional components of the object tracking device 1 in FIG. 3 with arrows is understood as an embodiment of the object tracking method according to the present invention.

  The camera 2 is a photographing device for visible light, near-infrared light, or infrared light provided with a solid-state image sensor such as a CCD image sensor or a CMOS image sensor. Further, the camera 2 or a camera control device (not shown) generates captured image data including an image of an object captured by the camera 2 and transmits the data to the object tracking device 1 in a time series or in a batch. Has functions. It is also preferable that the camera 2 is movable and can change the installation position, the shooting direction and the height, and has a function of receiving and processing a control signal for this change.

  The communication interface 101 receives captured image data as a time-series image group from the camera 2 or the camera control device via a communication network. Control of transmission / reception and communication data processing using the communication interface 101 is performed by the communication control unit 121, and the acquired captured image data is stored in the image storage unit 102. Here, the captured image data may be obtained by calling and acquiring in chronological order from the camera 2 or the camera control device, or may be obtained by sequentially acquiring images captured at fixed time intervals in real time. Is also good.

  The object detection unit 111 is a functional unit that can detect an object that has appeared or has not been tracked in the image of the object identification target by the detection classifier 111a that has learned using a predetermined feature amount. Specifically, an image area corresponding to an object to be tracked is detected in the image stored in the image storage unit 102. Here, when a person is to be tracked, a feature amount suitable for detecting a person is used for learning. It is also preferable to use, for example, a HOG feature amount as the feature amount for object detection. The HOG feature amount is a vector amount in which the gradient direction of the luminance in the local region of the image is converted into a histogram and each frequency is used as a component. About the person detection technology using the HOG feature, for example, non-patent documents Dalal.N and Triggs.B, `` Histograms of Oriented Gradients for Human Detection '', proceedings of IEEE Computer Vision and Pattern Recognition (CVPR), pp .886-893, 2005.

  Further, when detecting an object from the image input from the image storage unit 102, the object detection unit 111 notifies the ID management unit 112 of information on the detected object that may be newly registered, and requests registration.

Further, the object detection unit 111 has a height calculation unit 111b. The height calculation unit 111b calculates a ground contact position as a position of the object in the real space based on the lowest position (for example, the lowest pixel position) of the detected image area of the tracking target object, and and position in real space is calculated based on the highest position of the image area (e.g., uppermost pixel position) of the object was, based on the calculated ground position, and calculates the height h ₀ of the object. Next, an embodiment of the calculation of the height h ₀ detail.

FIG. 4 is a schematic diagram illustrating an embodiment of a method for calculating the height h ₀ of the object in the height calculation unit 111b. In the present embodiment, first, the detection of the object is performed when the object has a standard shape and a portion ( ground contact position) in contact with the floor or the ground in the image is clear. For example, in the case of a person, only a person standing upright and showing his / her feet is detected. When an object is detected in the image, the height of the object in a standard shape in the real space is estimated.

の関係が成立する。上式（１）において、行列Ｐは予め決定された透視投影行列であり、sは未知のスカラ変数である。この際、各カメラ２の内部パラメータ及び外部パラメータをキャリブレーションによって予め設定しておけば、カメラ２の位置・姿勢が変わらない限り、透視投影行列Ｐは当初設定された値をとり続ける。 Here, the coordinates (u, v) in the image coordinate system spanned on the image as shown in FIG. 1 and the coordinates (g _x , g) in the world coordinate system spanned in the real space (observation target space) _y , g _z )

Is established. In the above equation (1), the matrix P is a predetermined perspective projection matrix, and s is an unknown scalar variable. At this time, if the internal parameters and the external parameters of each camera 2 are set in advance by calibration, the perspective projection matrix P keeps taking the initially set value unless the position and orientation of the camera 2 change.

When the coordinates in the three-dimensional world coordinate system are obtained from the coordinates in the two-dimensional image coordinate system using the above equation (1), the coordinates (u, v) in the image coordinate system and the perspective projection matrix P are determined. , The number of unknown parameters (4) is greater than the number of observation equations (3), so it is not possible to uniquely determine the coordinates (g _x , g _y , g _z ) in the world coordinate system. Can not.

上式（２）において、未知のパラメータはs及びh₀の２つのみであり、一方、観測方程式の数は３つであることから、この式を用いて実空間での高さh₀を求めることが可能となる。尚、この際、s及びh₀の値を、最終的に最小二乗法を用いて決定することも好ましい。 However, in the present embodiment, as shown in FIG. 4, the detected object grounded position in contact with the floor or the ground in the image for ^{_{(u b 0, v b 0}} ) is obtained. Therefore, by substituting the ground position (u _b ⁰ , v _b ⁰ ) and g _z = 0 into the equation (1), the position in the real space corresponding to the ground position (u _b ⁰ , v _b ⁰ ) ( g _x ⁰ , g _y ⁰ , 0) can be uniquely obtained. Here, when the height of the object in real space and h _0, and the position coordinates g _x ⁰ and g _y ⁰ in the floor or ground of the acquired real space, the top of the object on the screen coordinate system The following relationship holds between the point (u _h ⁰ , v _h ⁰ ).

In the above equation (2), there are only two unknown parameters, s and h ₀ , while the number of observation equations is three. Therefore, the height h ₀ in the real space is calculated using this equation. It is possible to ask. At this time, it is also preferable that the values of s and h ₀ are finally determined by using the least squares method.

As described above, the height calculation unit 111b of the object detection unit 111 includes, for example, 1. 1. When a person is detected (extracted) in the image; 2. Project the human model to the real space to determine the coordinates of the feet in the world coordinate system; Height of the person, that is, the height h ₀ of a standard shape can be calculated.

  Returning to FIG. 3, the ID management unit 112 has an object integration unit 112a and an object registration unit 112b. Among them, the object integration unit 112a compares the detected object notified from the object detection unit 111 with a known object to which an identifier ID has been given in the past, and determines that the detected object is the same object. It is determined that an identifier ID given to the determined known object is given.

The object integration unit 112a specifically includes:
(A) The distance d in the real space between the two objects calculated from the image of the object identification target (for example, obtained from the plurality of cameras 2) is the distance between the two objects at the present time in consideration of the moving speed v of the known object. Is less than the estimated distance, and if the similarity determined from the detected object region and the known object region is greater than a predetermined threshold, if the known object is not currently tracked, it is detected. It is determined that the same identifier ID as that assigned to the known object is assigned to the object.
(B) On the other hand, in (a), if the known object is currently being tracked, the notification from the object detection unit 111 is ignored, and no new registration is performed.
(C) In cases other than the above (a) and (b), it is determined to assign a new identifier ID to the detected object.

On the other hand, the object registration unit 112b assigns an identifier ID to the object for which the identifier ID is determined, and registers and manages the object. Here, it is also preferable that the information relating to the image area of the detected object and the assigned identifier ID be stored in the ID storage unit 103 in association with each other. Note that the similarity in (a) above may be calculated using a classifier corresponding to each object learned during tracking. As will be described in detail later, the evaluation function F corresponding to each object is calculated using a value of a function Ψ (x ^t | _yt ) for scoring the closeness of appearance (appearance). Is also preferred.

The candidate information calculation unit 113 calculates, as “object motion information” at one time t, at least (a) the change amounts Δp _x ^t and Δp from the previous time (t−1) in the position of the tracking target object in the real space. _y ^t
To calculate a plurality of “candidate object motion information” having different amounts of change at one time t. Here, it is possible to estimate the position of the tracking target object at the one time t by determining an optimum one from the plurality of “candidate object motion information” using the tracking discriminator 114a. You can.

In addition, as a change mode, the candidate information calculation unit 113 sets “object motion information” at one time t as
(B) the a variation Delta] p _x ^t and Delta] p _y ^t of (a), employs a variation Delta] h ^t from the previous time (t-1) at the height of the tracking target object, the one at time t A plurality of “candidate object motion information” in which at least one of the changes in may be calculated. further,
A variation Delta] p _x ^t and Delta] p _y ^t of (c) above (a), the change in Delta] h ^t of the (b), the variation .DELTA.a ^t from the previous time (t-1) in the slope of the tracked object It is also preferable to calculate a plurality of “candidate object motion information” in which at least one of the changes at this one time t is different by adopting the change from. Specific examples of the “object motion information” and the “candidate object motion information” will be described later in detail with reference to FIGS.

  Similarly, in FIG. 3, the object tracking unit 114 uses the tracking discriminator 114 a to acquire instantaneous position information of the tracking target object in the real space. Specifically, for example, using the tracking discriminator 114a for determining whether or not the tracking target object is reflected in a certain area in the acquired image, if the tracking target object is reflected in an unknown image, Object tracking is performed by estimating the region to be recognized.

Here, the tracking discriminator 114a is
(A) online learning is performed by a teacher data set including image information relating to the acquired image and object motion information including position information relating to the position of the object in the real space, which is regarded as the correct answer;
(B) For each image of the object to be tracked, by inputting image information related to the image, at least position information that is a correct answer in the real space of the tracked object is output.
The teacher data set (a) is generated by the teacher data set generation unit 114b.

  In this way, the tracking discriminator 114a repeatedly executes the above (a) and (b), so that each time a new image is read, the position information of the object at this reading time is learned while learning online. Enable output.

  FIG. 5 is a schematic diagram schematically showing the relationship between the acquired time-series images and the identification function of the tracking identification unit 114a.

  According to FIG. 5, the tracking discriminator 114a performs learning every moment using the acquired time-series images. The tracking discriminator can be constructed with various kinds of supervised machine learning capable of handling structural data. For example, the tracking discriminator is constructed by an algorithm of a structured support vector machine (SVM). Is also preferred.

  Specifically, the contents of the learning include adding a positive label to “object motion information” as a feature amount associated with the area of the tracking target object, and adding a negative label to “object motion information” associated with the other areas. And the feature amounts are arranged in the feature space. Next, an identification hyperplane that separates the sign of the label in the feature space is calculated. Thereafter, the determination can be performed with reference to the identification hyperplane acquired by the learning. For example, the determination of the image area at time t is performed using the tracking discriminator 114a that has performed online learning from time zero to time (t-1).

  Here, the distance d between the feature amount and the discrimination hyperplane in the feature space corresponds to a value (score) of an evaluation function F described later in detail. Next, the “object motion information” y will be described.

First, as an estimation function y = f (x),
(3) f (x) = argmax _y∈Y F (x, y)
Is adopted. Thus, when the image x is given, the estimation function f outputs y. Here, F (x, y) is the above-described evaluation function, and a specific form in the present embodiment is shown later in Expression (6).

In the present embodiment, an image at time t upon the x ^t, the object motion information y ^t at this time t,
^{(4) y t = (Δp} x t, Δp y t, Δh t, Δa t)
Is defined. In the above equation (4), the parameter Delta] p _x ^t is the change in position from the previous time (t-1) in G _x-axis direction of the global coordinate system in the tracking target object, the parameter Delta] p _y ^t is tracked This is a change in position of the target object in the world _y- axis direction of the world coordinate system from the previous time (t−1). The parameter Delta] h ^t is (at G _z-axis direction of the global coordinate system) in the tracked object is a change in the height. Furthermore, the parameter .DELTA.a ^t is the change from the previous time (t-1) at an angle of inclination of the tracking target object. The angle of the tilt corresponds to, for example, the tilt angle when bowing in the case of a person, and is an angle in a plane including the _Gz axis in the world coordinate system.

Incidentally, for the parameter .DELTA.a ^t, it may be used the values of the other parameters as well the real space (world coordinate system), but in the following embodiments, and those using the values in the image (image coordinate system) I do. That, .DELTA.a ^t is a change in the angle value when the tracking target object projected on the image coordinate system. Thus, by adopting the values in the image as .DELTA.a ^t (image coordinate system), it is possible to consider the angular variation in one dimension, extremes number of candidates during the estimation of the object motion information y ^t Can be avoided, and the amount of calculation can be suppressed. The parameter Delta] h ^t also, as with .DELTA.a ^t, can be tracked object is the height variation of the on the image when projected on the image coordinate system.

  Next, the relationship between these parameters (object motion information) and the corresponding image area in the image coordinate system will be described.

  FIG. 6 is a schematic diagram for explaining an embodiment in which a tracking target object is projected on an image coordinate system.

A candidate object motion information y ^t is a parameter vectors, which are candidates at a certain time t, and the acquired image x ^t, having a relationship as shown in FIG. Here, the preceding time (t-1) determined in (output) is the object of the optimal solution of the motion information ^{y t-1 * = (Δp} x t-1 *, Δp y t-1 *, Δh t-1 * , Δa ^{t-1 *} ).

As shown in FIG. 6, it is assumed that a three-dimensional object model corresponding to an object position to be tracked first exists at an object position that is a ground contact position with the floor or the ground. This object model represents a predetermined standard approximate shape of an object, and is a set of points in a three-dimensional space representing the model surface. The initial height (at time zero) of this object model is h ₀ , and the height h ^{t-1 *} of this object at time (t−1) is
(5) h ^{t-1 *} = h ₀ + ΣΔh ^{k *}
Becomes Here, Σ is a sum of k from 1 to t−2. The height h ^t of the object model at time t has a h ^{t-1 *} from Delta] h ^t only changed values ^{(h t-1 * -Δh t} ).

In the present embodiment, the upper part of the object model, which is within a range of the length αh ₀ from the upper end, is projected onto the image coordinate system. α is a predetermined positive constant (within the range of (0, 1)) equal to or less than 1. Here, projecting the object model (portion) onto the image coordinate system means that the object model (portion) is Further, the point set corresponding to the surface is transformed into the image coordinate system, and the image area resulting from projecting the object model (part) onto the image coordinate system is defined as a point set on the surface of (part of) the object model A region in an image that is surrounded by a corresponding transformed point set on the screen.

That is, tracing discriminator 114a (FIG. 3) is, as the image area according to the learning and tracking the target object when the judgment, position to which a predetermined ratio α below the height h ₀ from the upper end of the object in the real space The image area calculated by performing coordinate conversion on the object portion is adopted. In general, in a captured image, a place where an object is in contact with the floor or the ground is often hidden behind another object such as a desk, a table, a person, or a car. However, according to the present embodiment, since the top of the tracking target object is tracked even in a situation where the ground contact position is hidden and cannot be seen, it is possible to continue to recognize the position and height of the object.

Here, the image area at time (t-1) corresponding to the upper projected into the image coordinate system in the object model, the area is rotated on the image just .DELTA.a ^t mainly a certain reference point, or less, x ^t | _yt . That, x ^t | _yt is the object motion information in the image x ^t is _{^{y t = (Δp x t,}} Δp y t, Δh t, Δa t) If it is, corresponds to the upper part of the object model appearing in the image This is an image area.

Next, a search method for generating a plurality of pieces of candidate object motion information and determining an optimal solution y ^{t *} at time t will be described.

  FIG. 7 is a schematic diagram illustrating a relationship between an element related to a change in the real space in the object motion information and the object model.

Delta] p _x ^t in the object motion information y ^t, Delta] p _y ^t and Delta] h ^t, as already described, it is a change in object position and height between the previous time (t-1) to time t, as shown in FIG. 7, each variation in G _x-axis direction on the floor or ground, the variation in G _y-axis direction on the floor or ground, corresponding to the variation in G _z-axis direction.

FIG. 8 is a schematic diagram showing an embodiment of obtaining candidate object motion information by sampling a position in a real space. Incidentally, among a plurality of candidate object motion information shown below (FIGS. 8 to 10), one that satisfies the above equation (3) f (x) = argmax _y∈Y F (x, y) is the optimal solution (correct answer). It becomes.

According to FIG. 8, a plurality of different candidate object motion information y ^t position variation Delta] p _x ^t and Delta] p _y ^t in the real space have been acquired by the circular grid sampling.

Specifically, the set of Delta] p _x ^t and Delta] p _y ^t is the circular grid built around a position on the floor or ground at the previous time (t-1), corresponding to the lattice point is within a predetermined range A plurality of values are determined. For example, when the radius r is 3, 4, or 5 (predetermined unit) and the azimuth angle θ takes a value from 0 to 350 ° in increments of 10 °, it corresponds to a circular grid lattice point (r, θ). value pairs ^{_{(Δp x t, Δp y t}} ) may be the candidate object motion information y ^t with. Incidentally As modifications, the position change of the real space in a candidate object motion information y ^t in polar coordinates, i.e. may be displayed as [Delta] r ^t and [Delta] [theta] ^t. Regarding the value of the range of the radius r, the moving speed of the object model at the previous time (t−1) is calculated, and a value range that can be the time t is set based on this value. It is also preferable to determine it. For example, if the moving speed is zero, the radius r takes a set of values starting from zero.

As described above, in the human model determined as the candidate position change, only the upper αh ₀ portion is projected on the image coordinate system as shown in FIG. Such projected plurality of image areas in becomes the plurality of candidate areas in the image x ^t.

  FIG. 9 is a schematic diagram illustrating an embodiment of obtaining candidate object motion information by sampling the height in the real space.

According to FIG. 9, a plurality of different candidate object motion information y ^t of the height variation Delta] h ^t in the real space is acquired.

Specifically, Delta] h ^t is before the time (t-1) is the height change from the height h ^{t-1 *} in, taking a plurality of candidate values as the height variation of the multiple variations. For example, the fixed change Δh may be set in advance, and a plus / minus coefficient times Δh may be set as a plurality of candidates for the height change. At this time, the coefficient value is also changed within a predetermined range. As shown in FIG. 9, only the upper αh ₀ portion of the human model determined as the candidate height change is projected onto the image coordinate system. Such projected plurality of image areas in becomes the plurality of candidate areas in the image x ^t.

  FIG. 10 is a schematic diagram illustrating an embodiment of acquiring candidate object motion information by sampling the inclination in the image coordinate system.

According to FIG. 10, the upper portion of the image area x ^t of the object model projected on the image coordinate system | About _yt, different plurality of candidate object motion information y ^t of the slope of the variation .DELTA.a ^t are acquired. By considering such candidate object motion information, for example, when the tracking target object is a person, tracking can be performed in response to a shape change such as tilting the body from the waist.

Specifically, .DELTA.a ^t is the image region x ^t at the previous time (t-1) | is the slope change from the orientation of _yt, taking a plurality of candidate values as the slope variation of the multiple variations. For example, the fixed change Δa can be set in advance, and a plus or minus coefficient times Δa can be set as a plurality of candidates for the slope change. At this time, the coefficient value is also changed within a predetermined range. It should be noted that only the upper αh ₀ part of the human model determined as the candidate inclination change is projected on the image coordinate system. Such projected plurality of image areas in becomes the plurality of candidate areas in the image x ^t.

The generation of the candidate object motion information y ^t has been described above with reference to FIGS. 8 to _{10. The} optimal solution y ^{t *} that satisfies the above equation (3) f (x) = argmax _y∈Y F (x, y) in the search for obtaining, variation Delta] p _x ^t as described above, Delta] p _y ^t, Delta] h ^t and Δa value of the evaluation function F with respect to y (∈Y) are all combined in a candidate value of ^t, i.e. calculate the scores Then, y that derives the largest one of the calculated scores is set as the optimal solution yt ^* .

In this score calculation, the calculation cost increases as the number of candidates increases. Therefore, it is preferable to reduce the calculation cost by setting a predetermined premise in advance to limit the number of combinations of changes. For example, if it can be assumed that the shape change of the person is due to sitting from the situation in the observation target space, the height change due to seating and The setting of the inclination change candidate can be limited to the case where there is no change in the position on the floor or the ground. That is, in this ^{_{case, (Δp x t, Δp y}} t) only candidate, since setting a plurality of candidate values on Delta] h ^t and .DELTA.a ^t, it is possible to reduce the total number of candidate y (∈Y).

  Hereinafter, tracking of the tracking target object using the evaluation function F will be described.

Returning to FIG. 3, the tracking discriminator 114a of the object tracking unit 114 is, as one embodiment,
(A) the term of the probability density function P _p for a variation Delta] p _x ^t and Delta] p _y ^t position in the real space of the tracking target object as a variable,
(B) a term for evaluating the closeness of the appearance (appearance) of the image area x ^t | _yt calculated from the candidate object motion information with respect to the image area relating to the tracking target object; candidate object motion information ^{_{(Δp x t, Δp y t}} ) and applies the image information relating to an image at time t, evaluating candidate object motion information that maximizes the score function F, the real of the object at time t it may be output as position information of the correct Delta] p _x ^{t *} and Delta] p _y ^{t *} according to the position in space.

Further, the correct answer as a modification, the candidate object motion information that applies to the evaluation function F as _{^{(Δp x t, Δp y t}} , Δa t), tracing identifier 114a is, according to the inclination of the tracking target object at time t It is also preferable to output information including the information Δat ^* .

Further, as another embodiment, the tracking discriminator 114a includes the above item (a), the above item (b), and
(C) with respect to the evaluation function F and a term relating to the probability density function P _h to the variable height of the change in Delta] h ^t of the tracking target object, a plurality of candidate object motion information input (Δp _x ^t, Δp _y ^t , Δh ^t ) and the image information of the image at time t, applying the candidate object motion information that maximizes the score of the evaluation function F to the position and height of the object in the real space at time t. according correct answer information Δp _x ^{t *,} may be output as Delta] p _y ^{t *} and Delta] h ^{t *.}

Further, as a modification, the candidate object motion information that applies to the evaluation function ^{_{F (Δp x t, Δp y}} t, Δh t, Δa t) and, tracing identifier 114a is, inclination of the tracking target object at time t It is also preferable to output information including the correct information Δat ^{* according} to the above.

Further, as another embodiment, the tracking discriminator 114a includes the above item (a), the above item (b), the above item (c), and
(D) For an evaluation function F having a term that evaluates the degree to which the change due to the motion of the object in the image area of the object to be tracked matches the amount of change related to the object motion information, candidate object motion information _{^{(Δp x t, Δp y t}} , Δh t) and applies the image information relating to an image at time t, the candidate object motion information that maximizes the score of the evaluation function F, the at time t ^* object in the real space at the position and height information of correct answers according to the s Δp _x ^t, may be output as Delta] p _y ^{t *} and Delta] h ^{t *.}

Further, as a modification, the candidate object motion information that applies to the evaluation function ^{_{F (Δp x t, Δp y}} t, Δh t, Δa t) and, tracing identifier 114a is, inclination of the tracking target object at time t It is also preferable to output information including the correct information Δat ^{* according} to the above.

Here, in the embodiment described below, as the evaluation function F, the following equation (6) including all of the terms (a) to (d) is given by F (x ^t , y ^t ) = w _p P _p ( Δp ^{t-1 *} , Δp _x ^t , Δp _y ^t ) + w _h P _h (Δh ^{t-1 *} , Δh ^t )
+ W _b Φ (x ^t-1 | _yt-1 , x ^t | _yt ) + w _s Ψ (x ^t | _yt )
Is adopted. The coefficients w _p , w _h , w _b, and w _s are weight parameters determined by learning. As this function value (score) is large, y ^t is the more suitable solutions (solutions closer to the correct answer). Next, the terms on the right side of the above equation (6) will be sequentially described.

FIG. 11 is a graph showing an example of the probability density function P _p for the position change in the evaluation function F.

P _p in the first term of the evaluation function F of the equation ^{(6) (Δp t-1} *, Δp x t, Δp y t) , as shown in FIG. 11, the position at the previous time (t-1) the variation ^{Δp t-1 * = (Δp} x t-1 *, Δp y t-1 *) is the probability density function for the position change amount Delta] p in is calculated from the time t _x ^t and Delta] p _y ^t. Specifically, a probability density function P _p is defined as a bivariate normal distribution N (Δp _x ^{t-1 *} , Σ) using Δp _x ^{t-1 *} as an average using a predetermined variance-covariance matrix ^-1. can do.

By adopting the probability density function P _p based on such a predetermined probability model as the evaluation function F, the time at the time t at which the probability that can occur is sufficiently high when viewed from the movement amount at the previous time (t−1) is considered. It is possible to estimate the moving amount.

Figure 12 is a graph showing an example of a probability density function P _h height variation in the evaluation function F.

^{_{P h (Δh t-1 *}} , Δh t) in the second term of the evaluation function F of the equation (6), as shown in FIG. 12, the previous time (t-1) at the height of the change in Delta] h it is the probability density function for the height variation Delta] h ^t at ^{t-1 *} is calculated from the time t. Specifically, it is possible to define a probability density function P _h as using a dispersion sigma predetermined Δh ^{t-1 *} Average and the bivariate normal distribution ^{N (Δh t-1 *,} σ).

By adopting a probability density function P _h based on such a predetermined probability model in the evaluation function F, the previous time as viewed from the height variation in the (t-1), possible sufficiently high time t probability It is possible to estimate the amount of change in height at.

Next, [Phi in the third term of the evaluation function F of the equation ^{(6) (x t-1} | yt-1, x t | yt) is defined by the parameter Δp _x ^t, Δp _y ^t and Delta] h ^t the three-dimensional object model image region x ^t is rotated by .DELTA.a ^t the region as a result projected on the image coordinate system | to _yt, a function to evaluate the movement in the image by using a differential image The function to evaluate.

Here, the difference image is an image composed of points having a brightness value corresponding to the difference in luminance value between the corresponding point on the image x ^t-1 on the point and the image x ^t. Point in the image x ^t at time t (u, v) the luminance value x ^t (u, v) and when the brightness of each point in the difference image of the image x ^t-1 and the image x ^t (u, v) The value x _bg ^{t-1, t} is given by the following equation (7): x _bg ^{t-1, t} (u, v) = | x ^t-1 (u, v) −x ^t (u, v) |
Is defined by Since the luminance value of an image is often defined to be in the range of 0 to 255 ([0,255]), dividing the luminance value of each point in the difference image by 255 It may be a value normalized to the range ([0,1]) from to.

  FIG. 13 is an image diagram showing one embodiment of the difference image. As shown in FIG. 13, in the difference image, a luminance distribution reflecting the movement of the object on the acquired image is observed.

で定義される。式（８）においては、画像領域Aから画像領域Bでもある点を除いた領域をA−Bとし、領域Cの面積（ピクセル数）を|C|としている。Φ(x^t-1|_yt-1, x^t|_yt)は、領域x^t-1|_yt-1であって領域x^t|_ytではない面積|x^t-1|_yt-1−x^t|_yt|がゼロでない場合、この面積部分の合計輝度値をこの面積（ピクセル数）で割り算した値、即ち差分画像における平均輝度値をとる。ここで、一般に、輝度値の差分は動きのあった領域で大きくなることから、領域x^t|_ytが実際に時刻tにおける物体相当の画像領域に近いほど、関数Φの値は大きくなる。その結果、関数Φによって、画像内での物体の移動に対し実空間での物体の移動が合致している度合いを評価することができる。 Φ (x ^t-1 | _yt-1 , x ^t | _yt ) related to such a difference image is expressed by the following equation.

Is defined by In equation (8), the area excluding the point that is also the image area B from the image area A is defined as AB, and the area (number of pixels) of the area C is defined as | C |. _{^{Φ (x t-1 | yt}} -1, x t | yt) , the region x ^t-1 | area a _yt-1 x ^t | not _yt area _{^{| x t-1 | yt-}} 1 -x t If | _yt | is not zero, the value obtained by dividing the total luminance value of this area by this area (the number of pixels), that is, the average luminance value in the difference image, is taken. Here, in general, since the difference between the luminance value becomes larger by made the motion region, the region x ^t | closer to the image area of the object corresponding in _yt actually time t, the value of the function Φ increases. As a result, the degree to which the movement of the object in the real space matches the movement of the object in the image can be evaluated by the function Φ.

Finally, [psi in the fourth term of the evaluation function F of the equation _{^{(6) (x t | yt}} ) , the parameter Δp _x ^t, Δp _y ^t and the image coordinate system a three-dimensional object model defined by Delta] h ^t to _yt, a function for evaluating the image area x ^t for the image area of the tracking target object | | the image area x ^t which is rotated by .DELTA.a ^t the region as a result of the projection of _yt apparent (appearance) It is a function that evaluates closeness.

Image region x ^t | A model of apparent calculated from _yt, can be used, for example color histograms or Haar-Like features in the region. At this time, the appearance in the region is converted into a feature vector, and the closeness is evaluated. For Haar-Like features, for example, Non-Patent Documents Viola, P and Jones, M, `` Rapid object detection using a boosted cascade of simple features '', proceedings of IEEE Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 511-518, 2001.

14, the image region x ^t | is a schematic diagram showing an embodiment of a feature vector of the apparent _yt.

According to FIG 14, the image region x ^t | to _yt, luminance histogram for pixels of the region is generated. The luminance histogram generated in the present embodiment is obtained by dividing the range of luminance values 0 to 255 into a plurality of sections, and indicating the number (frequency) of pixels having luminance values belonging to each section in a columnar graph. FIG. 14 shows an example in which the luminance range is divided into six sections.

Here, the feature vector is represented by a vector having the frequency (pixel number) of each luminance section as a component. In the example of FIG. 14, a six-dimensional feature amount vector is generated. Note that, naturally, the contents and dimensions of the feature vector are not limited to this example. Image region x ^t | if the amount representing the feature of _yt, it is possible to employ various ones as components of a feature vector.

  Next, a learning process in the tracking discriminator 114a of the object tracking unit 114 will be described.

Returning to FIG. 3, the tracking discriminator 114a of the object tracking unit 114 learns online using the teacher data set generated by the teacher data set generation unit 114b. Specifically, tracing identifier 114a is, when the detection time is zero, at time t acquired image x ^t of the object tracked, earlier time 1, in ... and t-1 The learning is performed by using the correct answer data every time and the update is repeated.

Specifically, a learning approach is used for learning. Structural learning is a type of machine learning in which learning is performed on a function that outputs data having an appropriate structural relationship (dependency) from an unknown input. The tracking discriminator 114a learns the structural relationship between the position / height information in the real space and the image area when the object is projected into the image, for the tracking target object. In the present embodiment, the above equation (3) is structured (Structured) as an algorithm for learning the transformation F: X → Y by the evaluation function F (x, y) of f (x) = argmax _y∈Y F (x, y). 3.) Use SVM. Incidentally, for structured SVM, for example, non-patent literature Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann and Yasemin Altun, "Large Margin Methods for Structured and Interdependent Output Variables", Journal of Machine Learning Research 6, pp. 1453-1484 , 2005.

As a training data set for learning, for example,
(A) A set ( ^xj , ^{yj *} , 1) of an image ^xj , correct object motion information ^{yj *} that maximizes the value of the evaluation function F, and 1 that is a positive label as a correct answer, and (B) A pair (x ^k , y ^k , y ^k , y ^k , y ^k , y ^k , y ^k ) of an image x ^k , non-correct object motion information y ^k that minimizes the value of the evaluation function F, and a negative label −1 as a non-correct solution -1)
Can be used. Here, the total number of sets (x ^j , y ^{j *} , 1) and sets (x ^k , y ^k , −1), that is, the number of training data sets is n, and (x ^j , y ^{j *)} ) And (x ^k , y ^k ) are represented as (x ⁱ , y ⁱ ) (i = 1, 2,..., N). In the above (b), as the object motion information y ^k is not a correct answer, it is also possible to use a y ^k which are not the maximum F value.

の形で定義される目的関数の最適化によって導出される。上式（９）においてL(y)は損失関数であって、次式

によって定義される。上式（１０）においてy^*は入力xに対する正解データである。この損失関数L(y)は、y＝y^*の場合のみゼロ値をとり、それ以外の場合、yと正解y*とのズレが大きいほど大きな正値をとるものであり、yの構造関係を反映した形となっている。 Here, the weight parameter w = (w _p , w _h , w _b , w _s ) of the evaluation function F is expressed by the following equation.

Is derived by optimizing an objective function defined in the form In the above equation (9), L (y) is a loss function.

Defined by In the above equation (10), y ^* is the correct answer data for the input x. This loss function L (y) takes a zero value only when y = y ^* , otherwise, the larger the difference between y and the correct answer y *, the larger the positive value, and the structural relationship of y It is a form that reflects.

In this way, the tracking discriminator 114a determines the weight parameter w = (w _p , w _h , w _b , w _s ) of each term of the evaluation function F by online structure learning, and determines the determined weight parameter. The input image ^xt is processed using the evaluation function F having w, and the output object motion information yt ^* is calculated. Incidentally, the weight parameter w determined by learning defines the identification hyperplane shown in FIG.

To summarize the learning and determination described above, the tracking discriminator 114a according to the present embodiment uses the data generated using the object motion information y ^{t-1 *} output as the correct answer at the previous time (t−1). learns by the set, and inputs the image x ^t obtained at time t, the image x ^t, the parameters determined by the structure learning evaluation function _{F w = (w p, w} h, w b, w s ) To output the correct object motion information y ^{t *} at time t. As a result, even when the position of the tracking target object in contact with the floor or the ground in the image cannot be specified, or even when the shape of the object changes or the height of the object changes, the actual image group is obtained using the acquired image group. The object can be tracked while maintaining a high positional accuracy in the space and continuously giving a unique identifier ID.

The object position / shape estimating unit 115 performs a predetermined time based on the object motion information y ^{t *} input from the object tracking unit 114 or the instantaneous position, height, and / or tilt information of the tracking target object in the real space. A change in position, height and / or tilt of the tracked object in real space in the range is determined. It is also preferable to determine the information, the flow line of the tracking target object, and events such as seating and bowing on the flow line, and store them in the tracking object management unit 104. In addition, such an object position / shape estimation result may be transmitted to the external information processing device via the communication control unit 121 and the communication interface 101, for example, in response to a request from the external information processing device. preferable.

  As described above in detail, according to the present invention, tracking is performed in consideration of not only image information relating to an acquired image but also “object motion information” including a restriction in a real space. As a result, the object can be tracked while maintaining high positional accuracy in the real space while using the acquired image group.

  In addition, even when the shape of the object changes or the height of the object changes by incorporating the height change and the tilt change as well as the position change of the tracking target object into the "object motion information", The object can be tracked while maintaining high position accuracy in the real space. Further, it is possible to estimate not only the instantaneous position of the object but also the instantaneous height and shape.

  In addition, the configuration and method of the present invention include, for example, a monitoring system for monitoring a place where a person moves, sits, and bends, and enters, breaks, watches, and events of a person in a shopping street or a commercial / service facility. The present invention is applicable to various systems such as a marketing research system for investigating participation and movement situations.

  In the above-described various embodiments of the present invention, various changes, modifications, and omissions of the scope of the technical idea and viewpoint of the present invention can be easily performed by those skilled in the art. The above description is merely an example and is not intended to be limiting. The invention is limited only as defined by the following claims and equivalents thereof.

Reference Signs List 1 object tracking device 101 communication interface 102 image storage unit 103 ID storage unit 104 tracking object management unit 111 object detection unit 111a detection discriminator 111b height calculation unit 112 ID management unit 112a object integration unit 112b object registration unit 113 candidate information calculation Unit 114 Object tracking unit 114a Tracking discriminator 114b Teacher data set generation unit 115 Object position / shape estimation unit 121 Communication control unit 2 Camera

Claims (11)

An apparatus capable of tracking an object to be tracked using a time-series image group obtained from one or more cameras capable of capturing the object,
As the object motion information which is information including the position information on the position of the object in the real space at one time, at least the change from the previous time in the position of the object in the real space is adopted, and the one time Candidate information calculation means for calculating a plurality of candidate object motion information that is object motion information different in the amount of change from each other,
And image information according to the acquired image, a classifier to learn the data set comprising a the object motion information that is the correct answer, the probability that a change in the position of the real space of the object a variable density For the evaluation function having a term relating to the function and a term for evaluating the closeness of the appearance (appearance) of the image area calculated from the candidate object motion information with respect to the image area relating to the object, Applying the candidate object motion information and the image information related to the image at the one time point, the candidate object motion information that maximizes the score of the evaluation function is calculated based on the position of the object in the real space at the one time point. An object tracking device, comprising: an object tracking unit configured to acquire instantaneous position information of the object in a real space by a discriminator that outputs the position information as a correct answer. The candidate information calculating means includes, as the object motion information at the one time point , a change amount of the position of the object in the real space from the previous time point and a change amount of the height of the object from the previous time point. And calculating a plurality of candidate object motion information in which at least one of the changes at the one time point is different,
The discriminator of the object tracking means includes a term relating to a probability density function having a change in the position of the object in real space as a variable, and a term relating to a probability density function having a change in the height of the object as a variable. And a term for evaluating the apparent proximity of the image area calculated from the candidate object motion information with respect to the image area of the object, the input plurality of candidate object motion information and the 1 Applying the image information relating to the image at one point in time , the candidate object motion information for maximizing the score of the evaluation function is converted to the correct answer relating to the position of the object in real space and the height of the object at the one point in time 2. The object tracking device according to claim 1 , wherein the object tracking device outputs the information. The candidate information calculating means includes, as the object motion information at the one time point , a change amount of the position of the object in the real space from the previous time point and a change amount of the height of the object from the previous time point. And calculating a plurality of candidate object motion information in which at least one of the changes at the one time point is different,
The discriminator of the object tracking means includes a term relating to a probability density function having a change in the position of the object in real space as a variable, and a term relating to a probability density function having a change in the height of the object as a variable. And a term for evaluating the degree to which a change due to the motion of the object in the image area related to the object matches a change amount related to the object motion information, and the candidate object motion information for the image area related to the object. Applying the input plurality of candidate object motion information and the image information of the image at the one point in time to an evaluation function having a term for evaluating the apparent proximity of the image area calculated from the candidate object motion information that maximizes the score of the evaluation function according to claim 1, and outputs as information correct answers according to the position and height of the object in the real space of the object in the one time or 2 Object tracking apparatus according. The candidate information calculation means further employs, as the object motion information at the one time point , a change in inclination of the object from a previous time point ,
The discriminator of the object tracking means outputs the candidate object motion information that maximizes the score of the evaluation function as information including information on the correct answer related to the tilt of the object at the one point in time . object tracking apparatus according to any one of claims 1 to 3. Detecting the object based on the acquired image, calculating a ground contact position of the object as a position in the real space of the object based on the lowest position of the detected image area of the object, and detecting The position in the real space calculated based on the uppermost position of the image area related to the object, and, based on the calculated ground contact position, further comprising an object detection unit that calculates the height of the object The object tracking device according to any one of claims 1 to 4 , wherein The discriminator of the object tracking means determines the weighting factor of each term of the evaluation function by learning, and uses the evaluation function having the determined weighting factor to process the input image information of the image, The object tracking device according to any one of claims 1 to 5 , wherein the object motion information to be output is calculated. Identifier of the object tracking means, before the time of one time point, the data set generated by using the object motion information outputted as the correct learns, image information relating to the image in the one time point type, the image information, processed using the parameters determined by the learning, either 6 from claim 1 and outputs the object motion information to be correct in the one time point 1 Item tracking device according to Item. The discriminator of the object tracking means is calculated by performing coordinate conversion on an object portion from an upper end of the object in a real space to a position below a predetermined ratio of the height of the object as an image area of the object. object tracking apparatus according to any one of claims 1 to 7, characterized in employing an image area. Identifier of the object tracking means, object tracking apparatus according to any one of claims 1 to 8, characterized in that it is constructed by the algorithm of the structured SVM (Structured Support Vector Machine). A program that causes a computer mounted on a device capable of tracking an object to be tracked using a time-series image group obtained from one or more cameras capable of capturing the object to be tracked,
As the object motion information which is information including the position information on the position of the object in the real space at one time, at least the change from the previous time in the position of the object in the real space is adopted, and the one time Candidate information calculation means for calculating a plurality of candidate object motion information that is object motion information different in the amount of change from each other,
And image information according to the acquired image, a classifier to learn the data set comprising a the object motion information that is the correct answer, the probability that a change in the position of the real space of the object a variable density For the evaluation function having a term relating to the function and a term for evaluating the closeness of the appearance (appearance) of the image area calculated from the candidate object motion information with respect to the image area relating to the object, Applying the candidate object motion information and the image information related to the image at the one time point, the candidate object motion information that maximizes the score of the evaluation function is calculated based on the position of the object in the real space at the one time point. the discriminator to output as the position information of the correct, object tracking flop for causing a computer to function as <br/> the object tracking means for acquiring every moment the position information of the real space of the object Program. A method of tracking an object to be tracked by a computer using a time-series image group obtained from one or more cameras capable of capturing the object to be tracked,
As the object motion information which is information including the position information on the position of the object in the real space at one time, at least the change from the previous time in the position of the object in the real space is adopted, and the one time Calculating a plurality of candidate object motion information, which are different object motion information from each other,
A probability classifier that learns from a data set including image information of an acquired image and the object motion information that is regarded as a correct answer, and uses a change in position of the object in real space as a variable. , And a term for evaluating the closeness of the appearance (appearance) of the image region calculated from the candidate object motion information with respect to the image region of the object, The candidate object motion information that maximizes the score of the evaluation function is obtained by applying the object motion information and the image information of the image at the one time point to the correct answer based on the position of the object in the real space at the one time point. Repeating the step of determining the position information of the object in the real space at the one point in time by the discriminator that outputs the position information of the object. An object tracking method characterized by acquiring.

Images