JP2023081582A - Information processing device, information processing method, and program - Google Patents

Abstract

To enable acquisition of a position and trajectory of a target object in a three-dimensional space.SOLUTION: An information processing device detects a prescribed target object from images photographed by each of a plurality of photographing means, and based on a detection result of the target object for the images photographed at a first point in time, for each photographing means, predicts a state of the target object at a second point in time after the first point in time. Then, the information processing device updates, for each photographing means, the predicted state of the target object based on the detection result of detection means and a prediction result of prediction means, and estimates a photographing timing for each photographing means based on the updated state of the target object and the detection result of the detection means.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing technology that can be applied when an object is imaged by a plurality of imaging devices and tracked.

There is a technique for estimating the position of a subject in a three-dimensional space based on images captured by a plurality of fixed cameras. In this technology, the identity of a subject is determined in multiple temporally consecutive images (hereinafter referred to as frames) captured by multiple synchronized cameras, and the trajectory of that subject in 3D space is estimated. .

Patent Literature 1 discloses a technique for estimating the trajectories of a plurality of persons on a plane. In the technique described in Patent Document 1, a large number of particles are arranged on a bird's-eye view plane looking down on a field such as a soccer field, and the next position is predicted from a model of a person's movement on the bird's-eye view plane. In the technique of Patent Document 1, the predicted particles are projected onto a camera frame, and the trajectories of a plurality of people on a plane are estimated by a particle filter that rearranges them in a likely foreground area of the frame. In addition, in Patent Document 2, a particle filter that treats the position and face orientation of a person in a three-dimensional space as state variables and rearranges particles using a classifier suitable for the state of the face orientation of the person. A technique for estimating the trajectory of is disclosed.

JP 2013-58132 A JP 2008-26974 A

By the way, when the object to be tracked is an object whose acceleration greatly changes and becomes high speed, such as a ball in a sports game, the tracking may fail. This also occurs in the techniques disclosed in Patent Literature 1 and Patent Literature 2 in the same way, and the tracking may fail because the position and trajectory of the object to be tracked in the three-dimensional space cannot be obtained. This is because the frame rate of multiple cameras is not sufficient to track a fast-moving object, or the imaging timing of each camera is not synchronized, resulting in the position and trajectory in 3D space. It is thought that this occurs due to the inability to obtain

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to enable acquisition of the position and trajectory of an object in a three-dimensional space.

The information processing apparatus of the present invention includes detection means for detecting a predetermined target object from images respectively captured by a plurality of imaging means, and a detection result of the target object for the image captured at a first point in time for each of the imaging means. prediction means for predicting the state of the object at a second time point after the first time point based on the detection means for detecting the predicted state of the object for each of the imaging means; estimating an imaging timing for each of the imaging means based on updating means for updating based on the result and the prediction result by the predicting means; and on the basis of the updated state of the object and the detection result of the detecting means. and estimating means.

According to the present invention, it is possible to acquire the position and trajectory of an object in a three-dimensional space.

It is a figure which shows the functional structure of the three-dimensional tracking apparatus of embodiment. FIG. 3 is a diagram showing an example of arrangement of a camera and an object and an example of an image captured by the camera; 4 is a flow chart showing the flow of information processing according to the first and second embodiments; It is a figure explaining the FOV of a detector, and the number of FOV which overlaps in space. It is a figure which shows a detector FOV, an object arrangement|positioning, a frame, and an example of a detector output. It is a figure explaining the method of assigning an observation value to an object. It is a figure explaining the imaging timing for every camera. It is a figure explaining the posterior probability distribution of the imaging timing for every camera. 10 is a flow chart showing the flow of information processing according to the third embodiment;

Hereinafter, this embodiment will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all combinations of features described in the embodiments are essential to the solution of the present invention. The configuration of the embodiment can be appropriately modified or changed according to the specifications of the device to which the present invention is applied and various conditions (use conditions, use environment, etc.). Also, a part of each embodiment described later may be appropriately combined. In each of the following embodiments, the same configurations are given the same reference numerals.

In this embodiment, the position and trajectory of an object to be tracked in a three-dimensional space are estimated based on images captured by a plurality of imaging devices (hereinafter referred to as cameras), and the object is tracked. will be described. In this embodiment, examples of tracking targets include players and balls during a sports match.

Here, if the object to be tracked is, for example, a ball whose acceleration changes greatly and becomes fast, tracking with a general frame rate camera for the purpose of obtaining images for viewing will fail in tracking. Sometimes. In the case of the techniques disclosed in Patent Document 1 and Patent Document 2 described above, the time step width of the prior distribution that predicts the three-dimensional spatial position at the next time from the three-dimensional spatial position of the tracking target at a certain time is determined by the camera is the same as the time interval of the imaged frames. That is, the frequency of the tracking result (tracking rate) is the same as the predetermined period for imaging by the camera, that is, the frame rate. For example, when an object moving at a speed of 50 km/h is imaged with a camera of 30 frames per second (hereinafter referred to as fps), a movement of about 0.5 m occurs before and after the frame in the three-dimensional space. ing. As a result, the likelihood calculated from object detection positions in later frames may deviate significantly from the prediction of the prior distribution, resulting in tracking failure. On the other hand, if a high-speed camera that captures images at 30 fps or more is used, it is possible to improve the tracking rate during tracking, but this requires an expensive system. Further, in the techniques disclosed in Patent Documents 1 and 2, when a plurality of cameras constituting an imaging system are connected to each other via a local area network (LAN), the imaging timing of each camera is not synchronized, and tracking fails. I have something to do.

Therefore, in the present embodiment, in a system in which a plurality of cameras with a general-purpose frame rate such as 30 fps are connected via a LAN or the like, an object can be tracked at a tracking rate exceeding the frame rate of each camera. It has an information processing device that That is, according to the information processing apparatus of the present embodiment, even if multiple cameras with general-purpose frame rates are not synchronized, the three-dimensional position and trajectory of the tracking target can be estimated with high accuracy. Allows tracking.

<First Embodiment>
Hereinafter, as a first embodiment, an imaging system in which a plurality of cameras are arranged around a three-dimensional space such as a stadium or a gymnasium will be described as an example. The information processing apparatus of the first embodiment estimates the imaging timings of the plurality of cameras, and enables tracking of an object in a three-dimensional space with a time resolution equal to or higher than the frame rate of the cameras.

Here, the imaging timing of the cameras means that when the time at which the shutter is released in one of the multiple cameras is set as the reference time, the shutter release of each of the other multiple cameras is performed. It is the time difference (in seconds) from the calculated time. In the present embodiment, the time refers to a point in time on a continuous time axis, and each time at which a shutter release is performed in each of a plurality of cameras is each point in time represented on the same time axis. and Also, when the frame rate of each camera is 30 fps, the difference between the largest imaging timing and the smallest imaging timing among the imaging timings of a plurality of cameras is the elapsed time of one frame (in the case of 30 fps, about 0.033 seconds) or less.

The information processing apparatus of this embodiment estimates the imaging timing of each camera based on the position of the target detected from the image captured by each camera and the state of the target in the three-dimensional space. In this embodiment, the state of the object includes at least the position and speed of the object in the three-dimensional space, and the direction of the object (the direction in which the face is facing) is important, such as the face of a person. In the case of an object that becomes , the orientation is included in the position and velocity. Then, the information processing apparatus of the present embodiment selects each camera (or selects an image of each camera) based on the estimated order of each imaging timing, performs object detection, and uses the detection result. track the position of the object in the three-dimensional space. Accordingly, in the information processing apparatus of the present embodiment, it is possible to track the object with a temporal resolution higher than the frame rate of the camera. Details of these will be described later. In the following description, an object to be tracked that is detected from an image will be referred to as an object as appropriate.

FIG. 1(a) is a diagram showing the functional configuration of the three-dimensional tracking device 200 of the first embodiment. Here, before describing the three-dimensional tracking device 200 according to the first embodiment, an example of an imaging system assumed in this embodiment will be described using FIGS. explain.

FIG. 2(a) is a diagram showing an example of a plurality of camera arrangements 100. FIG. A space 101 is a three-dimensional space in which a plurality of cameras are arranged and captured by each camera, and cameras 102 to 111 are arranged. FIG. 2A also shows an X-axis 113x, a Y-axis 114y, a Z-axis 115z of three-dimensional coordinates (world coordinates) in the space 101, and an origin 112. FIG. The plane formed by the X-axis 113x and the Z-axis 115z is the ground, the direction of the Y-axis 114y is the direction of the height from the ground, and the ground direction is positive (that is, the value of the height from the ground is negative). is performed). Each of the cameras 102 to 111 is fixed to a space wall portion (not shown) at a certain height from the ground, has a lens, an image sensor, etc., and can capture an imaging area including the ground of the space 101. is installed as In this embodiment, each of the cameras 102 to 111 is installed at a position capable of capturing an image of an object existing in the space 101 (object to be tracked). Further, it is assumed that each of the cameras 102 to 111 has undergone camera calibration using a known technique.

FIG. 2(b) is a diagram showing an example of an arrangement 120 of persons, etc. present in space at a certain time. Space 121 is the same space as space 101 in FIG. Assume that persons 122, 123, and 124 and a ball 125 exist as objects to be tracked in a space 121 illustrated in FIG. 2B.

FIG. 2C is a diagram showing an example of an image 131 of one frame 130 when the space 121 in FIG. An image 131 captured by the camera 104 includes persons 132, 133, and 134 and a ball 135. FIG. Persons 132, 133, and 134 respectively correspond to persons 122, 123, and 124 in arrangement 120 shown in FIG. 2(b). Ball 135 also corresponds to ball 125 in arrangement 120 shown in FIG. 2(b). The coordinate system of this frame 130 is a pixel coordinate system represented by an origin 138 and u-axis 136u and v-axis 137v. Note that an appropriately normalized coordinate system can also be used.

FIG. 2D is a diagram showing an example of an image 141 of one frame 140 when the space 121 in FIG. An image 141 captured by the camera 102 includes persons 142 and 143 and a ball 144 . Persons 142 and 143 correspond to persons 122 and 123 in arrangement 120 shown in FIG. 2(b), respectively. Ball 144 also corresponds to ball 125 in arrangement 120 shown in FIG. 2(b). The coordinate system of this frame 140 is a pixel coordinate system represented by an origin 145 and u-axis 146u and v-axis 147v.
Here, the example of the frame 130 in FIG. 2C taken by the camera 102 and the frame 140 in FIG. 2D taken by the camera 104 are given. The camera also performs imaging in the same manner as described above.

Next, the functional configuration of the three-dimensional tracking device 200 according to the first embodiment shown in FIG. 1(a) will be described with reference to FIG. 2(a).
The three-dimensional tracking device 200 of this embodiment has an imaging unit 220 , a processing unit 230 and a monitoring unit 240 .
The photographing unit 220 includes a plurality of moving image acquisition devices. In the case of the example of FIG. 1A, the imaging unit 220 has a total of K moving image capturing devices from the first moving image capturing unit 201 to the Kth moving image capturing unit 202 . In the example of FIG. 2(a), the cameras 102 to 111 are illustrated as K moving image acquisition devices, and in FIG. there is In the following description, the K moving image acquisition units of the first moving image acquiring unit 201 to the Kth moving image acquiring unit 202 are appropriately referred to as cameras. In other words, the camera described in the following description represents one of the moving image acquiring units 201 to 202. FIG. It is assumed that the imaging unit 220 is connected to the processing unit 230 via a communication path such as a local area network.

The processing unit 230 is an application example of the information processing apparatus of this embodiment. Details of each function of the processing unit 230 will be described below with reference to FIG. 3 and the like.
FIG. 3A is a flowchart showing the flow of information processing in the processing section 230 of the three-dimensional tracking device 200. FIG. First, an overview of the overall processing in the three-dimensional tracking device 200 will be described with reference to this flowchart.
In step S101, the initial value setting unit 205 sets the number of tracking target objects (person, ball) in the three-dimensional space, the ID (identification information) for identifying each object, and the state of the target object (object). Set the initial value, respectively. Further, the initial value setting unit 205 sets an initial value of an estimated value of the imaging timing of each of the K cameras of the first moving image acquisition unit 201 to the Kth moving image acquisition unit 202 of the imaging unit 220, which will be described later.

In loops L101 to L111, the processing unit 230 repeatedly performs time-related processing. Specifically, the processing unit 230 executes a repetitive process of giving an index t relating to time in order from 1 to T. FIG.
In loops L102 to L112, the processing unit 230 repeatedly executes processing related to the first moving image acquiring unit 201 to the Kth moving image acquiring unit 202. FIG. Specifically, the processing unit 230 executes a repetitive process of giving an index k related to the camera in order from 1 to K.

In step S102, the moving image acquisition selection unit 203 selects a camera from the first moving image acquisition unit 201 to the K-th camera of the imaging unit 220 based on the estimated value of the current imaging timing held by the timing estimation unit 210, which will be described later. Select from the moving image acquisition unit 202 . Specifically, the moving image acquisition selection unit 203 first selects the camera with the earliest imaging timing, and then sequentially selects cameras in order of earliest imaging timing. In other words, based on the estimated value of the current imaging timing, the moving image acquisition selection unit 203 first selects the image acquired by the camera with the earliest imaging timing among the images acquired by each of the K cameras. , and thereafter, are sequentially selected in order of earliest imaging timing.

Next, in step S103, the processing unit 230 captures the current frame (still image) captured by the moving image acquisition unit (hereinafter referred to as camera k) selected in the previous step among the moving image acquisition units included in the imaging unit 220. Get 1 The current frame image captured by the camera k is input to the detection unit 204 .

In step S104, the detection unit 204 detects an object appearing in the frame acquired in the previous step. The detection unit 204 detects each object when a plurality of objects are captured in the frame. Then, the detection unit 204 detects the position of each of the plurality of objects detected from within the frame and the score regarding the likelihood of each of the plurality of objects, and converts them to the actual detected objects from the frame captured by the camera. be the observed value of . In the case of this embodiment, the objects are a plurality of people and a ball as described above, so the detection unit 204 acquires their positions and scores as actual observed values for each object. Details of each position of the object (person and ball) and the score will be described later. Note that the ID association unit 207 assigns identification information for individually identifying each object to the observed value of each object detected by the detection unit 204 .

Next, in loops L103 to L113, the processing unit 230 repeats processing for objects. That is, the processing unit 230 executes a repetitive process of giving an index n for the object in order from 1 to N. Note that N in index n uses the value acquired as the initial value of the number of objects in step S101. In the soccer match exemplified in this embodiment, the objects are a plurality of persons and a ball, and an index n is given in order from 1 to N for each object of the plurality of persons and the ball. An object given an index n is hereinafter referred to as an object n.

Next, in step S105, when the current time is t (t≧1), the prediction unit 206 determines the state of the object n acquired at the past time t−1 (the state of the object n ), the process of predicting the probability distribution of the state of the object n at time t is performed. For example, when the time t-1 is the first time point and the time t after that is the second time point, the prediction unit 206 in step S105 determines the object obtained in the process at the first time point. Based on the state, the state of the object at the second time is predicted (probability distribution is predicted). Details of the processing here will be described later.

Next, in branch B101, the processing unit 230 determines that the object n is within the FOV of the camera k based on the state (position, speed, etc.) of the object n predicted in the previous step and information on the FOV overlapping map, which will be described later. The process branches depending on whether there is a probability of existence. If there is a probability that object n exists within the FOV of camera k, processing unit 230 advances the process to step S106, and if object n does not exist, advances the process to step S108.

Proceeding to step S106, the prediction unit 206 acquires the Predict the probability distribution of the observations that should be made. That is, when time t-1 is the first time point and time t is the second time point, in step S106 the prediction unit 206 calculates the second time point based on the state of the object obtained in the process at the first time point. Predict the probability distribution of observations that should be taken at . The details of the processing here will be described later.

Next, in step S107, the ID associating unit 207 combines the predicted probability distribution of observed values of object n with camera k at time t obtained as the prediction result in the previous step with the actual observed values of camera k obtained in step S104. Based on , we compute the likelihood of the actual observed value. Based on the likelihood of the observed value, the ID associating unit 207 associates the object n in the three-dimensional space with the observed value. Note that the correspondence between the object n in the three-dimensional space and the observed value is specifically the correspondence between the identification information given to the object n and the observed value. In this way, the detection result by the detection unit 204 and the prediction result by the prediction unit 206 are sent to the update unit 208 after being associated by the ID association unit 207 .

After proceeding to step S108, the updating unit 208 updates the state of the object n at time t using the observed values associated with the object n in the processing of steps S106 and S107. That is, when time t−1 is the first time point and time t is the second time point, the updating unit 208 calculates the object n update the state of That is, the updated state of object n corresponds to the state of object n at the third point in time. Details of the processing here will be described later.

Next, in step S109, the visualization unit 209 executes processing for visualizing the state of the object n updated in step S108. The result of this visualization processing represents the tracking status of the object, which is sent to the monitoring section 240 and displayed.
Next, in step S110, the timing estimation unit 210 estimates the imaging timing of each of the K cameras based on the state of the object n updated in step S108 and the detection result by the detection unit 204 in step S104. do. The detection result by the detection unit 204 is sent to the timing estimation unit 210 via the updating unit 208 after being associated by the ID associating unit 207 . Details of these processes will be described later.

Next, according to the flowchart shown in FIG. 3(a), more detailed and specific contents of the information processing in the processing unit 230 of the three-dimensional tracking device 200 shown in FIG. 1(a) will be described.
In this embodiment, the position, velocity, and orientation of the object to be tracked in the three-dimensional space are treated as unobservable hidden variables, and the position and score of the object detected from the image of the frame captured by the camera are treated as observed values. do. In this embodiment, the hidden variables (the position, velocity, and orientation of the object in the three-dimensional space) are estimated based on the observed values (the position and score of the object) detected from the image captured by the camera. using the framework of the state-space model. In the framework of this state-space model, the hidden variables are random variables called state variables. In this embodiment, a Gaussian state space model is used in which state variables are described only by means (first moment) and variance-covariance matrices (second moment).

In step S101, the initial value setting unit 205 sets the initial values of the estimated imaging timings of the K cameras, the number of objects (persons, balls) to be tracked in the three-dimensional space, IDs (identification information), and states. Sets the initial value of each variable. Here, the initial value of the estimated value of the imaging timing need not be a strict value and may be an arbitrary value, such as a random value. Here, however, the difference between the largest and smallest estimated values of the imaging timing of each camera shall be equal to or less than the elapsed time for one frame (0.033 seconds at 30 fps).

On the other hand, the initial values for the number of objects to be tracked, the IDs, and the state variables are different depending on whether the object is the only object in the three-dimensional space or there are multiple objects. If the object is the only one in the three-dimensional space, the initial value does not need to be strict, and an appropriate initial value may be set. This is because if there is only one object in the space, even if the initial value is an appropriate value, it can be expected to converge near the true value as the state space model is repeatedly updated. In this case, the first moment of the state variables should be an arbitrary value, and the second moment should be a sufficiently large positive semidefinite matrix. Since the ID is identification information for individually identifying each object, it is assumed that a natural number of 1 or more is assigned that does not overlap with objects of other types. A concrete example of a unique object in a three-dimensional space is a ball in a soccer game.

In the case of multiple objects existing in the three-dimensional space, the initial value of the number of objects must be set accurately. It is also necessary to estimate the position relatively accurately and use it as an initial value. When tracking multiple objects, it is necessary to solve an assignment problem that associates the position of each object at the next time predicted from the current position of each object with the position of the object detected from the image of each camera (this will be described in detail in step S107). Therefore, as the initial value of the position of each object, it is necessary to set the position with such accuracy that correct assignment can be made as a result of solving the assignment problem. A specific example of a plurality of objects existing in a three-dimensional space is a person such as a player in a soccer match.

Here, for example, at the start of a soccer match, it is considered that there are not many overlapping persons. In such a scene, by determining the initial values by the following method, it is possible to set the initial values with sufficient accuracy for the number, ID, and position of each person.

When setting the initial values for each of these objects, first, the height of the person's face is determined appropriately, such as 1.75 m from the ground, and the frame at that time is acquired from each of the K cameras. Then, each frame is subjected to face detection processing in step S104, which will be described later, and the detection position (u, v) on the pixel coordinates is projected onto the three-dimensional space using the calibration parameters of each camera. Get the position of At this time, the value on the Y axis is fixed at -1.75 m, and the values on the X and Z axes are estimated from (u, v). The reason why the value on the Y-axis is a negative value such as -1.75 m is that the ground direction is positive on the Y-axis as described with reference to FIG. 2(a).

Furthermore, the detection points on the three-dimensional space projected from each camera are clustered, the cluster of detection points is regarded as an object, an ID is given according to the number, and the position is calculated as the position of the object on the three-dimensional space. be the initial value of In setting the initial values for the objects, the ID obtained in this way is used as the ID of each object, and the position in the three-dimensional space is used as the value of the first moment of the state variable corresponding to the position of each object. . Note that the state variables of the state space model in this embodiment have components other than the position, but they may be set to zero. Also, the second-order moment may be a positive semidefinite matrix of an appropriate size. Details of the state variables will be described later.

Next, in loops L101 to L111, the processing unit 230 executes iterative processing relating to time. Also, in loops L102 to L112, the processing unit 230 repeatedly performs camera-related processing. A detailed description of these iterative processes is omitted.

Next, in step S102, the moving image acquisition selection unit 203 executes camera selection processing (movie acquisition unit selection processing) based on the estimated value of the imaging timing. In the present embodiment, the estimated value of the imaging timing is the difference (in seconds) between the imaging times of the reference camera and each of the other cameras, as described above. Also, the estimated value of the imaging timing is estimated and updated in step S110 described later, corresponding to each of the K moving image acquisition units in FIG. For this reason, the moving image acquisition selection unit 203 repeats loops L102 to L112 for each camera based on the current estimated value of the imaging timing of each camera, thereby selecting the cameras in order of the estimated value of the imaging timing. will have to choose. That is, in loops L102 to L112, the moving image acquisition selection unit 203 selects the camera having the smallest estimated value of the imaging timing when index k=1, and selects the camera having the largest estimated value of imaging timing when index k=K. to select. Details such as the estimation method for obtaining the estimated value of the imaging timing will be described later in step S110.

In the next step S<b>103 , the processing unit 230 acquires the frame at the current time from the camera k selected in the previous step, and inputs the frame image at the current time acquired from the camera k to the detection unit 204 . Here, in the case of this embodiment, each camera is assumed to have a resolution of so-called Full HD (1920×1080 pixels) and a speed of 30 frames per second. Note that the camera specifications are not limited to Full HD and the speed of 30 frames per second.

Here, the shutter timings of all the K cameras do not need to be synchronized by electrical signals such as trigger pulses and synchronization signals, and images are taken in an autonomous cycle by the clock of the microcontroller inside the cameras. . Also, the imaging unit 220 and the processing unit 230 are connected via a communication path such as a local area network. For this reason, the frames acquired and transmitted by the imaging unit 220 and received by the processing unit 230 may experience delays and dropped frames due to the performance of relay units such as switching hubs on the network path and bandwidth limitations. there is a possibility. That is, in the present embodiment, the frames acquired by the processing unit 230 may be frames in which asynchronization, dropped frames, or the like may occur. Frames captured by the imaging unit 220 may be acquired online using a communication path such as a local area network, or may be stored in an external storage device or the like and acquired therefrom.

Next, in step S104, the detection unit 204 performs processing for detecting an object from the image of the frame acquired in the previous step. In this embodiment, objects to be detected are a person and a ball. Here, when object tracking is performed as in this embodiment, it is desirable that a predetermined position such as the center of gravity of the object can be detected regardless of the position and orientation of the object, and that the detection position does not shift due to changes in appearance. Therefore, when a known object detector is used, the bounding rectangle of the object in the frame changes greatly due to changes in the position and orientation of the object. is desirable to be tracked. Therefore, a spherical ball is considered to be an ideally shaped object. In the case of a person, it is desirable to treat the head, which is nearly spherical, as the object to be detected. By handling an object that is close to a sphere, it is possible to reduce the displacement of the detection position of the object detected and output by the detection unit 204 in this step. If there is a detector that can detect the position of the object within the frame without deviation regardless of changes in the appearance or shape of the object, it may be used.

Further, in recent years, image recognition technology using convolutional neural networks (CNN) has been developed, and technology for simultaneously executing a plurality of image recognition processes at high speed and with high accuracy is being researched. Reference 1 discloses a technique that incorporates a layer for estimating object candidate regions into a CNN for object recognition, and realizes recognition processing for two tasks, object candidate region detection and multiple category classification, at an operating speed of 150 fps. It is In this embodiment, a high-speed object detector realized by such a CNN can be used as the detector 204 .

Reference 1: Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection", CVPR2016

This detector also outputs the position and score of the object. Here, the position is the position (u, v) on the pixel coordinates of the image. Also, the score (hereafter, the score is appropriately represented by q) is a normalized value of [0, 1] representing the likelihood of an object (in this embodiment, the ball or the head of a person), that is, the likelihood be. The closer the score is to 1, the more likely the detected position (u, v) is for that object. That is, the output of the detection unit 204 is represented by (u, v, q), which is called an observed value in this embodiment. The detection unit 204 may output a bounding box that matches the size of the object on the pixel coordinates as in reference 1, but in this embodiment, the center position of the bounding box is used as (u, v). .

Further, generally, a detector for detecting an object from an image has a limit to the size that can detect object-likeness. Therefore, even in the detection unit 204, there is a limit to the detection range in which the target object can be detected with respect to the image captured by the camera.
FIG. 4A is a diagram for explaining FOV (Feild of View) determined by the angle of view of the camera and the detection range of the detector. To simplify the drawing, in the example of FIG. 4, the plane including the optical center of the camera 401 represents the FOV. In FIG. 4A, the range (shaded area in the figure) where the angle of view 402 of the camera 401 and the detection range 403 of the detector intersect is the FOV 404 of the detector. That is, the FOV 404 forms a sector with a specific detection range and angle.

FIG. 4(b) is a diagram showing an FOV overlapping map in which the detector FOV is superimposed on the space 101 illustrated in FIG. 2(a). Each camera 102-111 corresponds to each camera 102-111 in FIG. 2(a). A fan-shaped FOV 411 is the FOV of the detector using the frame of the camera 109, the position 412 is the position of the point a in the three-dimensional space, and the position 413 is the position of the point b. In the FOV overlap map of FIG. 4B, a plurality of fan-shaped FOVs by a plurality of cameras are overlapped, and the number of overlaps by these FOVs is indicated by shading. At point a, the detector FOVs of the three cameras, cameras 109, 110 and 108, overlap. Also at point b, the detector FOVs of the five cameras 109, 110, 108, 102 and 106 overlap. In the example shown in FIG. 2A, since the camera arrangement is sparse, there are areas where the number of overlapping FOVs is small. there is By using such an FOV overlap map, the number of observations (the number of objects photographed) at each camera can be obtained when the placement of the objects is obtained.

FIG. 5 is a diagram showing an example of observations obtained by one camera 506 frame and detector.
FIG. 5(a) shows the positional relationship between the placement of a person and the FOV. A space 501 is a three-dimensional space in which a person captured by a camera is arranged. In FIG. 5(a), persons 502, 503 and 504 and a ball 505 are shown. FOV 507 is the detector FOV when the detector is applied to the frame acquired by camera 506 . In the example of this figure, people 502, 503 and ball 505 are within the area of detector FOV 507. FIG.

FIG. 5(b) is a visualization of the image of frame 511 acquired by the camera 506 of FIG. 5(a) and a portion of the detector output. In FIG. 5(b), persons 512, 513 and ball 514 are shown, with person 512 corresponding to person 502 shown in FIG. 5(a) and similarly person 513 shown in FIG. and the ball 514 corresponds to the ball 505 in FIG. 5(a). In addition, FIG. 5(b) shows a portion of the output of the detector in a visualized manner, and bounding boxes 515, 516, 517, and 518 are drawn as examples of the visualization, respectively.

The output of the detector used in this embodiment is the center (u,v) and width (h,w) of the bounding box and the score q, where the rectangles 515, 516, 517, 518 are (u,v,h, w) is the result of drawing the bounding box. Bounding boxes 515, 516, and 517 correspond to person 512, person 513, and ball 514, respectively. However, erroneous detection output may be obtained at the output of the detector. For example, there are cases where an object to be detected exists but is not detected, or a detection output is obtained even though an object to be detected does not exist. In this embodiment, the case where an object to be detected exists but is not detected is called "undetected", and the case where an object to be detected does not exist but is detected is called "overdetection". FIG. 5B shows a bounding box 518 as an example of a case where an object to be detected is over-detected as existing even though it does not exist. Also, as described above, in this embodiment, the detector outputs (u, v, q) are called observation values.

As a result, objects (heads of a plurality of persons and a ball) existing in the space 101 are detected by one or more detectors, the number of which varies depending on the position. Since an FOV overlapping map can be obtained by camera parameters and detector specifications, it is usually possible to know in advance which cameras and detectors will be used for detection at each point. In addition, a situation may occur in which a detection result indicating that an object is detected cannot be obtained from even one detector due to occurrence of concealment due to overlap of objects, non-detection by the detector, or the like.

Next, in loops L103 to L113, the processing unit 230 repeats processing for objects. That is, the processing unit 230 performs a repetitive process of giving indices n in order from 1 to N for a plurality of objects, as described above. For N, the number of objects (number of clusters) estimated in step S101 is used. A detailed description of this iterative process is omitted.

In this embodiment, when estimating a position in a three-dimensional space and tracking an object, the output (u, v, q) of the detector corresponding to each camera is used as an observed value as described above. Position (x, y, z) in space and six dimensions of velocity PV in the following equation are used as state variables. Among the objects, for the human head whose appearance on the image changes depending on the position and orientation, and the value of the score q of the detector changes, a total of 9 dimensions including orientation (φ, θ, ψ) are used as state variables. A state-space model with In this embodiment, state estimation and tracking are performed by an extended Kalman filter using this state space model.

Accordingly, in this embodiment, the observed value y and the state variable x of the head and the ball can be described as the following equations (1) to (3). Note that equation (2) is for the head, and equation (3) is for the state variables of the ball.

Note that (u, v, q) are the observed values that are the outputs of the detector, (x, y, z) are the positions of the object in the three-dimensional space, and the velocity PV is the x-axis and y-axis as shown in the above equation. , are values corresponding to the z-axis direction. Furthermore, (φ, θ, ψ) are values indicating the orientation of the human head. In equations (1) to (3), the subscript t is the time, k _j is the j-th observed value in the frame of camera k, and the subscript n is the n-th object (n of multiple existing objects). th object) and T is the transpose. The subscript n for an object can also be called the ID of the object.

Furthermore, after executing the process of step S107, which will be described later, n and _kj are associated, and yt _,kj is yt _,k,n = [ _ut,k,n , vt _,k,n , q _t,k,n ] ^T . where y _t,k,n means the observed value of object n with camera k at time t. At time t=1, the state variable x _t,n is assigned the initial value set in step S101. At time t=1, if this initial value is set to the first moment x _0|0,n and the second moment V _0|0,n of the initial filter distribution (posterior distribution) of the state variables to be described later, good.

In the next step S105, the prediction unit 206 performs a process of acquiring the predicted distribution of the state variables and the predicted distribution of the observed values.
First, the system equation used when obtaining the predicted distribution of the state variables will be described. In this embodiment, the following equation (4) is used as the system equation for the human head.

In Equation (4), Δt is the time step size (seconds) of one step, s _t is white Gaussian noise called process noise, and Q _t is the variance-covariance matrix of s _t . The system equations deal with position and velocity (PV) trend component models modeled with second order Markov processes for position (x, y, z). Also, the directions (φ, θ, ψ) are modeled as a first-order Markov process. For the time step size Δt, the sampling interval is normally used, but in this embodiment, the reciprocal of the product of the camera frame rate (fps) and the number of cameras (K) (1/(fps×K)) is used. For example, when the number of cameras (K) is 10 and the frame rate is 30 fps, the time step size Δt is 1/300. Therefore, in the case of this embodiment, the time resolution of object tracking is as fine as the number of cameras with respect to the camera frame rate.

Further, in this embodiment, the ball used in soccer is a sphere, and even if the position and orientation of the ball changes, the appearance of the shape does not change much. Equation (5) is used.

In Equation (5), Δt is the time width (seconds) of one step, s _t is process noise (white Gaussian noise), and Q _t is the variance-covariance matrix of s _t .

Next, an observation equation used when obtaining a predicted distribution of observed values, which is the output of the detector, will be described.
A point in the three-dimensional space can be projected onto the pixel coordinates of the camera by using camera parameters obtained by calibration in advance. This projection is described by equation (6) below.

p _xx,k in equation (6) are each element of the perspective projection matrix of camera k, and these elements are obtained in advance by camera calibration. γ is a parameter of the homogeneous coordinate system.
Based on this projection, the observation equation for the position (u, v) in the observed value is described by Equations (7) and (8) below. In the present embodiment, observations are modeled by equations (7) and (8) in common for the human head and the ball.

These equations (7) and (8) are equations that model the process in which a position (x, y, z) in a three-dimensional space is observed as pixel coordinates (u, v). As described above, in the photographing system having a plurality of cameras in this embodiment, each camera is asynchronous, and frame dropping may occur. The deviation of the detection position due to this asynchronism can be suppressed by the processing of this embodiment, in which the imaging timing is estimated and updated in that order. However, since a quantized constant value is used for the time step width Δt in the prediction, there is some deviation between the position of the object in the three-dimensional space and the position of the object in the pixel coordinates of the camera frame. It is quite possible that it will occur. Furthermore, positional deviation of the detection position due to the use of the object detector and positional deviation due to camera calibration error may occur. Due to these factors, an object in the three-dimensional space is considered to be observed with an error in its position on the detector. Equations (7) and (8) model the error as observation noise.

In this embodiment, the observation equation for the score q uses different models for the human head and the ball. First, let the observation equation of the head be the following equation (9).

q _t,kj =α ₀ +α ₁ ||x _t,n −C _k || ₂ +α ₂ cos(θ _x )+α ₃ cos(θ _y )+α ₄ cos(θ _z )+w _{t, q} formula (9)

_In equation ₍ 9), _{C k} is _the camera _{position of} camera k in the three-dimensional space, ||· _|| It is white Gaussian noise called observation noise. θ _x , θ _y , and θ _z are the elements of the matrix of formula (10) where R is the rotation matrix of the external parameters of the camera, and Ro is the rotation matrix obtained from the orientation of the human body (φ, θ, ψ). can be expressed as That is, it can be expressed as θ _x =asin(r ₃₂ ), θ _y =atan(−r ₃₁ /r ₃₃ ), θ _z =atan(r ₂₁ /r ₁₁ ).

Equation (9) is a multiple regression model that models the observation process regarding the score of the detector in the observed values. Detectors generally vary the score in relation to the size and orientation of the imaged object. The size of an object captured by a camera usually correlates with the distance from the camera, but assuming that there is no bias in the training data, image features such as texture are generally robust when the object is large (when it is close to the camera). and the score will be higher. Conversely, when the object is small (far from the camera), the texture tends to be distorted and the image feature quantity cannot be obtained stably, resulting in a low score. In particular, in the case of a person, when the camera is facing the front, the appearance of important organs (parts) such as the eyes, nose, and mouth is stable, so the score tends to be higher. . Conversely, when the back is facing the camera, there are fewer parts that can be used as identification clues, and the score tends to be lower.

In equation (9), the first term is a constant term, the second term is a term that models the relationship between the distance from the object to the camera and the score using a linear model, and the third, fourth, and fifth terms are the human body seen from the camera. , and the sixth term is a noise term.

On the other hand, the ball observation equation is the following equation (11).

q _t,kj =α ₀ +α ₁ ||x _t,n -C _k || ₂ +w _t,q formula (11)

The ball is supposed to be a sphere, and since it has invariance to rotation, the appearance of the shape does not change much depending on the position and orientation. Therefore, the observation equation for the score q is a linear regression model with only the distance from the camera to the object as an explanatory variable.

There are several methods for estimating the model parameters α ₀ , α ₁ , α ₂ , α ₃ , and α ₄ in Equations (9) and (11). One is to assign the correct value of orientation in 3D space to multiple images of people taken with a calibrated camera, obtain the score of the detector for the human body, and obtain multiple samples with orientation and score. is a method of estimating parameters by the method of least squares using The other is to use the likelihood function obtained by equation (12) described later, calculate the likelihood of the model for the observed values, obtain the log likelihood from a large number of observed values, and use the grid search to find the log likelihood This is a method of searching for a parameter that maximizes . The latter method is efficient because it does not require manual assignment of correct values. In addition, there are a recursive search method using the EM method and a method of forming a self-organizing model in which model parameters are also incorporated into the state space. Functionality is not significantly impaired.
The above equations (7), (8), (9), and (11) can be collectively expressed as the following equation (12).

y _t,kj =h _t,k (x _t,n )+w _t formula (12)

Here, let R _t be the variance-covariance matrix of the observation noise w _t in Equation (12). Also, the likelihood function P(y _t,kj |x _t,n ) can be obtained from this equation (12).
Based on the above system equation and observation equation, the following equations (13) to Equation (16).

where x _t|t-1,n is the first moment of the predicted distribution of the state variables, V _t|t-1,n is the second moment of the predicted distribution of the state variables, and y _{t|t-1,k, n} is the first moment of the prediction distribution of observations, U _t|t−1,k,n is the second moment of the prediction distribution of observations. Also, Q _t is the variance-covariance matrix of process noise, and R _t is the variance-covariance matrix of observation noise. F _t is the matrix of the system equations (equations (4) and (5)), and H _t,k is the Jacobian matrix of h _t,k (x _t,n ) of the observation equation (equation (12)).

Hereinafter, for simplicity, the predicted distribution of the state variables following the Gaussian distribution with the above first and second moments is expressed as P(x _t,n |Y _t−1 ), and the predicted distribution of the observed values is expressed as P(y _{t ,k,n} |Y _t-1 ). Y _t-1 is the set Y _t-1 of the observed values up to time t-1 shown in the following equation (17).

y _t,k,n in Equation (17) is the observed value of the object with ID n at time t and camera k. Also, K _n is the number of cameras overlapping the position of the object n in the three-dimensional space.

The prediction unit 206 predicts the state variables of the object n as described above in step S105. That is, the prediction unit 206 calculates the first-order moment x _t|t-1,n and the second-order moment V _t-1|t-1,n of the filter distribution of the state variables of the object n with respect to Equation (13) and Equation ( 14) apply respectively. Thereby, the prediction unit 206 acquires the first moment x _t|t-1,n and the second moment V _t|t-1,n of the prediction distribution.

In branch B101, the prediction unit 206 uses the FOV overlapping map information 410 shown in FIG. Branch the processing according to the If the position of the object predicted in the previous step exists within the FOV of camera k, the prediction unit 206 advances the process to step S106, and otherwise advances the process to step S108. In branch B101, the prediction unit 206 may advance the process to step S106 if the probability of camera k existing in the FOV of camera k is not substantially zero based on the first and second moments of the predicted distribution of the state variables.

After proceeding to step S106, the prediction unit 206 predicts the observed value of the object n in the camera k based on the state of the position and speed of the object predicted in step S105. That is, the prediction unit 206 predicts the first and second moments of the distribution of observation values obtained by the camera k for the object n. The prediction unit 206 uses equations (15) and (16) to obtain the following from the first and second moments x _t|t-1,n and V _t|t-1,n of the predicted distribution of the state variables of the object n. , obtain the first and second moments y _t|t-1,k,n and U _t|t-1,k,n of the prediction distribution of the observations of camera k.

Next, in step S107, the ID associating unit 207 performs a process of associating a plurality of (for example, J) observed values observed by camera k at time t with object n. That is, the ID associating unit 207 calculates the likelihood of the actual observed value based on the predicted probability distribution of the observed value of the object n with camera k at time t and the actual observed value with camera k. Based on the likelihood of the observed value, the ID associating unit 207 performs a process of associating the object n in the three-dimensional space with the observed value. In this embodiment, it is assumed that J observation values {y _t,k1 , y _t , _{k2 ,} . In the example of FIG. 5(b) described above, three observed values including one false detection (excessive detection) are obtained.

Here, according to the first-order and second-order moments of the predicted distribution of the observed values obtained by Expressions (15) and (16), the Gaussian distribution given by Expression (18) below can be described.

l _kj,n =N(y _t,kj ;y _t|t-1,k,n ,U _t|t-1,k,n ) Equation (18)

By giving the observed value y _t,kj to this function as an argument, the likelihood l kj,n of y _{t, kj} as the observed value of the object n can be calculated. Then, apply formula (18) to each of the plurality of observed values {y _t,k1 , y _{t, k2} , . With this, it is possible to associate the observed value with the object n.

In the matching, the observed value with the maximum likelihood among multiple observed values may be assigned as the observed value of a certain person (object n) based on the so-called greedy method, but the sum of the likelihoods is the maximum A linear programming may be used to compute a correspondence such that In that case, using the Mahalanobis distance calculated from the first and second moments of the observed value and the predicted distribution, the Hungarian method can be used to calculate the correspondence that minimizes the sum of the Mahalanobis distances, which maximizes the sum of the likelihoods. different assignments can be obtained.

At this time, due to detector underdetection and overdetection, the actual observations may not match the number of true observations that should be in the frame. Here, the true number of observed values is equal to the number of persons existing within the FOV of one camera, which is 2 in the example of FIG. 5(b).

FIG. 6 is a diagram for explaining the cost matrix used in the Hungarian method when non-detection and over-detection occur. Assume that there are objects assigned ID1, ID2, ID3, and ID4 in the three-dimensional space, and the objects ID2, ID3, and ID4 are imaged by camera k.
Figure 6(a) shows when the number of actual observed values and the number of true observed values match, Figure 6(b) shows when the actual number of observed values is small (undetected occurrence), and Figure 6(c) shows the actual number of observed values. It shows an example of the cost matrix used in the Hungarian method when the number of observed values is large (occurrence of overdetection).

In the example of FIG. 6A, P(yt _,k,2 |Yt _-1 ), P( _yt,k,3 |Yt _-1 ), P( _yt,k,4 | _{Yt −1} ) are the predicted distributions of observed values for objects with IDs ID2, ID3, and ID4, respectively. y _t,k1 , y _t,k2 , and y _t,k3 are the first, second, and third observed values of camera k at time t, respectively. Values 607 in FIG. 6A are Mahalanobis distances, and a matrix having Mahalanobis distances as elements is a cost matrix 608 . In cases such as the example of FIG. 6A where the number of actual observed values and the number of true observed values match, the Hungarian method may be applied based on this cost matrix.

FIG _. 6(b) shows an example of a case where non-detection occurs _. does not match. In such a case, by setting the false observed value y _t,k−1 and setting the Mahalanobis distance to infinity (∞), the cost matrix becomes a square matrix and the Hungarian method can be applied. As a result of this Hungarian method calculation, the prediction distribution of any ID is assigned to the false observation y _t,k-1 . This is a situation in which an observed value is missing, and processing different from normal processing is performed at the time of updating in step S108, which will be described later.

FIG. 6(c) shows an example of a situation in which overdetection occurs. In this case, a false prediction distribution P(y _t,k,-1 |Y _t-1 ) is set, and the Mahalanobis distance is The cost matrix can be squared by setting it to infinity (∞), and the Hungarian method can be applied.

It is conceivable that undetected and over-detected values may occur simultaneously in a certain frame, and the number of actual observed values and the number of true observed values seem to match, resulting in erroneous correspondence. In order to cope with such a case, a threshold is set for the Mahalanobis distance, and IDs associated with a Mahalanobis distance of 3 or more, for example, are treated in the same way as when the above observation value is missing. Mistakes can be reduced.

Next, in step S108, the updating unit 208 updates the predicted distribution of the state variables using the observed value at time t, and acquires the filter distribution (posterior distribution). Observed value y _t associated in the previous step from first moment x _t|t-1,n and second moment V _t|t-1,n of predicted distribution of state variables of object n at time t in the extended Kalman filter _{, k, n} can be expressed by the following equations (19) to (21).

where K _t,k,n is the Kalman gain of time t, camera k, and object n, R _t is the variance-covariance matrix of observation noise, and H _t,k is h _t,k (x _t,n ).
Then, for an object n, if there is a corresponding observed value, the first and second moments of the filter distribution of the state variable corrected by the observed value are obtained by applying the update formulas shown in Equations (19) to (21). can get.

On the other hand, if there is no observed value corresponding to object n due to lack of observed value or existence outside the FOV of the detector of camera k, it is handled by replacing the current filter distribution with the predicted distribution of the state variable. can do. That is, the update formulas when there is no observed value are the following formulas (22) and (23).

x _t|t,n =x _t|t-1,n formula (22)
V _t|t,n =V _t|t-1,n Formula (23)

The update unit 208 updates the state variables of the object n by performing any one of the above-described equations (19), (20), (21), or (22) and (23). implement.

Next, in step S109, the visualization unit 209 generates the position estimation results of the N objects n (where n is 1 to N) in the three-dimensional space executed in the loops L102 to L112 and the time series of the estimated positions. Visualize.
In the visualization, each result may be drawn in a virtual three-dimensional space, or may be superimposed and displayed as a trajectory or points on an actual image acquired by a camera. Then, the visualization result is transmitted to the monitoring unit and can be viewed by the user.

Next, in step S110, the timing estimation unit 210 estimates the imaging timing of each of the K cameras.
Here, when the observation value of camera k includes the observation value of object n, the detection position corresponding to the observation value is (u _t,k,n , v _t,k,n ). A straight line connecting the optical center of the camera and the detection position at that time is given by the following formula (24) from the formula (7).

(u _t,k,n p _21,k −v _t,k,n p _11,k )X+(u _t,k,n p _22,k −v _t,k,n p _12,k )Y+(u _t,k,n p _23,k −v _t,k,n p _13,k ) Z + u _t,k,n p _24,k −v _t,k,n p _14,k =0 Equation (24)

Now, from the point (x _t,n , y _t,n , z _t,n ), which is the component with respect to the position of the first moment x _t|t,n of the filter distribution of object n, to the intersection point of the perpendicular to this straight line be p' _t,n and let e t,n be the difference between the point x _t,n =(x _t,n , y _t,n , z _t,n ) and p' _{t, n} . The estimated value d _t,n (seconds) of the imaging timing is a numerical value (seconds) obtained by dividing the difference between the current estimated position and the position estimated from the current observation value by the current estimated speed. That is, the estimated value d _t,n of the imaging timing is obtained by dividing each dimension of the difference e _t,n by the component shown in the equation (25) regarding the velocity of the first moment of the filter distribution. is the average value of

The timing estimating unit 210 estimates only a plurality of objects n associated with a plurality of observed values of the camera k, and averages them as an estimated value of the imaging timing of the camera k. Further, the timing estimating unit 210 calculates this for all K cameras and obtains an estimated value of the imaging timing of each camera. The acquired value is used as an estimated value of the imaging timing for each camera in step S102.

The three-dimensional tracking device 200 of the first embodiment performs the above processing to estimate the positions of a plurality of objects existing in the three-dimensional space, and to adjust the time resolution Δt to the number of cameras with respect to the camera frame rate. It is possible to realize finer tracking. This temporal resolution is the resolution exceeding the frame rate of the camera. Therefore, according to this embodiment, it is possible to acquire the position and trajectory in the three-dimensional space of a tracking object moving at high speed, such as a ball during a sports match, by using the object detection results of multiple cameras that capture images at different timings. As a result, tracking failures can be reduced.

<Second embodiment>
In the first embodiment, the estimated value of the imaging timing of each camera is obtained by direct calculation using the estimated position and speed of the object to be tracked in the three-dimensional space and the actual observed value. . In the second embodiment, the Bayesian framework is also applied to the estimation of the imaging timing of each camera, and the posterior probability distribution of the imaging timing is calculated. Furthermore, in the second embodiment, when the state variables of the object to be tracked are updated, correction weighted by the posterior probability distribution of the imaging timing is performed to reduce the number of missing observation values and to improve stability. Realize tracking with high temporal resolution.

FIG. 1B is a diagram showing the functional configuration of a three-dimensional tracking device 300 according to the second embodiment.
A three-dimensional tracking device 200 according to the second embodiment shown in FIG.
The imaging unit 320 includes a first moving image acquiring unit 301 to a Kth moving image acquiring unit 302 similar to the imaging unit 220 in FIG. 1(a). In the following description, in the same way as described above, when the K moving image acquisition units of the first moving image acquiring unit 301 to the Kth moving image acquiring unit 302 are not particularly specified, the moving image acquiring units may be simply referred to as cameras.

Unlike the functional configuration shown in FIG. 1A, the processing unit 330 of the second embodiment does not have the moving image acquisition selection unit 203 described above, but has a temporary storage device 311, a timing prediction unit 314, and a timing updating unit 313 . Details will be described with reference to FIG. 3 and the like, but some functions of the functional configuration of the processing unit 330 of the second embodiment are different from those of the first embodiment.

FIG. 3B is a flowchart showing the flow of information processing in the processing section 330 of the three-dimensional tracking device 300. As shown in FIG. The outline of the overall processing in the three-dimensional tracking device 300 of the second embodiment will be described below with reference to the flowchart of FIG. 3(b).

In step S201, the initial value setting unit 305 sets initial values for the number, ID, and state of tracking target objects (persons, balls) in the three-dimensional space. Further, the initial value setting unit 305 sets the initial values of the imaging timings of the K cameras of the imaging unit 320 as state variables. In step S201 of the second embodiment, the initial value setting unit 305 sets the initial values of the first and second moments of the filter distribution of the state variables of the object n, as well as the filter of the imaging timing, which is the state variable of the camera k. Sets initial values for the first and second moments of the distribution. Regarding the processing of step S201, differences from step S101 in the first embodiment will be described later.

In loops L201 to L211, the processing unit 330 repeatedly performs time-related processing. That is, the processing unit 330 executes a repetitive process of giving the index t regarding the time in order from 1 to T. FIG.
Next, in loops L202 to L212, the processing unit 330 repeats processing related to the moving image acquisition unit of the imaging unit 320. FIG. That is, the processing unit 330 performs a repetitive process of giving the index k related to the camera in order from 1 to K. FIG.
The processing of loops L201 to L211 and L202 to L212 is the same as the processing of loops L101 to L111 and L102 to L112 according to the first embodiment, so detailed description thereof will be omitted.

Next, in step S202, one of the camera k (video acquisition unit k) of the first video acquisition unit 301 to the K-th video acquisition unit 302 of the imaging unit 320 acquires one current frame (still image).

Next, in step S203, the detection unit 304 detects the positions and scores of the objects (person and ball) appearing in the frame acquired in the previous step. When there are multiple persons in the frame, as in the example of soccer described above, basically multiple positions and scores corresponding to each number of persons are detected. Since this process is the same as step S104 in the first embodiment, detailed description thereof will be omitted.

Next, in step S<b>204 , the detection unit 304 saves the object detection results (positions and scores of the person and the ball) acquired in the previous step in the temporary storage device 311 .

In loops L203 to L213, processing unit 330 repeatedly executes processing for objects. That is, the processing unit 330 executes a repetitive process of giving indices n in order from 1 to N for a plurality of objects. For N, the value obtained as the initial value of the number of objects in step S201 is used.

Next, in step S205, the prediction unit 306 sets the current time to t (t≧1), and predicts the probability distribution of the state at time t based on the state of the object n at time t−1, such as the position and speed. process. This process is similar to step S105 in the first embodiment, except that the position, speed, and other states of the object temporarily stored in the temporary storage device 311 are used, so detailed description thereof will be omitted. .

In loops L204 to L214, the processing unit 330 repeatedly performs processing related to the camera (video acquisition unit) of the imaging unit 320. FIG. That is, the processing unit 330 performs a repetitive process of giving the index k related to the camera in order from 1 to K. FIG. This process is the same as the loops L102 to L112 of the first embodiment and the loops L202 to L212 of the second embodiment, so detailed description thereof will be omitted.

Next, in branch B201, the prediction unit 306 branches the processing depending on whether or not object n exists within the FOV of camera k based on the position of object n predicted in the previous step and the FOV overlapping map. If there is a probability that the object n exists within the FOV of the camera k, the prediction unit 306 advances the process to step S206, and if the object n does not exist, advances to the next repeated process. Since this process is the same process as the branch B101 in the first embodiment, detailed description thereof will be omitted.

Proceeding to step S206, the prediction unit 306 sets the current time to t (t≧1), and determines the observation value to be acquired at time t based on the state of the object n at time t−1, such as the position and speed. Predict probability distributions. Since this process is the same process as step S106 in the first embodiment, detailed description will be omitted.

Next, in step S<b>207 , the ID association unit 307 reads the object detection result corresponding to the frame acquired by the camera k, stored in the temporary storage device 311 .
Further, in step S208, the ID associating unit 307 calculates the predicted probability distribution of the observed values of the object n with the camera k predicted at the time t in the previous step and the actual observed values with the camera k at the time t. Compute the likelihood of actual observations. Based on the likelihood of the observed value, the ID associating unit 307 associates the object n in the three-dimensional space with the observed value. Since this association processing is the same processing as step S107 in the first embodiment, detailed description thereof will be omitted.

Next, in step S209, the updating unit 308 calculates weights to be used for updating the state variables of the object n from the posterior probability distribution of the state variables at the imaging timing.
Furthermore, in step S210, the updating unit 308 updates the state variables of the object n using the weights acquired in the previous step. Details of the processing of this step will be described later.

In loops L205 to L215, the processing unit 330 repeatedly performs processing related to the camera (video acquisition unit) of the imaging unit 302. FIG. That is, the processing unit 330 performs a repetitive process of giving the camera-related index k in order from 1 to K. FIG. This process is the same as the loops L102 to L112 of the first embodiment, the loops L202 to L212, and the loops L204 to L214 of the present embodiment, so detailed description thereof will be omitted.

Next, in step S211, the timing prediction unit 314 sets the current time to t (t≧1), and predicts the probability distribution of the state at time t based on the imaging timing (state variable) of camera k at time t−1. process.
Furthermore, in step S212, the timing updating unit 313 reads the object detection result corresponding to the frame acquired by the camera k, which is stored in the temporary storage device 311. FIG. Since this process is the same process as step S207, detailed description thereof will be omitted.
Then, in step S213, the timing update unit 313 uses the detection result obtained in the previous step to update the imaging timing state of the camera k.

Next, in step S<b>214 , the visualization unit 309 executes processing for visualizing the updated state of the object, and further displays the processing result on the monitoring unit 340 . Since this process is the same as step S109 in the first embodiment, detailed description is omitted.

Next, according to the flowchart shown in FIG. 3(b), more detailed and specific processing contents will be described for the processing in each functional unit of the processing unit 330 of the three-dimensional tracking device 300 shown in FIG. 1(b). . Note that loops L201 to L211, L202 to L212, L203 to L213, L204 to L214, and L205 to L215 are the same processing as in the first embodiment, so detailed description thereof will be omitted. Also, steps S203, S205, S206, S208, S214, and branch B201 are the same processes as in the first embodiment, so detailed description thereof will be omitted. Furthermore, since step S212 is the same processing as step S207 of the second embodiment, detailed description thereof will be omitted.

In this embodiment, the position, velocity, and orientation of each object to be tracked are treated as state variables, and the state space model is expanded to treat the imaging timing of each camera as state variables. Therefore, in the second embodiment, a system equation describing the temporal transition of the imaging timing and an observation equation describing the process of observing the imaging timing are introduced.

In this embodiment, the imaging timing of camera k is dt _,k , and a first-order random walk model is used. The imaging timing d _t,k is represented by the following equation (27).

d _t,k =Dd _t-1,k +s _t,n Formula (27)

where s _t,n is process noise (white Gaussian noise). Also, when the time k, the position of the object n in the three-dimensional space is x _t,n =(x _t,n , y _t,n , z _t,n ), and the velocity is represented by Equation (28), the camera The position in the three-dimensional space due to the imaging timing of k can be described as in Equation (29).

Further, based on these equations, equations (7) and (8), the following equations (30) and (31) are used as observation equations. Note that w _t,u and w _t,v are observation noises, respectively.

Returning to the flowchart of FIG. 3(b). In the case of the second embodiment, in step S201, the initial value setting unit 305 sets the initial values of the first and second moments of the filter distribution of the state variables of the object n, as well as the filter of the imaging timing, which is the state variable of the camera k. We also set initial values for the first and second moments of the distribution.

In the case of the second embodiment, in step S210, the updating unit 308 uses, as a weight, the value obtained by integrating the posterior distribution of the imaging timing acquired in the previous step for each time step to update the state variable of the object n. . For the update process of the extended Kalman filter in this case, a method of weighted sum of Kalman feedbacks and represented by the following equations (32) to (34) is used. Note that w _t,k,n is the integrated value of the imaging timing acquired in the previous step.

Next, in step S211, the timing prediction unit 314 predicts the probability distribution based on the imaging timing of camera k. In the case of the second embodiment, the timing prediction unit 314 models observations common to the human head and the ball using the observation equation of the position (u, v) in the observation values described above.

FIG. 7 is a diagram showing an example of true imaging timing assumed in this embodiment. In the example of FIG. 7, 10 cameras are assumed as a plurality of cameras, and one camera is used as a reference, and deviations in shooting timings of the remaining 9 cameras within one cycle are shown. T indicates the timing of the frame cycle of the camera. Timing 710 represents the timing of the reference camera, and timings 701, 702, 703, 704, 705, 706, 707, 708, and 709 represent the timings of the remaining nine cameras in descending order from the timing 710. . That is, the timings 701, 702, 703, 704, 705, 706, 707, 708, and 709 are shifted from the timing 710 on the time axis.

In this embodiment, the true imaging timing shown in FIG. 7 is treated as a state variable because it is a value that cannot be directly observed. That is, the imaging timing is estimated as a posterior distribution within the framework of the state space model from the detection results of each camera, which are observable values.
FIG. 8 is a diagram showing the posterior distribution of imaging timings estimated by the above-described state space model given observation values (u, v) of each camera. In step S210 of this embodiment, the imaging timings of the ten cameras shown in FIG. 7 are estimated as the probability distribution (Gaussian distribution) 801 shown in FIG.

As described above, in the second embodiment, the Bayesian framework is also applied to the estimation of the imaging timing of each camera, the posterior probability distribution of the imaging timing is calculated, and the state variables of the tracking target object are updated. , weighted by the posterior probability distribution of the imaging timing. As a result, according to the second embodiment, the number of missing observation values can be reduced, and more stable tracking with high temporal resolution can be performed.

<Third Embodiment>
In the first and second embodiments, a plurality of types of objects are assumed as tracking targets. Among these, there are objects that move relatively slowly, such as people, and objects that move at high speed, such as balls. Furthermore, the ball moves at high speed the moment it is kicked by a player during a sports match such as soccer, but moves at the same speed as a person during dribbling. In general, deviations in object detection positions on the frame of each network camera caused by deviations in imaging timing due to asynchronism of a plurality of network cameras become conspicuous when the object to be detected moves at high speed. Furthermore, due to the position and orientation of the camera, the deviation of the object detection position increases as the speed component parallel to the plane of the pixel coordinates of the camera increases at a short distance from the camera. In the third embodiment, a case will be described in which these properties are focused on and the accuracy of imaging timing estimation is improved. In the third embodiment, when updating the imaging timing, an index for quantifying a suitable condition for the detection result of the detection unit 304 is calculated. Then, the index is used as a weight for updating the state space model of the imaging timing, and an update amount for updating the imaging timing is set to improve the estimation accuracy of the imaging timing.

The functional configuration of the three-dimensional tracking device of the third embodiment is the same as the functional configuration of the three-dimensional tracking device 300 of the second embodiment shown in FIG. 1(b), so illustration thereof is omitted.
FIG. 9 is a flow chart showing the flow of information processing in the processing section of the three-dimensional tracking device. First, using the flowchart of FIG. 9, an overview of the overall processing according to the third embodiment will be described.

First, in step S301, the initial value setting unit 305 sets initial values for the number, ID, and state of tracking target objects (persons, balls) in the three-dimensional space. Furthermore, in this embodiment, the imaging timing of the K cameras (moving image acquisition units 301 and 302) of the imaging unit 320 is set as a state variable, and its initial value is set. Since this process is the same process as step S201 in the second embodiment, detailed description will be omitted.

In loops L301 to L311, processing unit 330 repeatedly performs time-related processing. That is, the processing unit 330 gives an index t relating to time in order from 1 to T, and repeats the process.
Next, in loops L302 to L312, the processing unit 330 repeatedly performs camera-related processing. That is, the processing unit 330 gives the camera-related index k in order from 1 to K, and repeats the process.
The processing of loops L301 to L311 and L302 to L312 is the same as the processing of loops L201 to L211 and L202 to L212 of the second embodiment, so detailed description thereof will be omitted.

In step S302, one of the cameras k (image acquiring unit k) of the photographing unit 320 acquires one current frame (still image). Since this process is the same process as step S202 in the second embodiment, detailed description will be omitted.

Next, in step S303, the detection unit 304 detects the positions and scores of the objects (person and ball) appearing in the frame acquired in the previous step. When a plurality of objects (a plurality of persons) exist within the frame, the detection unit 304 basically detects a plurality of positions and scores corresponding to the number of the objects (number of persons). Since this process is the same process as step S203 in the second embodiment, detailed description will be omitted.

Next, in step S<b>304 , the detection unit 304 saves the object detection result acquired in the previous step in the temporary storage device 311 . Since this process is the same process as step S204 in the second embodiment, detailed description will be omitted.

Next, in loops L303 to L313, the processing unit 330 repeats processing regarding the object. That is, the processing unit 330 gives an index n related to the object in order from 1 to N, and repeats the process. For N, the value obtained as the initial value of the number of objects in step S301 is used. This processing is the same processing as the loops L203 to L213 in the second embodiment, so detailed description will be omitted.

In step S305, the prediction unit 306 sets the current time to t (t≧1), and predicts the probability distribution of the state at time t based on the state of the object n at time t−1, such as the position and speed. conduct. Since this process is the same process as step S205 in the second embodiment, detailed description will be omitted.

Next, in loops L304 to L314, the processing unit 330 repeats processing related to the moving image acquisition unit. That is, the processing unit 330 gives the camera-related index k in order from 1 to K, and repeats the process. Since this processing is the same processing as the loops L204 to L214 of the second embodiment, detailed description thereof will be omitted.

In branch B301, the prediction unit 306 performs processing according to the determination result of whether or not there is a probability that object n exists within the FOV of camera k, based on the position of object n predicted in the previous step and the FOV overlapping map. branch. In branch B301, the prediction unit 306 proceeds to the process of step S306 if there is a probability that the object n exists within the FOV of the camera k, and proceeds to the next iterative process if not. Since this processing is the same processing as the branch B201 in the first embodiment, detailed description will be omitted.

Proceeding to step S306, the prediction unit 306 sets the current time to t (t≧1), and determines the observation value to be acquired at time t based on the state of the object n at time t−1, such as the position and speed. Predict probability distributions. Since this process is the same process as step S206 in the second embodiment, detailed description will be omitted.

Next, in step S<b>307 , the ID association unit 307 reads the object detection result corresponding to the frame acquired by the camera k, stored in the temporary storage device 311 . Since this process is the same process as step S207 in the second embodiment, detailed description will be omitted.

Next, in step S308, the ID associating unit 307 bases the predicted probability distribution of observed values of object n on camera k predicted at time t in the previous step and the likelihood of actual observed values on camera k at time t. is associated with the object n in the three-dimensional space and the observed value. Since this process is the same process as step S208 in the second embodiment, detailed description will be omitted.

Next, in step S309, the updating unit 308 calculates weights to be used for updating the state variables of the object n from the posterior probability distribution of the state variables at the imaging timing. Since this process is the same process as step S209 in the second embodiment, detailed description will be omitted.

Next, in step S310, the updating unit 308 updates the state variables of the object n using the weights acquired in the previous step. Since this process is the same process as step S210 in the second embodiment, detailed description will be omitted.

In loops L305 to L315, the processing unit 330 repeatedly performs camera-related processing. That is, the processing unit 330 gives the camera-related index k in order from 1 to K, and repeats the process. Since this processing is the same processing as the loops L205 to L215 of the second embodiment, detailed description thereof will be omitted.

Next, in step S311, the timing prediction unit 314 sets the current time to t (t≧1), and based on the imaging timing (state variable) of the moving image acquisition unit k at time t−1, the probability distribution of the state at time t Perform processing to predict Since this process is the same process as step S211 in the second embodiment, detailed description is omitted.

Next, in step S<b>312 , the timing updating unit 313 reads the object detection result corresponding to the frame acquired by the camera k, which is stored in the temporary storage device 311 . Since this process is the same process as step S212 in the second embodiment, detailed description is omitted.

Next, in step S313, the timing updating unit 313 calculates an index representing a suitable condition for updating the imaging timing, that is, an index for quantifying a suitable condition for the detection result of the detecting unit 304, and performs imaging based on the index. Compute weights for timing updates.
Further, in step S314, the timing updating unit 313 updates the state of the imaging timing of camera k using the weight for updating the imaging timing acquired in the previous step and the detection result acquired in step S311. That is, the timing updating unit 313 sets the weight for updating the state space model of the imaging timing as an index for quantifying the conditions suitable for estimating the imaging timing, and sets the update amount for updating the imaging timing. Update the timing.

Thereafter, in step S<b>315 , the visualization unit 309 executes processing for visualizing the updated state of the object, and further displays the processing result on the monitoring unit 340 . Since this process is the same process as step S214 in the second embodiment, detailed description will be omitted.

Hereinafter, according to the flowchart shown in FIG. 9, more detailed and specific processing contents will be described with respect to the processing in each functional unit of the processing unit 330 of the third embodiment. As described above, each step other than steps S312 and S313 is the same as the processing of the second embodiment, so detailed description thereof will be omitted.

In step S313, the timing update unit 313 calculates an index that quantifies conditions suitable for estimating the imaging timing, and uses the index as a weight when updating the state space model of the imaging timing. Updating the imaging timing is more suitable when the angle formed by the velocity vector of the object and the image plane on the world coordinates is close to a right angle and the positions of the object and the camera are close.

Therefore, the following formula (35) is used as an index for quantifying the suitability of updating the imaging timing from the multiplier element of the imaging timing d _t, k of the camera k in the above-described formulas (30) and (31). .

Here, α _t,k,n takes a larger real value when the angle between the velocity vector of the object and the image plane on the world coordinates is close to a right angle and the positions of the object and the camera are close. In this step, the timing updating unit 313 acquires the index α _t,k,n for each object n during the repetition of camera k at time t.

Next, the timing update unit 313 performs normalization using the following equation (36) in order to use this index as a weight when updating the state space model of the imaging timing.

Then, in step S314, the timing updating unit 313 applies the weights calculated in this way to the update equations for weighted sum of the Kalman feedbacks already shown in equations (32), (33), and (34), Update the imaging timing.

In the third embodiment, an index for quantifying a condition suitable for measuring the imaging timing shift is calculated, and the index is used as a weight when updating the state space model of the imaging timing, thereby estimating the imaging timing. It is possible to improve the accuracy.

The information processing apparatus (particularly, the processing unit of the three-dimensional tracking apparatus) of each embodiment described above may be realized by a personal computer or the like connected to K cameras (moving image acquisition units) and monitoring units. Then, the computer performs each information processing from object detection to visualization processing as described in each of the above-described embodiments. The computer in this example executes program code of software that implements the information processing of this embodiment. Although illustration of the hardware configuration is omitted, a computer that realizes the information processing apparatus of this embodiment includes a CPU, a ROM, a RAM (random access memory), an auxiliary storage device, a display unit (monitoring unit), an operation unit, a communication I /F, and a bus. The CPU controls the entire computer using computer programs and data stored in the ROM and RAM, and executes each information processing from object detection to visualization described above. Further, the information processing apparatus of the present embodiment may have one or a plurality of pieces of dedicated hardware different from the CPU, and may be configured such that at least part of the processing by the CPU is executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM stores programs and the like that do not require modification. The RAM temporarily stores programs and data supplied from the auxiliary storage device and data supplied from outside such as a camera via the communication I/F. The auxiliary storage device is composed of an HDD or the like, and stores various data such as image data and calibration information. The display unit (monitoring unit) is configured by, for example, a liquid crystal display or an LED display, and displays a GUI or the like for the user to operate the information processing device. The operation unit is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and inputs various instructions to the CPU in response to user's operations. The CPU also operates as a display control section that controls the display section and as an operation control section that controls the operation section. The communication I/F is used for communication with an external device of the information processing device. For example, when the information processing device is further connected to an external device by wire, a communication cable is connected to the communication I/F. If the information processing device has a function of wirelessly communicating with an external device, the communication I/F has an antenna. The bus connects each part of the information processing device to transmit information. Note that in the case of this embodiment, the external device connected to the information processing device is the above-described camera, other information processing device, or the like. In addition, although the display unit and the operation unit are assumed to exist inside the information processing apparatus, the display unit may exist as a monitoring unit and the operation unit as an input device, and may exist as separate devices outside the information processing apparatus. good.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.
All of the above-described embodiments merely show specific examples for carrying out the present invention, and the technical scope of the present invention should not be construed to be limited by these. That is, the present invention can be embodied in various forms without departing from its technical concept or main features.

200: three-dimensional tracking device, 201, 202: video acquisition unit, 230: processing unit, 203: video acquisition selection unit, 204: detection unit, 205: initial value setting unit, 206: prediction unit, 207: ID association unit , 208: updating unit, 209: visualization unit, 210: timing estimation unit

Claims (24)

a detection means for detecting a predetermined target object from images respectively captured by a plurality of imaging means;
prediction means for predicting the state of the object at a second time after the first time based on the detection result of the object for the image captured at the first time for each of the imaging means;
updating means for updating the predicted state of the object for each imaging means based on the detection result by the detection means and the prediction result by the prediction means;
estimating means for estimating imaging timing for each imaging means based on the updated state of the object and the detection result of the detecting means;
An information processing device comprising: 2. The information processing apparatus according to claim 1, wherein said updating means updates the predicted state of the object to the state of the object at a third point in time. The imaging timing is the difference between the reference imaging time and the imaging time of each of the other imaging means when the imaging time of one of the plurality of imaging means is used as a reference. 3. The information processing apparatus according to claim 1, wherein: 4. Information according to any one of claims 1 to 3, wherein the state of the object is all of the position, velocity, and orientation of the object in a three-dimensional space, or the position and velocity. processing equipment. 5. The information processing apparatus according to claim 1, wherein said prediction means predicts the state of said object as a probability distribution. 6. The method according to claim 5, wherein the prediction means predicts the state of the object as the probability distribution at a time interval obtained by dividing a time interval of imaging by the imaging means by the number of the imaging means. Information processing equipment. The update means updates the predicted state of the object as the probability distribution based on the detection result of the detection means for the image acquired at the second time point and the prediction result of the prediction means. 7. The information processing apparatus according to claim 5, wherein: 8. The apparatus according to any one of claims 1 to 7, further comprising selecting means for selecting an imaging means for imaging at the first time point from among the plurality of imaging means based on the imaging timing estimated by the estimating means. The information processing device according to item 1. 9. The information processing apparatus according to any one of claims 1 to 8, further comprising associating means for associating identification information with the object detected by the detecting means. The associating means obtains a likelihood of the detection result of the object by the detection means based on the detection result of the object by the detection means and the prediction result by the prediction means, and calculates the likelihood based on the likelihood. 10. The information processing apparatus according to claim 9, wherein the result of detection of said object by said detection means and the result of prediction by said prediction means are associated with each other. a detection means for detecting a predetermined target object from images respectively captured by a plurality of imaging means;
a timing prediction means for predicting an imaging timing for each imaging means;
and timing update means for updating the imaging timing based on the detection result of the object by the detection means and the prediction result of the imaging timing by the timing prediction means. prediction means for predicting the state of the object at a second time after the first time based on the detection result of the object for the image captured at the first time for each of the imaging means;
updating means for updating the predicted state of the object for each imaging means based on the detection result by the detection means and the prediction result by the prediction means;
12. The information processing apparatus according to claim 11, comprising: 13. The information processing apparatus according to claim 12, wherein said updating means updates the predicted state of the object to the state of the object at a third point in time. 13. The updating means calculates a weight for each imaging means based on the imaging timing updated by the timing updating means, and updates the state of the object based on the weight. 14. The information processing device according to 13. 15. The information processing apparatus according to claim 14, wherein said updating means updates the state of said object based on a weighted sum using said weight for each said imaging means. 16. The information processing apparatus according to any one of claims 12 to 15, further comprising associating means for associating identification information with the object detected by the detecting means. The associating means obtains a likelihood of the detection result of the object by the detection means based on the detection result of the object by the detection means and the prediction result by the prediction means, and calculates the likelihood based on the likelihood. 17. The information processing apparatus according to claim 16, wherein the result of detection of said object by said detection means and the result of prediction by said prediction means are associated with each other. 18. The information processing apparatus according to any one of claims 11 to 17, wherein said timing prediction means predicts said imaging timing as a probability distribution. The timing updating means calculates an index for quantifying the conditions under which the detecting means obtains the detection result of the object, and uses the index as a weight to set an amount for updating the imaging timing. Item 19. The information processing apparatus according to any one of Items 11 to 18. 20. The method according to any one of claims 1 to 19, wherein the detection means acquires at least one of a position of the object and a score representing likelihood of the position as the detection result of the object. The information processing device described. 21. The information processing apparatus according to any one of claims 1 to 20, further comprising visualization means for visualizing the positions of said objects in chronological order. a detection step of detecting a predetermined object from images captured by a plurality of imaging means;
a prediction step of predicting the state of the object at a second time after the first time based on the detection result of the object for the image captured at the first time for each of the imaging means;
an update step of updating the predicted state of the target object based on the detection result of the detection step and the prediction result of the prediction step for each of the imaging means;
an estimating step of estimating an imaging timing for each of the imaging means based on the updated state of the object and the detection result of the detecting step;
An information processing method characterized by having a detection step of detecting a predetermined object from images captured by a plurality of imaging means;
a timing prediction step of predicting an imaging timing for each of the imaging means;
An information processing method, comprising: a timing update step of updating the imaging timing based on a detection result of the object by the detection step and a prediction result of the imaging timing by the timing prediction step. A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 20.

Images