JP2020107349A - Object tracking device, object tracking system, and program - Google Patents

Abstract

To provide a technique capable of tracking an object with higher precision.SOLUTION: An object tracking system comprises a plurality of detection means and integrated tracking means. The integrated tracking means outputs position information on an object of a calculated common coordinate system. The detection means is configured to: convert the position information on the object on the common coordinate system into position information represented with an individual coordinate system characteristic of a camera outputting an image of the object serving as a target of detection; track the object on the individual coordinate system; detect the object based upon the position information represented with the individual coordinate system; and convert the position information on the object, detected based on the position information represented with the individual coordinate system, into position information represented with the common coordinate system.SELECTED DRAWING: Figure 1

Description

The present invention relates to an object tracking device, an object tracking system, an object tracking method, a display control device, an object detection device, a program and a recording medium.

In recent years, a system for tracking a person using a plurality of cameras has been developed. For example, the moving body tracking system described in Patent Document 1 uses a plurality of in-camera tracking means for tracking a person in a distributed manner for each camera. Then, the system tracks a moving body in cooperation with a plurality of in-camera tracking means. Further, Patent Document 2 describes a method of tracking the same object imaged between a plurality of imaging units based on the tracking result of each object.

In addition, as a related technique, Patent Document 3 describes a method of removing an object that does not need to be tracked from a tracking target earlier.

Further, Patent Document 4 describes an apparatus that detects a moving object in an image captured by one image capturing unit.

JP, 2004-72628, A Japanese Patent Publication No. 2009-510541 International Publication No. 2013/012091 JP, 2006-202047, A

However, in the technique described in Patent Document 1 or 2 described above, for example, when a camera exists at a position far away from the object, the tracking accuracy of the object (moving object) in the image captured by the camera may be low. is there. In this case, in the technique of Patent Document 1 or 2, the tracking accuracy of the camera is affected, and the tracking results of the objects cannot be integrated, or even if they are integrated, the detection of the position of the object required at the time of integration is detected. There were cases where the accuracy became low.

The present invention has been made in view of the above problems, and an object thereof is to provide a technique capable of tracking an object with higher accuracy.

An object tracking device according to an aspect of the present invention detects an object from output information of a sensor and outputs a detection result, and a plurality of detection results output by each of the detection means. Integrated tracking means for tracking the object based on the above, and generating tracking information of the object expressed in a common coordinate system, wherein the integrated tracking means detects the generated tracking information from the plurality of detecting means. And the detection means detects the object based on the tracking information.

An object tracking system according to an aspect of the present invention includes a sensor and an object tracking device that receives output information including information acquired by the sensor, and the object tracking device detects the object from the output information. Then, a plurality of detection means for outputting the detection result, and tracking the object based on the plurality of detection results output by each of the plurality of detection means, the tracking of the object represented in a common coordinate system Integrated tracking means for generating information, wherein the integrated tracking means outputs the generated tracking information to each of the plurality of detecting means, and the detecting means, based on the tracking information, the object. To detect.

An object tracking method according to an aspect of the present invention detects an object from output information of a sensor, outputs a detection result, and tracks the object based on the output results of the plurality of detections in a common coordinate system. The expressed tracking information of the object is generated, the generated tracking information is output, and the detection of the object detects the object based on the tracking information.

A display control device according to an aspect of the present invention is a display control device for displaying display data on a display device, wherein the display data searches the object based on tracking information of the object among output information of a sensor. A search range represented by a sensor-specific individual coordinate system that outputs the output information, wherein the object tracking information is the search range in the output information of each of the plurality of sensors. It is information indicating the result of tracking the object based on the detection result of the object detected from inside.

An object detection device according to an aspect of the present invention is tracking information indicating a tracking result of an object, which is output from each of a plurality of object detection devices and is tracked based on a plurality of detection results, and has a common coordinate system. The object is detected from the output information of the sensor based on the tracking information expressed by.

It should be noted that a computer program that implements the above-described devices, object tracking system, or object tracking method by a computer, and a computer-readable storage medium that stores the computer program are also included in the scope of the present invention.

According to the present invention, an object can be tracked with higher accuracy.

It is a functional block diagram which shows an example of a functional structure of the object tracking device which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the outline of the whole structure of the object tracking system which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the process which matches a target and a tracker. It is a functional block diagram which shows an example of a functional structure of the detection part of the object tracking device which concerns on the 1st Embodiment of this invention. It is a functional block diagram which shows an example of a functional structure of the object detection part in the detection part of the object tracking device which concerns on the 1st Embodiment of this invention. It is a functional block diagram which shows an example of a functional structure of the integrated tracking part of the object tracking device which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the sequential tracking process of the object which the integrated tracking part which concerns on the 1st Embodiment of this invention performs. It is a flow chart which shows an example of the flow of the object tracking processing of the object tracking device concerning a 1st embodiment of the present invention. It is a functional block diagram which shows an example of a functional structure of the object tracking device which concerns on the 2nd Embodiment of this invention. It is a functional block diagram which shows an example of a functional structure of the detection part of the object tracking apparatus which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating the application example of the object tracking apparatus which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating the application example of the object tracking apparatus which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating the application example of the object tracking apparatus which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating the application example of the object tracking apparatus which concerns on the 2nd Embodiment of this invention. It is a functional block diagram which shows an example of a functional structure of the detection part of the object tracking device which concerns on the 3rd Embodiment of this invention. It is a functional block diagram which shows an example of a functional structure of the object detection part in the detection part of the object tracking apparatus which concerns on the 3rd Embodiment of this invention. It is a functional block diagram which shows an example of a functional structure of the integrated tracking part of the object tracking apparatus which concerns on the 4th Embodiment of this invention. It is a figure for demonstrating the batch tracking process of the object which the integrated tracking part which concerns on the 4th Embodiment of this invention performs. It is a figure which illustrates the hardware constitutions of the computer (information processing apparatus) which can implement each embodiment of the present invention.

<First Embodiment>
A first embodiment of the present invention will be described in detail with reference to the drawings. First, with reference to FIG. 2, an overall configuration of an object tracking system (also simply called a system) of the present invention will be described. FIG. 2 is a diagram showing an example of a schematic overall configuration of the object tracking system 1 according to the present embodiment. As shown in FIG. 2, the object tracking system 1 according to the present embodiment includes an object tracking device 10, a plurality of cameras (20-1 to 20-N (N is a natural number)), and one or more display devices 30. Equipped with. In addition, in the present embodiment, when the plurality of cameras (20-1 to 20-N) are not distinguished from each other or are collectively referred to, they are referred to as a camera 20.

The object tracking device 10, the camera 20, and the display device 30 are communicatively connected to each other via a network 40. The display device 30 may not be included in the object tracking system 1. Further, the display device 30 may be directly connected to the object tracking device 10 without passing through the network 40.

The camera 20 functions as a sensor that detects an object. In the present embodiment, the case where the camera 20 is used as a sensor for detecting an object will be described as an example, but the present invention is not limited to this. The sensor is not limited to the camera, and may be a radio wave sensor or the like as long as the position can be measured. Alternatively, a sensor in which a plurality of sensors are mixed, such as a sensor in which a radio wave sensor and a camera are integrated, may be used. By using the camera 20 as a sensor, the object tracking device 10 can more appropriately acquire visual information such as color.

In this embodiment, the information acquired by the sensor will be described as an image captured by the camera.However, when the sensor is a radio wave sensor, the information acquired by the sensor is acquired by the radio wave sensor. It is a radio wave.

The object tracking device 10 is a device that tracks an object included in a video image captured by each of the plurality of cameras 20. The functional configuration of the object tracking device 10 will be described with reference to the drawings.

The display device 30 displays the tracking result of the object by the object tracking device 10. The display device 30 may display an image captured by the camera 20. Further, the display device 30 may display other information such as flow line information.

(Object tracking device 10)
Next, the function of the object tracking device 10 will be described. FIG. 1 is a functional block diagram showing an example of the functional configuration of the object tracking device 10 according to the present embodiment. As shown in FIG. 1, the object tracking device 10 includes a plurality of detection units (100-1 to 100-N) and an integrated tracking unit 200. In addition, in this Embodiment, when not distinguishing each of a some detection part (100-1-100-N), or when collectively calling, these are called the detection part 100.

(Detection unit 100)
The detection unit 100 outputs tracking information output from the camera 20 based on the tracking information output from the integrated tracking unit 200 described below, which is the tracking information of the object in the frame preceding the target frame for detecting an object (object). To detect. Here, in the present embodiment, the output information of the camera 20 indicates video data indicating a video image captured by the camera 20.

In the present embodiment, it is assumed that the plurality of detection units 100 and the plurality of cameras 20 are associated with each other one-on-one. For example, the detection unit 100-1 detects an object from the video image captured by the camera 20-1, and the detection unit 100-2 detects the object from the video image captured by the camera 20-2. Note that the present embodiment is not limited to this, and for example, the detection unit 100-1 may detect an object from the video imaged by the camera 20-N.

Further, the camera 20 and the detection unit 100 may not be associated with each other on a one-to-one basis. For example, the detection unit 100-1 may detect an object from the video captured by each of the plurality of cameras 20.

The operation of the detection unit 100 will be described below. The detection unit 100 receives, from the camera 20, video data indicating a video captured by the camera 20 (hereinafter, referred to as camera video). In FIG. 1, video data indicating a video image captured by the camera 20-n (n is 1 to N) is referred to as a camera video (n). Here, the camera image may be an image captured by the camera 20 such as a surveillance camera in real time, or the image captured by the camera 20 may be temporarily stored in a storage unit (not shown) and then stored. It may be decrypted (or reproduced) in. This video data includes time information indicating the time of shooting.

Further, the detection unit 100 receives the tracking information of the object in the previous frame from the integrated tracking unit 200. The previous frame may be the frame immediately before the frame (current frame) in which the object is to be detected, or may be the frame a predetermined number before the current frame. The number of the previous frame may be one or more. When the detecting unit 100 detects an object in the first frame, the detecting unit 100 does not receive (use) the tracking information of the object in the previous frame because the tracking information of the object in the previous frame does not exist.

The detection unit 100 uses the received camera image and the tracking information of the object in the previous frame to detect an object from the camera image (referred to as object detection). As described above, when the object detection is performed on the first frame, the detection unit 100 performs the detection without using the tracking information of the object in the previous frame. In addition, below, the detected object is called a target. That is, the object detection result (also referred to as the object detection result or simply the detection result) is a set of targets.

The object detection result includes, for example, information indicating the position of the target and information indicating the size of the target for each target. Specifically, the object detection result is, for example, information on the circumscribed rectangle of the area occupied by the target (target area), the coordinate value of the center of gravity of the target area, and the information indicating the width of the target in the frame in the video in which the object is detected. , And includes information indicating the height of the target. The object detection result is not limited to this. For example, the object detection result may include the coordinate value of the uppermost end or the coordinate value of the lowermost end of the target area instead of or in addition to the coordinate value of the center of gravity of the target area. The object detection result may include information indicating the position and size of the target for each target.

Note that in the present embodiment, the object detection result will be described by taking as an example that, for each target, the coordinate value of the lowermost end of the target and information indicating the circumscribed rectangle of the target are included. The coordinate value at the lowermost end of the target indicates the coordinate value of the point where the object contacts the floor surface (ground) and/or the coordinate value of the middle point of the lower side of the circumscribed rectangle of the object. Further, the coordinate value at the lowermost end of the target may be the coordinate value at the feet when the object is a person.

Then, the detection unit 100 converts the coordinate value included in the object detection result into the coordinate value in the common coordinate system defined in the space (shooting space) captured by the plurality of cameras 20, and the converted coordinate. The value is the object detection result.

Further, the object detection result may include information indicating the shape of the target in addition to the above-mentioned information. That is, the object detection result may include silhouette information representing the target area. Here, the silhouette information is information for distinguishing pixels inside and outside pixels of the target area, for example, image information in which the inside pixel value is set to 255 and the outside pixel value is set to 0, or the MPEG information. It is a value obtained by extracting a shape descriptor (shape feature amount) as standardized in −7 from the silhouette shape. Further, the object detection result may include a feature amount of the appearance of the object. For example, the object detection result may include a feature amount such as a color, a pattern, or a shape of the object.

Furthermore, the object detection result may include information (likelihood information of the target) describing the likelihood indicating the likelihood (accuracy) of the object detection for each target. The likelihood information of the target is information necessary for calculating the likelihood of the target, and is related to the accuracy of the object detection such as the score value at the time of detecting the object, the distance of the detected object from the camera, and the size. Information. Further, the detection unit 100 may calculate the likelihood of the target itself and use the calculated likelihood as the likelihood information of the target.

Then, the detection unit 100 outputs the object detection result to the integrated tracking unit 200.

(Integrated tracking unit 200)
The integrated tracking unit 200 receives the detection result output from each of the detection units 100. Then, the integrated tracking unit 200 tracks the object based on each detection result. Specifically, the integrated tracking unit 200 uses the object detection results for one or a plurality of objects detected by the detection units 100 from the images captured by the cameras 20 linked to the detection units 100, respectively. Track objects (do object tracking). Then, the integrated tracking unit 200 generates a tracking result (object tracking result) of the object expressed in the common coordinate system. In this way, the integrated tracking unit 200 integrates the object detection results detected by the respective detection units 100 from the video imaged by the camera 20 associated with the detection unit 100, and performs object tracking. Therefore, the object tracking performed by the integrated tracking unit 200 is also referred to as object integrated tracking.

Hereinafter, information for each object generated as an object tracking result will be referred to as a tracker. That is, it is assumed that the tracker includes information indicating the position of the tracked object, a motion model of the object, etc. as the information of the tracked object (object tracking result), but the present invention is not limited to this. is not. Since the tracked position of the object is the position of the object before (past) the current time, it is also referred to as the past position of the object.

That is, object tracking can be regarded as a process of associating objects between frames by associating a target detected by object detection with a tracker generated before the detection of this target. This object tracking will be described with reference to FIG. FIG. 3 is a diagram for explaining a process of associating a target with a tracker by the integrated tracking unit 200. As shown in FIG. 3, the number of targets is M and the number of trackers is K (M and K are integers of 0 or more). The integrated tracking unit 200 associates the M targets with the K trackers. When associating the target with the tracker, the integrated tracking unit 200 first predicts the current position of the object from the past position of the object indicated by the information included in the tracker, and represents the relationship between the target and the tracker. Correspondence is made using the index.

That is, the integrated tracking unit 200 predicts the position of the object on the current frame based on the position of the object detected in the previous frame and the motion model of the object calculated and held for each tracker. As this method, various existing methods such as a method using a Kalman filter or a method using a particle filter can be used.

Then, the integrated tracking unit 200 associates the object tracking result (tracker) in the previous frame with the object (target) included in the detection result, for example, based on the information listed in (1) to (3) below.
(1) The closeness of the distance between the position of the object predicted by the tracker on the current frame and the position of the target (2) Similarity of the appearance feature amount between the target and the object whose tracking result is shown by the tracker Gender (3) Likelihood of each target and tracker The process of this association can be reduced to a cost minimization problem of a bipartite graph as shown in FIG. Therefore, the integrated tracking unit 200 can solve this problem using an algorithm such as the Hungarian method.

In FIG. 3, a case where the target and the tracker are associated with each other is shown by using an arrow. That is, the top target is associated with the top tracker.

Then, when there is a target that does not correspond to the tracker, the integrated tracking unit 200 determines whether the target can be regarded as a newly appearing object. Then, when the integrated tracking unit 200 determines that there is a high possibility that the target has newly appeared, the integrated tracking unit 200 newly adds a tracker related to the target. In FIG. 3, it is assumed that the target indicated by reference sign m (referred to as target m) is a target that does not correspond to the tracker. At this time, the integrated tracking unit 200 determines whether the target m can be regarded as a newly appearing object, and if it can be regarded, newly creates a tracker related to the target m.

On the other hand, if there is a tracker that does not correspond to the target, the integrated tracking unit 200 determines whether or not the tracker is information regarding an object that has disappeared from the shooting space. Then, the integrated tracking unit 200 deletes the tracker when there is a high possibility that the tracker is information regarding an object that has disappeared from the shooting space. In FIG. 3, it is assumed that the tracker indicated by reference numeral k (called a tracker k) is a tracker that does not correspond to the target. At this time, the integrated tracking unit 200 determines whether or not the tracker k is information regarding the disappeared object, and if the tracker k is information regarding the disappeared object, deletes the tracker k.

The integrated tracking unit 200 performs object tracking by repeating these processes on a frame-by-frame basis. The integrated tracking unit 200 gives the tracker a unique ID (identifier) common to all the cameras 20, and manages the tracking result (tracker) by this ID. Further, the integrated tracking unit 200 includes a value (hereinafter referred to as a tracker's likelihood (weight)) that evaluates the certainty of the tracking result in the tracker as a parameter of the tracker. The position of the newest object, which is the position indicated by the information indicating the position of the tracked object included in the tracker, is called the position of the tracker. The size of the object at this time is also called the size of the tracker.

Further, the integrated tracking unit 200 updates the information indicating the position of each tracker, the information on the likelihood of the tracker, and the like based on the result of the association. The information on the position of the tracker is information expressed in a common coordinate system defined in the shooting space where the plurality of cameras 20 shoot. The information expressed in this common coordinate system is the coordinate value in the common coordinate system. The coordinate value in the common coordinate system is, for example, a coordinate system indicating the position of the floor in the real world when the plurality of cameras 20 are cameras installed in a certain store. On the other hand, the coordinate system unique to each camera 20 is called the individual coordinate system of the camera 20. This individual coordinate system is a coordinate system on a captured image of the camera 20. Hereinafter, the position information expressed in the common coordinate system will be described as coordinate values in the common coordinate system. Further, the position information represented by the individual coordinate system of the camera 20 will be described as the coordinate value of the individual coordinate system of the camera 20.

Then, the integrated tracking unit 200 generates a tracker whose information is updated based on the result of the association, as a new object tracking result. Then, the integrated tracking unit 200 includes the information indicating the position and/or the size of the tracker, the information about the likelihood of the tracker, and the like in the generated object tracking result (tracker), and the information indicating the tracking result of the object tracking. Output as (tracking information).

The tracking information includes coordinate values of the common coordinate system of each tracker as information indicating the position of each tracker. This tracking information is fed back to the detection unit 100. That is, the detection unit 100 receives the tracking information and uses the tracking information for object detection for the subsequent frames.

The integrated tracking unit 200 may be configured to output the object tracking result including the tracking information and other information of the tracker to the detection unit 100.

In this way, the object tracking device 10 according to the present embodiment uses the images captured by each of the plurality of cameras and integrates the object detection results for the images captured by each of the plurality of cameras 20. , Do object tracking. Then, the object tracking device 10 feeds back the obtained tracking information to the object detection in the next frame.

In this way, the object tracking device 10 detects the object from the video using the tracking result for the previous frame. For example, if there is an object that cannot be seen by a certain camera 20 but is seen by another camera 20, the object tracking device 10 also obtains a tracking result for an object that cannot be seen by a certain camera 20 from the object in the image of this certain camera 20. Used for detection. Accordingly, the detection unit 100 can preferably detect the same object when the same object appears in the range that can be seen from the certain camera 20. Therefore, the object tracking device 10 can accurately track an object for this object.

Therefore, the object tracking device 10 can improve the detection accuracy of the object as compared with the case where the tracking result for the previous frame is not used. Further, since the object tracking device 10 performs object tracking using all the detection results with high detection accuracy, the accuracy of the tracking result obtained as a whole is also improved.

(Details of the detection unit 100)
Next, the functions of the respective units of the object tracking device 10 will be described in more detail with reference to FIGS. 4 to 8. FIG. 4 is a functional block diagram showing an example of a more detailed functional configuration of the detection unit 100 of the object tracking device 10 according to this embodiment. As shown in FIG. 4, the detection unit 100 includes an object detection unit 110, a common coordinate conversion unit (second conversion unit) 120, and an individual coordinate conversion unit (first conversion unit) 130. In FIG. 4, the camera image (n) (n is 1 to N) received by the detection unit 100 is simply referred to as a camera image.

The individual coordinate conversion unit 130 receives the tracking information output from the integrated tracking unit 200 from the integrated tracking unit 200. Then, the individual coordinate conversion unit 130 converts the coordinate value of the common coordinate system of each tracker included in this tracking information into the coordinate value on the frame captured by each camera 20 (that is, in the individual coordinate system unique to each camera 20). Converted to expressed coordinate values). When the coordinate value of the common coordinate system is (X, Y, Z) and the coordinate value of the individual coordinate system of the camera 20 is (x, y), the individual coordinate conversion unit 130 indicates that the coordinate of the common coordinate system of the tracker. From the value (X, Y, Z), the coordinate value (x, y) of the individual coordinate system of the camera 20 associated with the detection unit 100 including the individual coordinate conversion unit 130 is obtained. At this time, it is preferable that the individual coordinate conversion unit 130 obtains at least camera parameters representing the camera position, orientation, etc. of the camera 20 linked to the detection unit 100 by calibration. As a result, the individual coordinate conversion unit 130 uses the obtained camera parameters to convert the coordinate values of the common coordinate system into the coordinate values of the individual coordinate system of the camera 20.

The camera parameters may be stored in a storage unit (not shown) or the like in the detection unit 100, or may be stored in a storage area in the individual coordinate conversion unit 130. In the latter case, the individual coordinate conversion unit 130 may be configured to supply the camera parameters to the common coordinate conversion unit 120.

For example, it is assumed that the object is a person and the information indicating the position of the tracker is the coordinate value indicating the foot position and the coordinate value indicating the crown position of the person. Then, the coordinate value of the foot position and the coordinate value of the crown position are respectively (X0, Y0, 0) and (X0, Y0, H) (H represents the height of the person). The camera 20 associated with the detection unit 100 including the individual coordinate conversion unit 130 is assumed to be the camera 20-1.

At this time, the individual coordinate conversion unit 130 uses the camera parameters for the camera 20-1 to determine the foot position (x0, y0) and the crown position (x1, y1) on the frame captured by the camera 20-1. Ask. If the information indicating the position of the tracker includes information indicating the circumscribing rectangle, the value indicating the width of the circumscribing rectangle is previously obtained by converting the coordinate value into the common coordinate system using the camera parameter. ing. Therefore, the individual coordinate conversion unit 130 may use the value converted using the camera parameter again as the width of the circumscribed rectangle.

In addition, not all the objects whose tracking results are shown by the tracker can be seen from one camera 20, and may exist outside the field of view of this camera 20. Therefore, when the information indicating the position of the tracker included in the tracking information is the information related to the object that cannot be seen from the camera 20 associated with the detection unit 100, the individual coordinate conversion unit 130 causes the individual coordinate conversion unit 130 to determine the individual coordinate system of the camera 20 of the object. The coordinate value of cannot be obtained. Therefore, the individual coordinate conversion unit 130 may not convert the coordinate value of the tracker's common coordinate system regarding the object that is not visible outside the angle of view of the camera 20 into the coordinate value of the individual coordinate system. At this time, the individual coordinate conversion unit 130 registers the range of coordinate values of the common coordinate system visible to each camera 20 in advance in a storage unit or the like (not shown) for each camera 20 so that each object can be stored therein. It may be determined whether or not there is. In addition, the individual coordinate conversion unit 130 actually converts the coordinate values into the individual coordinate system to obtain an abnormal value indicating the outside of the area monitored by the associated camera 20, or the value is obtained. If there is none, it may be determined that the object whose coordinate value has been converted is an object that cannot be seen by the camera 20.

Then, in the individual coordinate conversion unit 130, the coordinate value of the common coordinate system of the tracker included in the tracking information output from the integrated tracking unit 200 becomes the coordinate value of the individual coordinate system of the camera 20 linked to the detection unit 100. The converted result (tracking information) is output to the object detection unit 110. That is, the individual coordinate conversion unit 130 converts the tracking information expressed in the common coordinate system into the tracking information expressed in the individual coordinate system, and outputs the converted tracking information to the object detection unit 110. Hereinafter, when simply described as “coordinate value of individual coordinate system”, the coordinate value of the individual coordinate system of the camera 20 linked to the detection unit 100 is shown.

The object detection unit 110 receives a camera image from the camera 20 associated with the detection unit 100 including the object detection unit 110. Further, the object detection unit 110 receives the tracking information converted into the coordinate value of the individual coordinate system from the individual coordinate conversion unit 130. Then, the object detection unit 110 detects an object from the received camera image based on the tracking information.

Then, the object detection unit 110 generates a detection result. The object detection unit 110 outputs the generated detection result to the common coordinate conversion unit 120. The detection result is a detection result expressed in the individual coordinate system.

The configuration of the object detection unit 110 will be described in more detail with reference to FIG. FIG. 5 is a functional block diagram showing an example of the functional configuration of the object detection unit 110 according to this embodiment. As shown in FIG. 5, the object detection unit 110 includes a recognition-type object detection unit (first object detection unit) 111 and a search range setting unit 112.

The search range setting unit 112 receives the tracking information converted into the coordinate values of the individual coordinate system from the individual coordinate conversion unit 130. Then, the search range setting unit 112 uses the tracking information converted into the coordinate values of the individual coordinate system to obtain the area (search range) to be the target of detecting the object in the current frame. That is, the search range setting unit 112 predicts the position of the object in the current frame based on the tracking information that is converted into the coordinate values of the individual coordinate system and that is the tracking result of the previous frame. Then, the search range setting unit 112 obtains the detection range for searching the object from the predicted position. The search range is also called an object detection range.

Here, the tracking information received by the search range setting unit 112 is a result of object tracking in a past frame (also referred to as a past tracking result) when viewed from the time of a frame to be currently processed. Therefore, the search range setting unit 112 predicts the movement of each object and predicts the position of each object in the current frame. Hereinafter, the predicted position of the object will be referred to as a predicted position. Then, the search range setting unit 112 sets the vicinity of this predicted position as the search range of the object.

The search range setting unit 112 preferably predicts the movement of each object by using the movement model of each object calculated from the past tracking result. For example, the search range setting unit 112 determines that the object is stationary when the position of the object has not changed in the tracking result of the past several frames (may be two frames), and the object obtained by the tracking result is determined. The position of is directly used as the predicted position. Further, the search range setting unit 112 assumes that the object is moving at a constant speed when the object moves according to the tracking results of the past several frames, and considers the time difference from the past frame to predict the predicted position. May be asked.

The tracking result of the past several frames used when the search range setting unit 112 predicts the movement of each object may be included in the tracking information. Further, the motion model of the object obtained from the tracking result of the past several frames may be included in the tracking information.

In addition, this predicted position may be included in the tracking information. That is, the integrated tracking unit 200 may include the value obtained by the Kalman filter or the particle filter at the time of object tracking as the predicted position in the tracking information.

Further, for example, a new object may appear at the angle of view of the camera 20 in the outer edge portion of the range in which the camera 20 can shoot. Also, when the place where the camera 20 is shooting includes a place such as a doorway, a new object may appear in the angle of view of the camera 20. Therefore, it is preferable that the search range setting unit 112 also includes these areas (outer edge portion and/or entrance/exit portion of the frame) in the frame included in the image captured by the camera 20 in the object search range. ..

The search range setting unit 112 outputs information indicating the set search range of the object (search range information) to the recognition-type object detection unit 111.

The recognition-type object detection unit 111 receives the search range information from the search range setting unit 112. The recognition-type object detection unit 111 detects an object from the camera image input to the recognition-type object detection unit 111 based on the received search range information. The recognition-type object detection unit 111 temporarily stores the input camera video frame in a storage unit such as a buffer in the recognition-type object detection unit 111, and when receiving the search range information, applies this information. Perform object detection processing. Specifically, the recognition-type object detection unit 111 performs object detection on an area (search range) indicated by the search range information using a classifier that has learned the image characteristics of the object.

For example, when the object is a person, the recognition-type object detection unit 111 detects the person by applying a discriminator that has learned a characteristic part of the person (for example, the head or the upper half of the body). Further, the recognition-type object detection unit 111 may use a classifier that has learned the entire person as the classifier. The recognition-type object detection unit 111 can use various types of discriminators. For example, the recognition-type object detection unit 111 can use a discriminator obtained by learning images of the head, the upper half of the body, the whole body of a person, and the like with a CNN (Convolutional Neural Network). In addition, the recognition-type object detection unit 111 performs feature extraction such as HOG (Histogram Of Gaussian), SVM (Support Vector Machine; support vector machine), or GLVQ (Generalized Learning Vector Quantization) generalization learning; Alternatively, a discriminator such as the above may be used. In addition to the above, the recognition-type object detection unit 111 can use various existing recognition-based detection methods.

In this way, the recognition-type object detection unit 111 according to the present embodiment detects an object within the search range set by the search range setting unit 112. That is, the recognition type object detection unit 111 performs object detection within the search range narrowed down by the search range setting unit 112 using the tracking result in the previous frame. Therefore, the recognition-type object detection unit 111 does not detect an object in a range in which an object is unlikely to exist in a frame, and thus it is possible to reduce extra erroneous detection. Further, the recognition-type object detection unit 111 can speed up the process of object detection.

In addition, the search range in which the recognition-type object detection unit 111 performs object detection may include not only the area indicated by the search range information but also the area determined by the silhouette information obtained by the background difference or the like. Further, the recognition-type object detection unit 111 may set a common part of the area defined by the silhouette information and the area specified by the object search range information as an area (search range) in which the object detection is executed.

Then, the recognition-type object detection unit 111 generates a result (detection result) of object detection, and outputs the detection result to the common coordinate conversion unit 120. At this time, the coordinate value of the object included in the detection result is the coordinate value of the individual coordinate system.

Returning to FIG. 4, the function of the common coordinate conversion unit 120 of the detection unit 100 will be described. The common coordinate conversion unit 120 receives the object detection result expressed in the individual coordinate system from the object detection unit 110. Then, the individual coordinate conversion unit 130 converts the coordinate value of the individual coordinate system included in the received detection result into the coordinate value of the common coordinate system. Thereby, the common coordinate conversion unit 120 can generate information for integrating the detected positions of the objects with respect to the individual cameras 20.

Specifically, the common coordinate transformation unit 120 uses the camera parameters of the camera 20 associated with the detection unit 100 including the common coordinate transformation unit 120 to share the coordinate values of the individual coordinate system included in the object detection result. Convert to coordinate values in the coordinate system. For example, when the coordinates of the lower end of the object on the frame captured by the camera 20 are (x0, y0), the common coordinate conversion unit 120 converts the coordinates into (X0, Y0, 0) which is the coordinates of the common coordinate system. To do. Here, since the ground is a plane of Z=0, the component in the Z-axis direction is 0. Further, when the coordinates of the upper end of the object are (x1, y1) (here, it is assumed that the upper end of the object is directly above the lower end of the object (vertically upward)), the height of the object is H. At this time, the common coordinate conversion unit 120 converts the coordinates (x1, y1) of the upper end of this object into the coordinates (X0, Y0, H) of the common coordinate system. The common coordinate transformation unit 120 finds the height of the object by searching for H that satisfies this. In this way, the common coordinate transformation unit 120 obtains coordinate values (X, Y, Z) in the common coordinate system for each detected object.

When H is known, the common coordinate conversion unit 120 may use the known value as it is.

The common coordinate conversion unit 120 outputs the detection result including the coordinate values after coordinate conversion (coordinate values of the common coordinate system) to the integrated tracking unit 200. That is, the common coordinate conversion unit 120 outputs the detection result expressed in the common coordinate system to the integrated tracking unit 200. The common coordinate conversion unit 120 provides silhouette information and information such as the feature amount of the appearance feature (color, pattern, shape, etc.) of the object for each of the one or more targets (objects) included in the detection result. It may be included as information about the object. Then, the common coordinate conversion unit 120 may output the detection result including these pieces of information to the integrated tracking unit 200.

(Details of integrated tracking unit 200)
Next, the functional configuration of the integrated tracking unit 200 will be described in more detail with reference to FIG. FIG. 6 is a functional block diagram showing an example of a more detailed functional configuration of the integrated tracking unit 200 of the object tracking device 10 according to this embodiment. As illustrated in FIG. 6, the integrated tracking unit 200 includes a prediction unit 210, a storage unit 220, an associating unit 230, and an updating unit 240. The integrated tracking unit 200 in the present embodiment also sequentially calls the video from each camera 20 on a camera-by-camera basis, and is therefore also called a sequential tracking unit.

Sequential object tracking (also called object sequential tracking and sequential integrated tracking) performed by the integrated tracking unit 200 will be described with reference to FIG. 7. FIG. 7 is a diagram for explaining an object sequential tracking process performed by the integrated tracking unit 200 according to the present embodiment. FIG. 7 shows an example of timings at which images are acquired by each of the cameras A, B, and C when the number of cameras is three. In FIG. 7, the horizontal axis indicates the time axis, and the right side indicates that the time is later. As shown in FIG. 7, camera A acquires images at times t1, t5, and t8. Similarly, camera B acquires images at times t2, t4, t6, and t9, and camera C acquires images at times t3 and t7.

As shown in FIG. 7, the time (time stamp) of the frame (image) acquired by each camera 20 is not necessarily the same for all the cameras 20, and is usually different. In addition, the frame interval may be different for each camera 20, and the same camera 20 may be uneven.

Then, the detection unit 100 performs detection in time order from the camera images output asynchronously between the cameras 20, and outputs the detection result to the integrated tracking unit 200.

The integrated tracking unit 200 according to the present embodiment sequentially executes the tracking process in order from the image of the earliest time. That is, in the case of FIG. 7, the integrated tracking unit 200 first uses the object detection result of the image of the camera A for the time t1 to perform integration between a plurality of cameras to perform object tracking. After that, the integrated tracking unit 200 uses the object detection result in the order of the time t2 of the camera B, the time t3 of the camera C, the time t4 of the camera B,... And do object tracking. At this time, not all the cameras 20 are visible to all cameras 20, so the integrated tracking unit 200 performs object tracking on the objects that are likely to be viewed by each camera 20.

Returning to FIG. 6, each unit of the integrated tracking unit 200 will be described.

The storage unit 220 stores tracker information associated with an object (target) included in the detection result received by the integrated tracking unit 200. The tracker information stored in the storage unit 220 is managed by the update unit 240 using the tracker ID. The tracker information is, for example, information about an object whose tracking result is included in the tracker, parameters including the likelihood of the tracker, and the like, but the present invention is not limited to this. The information about the object includes information indicating a past position of the object, a motion model of the object, and the like, but the present invention is not limited thereto. The information about the object is included in the above-described object detection result. Information may be included.

In FIG. 6, the storage unit 220 is described as an example incorporated in the integrated tracking unit 200, but the present invention is not limited to this. The storage unit 220 may be provided in the object tracking device 10 separately from the integrated tracking unit 200. The storage unit 220 may be realized by a storage device or the like that is separate from the object tracking device 10.

When the storage unit 220 is not built in the integrated tracking unit 200, the storage unit 220 may be configured to store data or the like used in the object tracking device 10. For example, the storage unit 220 may store, for example, camera images captured by the cameras 20, camera parameters of the cameras 20, ranges of coordinate values of the common coordinate system visible by the cameras 20, and the like.

The prediction unit 210 refers to the storage unit 220 and predicts the position of the object on the current frame. Specifically, the prediction unit 210 predicts the current position of the object based on the motion model of the object using the tracking result (tracker) of the object in the previous frame. Here, the information indicating the position of the object is expressed in the common coordinate system.

The motion model of the object used by the prediction unit 210 to predict the position may be stored in the storage unit 220, or the prediction unit 210 may use the tracking result to predict the position of the object. It may be calculated before the execution.

For the prediction of the position of the object by the prediction unit 210, for example, a prediction process such as Kalman filter or particle filter can be applied. In addition, the prediction unit 210 simply calculates the speed of the object from the results of tracking several times in the past, and, assuming a uniform linear motion, predicts the amount of movement from the position in the previous frame from the speed to predict the previous frame. The current position may be predicted by adding to the position at.

Then, the prediction unit 210 outputs the prediction result to the association unit 230.

The associating unit 230 receives the detection result output from each of the detecting units 100. In addition, in FIG. 6, the detection result (n) (n is 1 to N) indicates the detection result output from the detection unit 100-n. The associating unit 230 also receives the prediction result from the prediction unit 210. Then, the associating unit 230 refers to the storage unit 220 and associates the target included in the detection result with the tracker using the prediction result.

The associating unit 230 obtains the combination with the highest accuracy as a whole associating. The likelihood that a certain target m is associated with a certain tracker k is obtained by multiplying the likelihoods Pm and ηk of the target m and the tracker k, respectively, and the likelihood qkm indicating the possibility that they are the same object. become. Therefore, the associating unit 230 calculates this value for each pair of the target and the tracker, and finds the maximum combination as a whole.

Here, the likelihood of the target (first likelihood) is a value that represents the probability (probability) of object detection. The accuracy of object detection depends on the size of the object to be detected (called a detected object) on the screen (on the frame), the distance from the camera 20 to the detection position of the object, the appearance of the object from the camera 20 and the like. To do.

For example, when the detected object is small and the size of the detected object is close to the limit of the size that can be detected, the accuracy of object detection is low. Further, when the size of the detected object deviates from the apparent size of the object assumed by the camera parameter, the accuracy of the object detection becomes low. In addition, if the detection position of the object is far from the camera 20 or if the illumination condition for the area where the object is present is poor and the object is difficult to detect, the accuracy of the object detection is low. In addition, the accuracy of object detection is low even when the data used for learning of the discriminator is significantly different from the actual appearance (for example, the angle is different).

The associating unit 230 calculates the likelihood of the target by reflecting such characteristics. Specifically, the associating unit 230 calculates the likelihood of the target using the likelihood information of the target included in the detection result received from the detecting unit 100. When the detecting unit 100 calculates the likelihood of the target and includes the calculated likelihood as the likelihood information of the target in the detection result, the associating unit 230 calculates the likelihood included in the likelihood information of the target. May be used as is. Note that the likelihood of the target does not have to reflect all the items described here, and may reflect only major factors.

The tracker's likelihood (second likelihood) is a value representing the likelihood (probability) of object tracking. The accuracy of object tracking changes depending on the tracking result of object tracking in the previous frame. For example, in the tracking results up to the previous (past) frame of the current frame, it can be said that the tracker that is surely associated with the target has high object tracking accuracy, and the tracker that is not well associated with the target tracking accuracy is Can be said to be low. Therefore, the associating unit 230 may change the likelihood based on the result of whether or not the target and the tracker are associated with each other in each frame. When the association is established, the likelihood of the tracker is increased and the association is performed. If not, the tracker's likelihood should be lowered.

Further, at this time, when the position of the tracker is far from the camera 20, it is considered that the error of the position of the tracker becomes large. As a result, such a tracker becomes hard to correspond to the object (target) included in the detection result. Therefore, the associating unit 230 may change the ratio for changing the likelihood of the tracker according to the distance between the tracker and the camera 20. Further, the associating unit 230, when the camera 20 looks at the object whose tracking result is shown by the tracker, according to the angle (depression angle or elevation angle) formed by the horizontal plane including the camera 20 and the line-of-sight direction, You may change the ratio which changes likelihood.

For example, when an object whose tracking result is shown by a tracker (hereinafter referred to as a tracker object) is close to the camera 20 and the depression angle of the camera 20 with respect to the object is larger than a predetermined angle, the accuracy of the position of the object is high. Therefore, the tracker and the target are easily associated with each other. Therefore, in such a case, the associating unit 230 increases the ratio of changing the likelihood of the tracker.

Further, for example, when the object of the tracker is far from the camera 20 and the depression angle of the camera 20 with respect to the object is shallower than a predetermined angle, the size of the object on the frame captured by the camera 20 becomes small. In addition, a slight positional shift on the image causes a large shift in the real space. Therefore, the accuracy of the detection position of the object is likely to be low. Therefore, in such a case, the association unit 230 reduces the ratio of changing the likelihood of the tracker. In this way, the association unit 230 calculates the likelihood of the tracker.

As described above, since the associating unit 230 can preferentially reflect the detection result detected by the camera 20 close to the object of the tracker to the likelihood of the tracker, it is possible to improve the tracking accuracy as a whole. Note that it is not necessary to reflect all the items described here in the likelihood of the tracker, and only the main factors may be reflected.

Further, the likelihood qkm indicating the identity between the target m and the tracker k indicates the probability that they are the same. When the object indicated by the target and the object of the tracker are the same object, the positions of the target and the object of the tracker are likely to be close to each other. Therefore, the association unit 230 changes the likelihood according to the distance between the target and the object of the tracker. That is, the associating unit 230 may increase the value of the likelihood qkm when the distance between the target m and the object of the tracker k is short, and decrease the value of the likelihood qkm when the distance is far. ..

At this time, when the target m is far from the camera 20 or the depression angle of the camera 20 with respect to the target m is shallower than a predetermined angle, the accuracy of the position of the target m is likely to be low. Therefore, the associating unit 230 does not calculate the distance using the simple Euclidean distance, but uses the Mahalanobis distance in consideration of the error (ambiguity) included in the detected position of the target, and the target and the object of the tracker. The distance between them may be obtained. In addition to the above method, the associating unit 230 may control the degree of change in the likelihood qkm depending on the distance, in consideration of ambiguity. That is, when the ambiguity is larger, the associating unit 230 reduces the change in the likelihood qkm according to the distance between the target m and the object of the tracker k. As a result, the associating unit 230 reduces the influence of the displacement of the target on the association.

Further, the associating unit 230 may also consider the similarity in appearance of the target and the tracker. That is, the associating unit 230 may extract features such as the color, pattern, and shape of the target and tracker objects, evaluate the similarity between them, and obtain the likelihood qkm.

For example, the associating unit 230 may calculate the color histogram of the object for both the target and the tracker objects, evaluate the similarity between them by overlapping the color histograms, and reflect the similarity in the likelihood qkm. .. Like the tracker's likelihood ηk and the target's likelihood Pm, the likelihood qkm indicating the identity of the target and the tracker does not need to reflect all the above-mentioned items, and only major factors should be reflected. You may

Further, the associating unit 230 may calculate each of the above likelihoods in consideration of the case where the object is not detected due to being out of the angle of view of the camera 20 or occluded by another object. Thereby, the integrated tracking unit 200 can track the object with high accuracy even when the object is not detected or goes out of the angle of view.

As described above, the problem of calculating each likelihood and finding the correspondence between the target and tracker that maximizes each likelihood as a whole is that each likelihood is converted into a cost by a monotonic non-increasing function and used. , Which results in the lowest cost allocation problem (which target is associated with which tracker). This allocation problem can be efficiently calculated by a method such as the Hungarian method.

Then, the associating unit 230 outputs the result of the association to the updating unit 240. The result of this association includes information indicating which target is associated with the tracker and each of the above-mentioned likelihoods including at least the likelihood of the tracker.

In addition, in the present embodiment, the association unit 230 performs association using both the likelihood of the target and the likelihood of the tracker, but uses either one of the likelihoods. Correlation may be performed.

The update unit 240 updates the tracker information. Then, the updating unit 240 generates this tracker as a new object tracking result. Specifically, the updating unit 240 receives the association result from the associating unit 230. Then, the updating unit 240 calculates the current position of the tracker object based on this result. Then, the updating unit 240 updates the tracker information stored in the storage unit 220. The information to be updated is, for example, parameters such as the position and/or size of the object whose tracking result is included in the tracker, the motion model of the object, and the likelihood of the tracker, but the present invention is not limited to this. Not a thing. The update unit 240 may update the updated information among the information stored in the storage unit 220.

First, the calculation of the current position of the tracker object by the update unit 240 will be described. The update unit 240 calculates the current position of the tracker object in consideration of the accuracy of the target position. For example, it is assumed that the update unit 240 weights the predicted position where the position of the object is predicted using the tracker and the detected position of the target associated with the tracker, and calculates the current position of the object of the tracker. .. In this case, the updating unit 240 may control the weight according to the accuracy of the position of the target.

For example, when the target is located away from the camera 20 and the depression angle of the camera 20 with respect to the target is shallow, the accuracy of the position of this target is likely to be low. In such a case, the updating unit 240 reduces the weight for the position of this target.

On the other hand, when the target is close to the camera 20 and the depression angle with respect to the camera 20 target is larger than a predetermined value, it is assumed that the accuracy of the position of this target is high. In such a case, the updating unit 240 increases the weight for the position of this target.

The update unit 240 calculates the current position of the tracker object using the predicted position and the position for which the weight is set.

In this way, the updating unit 240 determines the weight for the position of the target, so that the prediction result of the detected position of the object by the camera 20 close to the target is more strongly reflected. Therefore, the object tracking device 10 can improve the prediction accuracy of the position of the object.

Then, the updating unit 240 updates the latest position of the object included in the information about the object stored in the storage unit 220 to the calculated current position of the object.

Next, the updating of the likelihood of the tracker performed by the updating unit 240 will be described.

When a tracker object is located far from the camera 20, the size of the object is reduced. Therefore, the detection unit 100 has difficulty detecting such an object.

Further, a case will be described in which the recognition-type object detection unit 111 of the detection unit 100 performs recognition-type object detection when the object included in the frame has a different appearance from the object used for learning. When the object included in the frame has a different appearance from the object used for learning, for example, the depression angle of the camera 20 with respect to the object of the tracker, which is assumed from the position of the object, and the object used for learning are used. This is a case where the depression angle of the camera 20 with respect to the object is significantly different. In such a case, the recognition-type object detection unit 111 of the detection unit 100 becomes difficult to detect the object included in the frame.

In a situation where it is difficult to detect such an object, the object included in the frame may be undetected. In this case, there may not be a target associated with the tracker associated with this object.

Therefore, the update unit 240 suppresses the change in the likelihood of the tracker of the object in which the object is difficult to be detected among the trackers not associated with the target.

In this way, the update unit 240 can suppress the influence on tracking when an object is difficult to be detected, and can greatly reflect the detection result of the easily detected camera on the likelihood of the tracker. Then, when the object is tracked for the next frame, the associating unit 230 performs the matching based on the likelihood of this tracker, so that the integrated tracking unit 200 can further improve the accuracy of the object tracking. ..

Then, of the tracker likelihoods stored in the storage unit 220, the tracker likelihood calculated by the associating unit 230 and the tracker likelihood in which the above change is suppressed are updated.

Next, updating of the motion model of the tracker object by the updating unit 240 will be described. For example, a case will be described in which the integrated tracking unit 200 performs object tracking by predicting the position of an object using a Kalman filter. In this case, the updating unit 240 substitutes the position coordinates of the object of the tracker associated with the target as the detected position coordinates into the update formula of the state variable of the Kalman filter stored in the storage unit 220 to determine the state of the Kalman filter. To update.

In addition to the above, the update unit 240 updates the other parameters of the tracker stored in the storage unit 220.

For example, the object itself may change its posture. For example, when the object is a person, the apparent height of the object changes as the person crouches or bends. As described above, when the size of the object or the like is changed in addition to the motion model, the updating unit 240 updates the changed information in the information stored in the storage unit 220.

Further, when the tracker parameters include the probability of existence of the tracker object, the weight indicating the reliability of the tracking result, and the like, the weight changes according to the result of the association by the associating unit 230. Therefore, the updating unit 240 updates the parameters such as the weight.

The updating unit 240 also generates and deletes a tracker as a tracker update. First, the updating unit 240 determines whether or not there is a target that is not associated with the tracker after the association process by the association unit 230. The target that does not correspond to this tracker may be an object newly appearing within the range captured by the camera 20. Therefore, when there is a target that does not correspond to the tracker, the update unit 240 determines whether this target can be regarded as an object newly appearing within the range.

That is, when there is a target that does not correspond to the tracker, the update unit 240 evaluates the probability that this target exists. Then, the updating unit 240 determines whether or not this probability is equal to or higher than a predetermined value. Then, when this probability is equal to or higher than a predetermined value, the updating unit 240 determines that the object indicated by this target is an object newly appearing within the range. Then, the updating unit 240 newly creates a tracker related to the object (target) that is determined to be an object newly appearing within the range.

Further, the update unit 240 determines whether or not there is a tracker that does not correspond to the target. A tracker that does not correspond to this target may be a tracker for an object that has disappeared (moved from within the range) to within the range captured by the camera 20. Therefore, when there is a tracker that does not correspond to the target, the update unit 240 determines whether or not the target can be regarded as the tracker related to the object disappeared from the range.

That is, when there is a tracker that does not correspond to the target, the update unit 240 evaluates the probability that an object related to this tracker exists. Then, the updating unit 240 determines whether this probability is lower than a predetermined value. Then, when this probability falls below a predetermined value, the update unit 240 determines that the object related to this tracker is an object that has disappeared from the range. Then, the updating unit 240 deletes the tracker related to the object determined to be the object that has disappeared from the range.

The probability that an object exists is determined by the likelihood of the tracker. That is, when there is a tracker that does not correspond to the target, the update unit 240 reduces the likelihood of this tracker. Then, when the likelihood value of the tracker falls below a predetermined threshold value, the updating unit 240 deletes the tracker.

Then, the updating unit 240 generates the finally remaining tracker as the tracking result of the object in this frame. Then, the updating unit 240 outputs information indicating the position of the tracker, information regarding the size of the object on the tracker, and the like as information (tracking information) indicating the tracking result of object tracking.

As described above, according to the object tracking device 10 according to the present embodiment, object tracking is performed in order from the oldest time information included in the camera image output from each camera 20. The object detected from the video data of each of the plurality of cameras 20 is detected by integrating the detection results with high accuracy for each camera 20 and using the detection result and the tracking result in which the past tracking result is reflected. Therefore, the tracking accuracy of the object tracking performed by the integrated tracking unit 200 of the object tracking device 10 is improved.

Next, the flow of the object tracking process of the object tracking device 10 according to the present embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing an example of the flow of object tracking processing of the object tracking device 10 according to the present embodiment.

As shown in FIG. 8, first, the recognition-type object detection unit 111 of the detection unit 100 receives a camera image from the camera 20 associated with the detection unit 100 including the object detection unit 110 (step S81).

The detection unit 100 confirms whether the frame of the received camera image is the first frame output from the camera 20 (step S82), and if the frame is the first frame (YES in step S82), the process is performed. Proceed to S85.

When the frame of the received camera image is not the first frame (NO in step S82), the individual coordinate conversion unit 130 expresses the tracking information for the previous frame output from the integrated tracking unit 200 in the individual coordinate system. It is converted into the tracking information (step S83).

Then, the search range setting unit 112 of the object detection unit 110 uses the tracking information converted in step S83 to set the search range of the object for the current frame (step S84).

Then, the recognition-type object detection unit 111 detects an object from the received camera image (step S85).

Next, the individual coordinate conversion unit 130 converts the detection result obtained by the recognition-type object detection unit 111 into the detection result expressed in the common coordinate system (step S86).

Next, the prediction unit 210 of the integrated tracking unit 200 predicts the position of the object on the current frame using the tracker information (step S87).

Then, the associating unit 230 associates the object (target) included in the detection result with the tracker (step S88).

Next, the update unit 240 updates the tracker information such as the position of the tracker object and the motion model of the object (step S89).

Further, the updating unit 240 generates and/or deletes the tracker (step S90). Then, the object tracking device 10 repeats this process until no frame is input to the detection unit 100.

(effect)
As described above, the object tracking device 10 according to the present embodiment can track an object with higher accuracy. This is because the detection unit 100 detects an object from the output information of the camera 20 based on the tracking information for the output information before the output information (video frame). Then, the integrated tracking unit 200 tracks the object based on the plurality of detection results output by each detection unit 100, and generates tracking information of the object expressed in the common coordinate system.

For example, if there is an object that cannot be seen by a certain camera 20 but is seen by another camera 20, the object tracking device 10 also obtains a tracking result for an object that cannot be seen by a certain camera 20 from the object in the image of this certain camera 20. Used for detection. Accordingly, the detection unit 100 can preferably detect the same object when the same object appears in the range that can be seen from the certain camera 20. Therefore, the object tracking device 10 can accurately track an object for this object.

Therefore, the object tracking device 10 can improve the detection accuracy of the object as compared with the case where the tracking result for the previous frame is not used. Further, since the object tracking device 10 performs object tracking using all the detection results with high detection accuracy, the accuracy of the tracking result obtained as a whole is also improved.

In this way, the object tracking device 10 according to the present embodiment can extract the flow line of a person or the like that moves across the region photographed by each of the plurality of cameras 20. As a result, the tracking result by the object tracking device 10 can be used as basic information for marketing or changing the layout of the store by analyzing the behavior of the customer who travels around the store, for example. Further, this tracking result can be used for detection of a person who wanders between areas for security purposes.

Further, since the search range setting unit 112 uses the tracking result to set the search range for detecting an object in the camera image, the recognition-type object detection unit 111 can reduce extra false detection. Further, the recognition-type object detection unit 111 can speed up the process of object detection.

Further, the integrated tracking unit 200 performs object tracking using the likelihood of the target and/or the likelihood of the tracker, so that the object tracking device 10 can obtain a more reliable object tracking result. Further, since the object tracking device 10 searches for an object using the object tracking result obtained in this way, it is possible to further improve the accuracy of object tracking. Thereby, the object tracking device 10 can improve the accuracy of object tracking as a whole.

<Second Embodiment>
Next, a second embodiment of the present invention will be described with reference to the drawings. For convenience of description, members having the same functions as the members included in the drawings described in the above-described first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

The object tracking system 2 according to the present embodiment is configured to include an object tracking device 50 instead of the object tracking device 10 of the object tracking system 1 according to the first embodiment described with reference to FIG. The other system configuration of the object tracking system 2 is the same as that of the object tracking system 1 shown in FIG.

(Object tracking device 50)
The function of the object tracking device 50 will be described with reference to FIG. FIG. 9 is a functional block diagram showing an example of the functional configuration of the object tracking device 50 according to the present embodiment. As shown in FIG. 9, the object tracking device 50 includes a plurality of detection units (100-1 to 100-N), an integrated tracking unit 200, and a display control unit 300. In addition, like this 1st Embodiment mentioned above, in this Embodiment, when not distinguishing each of several object detection part (100-1-100-N), or when collectively calling, these are these. The detection unit 100 is called.

The display controller 300 controls an image (video) displayed on the display device 30. Specifically, the display control unit 300 converts the tracking information output by the integrated tracking unit 200 into data that can be displayed on the display device 30, generates display data, and transmits the display data to the display device 30. The tracking information output by the integrated tracking unit 200 is expressed in a common coordinate system. Therefore, the display control unit 300 generates display data that can be displayed on the display device 30 in the common coordinate system.

At this time, the integrated tracking unit 200 uses the information necessary for displaying the flow line of the object on the display device 30 (for example, the information indicating the past position of the object of the tracker) as the tracking information, and the display control unit 300. It is preferable to output to. This tracking information may be the same as or different from the information fed back to the detecting unit 100.

Then, the display device 30 displays the received display data on the screen. As a result, the object tracking device 50 can present the tracking result to the user.

Further, the detection unit 100 of the object tracking device 50 according to the present embodiment generates display data by converting the search range set by the search range setting unit 112 of the object detection unit 110 into data that can be displayed on the display device 30. Then, it is transmitted to the display device 30. The search range information output by the search range setting unit 112 is expressed in an individual coordinate system. Therefore, the display control unit 300 uses the camera parameters of the camera 20 associated with the detection unit 100 that has output the search range information to generate display data that can be displayed on the display device 30 in the individual coordinate system.

Then, the display device 30 displays the received display data on the screen. Thereby, the object tracking device 50 can present the search range to the user by using the search range information of the object output from each detection unit 100.

The display device 30 may be plural. For example, the display device 30 receives the display data displayed in the common coordinate system and the display data displayed in the individual coordinate system by different display devices 30, and displays the received display data on the screen. It may be configured. Further, the display device 30 may be configured to divide a display area of one screen and display a plurality of display data on the screen. As described above, the display data display method in display device 30 according to the present embodiment is not particularly limited.

Further, the display control unit 300 may be provided in each of the detection units 100. FIG. 10 is a functional block diagram showing an example of the functional configuration of the detection unit 100 of the object tracking device 50 according to this embodiment. As shown in FIG. 10, the detection unit 100 includes an object detection unit 110, a common coordinate conversion unit 120, an individual coordinate conversion unit 130, and a display control unit 150. The object detection unit 110 also includes a recognition-type object detection unit 111 and a search range setting unit 112, similar to the object detection unit 110 shown in FIG.

The search range setting unit 112 illustrated in FIG. 10 outputs search range information indicating the set search range to the display control unit 150. The display control unit 150 receives the search range information output from the search range setting unit 112 and, like the display control unit 300, generates display data converted into data that can be displayed on the display device 30. The search range information output by the search range setting unit 112 is expressed in an individual coordinate system. Therefore, the display control unit 150 uses the camera parameters of the camera 20 associated with the detection unit 100 including the display control unit 150 to generate display data that can be displayed on the display device 30 in the individual coordinate system.

Then, the display control unit 150 transmits the generated display data to the display device 30. Then, the display device 30 displays the received display data on the screen.

(Application example)
An application example of the object tracking device 50 according to the present embodiment will be described with reference to FIGS. 11 to 14. 11 to 14 are diagrams for explaining application examples of the object tracking device 50 according to the present embodiment.

First, FIG. 11 is a diagram showing an example of the room in which the shelf R1 and the shelf R2, and the room in which the plurality of cameras (A to F) are installed are viewed from the direction opposite to the gravity direction. As shown in FIG. 11, the horizontal direction of FIG. 11 is the X axis in the common coordinate system, and the vertical direction is the Y axis. The rack R1 and the rack R2 are arranged side by side on the X axis so that the longitudinal direction is parallel to the Y axis direction.

The camera A is installed at a position close to the doorway of this room. In the present embodiment, it is assumed that the interior of this room is photographed by cameras A to F. That is, as shown in FIG. 11, the indoor space in which the plurality of cameras (A to F) are installed becomes a shooting space. Further, the cameras A to F are photographing the common place.

FIG. 12 is a diagram showing an example of an image captured by each of the camera A and the camera B. The upper diagram of FIG. 12 is a diagram showing a certain frame of a video image captured by the camera A, and the lower diagram is a diagram showing a certain frame of a video image captured by the camera B. The coordinate values in these frames are represented by the coordinate values in the individual coordinate system for each camera.

Note that the object tracking device 50 in the present embodiment may be configured to display the image captured by the camera 20 on the display device 30.

As shown in FIG. 12, the image captured by the camera A includes the person C1. Further, since the camera A is installed near the doorway, this image includes the doorway. In addition, the image captured by the camera B includes the person C1 and the person C2.

When viewed from the camera A, the person C2 is hidden behind the shelf R1. Therefore, at the time of the image in FIG. 12, the person C2 is an object invisible to the camera A. If the frames of these images are the first frames of the images captured by the cameras A and B, since there is no tracking information in the previous frame, the object tracking device 50 detects the object from these frames and performs tracking. Generate information.

Then, the search range setting unit 112 uses the tracking information expressed in the individual coordinate system to set the search range of the object for the next frame in the video image captured by the camera A. Similarly, the search range setting unit 112 uses the tracking information expressed in the individual coordinate system to set the search range of the object in the next frame of the image captured by the camera B.

FIG. 13 is a diagram showing an example of the search range displayed on the display device 30. The upper diagram of FIG. 13 is a diagram showing an example of an object search range with respect to a frame output from camera A, and the lower diagram is an example of an object search range with respect to a frame output from camera B. FIG.

As shown in the upper diagram of FIG. 13, the search range setting unit 112 obtains the search range A1 from the position of the person C1 in the upper diagram of FIG. Further, the search range setting unit 112 obtains the area of the entrance/exit of the room as the search range N1. Further, the search range setting unit 112 finds the outer edge of the frame as the search ranges N2 and N3. Then, the search range setting unit 112 outputs the information obtained by summarizing the obtained search ranges A1, N1 to N3 as search range information to the recognition-type object detection unit 111 and the display control unit 150 or the display control unit 300. Then, the display control unit 150 or the display control unit 300 converts the search range indicated by the search range information on the display device 30 into display data that can be displayed on the screen, and transmits the display data to the display device 30.

The display device 30, which has received the display data from the display control unit 150 or the display control unit 300, displays the search range on the screen, as shown in the upper diagram of FIG. 13.

Next, the lower diagram of FIG. 13 will be described. As shown in the lower diagram of FIG. 13, the search range setting unit 112 obtains the search range B1 and the search range B2 from the positions of the person C1 and the person C2 in the lower diagram of FIG. 12, respectively. Further, the search range setting unit 112 obtains the outer edge portion of the frame as the search ranges N4 to N7. Then, the search range setting unit 112 outputs the information obtained by summarizing the obtained search ranges B1, B2, and N4 to N7 as search range information to the recognition-type object detection unit 111 and the display control unit 150 or the display control unit 300. Then, the display control unit 150 or the display control unit 300 converts the search range indicated by the search range information on the display device 30 into display data that can be displayed on the screen, and transmits the display data to the display device 30.

The display device 30, which has received the display data from the display control unit 150 or the display control unit 300, displays the search range on the screen, as shown in the lower diagram of FIG. 13.

Note that the display device 30 may display the search range so that the search range is different for each area. For example, the display device 30 may display the search range for the already detected object and the search range for the outer edge of the frame in mutually different colors.

Then, the integrated tracking unit 200 generates a tracker regarding the person C1 and the person C2 detected in the subsequent frame. Then, the integrated tracking unit 200 outputs the information necessary for displaying the flow lines of the person C1 and the person C2 to the display control unit 300 as the tracking information.

Upon receiving the tracking information from the integrated tracking unit 200, the display control unit 300 converts the tracking information into display data that can be displayed on the display device 30, and transmits the display data to the display device 30.

Then, the display device 30 displays the display data received from the display control unit 300 on the screen. FIG. 14 is a diagram showing an example when the display device 30 displays, on a screen (display screen), flow lines indicating the tracking results of the person C1 and the person C2. As shown in FIG. 14, in this application example, it is assumed that the display screen displays the object tracking result on the XY plane in the common coordinate system. In FIG. 14, the flow line of the person C1 is displayed as a solid line, and the flow line of the person C2 is displayed as a one-dot chain line. In this way, the display device 30 can display the tracking result of the object on the screen.

<Third Embodiment>
Next, a third embodiment of the present invention will be described with reference to the drawings. For convenience of description, members having the same functions as the members included in the drawings described in the above-described first and second embodiments are designated by the same reference numerals, and the description thereof will be omitted.

The object tracking device 10 according to the present embodiment is configured to include a detection unit 400 instead of the detection unit 100 of the object tracking device 10 shown in FIG. The configuration of the detection unit 400 will be described with reference to FIG. FIG. 15 is a functional block diagram showing an example of the functional configuration of the detection unit 400 of the object tracking device 10 according to this embodiment.

The detection unit 400 includes an object detection unit 140 instead of the object detection unit 110 of the detection unit 100 shown in FIGS. 4 and 5. Further, the detection unit 400 is configured to further include the storage unit 160. That is, the detection unit 400 according to the present embodiment includes an object detection unit 140, a common coordinate conversion unit 120, an individual coordinate conversion unit 130, and a storage unit 160, as shown in FIG.

In the present embodiment, a configuration including an object detection unit 140 instead of the object detection unit 110 of the detection unit 100 of the object tracking device 10 according to the first embodiment will be described as an example. The present invention is not limited to this, and the object detection unit 140 may be provided instead of the object detection unit 110 of the detection unit 100 of the object tracking device 50 according to the second embodiment. That is, the detection unit 100 according to the present embodiment may be configured to output the data to be displayed to the display control unit 150 or the display control unit 300.

The storage unit 160 stores camera parameters for each camera 20, which are used when the coordinate system is converted. Further, the storage unit 160 is used when the individual coordinate conversion unit 130 confirms whether or not the coordinate value of the common coordinate system included in the tracking information is included in the range photographed by the camera 20 associated with it. Information indicating the range of coordinate values of the common coordinate system is stored. Further, the storage unit 160 may store a video image captured by the camera 20. The video may be temporarily stored.

In addition, in FIG. 15, the storage unit 160 is described as an example that is incorporated in the detection unit 400, but the present invention is not limited to this. The storage unit 160 may be provided in the object tracking device 10 separately from the detection unit 400. Further, the storage unit 160 may be realized by a storage device or the like separate from the object tracking device 10.

Next, a detailed functional configuration of the object detection unit 140 of the detection unit 400 will be described with reference to FIG. FIG. 16 is a functional block diagram showing an example of the functional configuration of the object detection unit 140 of the detection unit 400 according to this embodiment. As shown in FIG. 16, the object detection unit 140 includes a recognition-type object detection unit (first object detection unit) 141, a non-recognition-type object detection unit (second object detection unit) 142, and a detection parameter update unit 143. And a detection result integration unit 144.

In the present embodiment, object detection using a dictionary (identifier) or the like is called “recognition-type object detection”. On the other hand, object detection that does not use a discriminator or the like is called “non-recognition type object detection”.

The recognition-type object detection unit 141 detects an object from the camera image input to the recognition-type object detection unit 141. The recognition-type object detection unit 141 detects an object in the entire frame. Note that the recognition-type object detection unit 141 may perform object detection based on the search range information output from the detection parameter update unit 143 described below, as indicated by the broken line in FIG. At this time, the recognition type object detection unit 141 performs object detection by the same method as the recognition type object detection unit 111 described in the first embodiment.

Further, when the search range information is not output from the detection parameter update unit 143, the recognition-type object detection unit 141 does not perform object detection for the entire frame, but may perform object detection using another criterion. Good. For example, the recognition-type object detection unit 141 may use the silhouette information to detect an object only in a region having a silhouette and a region surrounding the region.

The recognition-type object detection unit 141 outputs the detection result of object detection as the first detection result to the detection result integration unit 144.

In addition, the recognition-type object detection unit 141 may extract the appearance feature of the object at this point in preparation for the object detection by the non-recognition-type object detection unit 142 described later. The appearance feature of the object includes information such as the color, pattern, and shape of the object, but the present invention is not limited to this. The recognition-type object detection unit 141 extracts these feature quantities as appearance features of the object. At this time, the area used for object detection by the recognition-type object detection unit 141 and the area used for object detection by the non-recognition-type object detection unit 142 do not have to be the same. For example, when the object is a person, the recognition type object detection unit 141 detects the head, and the non-recognition type object detection unit 142 detects the object up to the clothing area. At this time, the recognition type object detection unit 141 extracts the feature amount of the appearance feature of the object so as to include the area of the clothes. Then, the recognition-type object detection unit 141 may output the extracted feature amount as template information together with information indicating an area used for extraction (extracted area information). Further, the recognition-type object detection unit 141 may hold the feature amount itself inside the recognition-type object detection unit 141 and output only the information for identifying the feature amount.

The detection parameter update unit 143 receives the tracking information represented by the individual coordinate system from the individual coordinate conversion unit 130. Then, the detection parameter update unit 143 uses this tracking information to obtain a parameter (referred to as a detection parameter) necessary for object detection. The detection parameters are parameters necessary for the object detection processing. The detection parameters include, for example, the predicted position (predicted area) of the object in the current frame, the search range to which the object detection is applied, the size of the template used for template matching, the feature amount of the target template previously associated with the tracker (template Information) etc. are included. It should be noted that the detection parameter does not have to include all of this information as long as it includes the parameters necessary for the non-recognition-type object detection unit 142 to detect an object. Further, the detection parameter may include target information associated with the tracker in the tracking result of the object in the previous frame.

For example, the detection parameter updating unit 143 detects the position of the object in the current frame as the predicted position for the target (object) detected in the previous frame and associated with the tracker, based on the tracking information of the object. Ask. This prediction process is the same as the prediction position prediction process in the search range setting unit 112 according to the first embodiment. The detection parameter updating unit 143 may obtain the area including the predicted position as the predicted area.

Further, for example, the detection parameter updating unit 143 may obtain a range to which the object detection by the template matching is applied, centered on the prediction region, and include this range as the search range of the objects included in the detection parameter.

The detection parameter updating unit 143 obtains the detection parameter for each object included in the tracking information. Then, the detection parameter updating unit 143 updates the obtained detection parameter as a detection parameter used in the object detection process. Then, the detection parameter update unit 143 outputs this detection parameter to the non-recognition type object detection unit 142.

Note that the detection parameter updating unit 143 uses the tracking information converted into the coordinate values of the individual coordinate system, similarly to the search range setting unit 112 of the detection unit 100 according to the above-described first embodiment, to detect the object. The search range may be obtained. Then, the detection parameter updating unit 143 may output search range information indicating the obtained search range of the object to the recognition-type object detection unit 141.

The non-recognition type object detection unit 142 receives the detection parameter from the detection parameter update unit 143. Then, the non-recognition type object detection unit 142 detects an object from the camera image input to the non-recognition type object detection unit 142 based on the received detection parameter. Unlike the recognition-type object detection unit 141, the non-recognition-type object detection unit 142 detects an object based on the appearance similarity of the object detected in the previous frame.

That is, when the object is detected in the previous frame, the non-recognition type object detection unit 142 stores the image feature of the area (or the partial image of the detection area itself) as a template. Then, the non-recognition type object detection unit 142 detects an object by checking by template matching whether or not an area similar to the stored template exists in the current frame. As the characteristics of the image used at this time, for example, information on color patterns and distributions, distribution information on edges and luminance gradients, or characteristics formed by combining these can be used.

The detection parameter used when the non-recognition type object detection unit 142 detects an object is controlled by the detection parameter output from the detection parameter update unit 143. Specifically, the non-recognition type object detection unit 142 performs object detection by template matching on the predicted position (predicted region) of the object predicted by the detection parameter updating unit 143 and its vicinity. That is, the non-recognition type object detection unit 142 sets the search range of template matching centering on the predicted object existence range (prediction area), and performs template matching on the periphery thereof. At this time, the non-recognition type object detection unit 142 may also consider that the apparent size of the object changes due to the movement of the position of the object. This change can be calculated by using camera parameters. Therefore, the non-recognition type object detection unit 142 may perform template matching after calculating the change in the size of the object and reflecting it in the template.

Further, the information of the template for which the unrecognized object detection unit 142 performs template matching may be the feature amount extracted by the recognized object detection unit 141 in the object detection process in the previous frame.

As described above, the non-recognition-type object detection unit 142 performs object detection based on the tracking result for the object tracked by the integrated tracking unit 200, and thus the detection accuracy is improved as compared with the case where the tracking result is not used. Can be made.

Then, the non-recognition type object detection unit 142 outputs the detection result of the object detection as the second detection result to the detection result integration unit 144.

The detection result integration unit 144 receives the first detection result from the recognition-type object detection unit 141. Further, the detection result integration unit 144 receives the second detection result from the non-recognition type object detection unit 142. Then, the detection result integration unit 144 integrates the first detection result and the second detection result. Then, the detection result integration unit 144 outputs the integrated result to the common coordinate conversion unit 120 as a detection result of object detection by the object detection unit 140.

There may be an object included in both the first detection result and the second detection result, and an object included in only one of them. Therefore, the detection result integration unit 144 associates and integrates the objects included in each of the first detection result and the second detection result. For this association, for example, the degree of overlapping of object areas can be used.

That is, the detection result integration unit 144 calculates the overlapping ratio of the object regions (for example, the overlapping ratio of the object circumscribing rectangles), and when the overlapping ratio is larger than a predetermined value, the object included in the first detection result and the first The object included in the detection result of No. 2 is associated.

In addition, the detection result integration unit 144 may formulate a graph problem with the overlapping ratio of regions between objects as a weight, and associate the objects with each other. For example, the detection result integration unit 144 associates the objects by converting the overlap ratio into a cost by a monotonous non-increasing function and then calculating the optimal association using the Hungarian method or the like.

As a result of the association, the detection result integration unit 144 may merge at this time a value (for example, an overlapping ratio or a cost) used in the association that is larger than a predetermined value. Further, the detection result integration unit 144 may generate the information indicating that the correspondences are not merged at this point. Then, the detection result integration unit 144 outputs, as the detection result of the object detection unit 140, the detection result obtained by combining the first detection result and the second detection result with the information that the correspondence is associated, You may make it track using the information of an association at the time of integrated tracking.

In addition, the non-recognition type object detection unit 142 outputs the second detection result based on the tracking result of the object with respect to the previous frame. Therefore, the second detection result may be generated later than the first detection result. In such a case, the detection result integration unit 144 temporarily stores the first detection result in the storage unit such as a buffer in the detection result integration unit 144 or the storage unit 160. Then, the detection result integration unit 144 may integrate both results at the time when the second detection result for the frame corresponding to the frame for which the first detection result is generated is received.

As described above, the detection unit 400 of the object tracking device 10 according to the present embodiment uses the result (first detection result) of object detection by the recognition-type object detection unit 141 and the object by the non-recognition-type object detection unit 142. The result obtained by integrating the detection result (second detection result) is output as the detection result. At this time, the non-recognition type object detection unit 142 detects the object by performing template matching based on the tracking result for the object tracked by the integrated tracking unit 200. As a result, the detection unit 400 can further improve the accuracy of object detection as compared with the case where only object detection by identifying an object (recognition-type object detection) is performed.

Therefore, the object tracking device 10 can track the object with higher accuracy.

<Fourth Embodiment>
Next, a fourth embodiment of the present invention will be described with reference to the drawings. For convenience of description, members having the same functions as the members included in the drawings described in the above-described embodiments are designated by the same reference numerals, and the description thereof will be omitted.

The object tracking device 10 according to the present embodiment is configured to include an integrated tracking part 500 instead of the integrated tracking part 200 of the object tracking device 10 shown in FIG. The configuration of the integrated tracking unit 500 will be described with reference to FIG. FIG. 17 is a functional block diagram showing an example of the functional configuration of the integrated tracking unit 500 of the object tracking device 10 according to this embodiment. As shown in FIG. 17, the integrated tracking unit 500 includes a buffer unit 510, a prediction unit 210, a storage unit 220, an associating unit 530, and an updating unit 240.

In the present embodiment, a configuration including an integrated tracking unit 500 instead of the integrated tracking unit 200 of the object tracking device 10 according to the first embodiment will be described as an example. It should be noted that the present invention is not limited to this, and an integrated tracking unit 500 may be provided instead of the integrated tracking unit 200 of the object tracking device 50 according to the second embodiment. That is, the integrated tracking unit 500 according to the present embodiment may be configured to output the data to be displayed to the display control unit 300.

Further, the detection unit that outputs the detection result to the integrated tracking unit 500 according to the present embodiment may be the detection unit 400 described in the third embodiment.

The buffer unit 510 is a unit that temporarily stores the detection result expressed by the common coordinate system output from the detection unit 100. Then, among the data (detection result) buffered in the buffer unit 510, the data in which the time information included in the detected camera image is within the predetermined period is acquired by the associating unit 530. This predetermined period is a periodic period. Then, the associating unit 530 performs object tracking using one or a plurality of detection results acquired in a certain cycle. As described above, the integrated tracking unit 500 according to the present embodiment performs object tracking using images from a plurality of cameras among images from each camera 20, and is therefore also referred to as a collective tracking unit.

Object tracking (also called collective integrated tracking) performed by the integrated tracking unit 500 will be described with reference to FIG. FIG. 18 is a diagram for explaining the batch tracking process of objects performed by the integrated tracking unit 500 according to the present embodiment. Similar to FIG. 7, FIG. 18 illustrates an example of the timings at which images are acquired by each of the cameras A, B, and C when the number of cameras is three. In FIG. 18, the horizontal axis indicates the time axis, and the closer to the right, the later in time. As shown in FIG. 18, camera A acquires images at times t1, t5, and t8. Similarly, camera B acquires images at times t2, t4, t6, and t9, and camera C acquires images at times t3 and t7.

Then, objects are sequentially detected from the images acquired at each of these timings. In the following, for convenience of description, it is assumed that the time shown in FIG. 18 is substantially the same as the time when the detection result of the object is output. That is, it is assumed that the time t1 is the time when the detection result for the frame of the video image captured by the camera A is output from the detection unit 100 and buffered in the buffer unit 510.

The lowermost time axis in FIG. 18 shows an example of a periodic period.

The associating unit 530 acquires, from the one or more detection results buffered in the buffer unit 510, a detection result in which the time information included in the camera image that is the object detection target is within a predetermined period. As described above, this predetermined period is a periodic period. Further, in the present embodiment, it is assumed that the buffered time and the camera image time are the same. Therefore, it can be said that the associating unit 530 acquires one or a plurality of detection results buffered in the buffer unit 510 at a predetermined cycle.

Specifically, the associating unit 530 first acquires the detection result buffered in the first period T1. That is, the associating unit 530 acquires the detection results buffered at the times t1, t2, and t3. The detection result buffered at time t1 is the detection result for the frame of the video image captured by the camera A. The detection result buffered at time t2 is the detection result for the frame of the image captured by camera B, and the detection result buffered at time t3 is the detection result for the frame of the image captured by camera C. The result. Therefore, the associating unit 530 is a detection result buffered in the buffer unit 510 within a predetermined period (T1 in this case), and detects a plurality of detection results for frames of video images captured by each of the plurality of cameras 20. From the buffer unit 510. Then, the associating unit 530 uses the acquired detection result to perform object tracking.

Similarly, the associating unit 530 also obtains the buffered detection results within this periodic period in the periods T2, T3, and T4, and performs object tracking.

In the present embodiment, a configuration will be described in which associating unit 530 acquires data (a plurality of detection results) buffered in buffer unit 510 from buffer unit 510 at a predetermined cycle, but associating unit 530. May be configured to receive this data from the buffer unit 510 at a predetermined cycle. That is, the buffer unit 510 may have a function of outputting this data to the associating unit 530 at a predetermined cycle.

Returning to FIG. 17, the associating unit 530 of the integrated tracking unit 500 will be further described.

The associating unit 530 obtains the distance between the targets from the positions of the targets included in the acquired detection results, and associates the targets having the short distances with each other. At this time, the associating unit 530 uses the distance between the targets and makes the association by a method such as the Hungarian method. In addition to the distance between the targets, the associating unit 530 may also use the similarity of the appearance features of the targets at the same time. For example, targets that are close in position and have similar colors are likely to be the same object. Therefore, the associating unit 530 may make the association using such a feature. The feature for determining the similarity of the appearance features is not limited to the color, and may be, for example, the pattern of the target.

Then, the associating unit 530 integrates the detection results of the targets associated with each other. That is, the association unit 530 obtains the position of the object by using the detection result of each of the associated targets after performing the association between the targets. At this time, the associating unit 530 may evaluate the likelihood of each target and/or the accuracy of the predicted position, and use the position with the highest accuracy as the position of the object.

The associating unit 530 also weights the position of each target based on the accuracy of the predicted position determined by the angle (depression angle or elevation angle) of the camera 20 with respect to each target and the distance from the camera 20 to the target. May be. Then, the associating unit 530 may calculate a statistic such as an average value from the weighted positions, and use the position indicated by the calculated statistic as the position of the object.

Then, the associating unit 530 sets the obtained position of the object as the position of the target with respect to the cycle in which the detection result is acquired. The associating unit 530 uses the position of this target to perform the association in the same manner as the associating unit 230 according to the first embodiment. Further, the association processing and the subsequent processing by the integrated tracking unit 500 are the same as the processing by the integrated tracking unit 200 described in the first embodiment, and thus the description thereof will be omitted.

Further, the associating unit 530 may associate and integrate each target with the tracker before performing the association between the targets. That is, when there are a plurality of targets associated with the same tracker, the associating unit 530 performs merge processing between these targets. In this case, the associating unit 530 may preferentially merge the targets with higher likelihood and/or accuracy of the predicted position of each target to perform the merging. In this way, the associating unit 530 may evaluate the detection result based on these pieces of information and integrate the targets associated with the same tracker.

As described above, the object tracking device 10 according to the present embodiment performs object tracking using all the detection results for the camera images captured by each camera 20 within a predetermined period. As a result, the object tracking device 10 can easily apply a process of performing object tracking by prioritizing search results of objects among a plurality of cameras 20.

Further, for example, when the frame rates of all the cameras 20 are stable and the same, the frames of all the cameras 20 are included in the predetermined period by setting the predetermined period according to the frame interval. Therefore, the object tracking device 10 according to the present embodiment can perform object tracking on the frames for all the cameras 20. As a result, the object tracking device 10 can simultaneously evaluate the detection results for the objects that are simultaneously seen by the plurality of cameras 20, and thus the reliability of the detection results can be directly reflected in the tracking.

<Fifth Embodiment>
A fifth embodiment of the present invention will be described. In this embodiment, the minimum configuration that solves the problems of the present invention will be described.

The object tracking device 10 according to the present embodiment has the same configuration as the object tracking device 10 shown in FIG. 1 described in the first embodiment, and therefore will be described with reference to FIG.

As shown in FIG. 1, the object tracking device 10 according to the present embodiment includes a plurality of detection units (100-1 to 100-N) (N is a natural number) and an integrated tracking unit 200. In addition, in this Embodiment, when not distinguishing each of a some detection part (100-1-100-N), or when collectively calling, these are called the detection part 100.

Each of the plurality of detection units 100 detects an object from the output information output from the sensor. Although the sensor is a camera and the output information of the sensor is a camera image (image data) in FIG. 1, the sensor is not limited to the camera. Specifically, the detection unit 100 detects an object based on the tracking information output from the integrated tracking unit 200. The detection unit 100 outputs the detection result to the integrated tracking unit 200.

Based on a plurality of detection results output by each of the plurality of detection units (100-1 to 100-N), the integrated tracking unit 200 tracks each of one or a plurality of objects indicated by the detection results. .. Then, the integrated tracking unit 200 generates tracking information of the object expressed in the common coordinate system as the tracking result. Then, the integrated tracking unit 200 outputs to each of the plurality of detection units (100-1 to 100-N).

As described above, the detection unit 100 of the object tracking device 10 according to the present embodiment detects an object from the output information output from the sensor based on the tracking result for the object tracked by the integrated tracking unit 200.

In this way, the object tracking device 10 detects the object from the video by using the tracking result for the previous frame, so that it is possible to improve the detection accuracy of the object as compared with the case where the tracking result is not used. Further, since the object tracking is performed by using all the detection results of the objects in the images captured by the cameras 20, the object tracking device 10 can improve the tracking accuracy as compared with the case where the object tracking is performed for each camera independently. .. Further, since the object tracking device 10 performs object tracking using all the detection results with high detection accuracy, it is possible to track the object with higher accuracy.

In each of the above-described embodiments, the object tracking device 10 has been described as an example including the detection unit (100, 400) and the integrated tracking unit (200, 500), but the detection unit and the integrated tracking unit are described. And may be realized by separate devices. That is, the detection unit (100, 400) may be realized by a separate device as the object detection device, and the integrated tracking unit (200, 500) may be realized by a separate device. Also, the display control unit 300 may be realized by a display control device separate from the object tracking device 50. The display control device may be built in the display device 30.
<Sixth Embodiment>
A sixth embodiment of the present invention will be described. The object tracking device 10 according to the present embodiment has the same configuration as the object tracking device 10 shown in FIG. 1 described in the first embodiment, and therefore will be described with reference to FIG. Note that the object tracking device 10 according to the present embodiment has a configuration in which the functions described below are further added to the object tracking device 10 according to the first embodiment, but the present invention is not limited to this. It is not something that will be done. The object tracking device 10 according to the present embodiment is also applicable to the object tracking devices according to the second to fifth embodiments described above.

In the present embodiment, the integrated tracking unit 200 further acquires information regarding the appearance of the object and includes the acquired information in the tracking information. Then, the detection unit 100 controls the detection of the object by using the information regarding the appearance of each object included in the tracking information output from the integrated tracking unit 200.

The information regarding the appearance of the object (hereinafter, appearance information) is information regarding how each object looks when viewed from the position of each camera 20, and depends on the position of the object of each tracker. It is fixed.

For example, consider a case where an object is in front of another object (on the side of the camera 20) when an object and another object are viewed from a camera 20. In this case, the object on the back side (other object) is hidden by the object on the front side (a certain object), and there is a high possibility that it cannot be seen from the camera 20. At this time, the integrated tracking unit 200 includes information indicating such overlapping of objects as appearance information in a tracker (tracking result) related to another object, and outputs the tracking result.

Next, a specific operation of each unit of the object tracking device 10 according to the present embodiment will be described.

The integrated tracking unit 200 stores information about the arrangement of the cameras 20 in, for example, the storage unit 220 illustrated in FIG. The information regarding the arrangement of the cameras 20 is, for example, information indicating at which position each camera 20 is arranged, in which direction the camera 20 is photographing, and the like. Further, the integrated tracking unit 200 may include, as the information on the arrangement of the camera 20, information on the position and direction of the illumination in the shooting space, information on the characteristics of the illumination, and information on the lighting conditions such as which position in the shooting space is bright or dark. .. Further, the integrated tracking unit 200 may also hold direction information of the shooting space as information on the arrangement of the cameras 20.

Similar to the integrated tracking unit 200 according to each of the above-described embodiments, the integrated tracking unit 200 predicts the movement of the object indicated by each tracker on the current frame, and associates the target with the tracker to determine the position of the tracker. Ask for.

Then, the integrated tracking unit 200 predicts the position of the object indicated by each tracker on the captured image captured by each camera 20, based on the obtained position of the tracker and the information on the movement of each tracker.

Then, the integrated tracking unit 200 refers to the information on the arrangement of the cameras 20 and predicts how each object at the predicted position will look (appearance) when viewed from each camera 20. That is, the integrated tracking unit 200 predicts whether or not the objects indicated by the trackers overlap each other at each of the plurality of cameras 20 at the next shooting timing.

Then, when the integrated tracking unit 200 determines that the objects overlap each other on the captured image of a certain camera 20 based on the predicted appearance, it is possible that the objects may not overlap and are not visible. The information (visual appearance information) is generated.

For example, the integrated tracking unit 200 determines whether or not there is another predicted object between the line segment connecting the certain camera 20 and the predicted certain object. When another predicted object is between the line segments, there is a high possibility that this certain object will overlap with another object. Therefore, the integrated tracking unit 200 obtains information about hidden (overlapping) objects and the degree of overlap (likelihood) as appearance information for the certain object.

Then, the integrated tracking unit 200 may include the determination result as the appearance information in the tracking result of the certain object.

Then, the integrated tracking unit 200 includes the generated appearance information in the tracking information of the object that is likely to be invisible in the captured image captured by the certain camera 20. At this time, it is preferable that the appearance information includes information indicating the camera 20 in which the object is likely to be invisible.

Then, the integrated tracking unit 200 outputs the tracking information including the appearance information to each detection unit 100. The integrated tracking unit 200 may output the tracking information including the appearance information to the detection unit 100 associated with the camera 20 (a certain camera 20) in which the objects are highly likely to be invisible. Then, the integrated tracking unit 200 may output tracking information not including the appearance information to the object tracking device 10 associated with another camera 20.

When it is known that the way the light hits changes depending on the position of the object, and the hue and brightness of the object change, the integrated tracking unit 200 changes the appearance depending on the position of the object. The information describing the above may be included in the tracking information.

For example, when the way the light hits is determined by the position of the object, the integrated tracking unit 200 predicts the way the light hits from that position, and provides information such as brighter, darker, and changed tint for each tracker. May be included in the tracking information.

When the position of the illumination in the space where the object is arranged is known, the integrated tracking unit 200 determines that the shadow of the object or another object arranged in this environment is different from the relationship between the illumination and the object. It is determined whether or not it overlaps the object. Then, when there is a possibility that the shadows overlap, the integrated tracking unit 200 calculates the possibility (likelihood) that the shadows will overlap for the objects where the shadows overlap, and includes this in the tracking information.

Further, even when moving like the sun, the integrated tracking unit 200 obtains the position of the sun from the time and the information on the direction of the site, predicts the direction in which the shadow is formed, and gives it to the appearance of the object. You may consider the influence. For example, the integrated tracking unit 200 obtains the current position of the sun from the time information, and, together with the direction information, predicts in which direction the shadow will form. Then, the integrated tracking unit 200 may calculate the possibility (likelihood) of overlapping shadows when the shadows of other objects are likely to be included, and include them in the tracking information.

Next, the operation of the detection unit 100 will be described. The detection unit 100 detects an object based on the tracking information, as in each of the above-described embodiments. At this time, the detection unit 100 of the object tracking device 10 according to the present embodiment controls the detection of the object using the information regarding the appearance of each object included in the tracking information. Specifically, the detection unit 100 does not detect an object that is likely to be invisible because it overlaps with another object. For example, the detection unit 100 does not set the search range for the object that is likely to be invisible.

If the tracking information includes information that changes in brightness or color tone, the detection unit 100 may correct the effect of the illumination when performing the search and perform detection. For example, in a dark area, the detection unit 100 may detect the pixel value of the area after correcting the pixel value to be lighter.

Further, when the hue changes, the detection unit 100 considers the change in the hue when updating the matching parameter used in the template matching (that is, the above-described detection parameter), and detects the color of the parameter. The information may be corrected. If the change in hue is large, the detection unit 100 may not use the color information. Further, the detection unit 100 may perform matching by reducing the weight of color information and increasing the weight of other features such as edges among the features of the template.

Further, when the detection unit 100 updates the detection parameters used in the object detection process, if it is determined that the objects are likely to overlap, the detection parameters may not be updated. ..

As described above, the detection unit 100 controls the detection of the object based on the appearance information included in the tracking information. As a result, the detection unit 100 can reduce the processing of detecting an invisible object and the processing of updating parameters such as template matching. Thereby, the detection unit 100 can reduce the possibility of false detection and incorrect parameter update.

Similarly, when there is a high possibility that the lighting condition has changed, the detection unit 100 may perform control so as to correct the effect and update the parameter, or control not to update the parameter. May be.

In addition, when a plurality of detection algorithms can be switched, the object tracking device 10 may use, as the detection unit 100, a detector that is stronger in overlapping. Specifically, the object tracking device 10 normally uses a detector that detects the entire head, but when overlapping, uses a detector that detects only a part of the head instead of the entire head. You may do it. Thereby, the object tracking device 10 can use a simple detector at normal times, and can use a more detailed detector when there is a possibility of overlapping. As a result, the object tracking device 10 can perform highly accurate detection while maintaining efficiency. Similarly, when the lighting condition changes, the object tracking device 10 may control the detection by using a detector (feature amount) having high robustness to the lighting condition.

As described above, in the object tracking device 10 according to the present embodiment, the integrated tracking unit 200 uses the information of other cameras to determine the appearance of the object. Therefore, the integrated tracking unit 200 can determine the appearance with a high degree of accuracy from a certain camera 20 even when the determination is difficult with the camera 20 alone, such as when the objects overlap each other. Then, the integrated tracking unit 200 can feed back the result to the detection unit 100. Thereby, the detection unit 100 can reduce erroneous detection of the object. Then, since the integrated tracking unit 200 uses the detection result to track the object, the tracking accuracy of the object can be improved.

<Example of hardware configuration>
Here, a configuration example of hardware capable of realizing the object tracking device (10, 50) according to each of the above-described embodiments will be described. The object tracking device (10, 50) described above may be realized as a dedicated device, or may be realized using a computer (information processing device).

FIG. 19 is a diagram illustrating a hardware configuration of a computer (information processing device) that can implement each embodiment of the present invention.

The hardware of the information processing apparatus (computer) 700 shown in FIG. 19 includes a CPU (Central Processing Unit) 11, a communication interface (I/F) 12, an input/output user interface 13, a ROM (Read Only Memory) 14, and a RAM ( A random access memory 15, a storage device 17, and a drive device 18 for a computer-readable storage medium 19 are provided, and these are connected via a bus 16. The input/output user interface 13 is a man-machine interface such as a keyboard as an example of an input device and a display as an output device. The communication interface 12 is a general communication means for allowing the devices (FIGS. 1 and 9) according to the above-described embodiments to communicate with external devices via the communication network 600. In such a hardware configuration, the CPU 11 controls the entire operation of the information processing device 700 that realizes the object tracking device (10, 50) according to each embodiment.

The present invention described in each of the above-described embodiments as an example supplies the information processing apparatus 700 shown in FIG. 19 with a program (computer program) capable of implementing the processing described in each of the above-described embodiments. After that, the program is read out to the CPU 11 and executed. Note that the program is, for example, various processing described in the flowchart (FIG. 8) referred to in the description of each of the above-described embodiments, or in FIG. 1, FIG. 4 to FIG. 6, FIG. 9, and FIG. It may be a program that can realize each unit (each block) shown in the apparatus in the block diagram shown.

Further, the program supplied into the information processing device 700 may be stored in a readable/writable temporary storage memory (15) or a non-volatile storage device (17) such as a hard disk drive. That is, in the storage device 17, the program group 17A is, for example, a program that can realize the function of each unit shown in the object tracking device (10, 50) in each of the above-described embodiments. Further, the various stored information 17B is, for example, the object tracking result, the camera image, the camera parameter, the range of the coordinate value of the common coordinate system seen by each camera 20 in each of the above-described embodiments. However, when the program is installed in the information processing device 700, the constituent units of the individual program modules are the blocks shown in the block diagrams (FIG. 1, FIG. 4 to FIG. 6, FIG. 9, and FIG. 15 to FIG. 17). The division is not limited, and those skilled in the art may appropriately select when mounting.

In the above case, the method of supplying the program into the device is installed in the device via various computer-readable recording media (19) such as a CD (Compact Disk)-ROM and a flash memory. A general procedure can be adopted at present, such as a method or a method of downloading from the outside through a communication line (600) such as the Internet. In such a case, the present invention can be considered to be configured by the code (program group 17A) configuring the computer program or the storage medium (19) storing the code.

The present invention has been described above as an example applied to the exemplary embodiment described above. However, the technical scope of the present invention is not limited to the scope described in each of the above-described embodiments. It is obvious to those skilled in the art that various modifications and improvements can be added to the embodiment. In such a case, a new embodiment with such changes or improvements may be included in the technical scope of the present invention. And this is clear from the matters described in the claims.

DESCRIPTION OF SYMBOLS 1 Object tracking system 2 Object tracking system 10 Object tracking device 100 Detection part 110 Object detection part 111 Recognition type object detection part 112 Search range setting part 120 Common coordinate conversion part 130 Individual coordinate conversion part 140 Object detection part 141 Recognition type object detection part 142 Non-recognition type object detection unit 143 Detection parameter update unit 144 Detection result integration unit 150 Display control unit 160 Storage unit 200 Integrated tracking unit 210 Prediction unit 220 Storage unit 230 Correlation unit 240 Update unit 300 Display control unit 400 Detection unit 500 Integration Tracking unit 510 Buffer unit 530 Matching unit 20 Camera 30 Display device 40 Network 50 Object tracking device

Claims (18)

A plurality of detecting means for detecting an object from the captured image and outputting the detection result;
Integrated tracking means for calculating the position information of the object expressed in a common coordinate system, based on the plurality of detection results output by each of the plurality of detecting means,
The integrated tracking means outputs position information of the object in the calculated common coordinate system,
The detection means converts the position information of the object in the common coordinate system into position information expressed in an individual coordinate system specific to a camera that outputs an image that is a detection target of the object, and in the individual coordinate system. Tracking the object,
Detecting the object based on the position information expressed in the individual coordinate system,
Converting the position information of the object detected based on the position information expressed in the individual coordinate system into position information expressed in the common coordinate system,
Object tracking system. In the detection of the object based on the position information represented by the individual coordinate system, the detection means,
A camera-specific individual coordinate system that outputs the image, which is a search range for searching the object in the image that is the detection target of the object, and is specified based on the position information expressed in the individual coordinate system. The object is detected within the search range expressed by
The object tracking system according to claim 1. The object tracking system according to claim 1, wherein the detection of the object is performed by using a discriminator that has learned image characteristics of the object. The detection result of the detection of the object is a result of integrating the detection result by the discriminator and the detection result by template matching with the object region detected in the previous frame,
The object tracking system according to claim 3. A plurality of detection means for detecting an object from the output information of the sensor and outputting the detection result,
An integrated tracking unit that outputs the tracking information of the object represented by a common coordinate system, tracking the object based on the plurality of detection results output by each of the plurality of detection units,
The integrated tracking means outputs the generated tracking information to each of the plurality of detection means,
The detection means generates a first detection result, which is a result of object identification, and a second detection result, which detects an object by template matching with a previously detected object region, based on the tracking information. , An object tracking system for detecting the object by integrating. A plurality of detection means for detecting an object from the output information of the sensor and outputting the detection result,
An integrated tracking unit that outputs the tracking information of the object represented by a common coordinate system, tracking the object based on the plurality of detection results output by each of the plurality of detection units,
The integrated tracking means outputs the generated tracking information to each of the plurality of detection means,
The detection means detects the object based on the tracking information,
An object tracking system in which the integrated tracking unit calculates the position by collectively collecting the detection results output from the plurality of detection units within a predetermined period, and generates tracking information. Detects an object from the captured image, outputs the detection result,
Based on the output results of the plurality of detected, to calculate the position information of the object represented in a common coordinate system,
Outputting position information of the object in the calculated common coordinate system,
Position information of the object in the common coordinate system is converted into position information expressed in an individual coordinate system specific to a camera that outputs an image that is a detection target of the object, and the object is tracked in the individual coordinate system. ,
Detecting the object based on the position information expressed in the individual coordinate system,
Converting the position information of the object detected based on the position information expressed in the individual coordinate system into position information expressed in the common coordinate system,
Object tracking method. In the detection of the object based on the position information expressed in the individual coordinate system,
A camera-specific individual coordinate system for outputting the image, which is a search range for searching the object in the image to be detected by the object and is specified based on position information expressed in the individual coordinate system. The object is detected within the search range expressed by
The object tracking method according to claim 7. 9. The object tracking method according to claim 7, wherein the detection of the object is performed using a classifier that has learned the image feature of the object. The detection result of the detection of the object is a result of integrating the detection result by the discriminator and the detection result by template matching with the object region detected in the previous frame,
The object tracking method according to claim 9. The object is detected from the output information of the sensor, the detection result is output,
Tracking the object based on the output of the plurality of detection results, to generate tracking information of the object represented in a common coordinate system,
Output the generated tracking information,
By generating and integrating a first detection result which is a result of object identification and a second detection result which detects an object by template matching with a previously detected object region based on the tracking information. An object tracking method for detecting the object. The object is detected from the output information of the sensor, the detection result is output,
Tracking the object based on the output of the plurality of detection results, to generate tracking information of the object represented in a common coordinate system,
Output the generated tracking information,
Detecting the object based on the tracking information,
An object tracking method in which a plurality of detection results output within a predetermined period are combined to calculate a position and generate tracking information. A process of detecting an object from a captured image,
The process of outputting the detection result,
Based on the output results of the plurality of detection, a process of calculating the position information of the object represented in a common coordinate system,
A process of outputting the position information of the object in the calculated common coordinate system;
Position information of the object in the common coordinate system is converted into position information expressed in an individual coordinate system unique to a camera that outputs an image that is a detection target of the object, and the object is tracked in the individual coordinate system. Processing and
A process of detecting the object based on the position information expressed in the individual coordinate system,
A process of converting the position information of the object detected based on the position information expressed in the individual coordinate system into the position information expressed in the common coordinate system,
Programs that include. In the process of detecting the object based on the position information expressed in the individual coordinate system,
A camera-specific individual coordinate system that outputs the image, which is a search range for searching the object in the image that is the detection target of the object, and is specified based on the position information expressed in the individual coordinate system. In the search range represented by, a process of detecting the object,
14. The program according to claim 13, including. The program according to claim 13 or 14, wherein the processing of detecting the object is performed using a classifier that has learned the image characteristics of the object. The detection result of the process of detecting the object is a result of integrating the detection result by the discriminator and the detection result by template matching with the object region detected in the previous frame,
The program according to claim 15. The process of detecting an object from the output information of the sensor,
The process of outputting the detection result,
A process of tracking the object based on the output results of the plurality of detections, and generating tracking information of the object expressed in a common coordinate system;
A process of outputting the generated tracking information,
By generating and integrating a first detection result which is a result of object identification and a second detection result which detects an object by template matching with a previously detected object region based on the tracking information. A process of detecting the object,
Programs that include. The process of detecting an object from the output information of the sensor,
The process of outputting the detection result,
A process of tracking the object based on the output results of the plurality of detections, and generating tracking information of the object expressed in a common coordinate system;
A process of outputting the generated tracking information,
A process of detecting the object based on the tracking information;
A process of collectively calculating a plurality of detection results output within a predetermined period and generating tracking information,
Programs that include.

Images