JP7004017B2 - Object tracking system, object tracking method, program - Google Patents

Description

The present invention relates to an object tracking device, an object tracking system, an object tracking method, a display control device, an object detection device, a program, and a recording medium.

In recent years, a system for tracking a person has been developed using a plurality of cameras and the like. For example, the mobile tracking system described in Patent Document 1 uses a plurality of in-camera tracking means for tracking a person in a distributed manner for each camera. The system then tracks the moving object in cooperation with the plurality of in-camera tracking means. Further, Patent Document 2 describes a method of tracking the same object imaged between a plurality of imaging units based on the tracking result of each object.

Further, as a related technique, Patent Document 3 describes a method of early removing an object that does not need to be tracked from the tracking target.

Further, Patent Document 4 describes a device for detecting a moving body in an image taken by one photographing means.

Japanese Unexamined Patent Publication No. 2004-72628 Special Table 2009-510541A International Publication No. 2013/012091 Japanese Unexamined Patent Publication No. 2006-202047

However, in the technique described in Patent Document 1 or 2 described above, for example, when the camera is located far away from the object, the tracking accuracy of the object (moving body) in the image taken by the camera may be low. be. In this case, in the technique of Patent Document 1 or 2, due to the influence of the tracking accuracy of the camera, the tracking results of the objects cannot be integrated, or even if they can be integrated, the detection of the position of the object required at the time of integration can be detected. In some cases, the accuracy was low.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique capable of tracking an object with higher accuracy.

The object tracking device according to one aspect of the present invention has a plurality of detection means that detect an object from the output information of the sensor and output the detection result, and a plurality of the detection results output by each of the plurality of detection means. The integrated tracking means comprises an integrated tracking means that tracks the object based on the above and generates tracking information of the object expressed in a common coordinate system, and the integrated tracking means detects the generated tracking information by the plurality of detection means. The detection means detects the object based on the tracking information.

The object tracking system according to one aspect of the present invention includes a sensor and an object tracking device that receives output information consisting of information acquired by the sensor, and the object tracking device detects the object from the output information. The object is tracked based on the plurality of detection means that output the detection results and the plurality of detection results output by each of the plurality of detection means, and the object is tracked represented by the common coordinate system. The integrated tracking means includes an integrated tracking means for generating information, the integrated tracking means outputs the generated tracking information to each of the plurality of detection means, and the detection means is based on the tracking information to the object. Is detected.

In the object tracking method according to one aspect of the present invention, an object is detected from the output information of the sensor, the detection result is output, the object is tracked based on the plurality of output detection results, and the object is tracked in a common coordinate system. The tracked information of the expressed object is generated, the generated tracking information is output, and the detection of the object detects the object based on the tracking information.

The display control device according to one aspect of the present invention is a display control device for displaying display data on the display device, and the display data searches for the object based on the tracking information of the object among the output information of the sensor. The search range is a search range expressed by a sensor-specific individual coordinate system that outputs the output information, and the tracking information of the object is the search range in the output information of each of the plurality of sensors. It is information which shows the result of having tracked the object based on the detection result of the object detected from the inside.

The object detection device according to one aspect of the present invention is tracking information indicating the tracking result of an object, which is output from each of the plurality of object detection devices and is tracked based on the plurality of detection results, and is a common coordinate system. Based on the tracking information represented by, the object is detected from the output information of the sensor.

A computer program that realizes each of the above devices, an object tracking system, or an object tracking method by a computer, and a computer-readable storage medium in which the computer program is stored are also included in the scope of the present invention.

According to the present invention, it is possible to track an object with higher accuracy.

It is a functional block diagram which shows an example of the functional structure of the object tracking apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows an example of outline of the whole structure of the object tracking system which concerns on 1st Embodiment of this invention. It is a figure for demonstrating the process of associating a target with a tracker. It is a functional block diagram which shows an example of the functional structure of the detection part of the object tracking apparatus which concerns on 1st Embodiment of this invention. It is a functional block diagram which shows an example of the functional structure of the object detection part in the detection part of the object tracking apparatus which concerns on 1st Embodiment of this invention. It is a functional block diagram which shows an example of the functional structure of the integrated tracking part of the object tracking apparatus which concerns on 1st Embodiment of this invention. It is a figure for demonstrating the sequential tracking process of an object performed by the integrated tracking unit which concerns on 1st Embodiment of this invention. It is a flowchart which shows an example of the flow of the object tracking process of the object tracking apparatus which concerns on 1st Embodiment of this invention. It is a functional block diagram which shows an example of the functional structure of the object tracking apparatus which concerns on 2nd Embodiment of this invention. It is a functional block diagram which shows an example of the functional structure of the detection part of the object tracking apparatus which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the application example of the object tracking apparatus which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the application example of the object tracking apparatus which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the application example of the object tracking apparatus which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the application example of the object tracking apparatus which concerns on 2nd Embodiment of this invention. It is a functional block diagram which shows an example of the functional structure of the detection part of the object tracking apparatus which concerns on 3rd Embodiment of this invention. It is a functional block diagram which shows an example of the functional structure of the object detection part in the detection part of the object tracking apparatus which concerns on 3rd Embodiment of this invention. It is a functional block diagram which shows an example of the functional structure of the integrated tracking part of the object tracking apparatus which concerns on 4th Embodiment of this invention. It is a figure for demonstrating the batch tracking process of an object performed by the integrated tracking unit which concerns on 4th Embodiment of this invention. It is a figure which illustrates the hardware configuration of the computer (information processing apparatus) which can realize each embodiment of this invention.

<First Embodiment>
The first embodiment of the present invention will be described in detail with reference to the drawings. First, with reference to FIG. 2, the overall configuration of the object tracking system (also simply referred to as a system) of the present invention will be described. FIG. 2 is a diagram showing an outline example of the overall configuration of the object tracking system 1 according to the present embodiment. As shown in FIG. 2, the object tracking system 1 according to the present embodiment includes an object tracking device 10, a plurality of cameras (20-1 to 20-N (N is a natural number)), and one or more display devices 30. It is equipped with. In the present embodiment, when the plurality of cameras (20-1 to 20-N) are not distinguished from each other, or when they are generically referred to, they are referred to as cameras 20.

The object tracking device 10, the camera 20, and the display device 30 are communicably connected to each other via the network 40. The display device 30 may not be included in the object tracking system 1. Further, the display device 30 may be configured to be directly connected to the object tracking device 10 without going through the network 40.

The camera 20 functions as a sensor for detecting an object. In the present embodiment, a case where a camera 20 is used as a sensor for detecting an object will be described as an example, but the present invention is not limited thereto. The sensor is not limited to the camera, and may be any sensor such as a radio wave sensor that can perform positioning. Further, a sensor in which a plurality of sensors coexist, such as a sensor in which a radio wave sensor and a camera are integrated, may be used. By using the camera 20 as a sensor, the object tracking device 10 can more preferably acquire visual information such as colors.

Further, in the present embodiment, the information acquired by the sensor is described as an image taken by the camera, but when the sensor is a radio wave sensor, the information acquired by the sensor is acquired by the radio wave sensor. It is a radio wave.

The object tracking device 10 is a device for tracking an object included in an image taken by each of a plurality of cameras 20. The functional configuration of the object tracking device 10 will be described by changing the drawings.

The display device 30 displays the tracking result of the object by the object tracking device 10. The display device 30 may display an image taken by the camera 20. Further, the display device 30 may display other information such as flow line information.

(Object tracking device 10)
Next, the function of the object tracking device 10 will be described. FIG. 1 is a functional block diagram showing an example of the functional configuration of the object tracking device 10 according to the present embodiment. As shown in FIG. 1, the object tracking device 10 includes a plurality of detection units (100-1 to 100-N) and an integrated tracking unit 200. In the present embodiment, when the plurality of detection units (100-1 to 100-N) are not distinguished from each other, or when they are generically referred to, these are referred to as detection units 100.

(Detection unit 100)
The detection unit 100 is tracking information output from the integrated tracking unit 200 described later, and is an object from the output information of the camera 20 based on the tracking information of the object in the frame before the target frame for detecting the object (object). Is detected. Here, in the present embodiment, the output information of the camera 20 indicates video data indicating a video captured by the camera 20.

In the present embodiment, it is assumed that the plurality of detection units 100 and the plurality of cameras 20 are associated with each other on a one-to-one basis. For example, the detection unit 100-1 detects an object from the image captured by the camera 20-1, and the detection unit 100-2 detects the object from the image captured by the camera 20-2. The present embodiment is not limited to this, and for example, the detection unit 100-1 may detect an object from the image captured by the camera 20-N.

Further, the camera 20 and the detection unit 100 may not be associated with each other on a one-to-one basis. For example, the detection unit 100-1 may detect an object from the images taken by each of the plurality of cameras 20.

Hereinafter, the operation of the detection unit 100 will be described. The detection unit 100 receives from the camera 20 video data (hereinafter referred to as a camera video) indicating the video captured by the camera 20. In FIG. 1, video data showing images taken by cameras 20-n (n are 1 to N) is referred to as camera images (n). Here, the camera image may be an image taken by a camera 20 such as a surveillance camera acquired in real time, or an image taken by the camera 20 may be temporarily stored in a storage unit or the like (not shown) and stored later. It may be decoded (or reproduced) by. This video data includes time information indicating the time when the image was taken.

Further, the detection unit 100 receives the tracking information of the object in the previous frame from the integrated tracking unit 200. The previous frame may be the frame immediately before the frame (current frame) for which the object is detected, or may be a predetermined number of frames before the current frame. Further, the front frame may be one or a plurality. When the detection unit 100 detects an object for the first frame, the detection unit 100 does not receive (do not use) the tracking information of the object in the previous frame because the tracking information of the object in the previous frame does not exist.

The detection unit 100 detects an object (referred to as object detection) from the camera image using the received camera image and the tracking information of the object in the previous frame. As described above, when the object is detected for the first frame, the detection unit 100 detects the object without using the tracking information of the object in the previous frame. In the following, the detected object will be referred to as a target. That is, the object detection result (also called the object detection result or simply the detection result) is a set of targets.

The object detection result includes, for example, information indicating the position of the target, information indicating the size of the target, and the like for each target. Specifically, the object detection result is, for example, information on the circumscribing rectangle of the area occupied by the target (target area) in the frame in the video in which the object is detected, the coordinate value of the center of gravity of the target area, and the information indicating the width of the target. , Information indicating the height of the target, etc. are included. The object detection result is not limited to this. For example, the object detection result may include, or in addition to, the coordinate value of the uppermost end of the target area, the coordinate value of the lowermost end, or the like in place of the coordinate value of the center of gravity of the target area. The object detection result may include information indicating the position, size, etc. of the target for each target.

In the present embodiment, the object detection result will be described by taking as an example the coordinate value of the lowermost end of the target and the information indicating the circumscribed rectangle of the target for each target. The coordinate value at the lowermost end of the target indicates the coordinate value of the point where the object touches the floor (ground) and / or the coordinate value of the midpoint of the lower side of the circumscribing rectangle of the object. Further, the coordinate value at the lowermost end of the target may be the coordinate value at the feet when the object is a person.

Then, the detection unit 100 converts the coordinate values included in the object detection result into the coordinate values in the common common coordinate system defined in the space (shooting space) photographed by the plurality of cameras 20, and the converted coordinates. The value is the object detection result.

Further, the object detection result may include information representing the shape of the target in addition to the above-mentioned information. That is, the object detection result may include silhouette information representing the target area and the like. Here, the silhouette information is information that distinguishes between the internal pixel and the external pixel of the target area, and is, for example, image information in which the internal pixel value is set to 255 and the external pixel value is set to 0, or MPEG. It is a value obtained by extracting a shape descriptor (shape feature amount) as standardized in -7 from a silhouette shape. Further, the object detection result may include the feature amount of the appearance of the object. For example, the object detection result may include features such as the color, pattern, and shape of the object.

Further, the object detection result may include information (target likelihood information) describing the likelihood representing the certainty (probability) of object detection for each target. The target likelihood information is the information required to calculate the target likelihood, and is related to the accuracy of object detection, such as the value of the score at the time of object detection, the distance of the detected object from the camera, and the size. Information. Further, the detection unit 100 may calculate the likelihood of the target itself and use the calculated likelihood as the likelihood information of the target.

Then, the detection unit 100 outputs the object detection result to the integrated tracking unit 200.

(Integrated Tracking Unit 200)
The integrated tracking unit 200 receives the detection results output from each of the detection units 100. Then, the integrated tracking unit 200 tracks the object based on each of the detection results. Specifically, the integrated tracking unit 200 uses the object detection results for one or more objects detected from the video captured by the camera 20 associated with the detection unit 100 by each of the detection units 100. Track objects (perform object tracking). Then, the integrated tracking unit 200 generates a tracking result (object tracking result) of the object represented by the common coordinate system. In this way, the integrated tracking unit 200 integrates the object detection results detected by each detection unit 100 from the image captured by the camera 20 associated with the detection unit 100, and performs object tracking. Therefore, the object tracking performed by the integrated tracking unit 200 is also referred to as an object integrated tracking.

Hereinafter, the information for each object generated as the object tracking result is referred to as a tracker. That is, it is assumed that the tracker includes information indicating the position of the tracked object, a motion model of the object, and the like as information on the tracked object (object tracking result), but the present invention is limited to this. is not it. Since the position of the tracked object is the position of the object before (past) the present time, it is also called the past position of the object.

In other words, object tracking can be regarded as a process of associating objects between frames by associating a target detected by object detection with a tracker generated before the detection of this target. This object tracking will be described with reference to FIG. FIG. 3 is a diagram for explaining a process of associating a target with a tracker by the integrated tracking unit 200. As shown in FIG. 3, the number of targets is M and the number of trackers is K (M and K are integers of 0 or more). The integrated tracking unit 200 associates the M targets with the K trackers. When associating a target with a tracker, the integrated tracking unit 200 first predicts the current position of the object from the past position of the object indicated by the information contained in the tracker, and shows the relationship between the target and the tracker. Correspond using the index.

That is, the integrated tracking unit 200 predicts the position of the object on the current frame based on the position of the object detected in the previous frame and the motion model of the object calculated and held for each tracker. As this method, various existing methods such as a method using a Kalman filter or a method using a particle filter can be used.

Then, the integrated tracking unit 200 associates the object tracking result (tracker) in the previous frame with the object (target) included in the detection result, for example, based on the information listed in (1) to (3) below.
(1) Closeness of the distance between the position of the object on the current frame predicted using the tracker and the position of the target (2) Similarity of appearance features between the target and the object whose tracking result is shown by the tracker. (3) Likelihood of each target and tracker The processing of this association can be reduced to the cost minimization problem of the bipartite graph as shown in FIG. Therefore, the integrated tracking unit 200 can solve this problem by an algorithm such as the Hungarian method.

In FIG. 3, the case where the target and the tracker are associated with each other is shown by using an arrow. That is, the top target is associated with the top tracker.

Then, when there is a target that does not correspond to the tracker, the integrated tracking unit 200 determines whether or not the target can be regarded as a newly appearing object. Then, when the integrated tracking unit 200 determines that the target is likely to newly appear, the integrated tracking unit 200 newly adds a tracker related to the target. In FIG. 3, it is assumed that the target indicated by the symbol m (referred to as the target m) is a target that does not correspond to the tracker. At this time, the integrated tracking unit 200 determines whether or not the target m can be regarded as a newly appearing object, and if it can be regarded, a new tracker related to the target m is created.

On the other hand, when there is a tracker that does not correspond to the target, the integrated tracking unit 200 determines whether or not the tracker is information about an object that has disappeared from the shooting space. Then, the integrated tracking unit 200 deletes the tracker when it is highly possible that the tracker is information about an object that has disappeared from the shooting space. In FIG. 3, it is assumed that the tracker represented by the reference numeral k (referred to as tracker k) is a tracker that does not correspond to the target. At this time, the integrated tracking unit 200 determines whether or not the tracker k is information about the disappeared object, and if the tracker k is information about the disappeared object, the tracker k is deleted.

The integrated tracking unit 200 performs object tracking by repeating these processes on a frame-by-frame basis. The integrated tracking unit 200 gives the tracker a unique ID (identifier) common to all cameras 20, and manages the tracking result (tracker) by this ID. Further, the integrated tracking unit 200 includes a value evaluated for the certainty of the tracking result (hereinafter referred to as a tracker likelihood (weight)) as a tracker parameter in the tracker. The position indicated by the information indicating the position of the tracked object contained in the tracker, and the position of the newest object is referred to as the position of the tracker. The size of the object at this time is also called the size of the tracker.

Further, the integrated tracking unit 200 updates the information indicating the position of each tracker, the information on the likelihood of the tracker, and the like based on the result of the association. The tracker position information is information expressed in a common coordinate system defined in a shooting space shot by a plurality of cameras 20. The information expressed in this common coordinate system is a coordinate value in the common coordinate system. The coordinate value in this common coordinate system is, for example, a coordinate system indicating the position of the floor in the real world when a plurality of cameras 20 are cameras installed in a certain store. On the other hand, the coordinate system unique to each camera 20 is called the individual coordinate system of the camera 20. This individual coordinate system is a coordinate system on the captured image of the camera 20. Hereinafter, the position information represented by the common coordinate system will be described as the coordinate values of the common coordinate system. Further, the position information represented by the individual coordinate system of the camera 20 will be described as the coordinate values of the individual coordinate system of the camera 20.

Then, the integrated tracking unit 200 generates a tracker whose information is updated based on the result of the mapping as a new object tracking result. Then, the integrated tracking unit 200 includes information indicating the position and / or size of the tracker, information on the likelihood of the tracker, and the like among the generated object tracking results (trackers), and information indicating the tracking results of the object tracking. Output as (tracking information).

This tracking information includes the coordinate values of the common coordinate system of each tracker as the information indicating the position of each tracker. This tracking information is fed back to the detection unit 100. That is, the detection unit 100 receives this tracking information and uses the tracking information for object detection for subsequent frames.

The integrated tracking unit 200 may be configured to output the object tracking result including the tracking information and other information of the tracker to the detection unit 100.

As described above, the object tracking device 10 according to the present embodiment uses the images taken by each of the plurality of cameras and integrates the object detection results for the images taken by each of the plurality of cameras 20. , Perform object tracking. Then, the object tracking device 10 feeds back the obtained tracking information to the object detection in the next frame.

In this way, the object tracking device 10 detects an object from the video by using the tracking result for the previous frame. For example, if there is an object that is not visible to one camera 20 but visible to another camera 20, the object tracking device 10 will also track the object that is not visible to the camera 20 as an object in the image of this camera 20. Used for detection. As a result, the detection unit 100 can suitably detect this object when the same object appears in the range visible from the certain camera 20. Therefore, the object tracking device 10 can accurately track the object with respect to this object.

Therefore, the object tracking device 10 can improve the detection accuracy of the object as compared with the case where the tracking result for the previous frame is not used. Further, since the object tracking device 10 tracks the object using all the detection results having high detection accuracy, the accuracy of the tracking result obtained as a whole is also improved.

(Details of detection unit 100)
Next, with reference to FIGS. 4 to 8, the functions of each part of the object tracking device 10 will be described in more detail. FIG. 4 is a functional block diagram showing an example of a more detailed functional configuration of the detection unit 100 of the object tracking device 10 according to the present embodiment. As shown in FIG. 4, the detection unit 100 includes an object detection unit 110, a common coordinate conversion unit (second conversion unit) 120, and an individual coordinate conversion unit (first conversion unit) 130. In FIG. 4, the camera images (n) (n are 1 to N) received by the detection unit 100 are simply described as camera images.

The individual coordinate conversion unit 130 receives the tracking information output from the integrated tracking unit 200 from the integrated tracking unit 200. Then, the individual coordinate conversion unit 130 sets the coordinate values of the common coordinate system of each tracker included in this tracking information to the coordinate values on the frame taken by each camera 20 (that is, the individual coordinate system unique to each camera 20). Convert to the expressed coordinate value). When the coordinate value of the common coordinate system is (X, Y, Z) and the coordinate value of the individual coordinate system of the camera 20 is (x, y), the individual coordinate conversion unit 130 is the coordinates of the common coordinate system of the tracker. From the values (X, Y, Z), the coordinate values (x, y) of the individual coordinate system of the camera 20 associated with the detection unit 100 including the individual coordinate conversion unit 130 are obtained. At this time, it is preferable that the individual coordinate conversion unit 130 obtains at least the camera parameters representing the camera position, posture, etc. of the camera 20 associated with the detection unit 100 by calibration. As a result, the individual coordinate conversion unit 130 converts the coordinate values of the common coordinate system into the coordinate values of the individual coordinate system of the camera 20 by using the obtained camera parameters.

The camera parameters may be stored in a storage unit (not shown) in the detection unit 100, or may be stored in a storage area in the individual coordinate conversion unit 130. In the latter case, the individual coordinate conversion unit 130 may be configured to supply camera parameters to the common coordinate conversion unit 120.

For example, suppose that the object is a person and the information indicating the position of the tracker is a coordinate value indicating the foot position of the person and a coordinate value indicating the position of the top of the head. Then, the coordinate value of the foot position and the coordinate value of the crown position are set to (X0, Y0,0) and (X0, Y0, H) (H represents the height of the person), respectively. Further, it is assumed that the camera 20 associated with the detection unit 100 including the individual coordinate conversion unit 130 is the camera 20-1.

At this time, the individual coordinate conversion unit 130 uses the camera parameters related to the camera 20-1 to set the foot position (x0, y0) and the crown position (x1, y1) on the frame taken by the camera 20-1, respectively. Ask. If the information indicating the position of the tracker includes the information indicating the circumscribed rectangle, the value indicating the width of the circumscribed rectangle was previously obtained by converting it into the coordinate values of the common coordinate system using the camera parameters. ing. Therefore, the individual coordinate conversion unit 130 may use the value converted using the above camera parameters again as the width of the circumscribed rectangle.

Also, not all of the objects whose tracking results are shown by the tracker are visible to one camera 20, and may be outside the field of view of this camera 20. Therefore, when the information indicating the position of the tracker included in the tracking information is information about an object that cannot be seen from the camera 20 associated with the detection unit 100, the individual coordinate conversion unit 130 uses the individual coordinate system of the camera 20 of the object. The coordinate value of is not available. Therefore, the individual coordinate conversion unit 130 may not convert the coordinate values of the tracker's common coordinate system for such an object that cannot be seen outside the angle of view of the camera 20 into the coordinate values of the individual coordinate system. At this time, the individual coordinate conversion unit 130 registers in advance the range of the coordinate values of the common coordinate system that can be seen by each camera 20 in a storage unit or the like (not shown) for each camera 20, and each object is included in this. It may be determined whether or not it is present. Further, the individual coordinate conversion unit 130 actually converts the coordinate values into the coordinate values of the individual coordinate system to obtain an abnormal value indicating the outside of the area monitored by the associated camera 20, or the value can be obtained. If not, it may be determined that the object whose coordinate values have been converted is an object that cannot be seen from the camera 20.

Then, in the individual coordinate conversion unit 130, the coordinate value of the common coordinate system of the tracker included in the tracking information output from the integrated tracking unit 200 becomes the coordinate value of the individual coordinate system of the camera 20 associated with the detection unit 100. The converted result (tracking information) is output to the object detection unit 110. That is, the individual coordinate conversion unit 130 converts the tracking information represented by the common coordinate system into the tracking information represented by the individual coordinate system, and outputs the converted tracking information to the object detection unit 110. Hereinafter, when simply described as "coordinate values of the individual coordinate system", the coordinate values of the individual coordinate system of the camera 20 associated with the detection unit 100 are shown.

The object detection unit 110 receives a camera image from the camera 20 associated with the detection unit 100 including the object detection unit 110. Further, the object detection unit 110 receives the tracking information converted into the coordinate values of the individual coordinate system from the individual coordinate conversion unit 130. Then, the object detection unit 110 detects an object from the received camera image based on the tracking information.

Then, the object detection unit 110 generates a detection result. The object detection unit 110 outputs the generated detection result to the common coordinate conversion unit 120. It should be noted that this detection result is a detection result expressed in an individual coordinate system.

The configuration of the object detection unit 110 will be described in more detail with reference to FIG. FIG. 5 is a functional block diagram showing an example of the functional configuration of the object detection unit 110 according to the present embodiment. As shown in FIG. 5, the object detection unit 110 includes a recognition type object detection unit (first object detection unit) 111 and a search range setting unit 112.

The search range setting unit 112 receives the tracking information converted into the coordinate values of the individual coordinate system from the individual coordinate conversion unit 130. Then, the search range setting unit 112 obtains an area (search range) for detecting an object for the current frame by using the tracking information converted into the coordinate values of the individual coordinate system. That is, the search range setting unit 112 predicts the position of the object in the current frame based on the tracking information consisting of the tracking results of the previous frame converted into the coordinate values of the individual coordinate system. Then, the search range setting unit 112 obtains the detection range for searching the object from the predicted position. This search range is also called the object detection range.

Here, the tracking information received by the search range setting unit 112 is the result of object tracking in the past frame (also referred to as the past tracking result) when viewed from the time of the frame to be currently processed. Therefore, the search range setting unit 112 predicts the movement of each object and predicts the position of each object in the current frame. Hereinafter, the predicted position of the object is referred to as a predicted position. Then, the search range setting unit 112 sets the neighborhood of the predicted position as the search range of the object.

It is preferable that the search range setting unit 112 predicts the movement of each object by using the motion model of each object calculated from the past tracking results. For example, the search range setting unit 112 determines that the object is stationary when the position of the object has not changed in the tracking result of the past several frames (may be 2 frames), and the object obtained by the tracking result. The position of is used as the predicted position as it is. Further, when the object is moving in the tracking result of the past several frames, the search range setting unit 112 assumes that the object is moving at a constant speed, and considers the time difference from the past frame to predict the predicted position. May be sought.

The tracking result of the past several frames used by the search range setting unit 112 to predict the movement of each object may be included in the tracking information. Further, the motion model of the object obtained from the tracking result of the past several frames may also be included in the tracking information.

Further, this predicted position may be included in the tracking information. That is, the integrated tracking unit 200 may include the value obtained by the Kalman filter or the particle filter at the time of object tracking as the predicted position in the tracking information.

Further, for example, a new object may appear at the angle of view of the camera 20 at the outer edge portion of the range in which the camera 20 can take a picture. Further, even when the place where the camera 20 is shooting includes a place such as an entrance / exit, a new object may appear at the angle of view of the camera 20. Therefore, it is preferable that the search range setting unit 112 also includes these areas (outer edge portion and / or entrance / exit portion of the frame) on the frame included in the image captured by the camera 20 in the object search range. ..

The search range setting unit 112 outputs information (search range information) indicating the search range of the set object to the recognition type object detection unit 111.

The recognition type object detection unit 111 receives the search range information from the search range setting unit 112. The recognition type object detection unit 111 detects an object from the camera image input to the recognition type object detection unit 111 based on the received search range information. The recognition type object detection unit 111 temporarily stores the input frame of the camera image in a storage means such as a buffer in the recognition type object detection unit 111, and when it receives the search range information, it applies this information. Execute object detection processing. Specifically, the recognition type object detection unit 111 detects an object in the area (search range) indicated by the search range information by using a discriminator trained in the image features of the object.

For example, when the object is a person, the recognition type object detection unit 111 detects the person by applying a discriminator trained in a characteristic part of the person (for example, the head or the upper body). Further, the recognition type object detection unit 111 may use a discriminator trained by the entire person as the discriminator. The recognition type object detection unit 111 can use various types of the classifier. For example, the recognition type object detection unit 111 can use a discriminator obtained by learning images of the head, upper body, whole body of a person, and the like with a CNN (Convolutional Neural Network). Further, the recognition type object detection unit 111 performs feature extraction such as HOG (Histogram Of Gaussian), and SVM (Support Vector Machine; support vector machine) or GLVQ (Generalized Learning Vector Quantization) generalization; Etc. may be used. In addition to the above, the recognition type object detection unit 111 can use various existing recognition-based detection methods.

As described above, the recognition type object detection unit 111 according to the present embodiment detects an object within the search range set by the search range setting unit 112. That is, the recognition type object detection unit 111 performs object detection within the search range narrowed down by the search range setting unit 112 using the tracking result in the previous frame. Therefore, the recognition type object detection unit 111 does not detect the object in the frame in the range where the possibility that the object exists is low, so that extra false detection can be reduced. Further, the recognition type object detection unit 111 can speed up the object detection process.

Further, the search range in which the recognition type object detection unit 111 detects an object may include not only the area indicated by the search range information but also the area determined by the silhouette information obtained by background subtraction and the like. Further, the recognition type object detection unit 111 may use a common portion of the area determined by the silhouette information and the area specified by the object search range information as an area (search range) for executing object detection.

Then, the recognition type object detection unit 111 generates a result (detection result) of performing object detection, and outputs the detection result to the common coordinate conversion unit 120. At this time, the coordinate values of the objects included in the detection result are the coordinate values of the individual coordinate system.

Returning to FIG. 4, the function of the common coordinate conversion unit 120 of the detection unit 100 will be described. The common coordinate conversion unit 120 receives the object detection result represented by the individual coordinate system from the object detection unit 110. Then, the individual coordinate conversion unit 130 converts the coordinate values of the individual coordinate system included in the received detection result into the coordinate values of the common coordinate system. As a result, the common coordinate conversion unit 120 can generate information for integrating the detection positions of the objects with respect to the individual cameras 20.

Specifically, the common coordinate conversion unit 120 shares the coordinate values of the individual coordinate system included in the object detection result by using the camera parameters of the camera 20 associated with the detection unit 100 including the common coordinate conversion unit 120. Convert to coordinate values in the coordinate system. For example, when the coordinates of the lower end of the object on the frame taken by the camera 20 are (x0, y0), the common coordinate conversion unit 120 converts them into the coordinates of the common coordinate system (X0, Y0,0). do. Here, since the ground is a plane of Z = 0, the component in the Z-axis direction is 0. Further, when the coordinates of the upper end of the object are (x1, y1) (here, it is assumed that the upper end of the object is directly above the lower end of the object (upper in the vertical direction)), the height of the object is H. At this time, the common coordinate conversion unit 120 converts the coordinates (x1, y1) of the upper end of this object into the coordinates (X0, Y0, H) of the common coordinate system. The common coordinate conversion unit 120 obtains the height of the object by searching for H that satisfies this. In this way, the common coordinate conversion unit 120 obtains the coordinate values (X, Y, Z) in the common coordinate system for each detected object.

When H is known, the common coordinate conversion unit 120 may use the known value as it is.

The common coordinate conversion unit 120 outputs the detection result including the coordinate values (coordinate values of the common coordinate system) after the coordinate conversion to the integrated tracking unit 200. That is, the common coordinate conversion unit 120 outputs the detection result represented by the common coordinate system to the integrated tracking unit 200. The common coordinate conversion unit 120 provides silhouette information and information such as feature quantities of appearance features (color, pattern, shape, etc.) of the object to each of one or a plurality of targets (objects) included in the detection result. It may be included as information about the object. Then, the common coordinate conversion unit 120 may output the detection result including these information to the integrated tracking unit 200.

(Details of Integrated Tracking Unit 200)
Next, with reference to FIG. 6, the functional configuration of the integrated tracking unit 200 will be described in more detail. FIG. 6 is a functional block diagram showing an example of a more detailed functional configuration of the integrated tracking unit 200 of the object tracking device 10 according to the present embodiment. As shown in FIG. 6, the integrated tracking unit 200 includes a prediction unit 210, a storage unit 220, an association unit 230, and an update unit 240. The integrated tracking unit 200 in the present embodiment is also referred to as a sequential tracking unit because the images from each camera 20 are sequentially tracked on a camera-by-camera basis.

Sequential object tracking (also referred to as sequential tracking of objects and sequential integrated tracking) performed by the integrated tracking unit 200 will be described with reference to FIG. 7. FIG. 7 is a diagram for explaining the sequential tracking process of objects performed by the integrated tracking unit 200 according to the present embodiment. FIG. 7 shows an example of timing for acquiring an image by each of the camera A, the camera B, and the camera C when the number of cameras is three. In FIG. 7, the horizontal axis indicates the time axis, and the right side indicates that the time is later. As shown in FIG. 7, the camera A acquires images at times t1, t5, and t8. Similarly, camera B acquires images at times t2, t4, t6 and t9, and camera C acquires images at times t3 and t7.

As shown in FIG. 7, the time (time stamp) of the frame (image) acquired by each camera 20 is not always the same in all the cameras 20, and is usually different. Further, the frame interval may be different for each camera 20, and may be non-uniform even in the same camera 20.

Then, the detection unit 100 detects the camera images asynchronously output between the cameras 20 in chronological order, and outputs the detection result to the integrated tracking unit 200.

The integrated tracking unit 200 according to the present embodiment executes sequential tracking processing in order from the image at the earliest time. That is, in the case of FIG. 7, the integrated tracking unit 200 first performs integration between a plurality of cameras and performs object tracking by using the object detection result for the image at time t1 of the camera A. After that, the integrated tracking unit 200 then integrates the plurality of cameras using the object detection result in the order of camera B time t2, camera C time t3, camera B time t4, and so on. And track the object. At this time, since not all the objects are visible from any camera 20, the integrated tracking unit 200 tracks the objects that are likely to be visible for each camera 20.

Returning to FIG. 6, each part of the integrated tracking unit 200 will be described.

The storage unit 220 stores tracker information associated with an object (target) included in the detection result received by the integrated tracking unit 200. The tracker information stored in the storage unit 220 is managed by the update unit 240 using the tracker ID. The information of the tracker is, for example, information about an object whose tracking result is included in the tracker, parameters including the likelihood of the tracker, and the like, but the present invention is not limited thereto. The information about the object includes information indicating the past position of the object, a motion model of the object, and the like, but the present invention is not limited thereto. The information about the object is included in the above-mentioned object detection result. Information may be included.

Note that FIG. 6 describes an example in which the storage unit 220 is built in the integrated tracking unit 200, but the present invention is not limited thereto. The storage unit 220 may be provided in the object tracking device 10 separately from the integrated tracking unit 200. Further, the storage unit 220 may be realized by a storage device or the like separate from the object tracking device 10.

When the storage unit 220 is not built in the integrated tracking unit 200, the storage unit 220 may be configured to store data or the like used in the object tracking device 10. For example, the storage unit 220 may store, for example, a camera image taken by the camera 20, a camera parameter of each camera 20, a range of coordinate values of a common coordinate system visible by each camera 20, and the like.

The prediction unit 210 refers to the storage unit 220 and predicts the position of the object on the current frame. Specifically, the prediction unit 210 predicts the current position of the object based on the motion model of the object by using the tracking result (tracker) of the object in the previous frame. Here, the information indicating the position of the object is expressed in the common coordinate system.

Further, the motion model of the object used by the prediction unit 210 to predict the position may be stored in the storage unit 220, or the prediction unit 210 uses the tracking result to predict the position of the object. It may be calculated before performing.

For example, a prediction process such as a Kalman filter or a particle filter can be applied to the prediction of the position of the object by the prediction unit 210. Further, the prediction unit 210 simply calculates the velocity of the object from the tracking results of the past several times, assumes constant velocity linear motion, predicts the amount of movement from the position in the previous frame from the velocity, and predicts the movement amount from the position in the previous frame. The current position may be predicted by adding to the position in.

Then, the prediction unit 210 outputs the prediction result to the matching unit 230.

The association unit 230 receives the detection result output from each of the detection units 100. In FIG. 6, the detection results (n) (n are 1 to N) indicate the detection results output from the detection unit 100-n. Further, the mapping unit 230 receives the prediction result from the prediction unit 210. Then, the matching unit 230 refers to the storage unit 220 and uses the prediction result to associate the target included in the detection result with the tracker.

The mapping unit 230 seeks the combination with the highest accuracy as a whole. The likelihood associated with a target m and a tracker k is the product of the likelihood Pm and ηk of the target m and the tracker k, respectively, and the likelihood qkm indicating that they are the same object. become. Therefore, the matching unit 230 calculates this value for each pair of the target and the tracker, and obtains the maximum combination as a whole.

Here, the likelihood of the target (first likelihood) is a value representing the certainty (accuracy) of object detection. The accuracy of object detection depends on the size of the object to be detected (called a detection object) on the screen (on the frame), the distance from the camera 20 to the detection position of the object, the appearance of the object from the camera 20, and the like. do.

For example, if the detected object is small and the size of the detected object is close to the limit of the size that can be detected, the accuracy of object detection is low. Further, if the size of the detected object deviates from the apparent size of the object assumed by the camera parameters, the accuracy of object detection becomes low. Further, when the detection position of the object is far from the camera 20, or the lighting condition for the area where the object exists is poor and the object is difficult to be detected, the accuracy of object detection is low. Further, when the actual appearance is significantly different from the data used for learning the classifier (for example, the angle is different), the accuracy of object detection is low.

The association unit 230 calculates the likelihood of the target by reflecting such characteristics. Specifically, the association unit 230 calculates the likelihood of the target using the likelihood information of the target included in the detection result received from the detection unit 100. When the detection unit 100 calculates the likelihood of the target and includes the calculated likelihood as the likelihood information of the target in the detection result, the matching unit 230 has the likelihood included in the likelihood information of the target. May be used as it is. It should be noted that the likelihood of the target does not have to reflect all the items described here, but may reflect only the main factors.

The tracker likelihood (second likelihood) is a value that represents the certainty (accuracy) of object tracking. The accuracy of object tracking depends on the tracking result of object tracking in the previous frame. For example, in the tracking results up to the (past) frame before the current frame, it can be said that the tracker that has a reliable correspondence with the target has a high accuracy of object tracking, and the tracker that does not have a good correspondence has a high accuracy of object tracking. Can be said to be low. Therefore, the mapping unit 230 may change the likelihood based on the result of whether or not the target and the tracker correspond to each other in each frame, and if they correspond, the likelihood of the tracker is increased and the correspondence is made. If it does not attach, the likelihood of the tracker may be lowered.

Further, at this time, if the position of the tracker is far from the camera 20, it is considered that the error in the position of the tracker becomes large. As a result, such a tracker becomes difficult to correspond to the object (target) included in the detection result. Therefore, the matching unit 230 may change the ratio of changing the likelihood of the tracker according to the position of the tracker and the distance from the camera 20. Further, the mapping unit 230 determines that the tracker has an angle (depression angle or elevation angle) between the horizontal plane including the camera 20 and the line-of-sight direction when the camera 20 sees an object whose tracking result is shown by the tracker. The ratio of changing the likelihood may be changed.

For example, when the object whose tracking result is shown by the tracker (hereinafter referred to as the tracker object) is close to the camera 20 and the depression angle of the camera 20 with respect to the object is larger than a predetermined angle, the accuracy of the position of the object is high. Therefore, the tracker and the target can easily correspond to each other. Therefore, in such a case, the matching unit 230 increases the ratio of changing the likelihood of the tracker.

Further, for example, when the tracker object is far from the camera 20 and the depression angle of the camera 20 with respect to the object is shallower than a predetermined angle, the size of the object on the frame taken by the camera 20 becomes small. In addition, a slight positional deviation on the image becomes a large deviation in the real space. Therefore, the accuracy of the detection position of the object is likely to be low. Therefore, in such a case, the matching unit 230 makes the ratio of changing the likelihood of the tracker smaller. In this way, the mapping unit 230 calculates the likelihood of the tracker.

As described above, the mapping unit 230 can preferentially reflect the detection result detected by the camera 20 close to the tracker's object to the tracker's likelihood, so that the tracking accuracy can be improved as a whole. It should be noted that the likelihood of the tracker does not have to reflect all the items described here, but may reflect only the main factors.

Further, the likelihood qkm representing the identity between the target m and the tracker k represents the probability that they are the same. If the object pointed to by the target and the object of the tracker are the same object, the positions of the target and the object of the tracker are likely to be close to each other. Therefore, the mapping unit 230 changes the likelihood according to the distance between the target and the tracker object. That is, the mapping unit 230 may increase the value of the likelihood qkm when the distance between the target m and the object of the tracker k is short, and decrease the value of the likelihood qkm when the distance is long. ..

At this time, if the target m is far from the camera 20 or the depression angle of the camera 20 with respect to the target m is shallower than a predetermined angle, the accuracy of the position of the target m is likely to be low. Therefore, the mapping unit 230 does not calculate the distance using a simple Euclidean distance, but uses the Mahalanobis distance in consideration of the error (ambiguity) included in the detection position of the target to obtain the target and the tracker object. You may find the distance between them. Further, in addition to the above method, the mapping unit 230 may control the degree of change in the likelihood qkm according to the distance in consideration of ambiguity. That is, when the ambiguity is larger, the mapping unit 230 makes the change in the likelihood qkm smaller according to the distance between the target m and the object of the tracker k. As a result, the mapping unit 230 reduces the influence of the misalignment of the target on the mapping.

In addition, the mapping unit 230 may also consider similarities in appearance between the target and the tracker. That is, the mapping unit 230 may extract features such as the color, pattern, and shape of the target and tracker objects, evaluate their similarity, and obtain the likelihood qkm.

For example, the mapping unit 230 may calculate the color histogram of the object for both the target and the tracker object, evaluate the similarity between them by overlapping the color histograms, and reflect the color histogram in the likelihood qkm. .. Like the tracker likelihood ηk and the target likelihood Pm, the likelihood qkm, which represents the identity of the target and tracker, does not have to reflect all of the above items, but only the main factors. You may do it.

Further, the mapping unit 230 may calculate each of the above likelihoods in consideration of the case where the object is out of the angle of view of the camera 20 or is shielded by another object and is not detected. As a result, the integrated tracking unit 200 can track the object with high accuracy even if the object is not detected or goes out of the angle of view.

As described above, the problem of calculating each likelihood and finding the correspondence between the target and the tracker that maximizes each likelihood as a whole is solved by converting each likelihood into a cost by a monotonous non-increasing function. , Can be reduced to the least costly allocation problem (which target corresponds to which tracker). This allocation problem can be efficiently calculated by, for example, a method such as the Hungarian method.

Then, the association unit 230 outputs the result of the association to the update unit 240. The result of this mapping includes information indicating which target corresponds to the tracker, and at least each of the above likelihoods including the likelihood of the tracker.

In the present embodiment, the mapping unit 230 performs the mapping using both the likelihood of the target and the likelihood of the tracker, but the mapping is performed using either of the likelihoods. You may make a mapping.

The update unit 240 updates the tracker information. Then, the update unit 240 generates this tracker as a new object tracking result. Specifically, the update unit 240 receives the result of the association from the association unit 230. Then, the update unit 240 calculates the current position of the tracker object based on this result. Then, the update unit 240 updates the tracker information stored in the storage unit 220. The information to be updated is, for example, parameters such as the position and / or size of the object whose tracker contains the tracking result, the motion model of the object, and the likelihood of the tracker, but the present invention is limited thereto. It's not a thing. The update unit 240 may update the updated information among the information stored in the storage unit 220.

First, the calculation of the current position of the tracker object by the update unit 240 will be described. The update unit 240 calculates the current position of the tracker object in consideration of the accuracy of the target position. For example, it is assumed that the update unit 240 weights the predicted position where the position of the object is predicted using the tracker and the detection position of the target corresponding to the tracker, and calculates the current position of the object of the tracker. .. In this case, the update unit 240 may control the weight depending on the accuracy of the position of the target.

For example, if the target is located away from the camera 20 and the depression angle of the camera 20 with respect to the target is shallow, the accuracy of the position of the target is likely to be low. In such a case, the update unit 240 makes the weight for the position of the target smaller.

On the other hand, when the target is located near the camera 20 and the depression angle with respect to the camera 20 target is larger than a predetermined value, the accuracy of the position of the target is assumed to be high. In such a case, the update unit 240 increases the weight for the position of the target.

The update unit 240 calculates the current position of the tracker object using the predicted position and the position where the weight is set.

In this way, the update unit 240 determines the weight for the position of the target, so that the prediction result of the detection position of the object by the camera 20 near the target is more strongly reflected. Therefore, the object tracking device 10 can improve the accuracy of predicting the position of the object.

Then, the update unit 240 updates the latest position of the object included in the information about the object stored in the storage unit 220 to the current position of the calculated object.

Next, the update of the likelihood of the tracker performed by the update unit 240 will be described.

If the tracker object is located away from the camera 20, the size of the object will be smaller. Therefore, it becomes difficult for the detection unit 100 to detect such an object.

Further, a case where the recognition type object detection unit 111 of the detection unit 100 performs recognition type object detection when the object included in the frame looks different from the appearance of the object used for learning will be described. When the object included in the frame looks different from the object used for learning, for example, the depression angle of the camera 20 with respect to the object assumed from the position of the object on the tracker and the object used for learning. This is a case where the depression angle of the camera 20 with respect to the object is significantly different. In such a case, the recognition type object detection unit 111 of the detection unit 100 has difficulty in detecting the object included in the frame.

In situations where it is difficult to detect such objects, the objects contained in the frame may be undetected. In this case, there may not be a target associated with the tracker associated with this object.

Therefore, the update unit 240 suppresses the change in the likelihood of the tracker of the object that is difficult to detect among the trackers that are not related to the target.

In this way, the update unit 240 can suppress the influence on the tracking when the object is difficult to be detected, and can greatly reflect the detection result by the camera which is easy to detect in the likelihood of the tracker. Then, when tracking the object for the next frame, the matching unit 230 performs the mapping based on the likelihood of this tracker, so that the integrated tracking unit 200 can further improve the accuracy of the object tracking. ..

Then, among the likelihoods of the trackers stored in the storage unit 220, the likelihood of the tracker calculated by the matching unit 230 and the likelihood of the tracker that suppresses the above change to a small value are updated.

Next, the update unit 240 will explain the update of the motion model of the tracker object. For example, a case where the integrated tracking unit 200 performs object tracking by predicting the position of an object using a Kalman filter will be described. In this case, the update unit 240 substitutes the position coordinates of the tracker object associated with the target into the update formula of the state variable of the Kalman filter stored in the storage unit 220 as the detected position coordinates, and the state of the Kalman filter. To update.

In addition to the above, the update unit 240 updates other parameters of the tracker stored in the storage unit 220.

For example, the object itself may change its posture. For example, when an object is a person, the apparent height of the object changes when the person crouches or bends down. As described above, when the size of the object or the like is changed in addition to the motion model, the updating unit 240 updates the changed information among the information stored in the storage unit 220.

Further, when the tracker parameters include the probability that the tracker object exists, the weight indicating the reliability of the tracking result, and the like, the weight changes according to the result of the mapping by the matching unit 230. Therefore, the update unit 240 updates the parameters such as the weight.

Further, the update unit 240 creates and deletes a tracker as an update of the tracker. First, the update unit 240 determines whether or not there is a target that does not correspond to the tracker after the mapping process by the association unit 230. The target that does not correspond to this tracker may be a newly appearing object within the range photographed by the camera 20. Therefore, when there is a target that does not correspond to the tracker, the update unit 240 determines whether or not this target can be regarded as a newly appearing object within the above range.

That is, when there is a target that does not correspond to the tracker, the update unit 240 evaluates the probability that this target exists. Then, the update unit 240 determines whether or not this probability is equal to or greater than a predetermined value. Then, when this probability is equal to or greater than a predetermined value, the update unit 240 determines that the object indicated by this target is an object newly appearing within the above range. Then, the update unit 240 newly creates a tracker related to the object (target) determined to be an object newly appearing within the above range.

Further, the update unit 240 determines whether or not there is a tracker that does not correspond to the target. The tracker that does not correspond to this target may be a tracker for an object that has disappeared (moved from within range to out of range) within the range captured by the camera 20. Therefore, if there is a tracker that does not correspond to the target, the update unit 240 determines whether or not this target can be regarded as a tracker for an object that has disappeared from the above range.

That is, if there is a tracker that does not correspond to the target, the update unit 240 evaluates the probability that an object related to this tracker exists. Then, the update unit 240 determines whether or not this probability is less than a predetermined value. Then, when this probability is less than a predetermined value, the update unit 240 determines that the object related to this tracker is an object that has disappeared from the above range. Then, the update unit 240 deletes the tracker related to the object determined to be an object that has disappeared from the above range.

The probability that an object exists is determined by the likelihood of the tracker. That is, if there is a tracker that does not correspond to the target, the update unit 240 reduces the likelihood of this tracker. Then, when the value of the likelihood of the tracker falls below a predetermined threshold value, the update unit 240 deletes the tracker.

Then, the update unit 240 generates the finally remaining tracker as the tracking result of the object in this frame. Then, the update unit 240 outputs the information indicating the position of the tracker, the information regarding the size of the object of the tracker, and the like as the information (tracking information) indicating the tracking result of the object tracking.

As described above, according to the object tracking device 10 according to the present embodiment, the object tracking is performed in order from the oldest time information included in the camera image output from each camera 20. Objects detected from the video data of each of the plurality of cameras 20 are detected by integrating the highly accurate detection results for each camera 20 and using the detection results and the tracking results reflecting the past tracking results. Therefore, the tracking accuracy of the object tracking performed by the integrated tracking unit 200 of the object tracking device 10 is improved.

Next, the flow of the object tracking process of the object tracking device 10 according to the present embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing an example of the flow of the object tracking process of the object tracking device 10 according to the present embodiment.

As shown in FIG. 8, first, the recognition type object detection unit 111 of the detection unit 100 receives the camera image from the camera 20 associated with the detection unit 100 including the object detection unit 110 (step S81).

The detection unit 100 confirms whether or not the frame of the received camera image is the first frame output from the camera 20 (step S82), and if it is the first frame (YES in step S82), the process is stepped. Proceed to S85.

When the frame of the received camera image is not the first frame (NO in step S82), the individual coordinate conversion unit 130 expresses the tracking information for the previous frame output from the integrated tracking unit 200 in the individual coordinate system. It is converted into tracking information (step S83).

Then, the search range setting unit 112 of the object detection unit 110 sets the search range of the object for the current frame using the tracking information converted in step S83 (step S84).

Then, the recognition type object detection unit 111 detects an object from the received camera image (step S85).

Next, the individual coordinate conversion unit 130 converts the detection result by the recognition type object detection unit 111 into the detection result represented by the common coordinate system (step S86).

Next, the prediction unit 210 of the integrated tracking unit 200 predicts the position of the object on the current frame using the information of the tracker (step S87).

Then, the matching unit 230 associates the object (target) included in the detection result with the tracker (step S88).

Next, the update unit 240 updates the tracker information such as the position of the tracker object and the motion model of the object (step S89).

Further, the update unit 240 creates and / or deletes the tracker (step S90). Then, the object tracking device 10 repeats this process until no frame is input to the detection unit 100.

(effect)
As described above, according to the object tracking device 10 according to the present embodiment, it is possible to track an object with higher accuracy. This is because the detection unit 100 detects an object from the output information of the camera 20 based on the tracking information for the output information before the output information (video frame). Then, the integrated tracking unit 200 tracks the object based on the plurality of detection results output by each detection unit 100, and generates the tracking information of the object represented by the common coordinate system.

For example, if there is an object that cannot be seen by one camera 20 but is visible by another camera 20, the object tracking device 10 can also track the object that is not visible by one camera 20 as an object in the image of this camera 20. Used for detection. As a result, the detection unit 100 can suitably detect this object when the same object appears in the range visible from the certain camera 20. Therefore, the object tracking device 10 can accurately track the object with respect to this object.

Therefore, the object tracking device 10 can improve the detection accuracy of the object as compared with the case where the tracking result for the previous frame is not used. Further, since the object tracking device 10 tracks the object using all the detection results having high detection accuracy, the accuracy of the tracking result obtained as a whole is also improved.

As described above, the object tracking device 10 according to the present embodiment can extract the flow line of a person or the like moving across the area photographed by each of the plurality of cameras 20. As a result, the tracking result by the object tracking device 10 can be used as basic information for marketing or changing the layout of the store by analyzing the behavior of the customer who wanders around the store, for example. In addition, this tracking result can be used to detect a person who roams between areas for security purposes.

Further, since the search range setting unit 112 sets the search range for detecting the object in the camera image by using the tracking result, the recognition type object detection unit 111 can reduce extra false detection. Further, the recognition type object detection unit 111 can speed up the object detection process.

Further, when the integrated tracking unit 200 performs object tracking using the likelihood of the target and / or the likelihood of the tracker, the object tracking device 10 can obtain a more reliable object tracking result. Further, since the object tracking device 10 performs an object search using the object tracking result obtained in this way, the accuracy of the object tracking can be further improved. As a result, the object tracking device 10 can improve the accuracy of object tracking as a whole.

<Second embodiment>
Next, a second embodiment of the present invention will be described with reference to the drawings. For convenience of explanation, the members having the same functions as the members included in the drawings described in the above-described first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

The object tracking system 2 according to the present embodiment is configured to include an object tracking device 50 instead of the object tracking device 10 of the object tracking system 1 according to the first embodiment described with reference to FIG. Since the other system configurations of the object tracking system 2 are the same as those of the object tracking system 1 shown in FIG. 2, the description thereof will be omitted.

(Object tracking device 50)
The function of the object tracking device 50 will be described with reference to FIG. FIG. 9 is a functional block diagram showing an example of the functional configuration of the object tracking device 50 according to the present embodiment. As shown in FIG. 9, the object tracking device 50 includes a plurality of detection units (100-1 to 100-N), an integrated tracking unit 200, and a display control unit 300. As in the first embodiment described above, in the present embodiment, when the plurality of object detection units (100-1 to 100-N) are not distinguished from each other, or when they are collectively referred to, these are referred to. It is called a detection unit 100.

The display control unit 300 controls an image (video) to be displayed on the display device 30. Specifically, the display control unit 300 generates display data obtained by converting the tracking information output by the integrated tracking unit 200 into data that can be displayed on the display device 30, and transmits the display data to the display device 30. The tracking information output by the integrated tracking unit 200 is represented by a common coordinate system. Therefore, the display control unit 300 generates display data that can be displayed on the display device 30 in the common coordinate system.

At this time, the integrated tracking unit 200 uses information necessary for displaying the flow line of the object on the display device 30 (for example, information indicating the past position of the tracker object) as tracking information, and the display control unit 300. It is preferable to output to. This tracking information may be the same as the information fed back to the detection unit 100, or may be different.

Then, the display device 30 displays the received display data on the screen. As a result, the object tracking device 50 can present the tracking result to the user.

Further, the detection unit 100 of the object tracking device 50 according to the present embodiment generates display data obtained by converting the search range set by the search range setting unit 112 of the object detection unit 110 into data that can be displayed on the display device 30. Then, it is transmitted to the display device 30. The search range information output by the search range setting unit 112 is represented by an individual coordinate system. Therefore, the display control unit 300 generates display data that can be displayed on the display device 30 in the individual coordinate system by using the camera parameters of the camera 20 associated with the detection unit 100 that outputs the search range information.

Then, the display device 30 displays the received display data on the screen. Thereby, the object tracking device 50 can present the search range to the user by using the search range information of the object output from each detection unit 100.

The number of display devices 30 may be plural. For example, the display device 30 receives the display data displayed in the common coordinate system and the display data displayed in the individual coordinate system by different display devices 30, and displays the received display data on the screen in each case. It may be configured. Further, the display device 30 may be configured to divide the display area of one screen and display a plurality of display data on the screen. As described above, the display data display method in the display device 30 according to the present embodiment is not particularly limited.

Further, the display control unit 300 may be provided in the detection unit 100, respectively. FIG. 10 is a functional block diagram showing an example of the functional configuration of the detection unit 100 of the object tracking device 50 according to the present embodiment. As shown in FIG. 10, the detection unit 100 includes an object detection unit 110, a common coordinate conversion unit 120, an individual coordinate conversion unit 130, and a display control unit 150. Further, the object detection unit 110 includes a recognition type object detection unit 111 and a search range setting unit 112, similarly to the object detection unit 110 shown in FIG.

The search range setting unit 112 shown in FIG. 10 outputs search range information indicating the set search range to the display control unit 150. The display control unit 150 receives the search range information output from the search range setting unit 112, and generates display data converted into data that can be displayed on the display device 30 in the same manner as the display control unit 300. The search range information output by the search range setting unit 112 is represented by an individual coordinate system. Therefore, the display control unit 150 generates display data that can be displayed on the display device 30 in the individual coordinate system by using the camera parameters of the camera 20 associated with the detection unit 100 including the display control unit 150.

Then, the display control unit 150 transmits the generated display data to the display device 30. Then, the display device 30 displays the received display data on the screen.

(Application example)
An application example of the object tracking device 50 according to the present embodiment will be described with reference to FIGS. 11 to 14. 11 to 14 are diagrams for explaining an application example of the object tracking device 50 according to the present embodiment.

First, FIG. 11 is a diagram showing an example of a room in which a shelf R1 and a shelf R2 and a plurality of cameras (A to F) are installed and viewed from a direction opposite to the direction of gravity. As shown in FIG. 11, the horizontal direction of FIG. 11 is the X axis in the common coordinate system, and the vertical direction is the Y axis. The shelves R1 and the shelves R2 are installed side by side on the X-axis so that the longitudinal direction is parallel to the Y-axis direction.

The camera A is installed at a position close to the doorway of this room. In the present embodiment, it is considered that the entire room is photographed by the cameras A to F. That is, as shown in FIG. 11, the indoor space in which a plurality of cameras (A to F) are installed becomes a shooting space. Further, the cameras A to F are photographing a place common to each other.

FIG. 12 is a diagram showing an example of images taken by each of the camera A and the camera B. The upper view of FIG. 12 is a diagram showing a frame having an image taken by the camera A, and the lower figure is a diagram showing a frame having an image taken by the camera B. The coordinate values in these frames are represented by the coordinate values of the individual coordinate system for each camera.

The object tracking device 50 in the present embodiment may be configured to display the image captured by the camera 20 on the display device 30.

As shown in FIG. 12, the image taken by the camera A includes the person C1. Further, since the camera A is installed near the doorway, the doorway is included in this image. Further, the image taken by the camera B includes the person C1 and the person C2.

The person C2 is hidden behind the shelf R1 when viewed from the camera A. Therefore, at the time of the image of FIG. 12, the person C2 is an object that cannot be seen from the camera A. If the frame of these images is the first frame of the images taken by the cameras A and B, the object tracking device 50 detects and tracks the object from these frames because there is no tracking information in the previous frame. Generate information.

Then, the search range setting unit 112 sets the search range of the object for the next frame of the image captured by the camera A by using the tracking information expressed in the individual coordinate system. Similarly, the search range setting unit 112 sets the search range of the object for the next frame of the image captured by the camera B by using the tracking information expressed in the individual coordinate system.

FIG. 13 is a diagram showing an example of the search range displayed on the display device 30. The upper diagram of FIG. 13 is a diagram showing an example of an object search range for a frame output from the camera A, and the lower diagram is an example of an object search range for a frame output from the camera B. It is a figure which shows.

As shown in the upper diagram of FIG. 13, the search range setting unit 112 obtains the search range A1 from the position of the person C1 in the upper diagram of FIG. 12. Further, the search range setting unit 112 obtains the area of the entrance / exit portion into the room as the search range N1. Further, the search range setting unit 112 obtains the outer edge portion of the frame as the search ranges N2 and N3. Then, the search range setting unit 112 outputs the information that summarizes the obtained search ranges A1 and N1 to N3 as the search range information to the recognition type object detection unit 111 and the display control unit 150 or the display control unit 300. Then, the display control unit 150 or the display control unit 300 converts the search range indicated by the search range information on the display device 30 into display data that can be displayed on the screen, and transmits the display data to the display device 30.

The display device 30 that has received the display data from the display control unit 150 or the display control unit 300 displays the search range on the screen as shown in the upper diagram of FIG.

Next, the lower figure of FIG. 13 will be described. As shown in the lower figure of FIG. 13, the search range setting unit 112 obtains the search range B1 and the search range B2 from the positions of the person C1 and the person C2 in the lower figure of FIG. 12, respectively. Further, the search range setting unit 112 obtains the outer edge portion of the frame by the search range N4 to N7. Then, the search range setting unit 112 outputs the information that summarizes the obtained search ranges B1, B2, N4 to N7 as the search range information to the recognition type object detection unit 111 and the display control unit 150 or the display control unit 300. Then, the display control unit 150 or the display control unit 300 converts the search range indicated by the search range information on the display device 30 into display data that can be displayed on the screen, and transmits the display data to the display device 30.

The display device 30 that has received the display data from the display control unit 150 or the display control unit 300 displays the search range on the screen as shown in the lower diagram of FIG.

The display device 30 may display the search range so as to be different for each area. For example, the display device 30 may display the search range for the already detected object and the search range for the outer edge of the frame in different colors.

Then, the integrated tracking unit 200 generates a tracker for the person C1 and the person C2 detected in the subsequent frame. Then, the integrated tracking unit 200 outputs the information necessary for displaying the flow lines of the person C1 and the person C2 as tracking information to the display control unit 300.

Upon receiving the tracking information from the integrated tracking unit 200, the display control unit 300 converts the tracking information into display data that can be displayed on the display device 30, and transmits the display data to the display device 30.

Then, the display device 30 displays the display data received from the display control unit 300 on the screen. FIG. 14 is a diagram showing an example when the display device 30 displays a flow line showing the tracking results of each of the person C1 and the person C2 on the screen (display screen). As shown in FIG. 14, in this application example, it is assumed that the tracking result of the object is displayed on the display screen in the XY plane in the common coordinate system. In FIG. 14, the flow line of the person C1 is displayed as a solid line, and the flow line of the person C2 is displayed as a alternate long and short dash line. In this way, the display device 30 can display the tracking result of the object on the screen.

<Third embodiment>
Next, a third embodiment of the present invention will be described with reference to the drawings. For convenience of explanation, the members having the same functions as the members included in the drawings described in the first and second embodiments described above are designated by the same reference numerals, and the description thereof will be omitted.

The object tracking device 10 according to the present embodiment is configured to include a detection unit 400 instead of the detection unit 100 of the object tracking device 10 shown in FIG. The configuration of the detection unit 400 will be described with reference to FIG. FIG. 15 is a functional block diagram showing an example of the functional configuration of the detection unit 400 of the object tracking device 10 according to the present embodiment.

The detection unit 400 includes an object detection unit 140 in place of the object detection unit 110 of the detection unit 100 shown in FIGS. 4 and 5. Further, the detection unit 400 is configured to further include a storage unit 160. That is, as shown in FIG. 15, the detection unit 400 according to the present embodiment includes an object detection unit 140, a common coordinate conversion unit 120, an individual coordinate conversion unit 130, and a storage unit 160.

In the present embodiment, the configuration in which the object detection unit 140 is provided instead of the object detection unit 110 of the detection unit 100 of the object tracking device 10 according to the first embodiment will be described as an example. The present invention is not limited to this, and the object detection unit 140 may be provided instead of the object detection unit 110 of the detection unit 100 of the object tracking device 50 according to the second embodiment. That is, the detection unit 100 according to the present embodiment may be configured to output data to be displayed to the display control unit 150 or the display control unit 300.

The storage unit 160 stores camera parameters for each camera 20 used when converting the coordinate system. Further, the storage unit 160 is used by the individual coordinate conversion unit 130 to confirm whether or not the coordinate values of the common coordinate system included in the tracking information are included in the range photographed by the associated camera 20. Information indicating the range of coordinate values of the common coordinate system is stored. Further, the image captured by the camera 20 may be stored in the storage unit 160. Note that this video may be temporarily stored.

Note that FIG. 15 describes an example in which the storage unit 160 is built in the detection unit 400, but the present invention is not limited thereto. The storage unit 160 may be provided in the object tracking device 10 separately from the detection unit 400. Further, the storage unit 160 may be realized by a storage device or the like separate from the object tracking device 10.

Next, the detailed functional configuration of the object detection unit 140 of the detection unit 400 will be described with reference to FIG. FIG. 16 is a functional block diagram showing an example of the functional configuration of the object detection unit 140 of the detection unit 400 according to the present embodiment. As shown in FIG. 16, the object detection unit 140 includes a recognition type object detection unit (first object detection means) 141, a non-recognition type object detection unit (second object detection means) 142, and a detection parameter update unit 143. And the detection result integration unit 144.

In this embodiment, object detection using a dictionary (identifier) or the like is referred to as "recognition-type object detection". On the other hand, object detection that does not use a classifier or the like is called "unrecognized object detection".

The recognition type object detection unit 141 detects an object from the camera image input to the recognition type object detection unit 141. The recognition type object detection unit 141 detects an object for the entire frame. As shown by the broken line in FIG. 16, the recognition type object detection unit 141 may perform object detection based on the search range information output from the detection parameter update unit 143, which will be described later. At this time, the recognition type object detection unit 141 performs object detection in the same manner as the recognition type object detection unit 111 described in the first embodiment.

Further, when the search range information is not output from the detection parameter update unit 143, the recognition type object detection unit 141 does not perform object detection for the entire frame, but may perform object detection using another standard. good. For example, the recognition type object detection unit 141 may detect an object only in a region having a silhouette and a region around the silhouette by using the silhouette information.

The recognition type object detection unit 141 outputs the detection result of the object detection as the first detection result to the detection result integration unit 144.

Further, the recognition type object detection unit 141 may extract the appearance characteristics of the object at this point in preparation for the object detection by the non-recognition type object detection unit 142 described later. The appearance characteristics of the object include, but are not limited to, information such as the color, pattern, and shape of the object. The recognition type object detection unit 141 extracts these feature quantities as the appearance features of the object. At this time, the area used for object detection by the recognition type object detection unit 141 and the area used for object detection by the non-recognition type object detection unit 142 do not have to be the same. For example, when the object is a person, the object detection by the recognition type object detection unit 141 detects the head, and the object detection by the non-recognition type object detection unit 142 detects the area up to the clothes. At this time, the recognition type object detection unit 141 extracts the feature amount of the appearance feature of the object so as to include the area of the clothes. Then, the recognition type object detection unit 141 may output the extracted feature amount as template information together with the information indicating the area used for the extraction (extraction area information). Further, the recognition type object detection unit 141 may hold the feature amount itself inside the recognition type object detection unit 141 and output only the information for identifying the feature amount.

The detection parameter update unit 143 receives the tracking information expressed in the individual coordinate system from the individual coordinate conversion unit 130. Then, the detection parameter update unit 143 obtains a parameter (referred to as a detection parameter) necessary for object detection by using this tracking information. These detection parameters are parameters required for the object detection process. The detection parameters include, for example, the predicted position (predicted area) of the object in the current frame, the search range to which the object detection is applied, the size of the template used for template matching, and the features of the target template previously associated with the tracker (template). Information) etc. are included. It should be noted that the detection parameters do not have to include all of these information, and may include parameters necessary for object detection by the non-recognition type object detection unit 142. In addition, the detection parameter may include the target information corresponding to the tracker in the tracking result of the object in the previous frame.

For example, the detection parameter update unit 143 uses the position where the object exists in the current frame as the predicted position for the target (object) detected in the previous frame and associated with the tracker based on the tracking information of the object. Ask. This prediction process is the same as the prediction process of the predicted position in the search range setting unit 112 according to the first embodiment. The detection parameter update unit 143 may obtain a region including this predicted position as a predicted region.

Further, for example, the detection parameter update unit 143 may obtain a range to which object detection by template matching is applied centering on the prediction area, and may include this range as a search range for objects included in the detection parameter.

The detection parameter update unit 143 obtains the detection parameter for each object included in the tracking information. Then, the detection parameter update unit 143 updates the obtained detection parameter as a detection parameter used for the object detection process. Then, the detection parameter update unit 143 outputs this detection parameter to the non-recognition type object detection unit 142.

Note that the detection parameter update unit 143 uses the tracking information converted into the coordinate values of the individual coordinate system to use the tracking information of the object, similarly to the search range setting unit 112 of the detection unit 100 according to the first embodiment described above. You may find the search range. Then, the detection parameter update unit 143 may output the search range information indicating the search range of the obtained object to the recognition type object detection unit 141.

The non-recognition type object detection unit 142 receives the detection parameter from the detection parameter update unit 143. Then, the non-recognition type object detection unit 142 detects an object from the camera image input to the non-recognition type object detection unit 142 based on the received detection parameter. Unlike the recognition type object detection unit 141, the non-recognition type object detection unit 142 detects an object based on the similarity in appearance of the objects detected in the previous frame.

That is, when the object is detected in the previous frame, the non-recognition type object detection unit 142 stores the image feature of the area (or the partial image itself of the detection area) as a template. Then, the non-recognition type object detection unit 142 performs object detection by checking whether or not an area similar to the stored template exists in the current frame by template matching. As the characteristics of the image used at this time, for example, information on the color pattern and distribution, distribution information of the edge and the luminance gradient, or a feature formed by combining these can be used.

The detection parameters used when the non-recognition type object detection unit 142 performs object detection are controlled by the detection parameters output from the detection parameter update unit 143. Specifically, the non-recognition type object detection unit 142 performs object detection by template matching for the predicted position (prediction area) of the object and its vicinity predicted by the detection parameter update unit 143. That is, the non-recognition type object detection unit 142 sets the search range for template matching around the predicted object existence range (prediction area), and performs template matching to the periphery thereof. At this time, the non-recognition type object detection unit 142 may also consider that the apparent size of the object changes due to the movement of the position of the object. This change can be calculated using camera parameters. Therefore, the non-recognition type object detection unit 142 may calculate the change in the size of the object, reflect it in the template, and then perform template matching.

Further, the template information for which the non-recognition type object detection unit 142 performs template matching may be the feature amount extracted by the recognition type object detection unit 141 in the object detection process in the previous frame.

As described above, since the non-recognition type object detection unit 142 performs object detection based on the tracking result for the object tracked by the integrated tracking unit 200, the detection accuracy is improved as compared with the case where the tracking result is not used. Can be made to.

Then, the non-recognition type object detection unit 142 outputs the detection result of the object detection as the second detection result to the detection result integration unit 144.

The detection result integration unit 144 receives the first detection result from the recognition type object detection unit 141. Further, the detection result integration unit 144 receives the second detection result from the non-recognition type object detection unit 142. Then, the detection result integration unit 144 integrates the first detection result and the second detection result. Then, the detection result integration unit 144 outputs the integrated result to the common coordinate conversion unit 120 as the detection result of the object detection in the object detection unit 140.

There may be an object contained in both the first detection result and the second detection result, and an object contained in only one of them. Therefore, the detection result integration unit 144 associates and integrates the objects included in each of the first detection result and the second detection result. For this correspondence, for example, the degree of overlap of the object areas can be used.

That is, the detection result integration unit 144 calculates the overlap ratio between the object areas (for example, the overlap ratio of the object circumscribed rectangles), and when this becomes larger than a predetermined value, the object included in the first detection result and the first It is associated with the object included in the detection result of 2.

Further, the detection result integration unit 144 may formulate a graph problem in which the overlap ratio of the areas between the objects is used as a weight, and associate the objects with each other. For example, the detection result integration unit 144 converts the overlap ratio into a cost by a monotonous non-increasing function, and then calculates the optimum correspondence by using the Hungarian method or the like to perform the correspondence between the objects.

As a result of the mapping, the detection result integration unit 144 may merge the values used at the time of the mapping (for example, the overlap ratio or the cost) larger than a predetermined value at this point. Further, the detection result integration unit 144 may generate information that the detection result integration unit 144 does not merge at this point and corresponds. Then, the detection result integration unit 144 outputs the result of adding the information that the first detection result and the second detection result are associated with the detection result as the detection result of the object detection unit 140. Tracking may be performed using the associated information at the time of integrated tracking.

Further, the non-recognition type object detection unit 142 outputs the second detection result based on the tracking result of the object with respect to the previous frame. Therefore, the second detection result may be generated later than the first detection result. In such a case, the detection result integration unit 144 temporarily stores the first detection result in a storage means such as a buffer or a storage unit 160 in the detection result integration unit 144. Then, the detection result integration unit 144 may integrate both results when the second detection result for the frame corresponding to the frame to be the target for generating the first detection result is received.

As described above, the detection unit 400 of the object tracking device 10 according to the present embodiment has the object detection result (first detection result) by the recognition type object detection unit 141 and the object by the non-recognition type object detection unit 142. The result of integrating the detection result (second detection result) is output as the detection result. At this time, the non-recognition type object detection unit 142 detects the object by performing template matching based on the tracking result for the object tracked by the integrated tracking unit 200. As a result, the detection unit 400 can further improve the accuracy of object detection as compared with the case where only object detection (recognition type object detection) by identifying an object is performed.

Therefore, the object tracking device 10 can track the object with higher accuracy.

<Fourth Embodiment>
Next, a fourth embodiment of the present invention will be described with reference to the drawings. For convenience of explanation, the members having the same functions as the members included in the drawings described in the above-described embodiments are designated by the same reference numerals, and the description thereof will be omitted.

The object tracking device 10 according to the present embodiment is configured to include an integrated tracking unit 500 instead of the integrated tracking unit 200 of the object tracking device 10 shown in FIG. The configuration of the integrated tracking unit 500 will be described with reference to FIG. FIG. 17 is a functional block diagram showing an example of the functional configuration of the integrated tracking unit 500 of the object tracking device 10 according to the present embodiment. As shown in FIG. 17, the integrated tracking unit 500 includes a buffer unit 510, a prediction unit 210, a storage unit 220, an association unit 530, and an update unit 240.

In the present embodiment, a configuration including an integrated tracking unit 500 instead of the integrated tracking unit 200 of the object tracking device 10 according to the first embodiment will be described as an example. The present invention is not limited to this, and may be configured to include the integrated tracking unit 500 instead of the integrated tracking unit 200 of the object tracking device 50 according to the second embodiment. That is, the integrated tracking unit 500 according to the present embodiment may be configured to output data to be displayed to the display control unit 300.

Further, the detection unit that outputs the detection result to the integrated tracking unit 500 according to the present embodiment may be the detection unit 400 described in the third embodiment.

The buffer unit 510 is a means for temporarily storing the detection result represented by the common coordinate system output from the detection unit 100. Then, among the data buffered in the buffer unit 510 (detection result), the data whose time information included in the detected camera image is within a predetermined period is acquired by the association unit 530. This predetermined period is a periodic period. Then, the matching unit 530 performs object tracking using one or a plurality of detection results acquired in a certain cycle. As described above, the integrated tracking unit 500 in the present embodiment is also referred to as a batch tracking unit because the object tracking is performed using the images of a plurality of cameras among the images from each camera 20.

The object tracking (also referred to as collective integrated tracking) performed by the integrated tracking unit 500 will be described with reference to FIG. FIG. 18 is a diagram for explaining the batch tracking process of objects performed by the integrated tracking unit 500 according to the present embodiment. FIG. 18 shows an example of timing for acquiring an image by each of the camera A, the camera B, and the camera C when the number of cameras is three, as in the case of FIG. 7. In FIG. 18, the horizontal axis indicates the time axis, and the right side indicates that the time is later. As shown in FIG. 18, camera A acquires images at times t1, t5, and t8. Similarly, camera B acquires images at times t2, t4, t6 and t9, and camera C acquires images at times t3 and t7.

Then, the images acquired at each of these timings are subject to object detection in order. In the following, for convenience of explanation, the time shown in FIG. 18 will be described assuming that the time shown in FIG. 18 is substantially the same as the time when the object detection result is output. That is, it is assumed that the time t1 is the time when the detection result for the frame of the image captured by the camera A is output from the detection unit 100 and buffered in the buffer unit 510.

The time axis at the bottom of FIG. 18 shows an example of a periodic period.

The association unit 530 acquires a detection result in which the time information included in the camera image for which the object is detected is within a predetermined period among the one or a plurality of detection results buffered in the buffer unit 510. As described above, this predetermined period is a periodic period. Further, in the present embodiment, the buffered time and the time of the camera image are considered to be the same. Therefore, it can be said that the association unit 530 acquires one or a plurality of detection results buffered in the buffer unit 510 at a predetermined cycle.

Specifically, the mapping unit 530 first acquires the detection result buffered in the first period T1. That is, the association unit 530 acquires the detection result buffered at the times t1, t2, and t3. The detection result buffered at time t1 is the detection result for the frame of the image captured by the camera A. Further, the detection result buffered at time t2 is the detection result for the frame of the image captured by the camera B, and the detection result buffered at time t3 is the detection result for the frame of the image captured by the camera C. The result. Therefore, the association unit 530 is a detection result buffered in the buffer unit 510 within a predetermined period (T1 in this case), and is a plurality of detection results for the frame of the image taken by each of the plurality of cameras 20. Is obtained from the buffer unit 510. Then, the mapping unit 530 performs object tracking using the acquired detection result.

Similarly, in the period T2, T3, and T4, the mapping unit 530 acquires the detection result buffered within this periodic period and performs object tracking.

In the present embodiment, the configuration in which the mapping unit 530 acquires the data (a plurality of detection results) buffered in the buffer unit 510 from the buffer unit 510 at a predetermined cycle will be described, but the mapping unit 530 will be described. May be configured to receive this data from the buffer unit 510 at a predetermined cycle. That is, the buffer unit 510 may have a function of outputting this data to the association unit 530 at a predetermined cycle.

Returning to FIG. 17, the mapping unit 530 of the integrated tracking unit 500 will be further described.

The association unit 530 obtains the distance between the targets from the positions of the targets included in each of the acquired detection results, and associates the targets with the short distances with each other. At this time, the mapping unit 530 performs mapping by a method such as the Hungarian method using the distance between the targets. In addition to the distance between the targets, the mapping unit 530 may also use the similarity of the appearance characteristics of the targets at the same time. For example, targets that are close in position and have similar colors are likely to be the same object. Therefore, the mapping unit 530 may perform mapping using such a feature. The feature for determining the similarity of the appearance feature is not limited to the color, and may be, for example, the pattern of the target.

Then, the association unit 530 integrates the detection results for the targets associated with each other. That is, the mapping unit 530 obtains the position of the object by using the detection results of the corresponding targets after the mapping between the targets. At this time, the association unit 530 evaluates the likelihood of each target and / or the accuracy of the predicted position, and the position where the accuracy is maximized may be the position of the object.

Further, the mapping unit 530 weights the position of each target based on the accuracy of the predicted position determined by the angle (depression angle or elevation angle) of the camera 20 with respect to each target and the distance from the camera 20 to the target. You may. Then, the mapping unit 530 may calculate a statistic such as an average value from the weighted position, and the position indicated by the calculated statistic may be the position of the object.

Then, the matching unit 530 sets the position of the obtained object as the position of the target with respect to the period in which the detection result is acquired. The mapping unit 530 uses the position of this target to perform mapping in the same manner as the mapping unit 230 according to the first embodiment. Further, the processing of the association and the subsequent processing by the integrated tracking unit 500 are the same as the processing in the integrated tracking unit 200 described in the first embodiment, and thus the description thereof will be omitted.

Further, the association unit 530 may associate each target with the tracker and integrate them before making the association between the targets. That is, when there are a plurality of targets associated with the same tracker, the association unit 530 performs a merge process between these targets. In this case, the matching unit 530 may perform merging by giving priority to those having higher likelihood of each target and / or accuracy of the predicted position. In this way, the mapping unit 530 may evaluate the detection result based on this information and integrate the targets associated with the same tracker.

As described above, the object tracking device 10 according to the present embodiment performs object tracking using all the detection results for the camera images taken by each camera 20 within a predetermined period. This makes it easier for the object tracking device 10 to apply a process of prioritizing object search results among a plurality of cameras 20 and performing object tracking.

Further, for example, when the frame rates of all the cameras 20 are stable and the same, the frames of all the cameras 20 are included in the predetermined period by setting the predetermined period according to the frame interval. Therefore, the object tracking device 10 according to the present embodiment can perform object tracking for frames for all cameras 20. As a result, the object tracking device 10 can evaluate the detection result for the objects that are simultaneously visible from the plurality of cameras 20 at the same time, so that the reliability of the detection result can be directly reflected by the tracking.

<Fifth Embodiment>
A fifth embodiment of the present invention will be described. In the present embodiment, the minimum configuration for solving the problem of the present invention will be described.

Since the object tracking device 10 according to the present embodiment has the same configuration as the object tracking device 10 shown in FIG. 1 described in the first embodiment, the description will be given with reference to FIG.

As shown in FIG. 1, the object tracking device 10 according to the present embodiment includes a plurality of detection units (100-1 to 100-N) (N is a natural number) and an integrated tracking unit 200. In the present embodiment, when the plurality of detection units (100-1 to 100-N) are not distinguished from each other, or when they are generically referred to, these are referred to as detection units 100.

Each of the plurality of detection units 100 detects an object from the output information output from the sensor. In FIG. 1, the sensor is used as a camera, and the output information of the sensor is described as a camera image (video data), but the sensor is not limited to the camera. Specifically, the detection unit 100 detects an object based on the tracking information output from the integrated tracking unit 200. The detection unit 100 outputs the detection result to the integrated tracking unit 200.

The integrated tracking unit 200 tracks each of the one or more objects indicated by the detection results based on the plurality of detection results output by each of the plurality of detection units (100-1 to 100-N). .. Then, the integrated tracking unit 200 generates tracking information of the object represented by the common coordinate system as the tracking result. Then, the integrated tracking unit 200 outputs to each of the plurality of detection units (100-1 to 100-N).

As described above, the detection unit 100 of the object tracking device 10 according to the present embodiment detects the object from the output information output from the sensor based on the tracking result for the object tracked by the integrated tracking unit 200.

As described above, since the object tracking device 10 detects the object from the video by using the tracking result for the previous frame, the detection accuracy of the object can be improved as compared with the case where the tracking result is not used. Further, since the object tracking is performed using all the object detection results for the images captured by the cameras 20, the object tracking device 10 can improve the tracking accuracy as compared with the case where the object tracking is performed independently for each camera. .. Further, since the object tracking device 10 performs object tracking using all the detection results having high detection accuracy, it is possible to track the object with higher accuracy.

In each of the above-described embodiments, the object tracking device 10 includes a detection unit (100, 400) and an integrated tracking unit (200, 500) as an example, but the detection unit and the integrated tracking unit have been described. And may be realized by separate devices. That is, the detection unit (100, 400) may be realized as an object detection device, and the integrated tracking unit (200, 500) may be realized as an integrated tracking device by separate devices. Further, the display control unit 300 may also be realized by a display control device separate from the object tracking device 50. This display control device may be built in the display device 30.
<Sixth Embodiment>
A sixth embodiment of the present invention will be described. Since the object tracking device 10 according to the present embodiment has the same configuration as the object tracking device 10 shown in FIG. 1 described in the first embodiment, the description will be given with reference to FIG. It is assumed that the object tracking device 10 according to the present embodiment has a configuration in which the function described below is further added to the object tracking device 10 according to the first embodiment, but the present invention is limited to this. It is not something that will be done. The object tracking device 10 according to the present embodiment can also be applied to the object tracking device according to the second to fifth embodiments described above.

In the present embodiment, the integrated tracking unit 200 further acquires information on the appearance of the object, and includes the acquired information in the tracking information. Then, the detection unit 100 controls the detection of the object by using the information on the appearance of each object included in the tracking information output from the integrated tracking unit 200.

The information on the appearance of the object (hereinafter referred to as the appearance information) is information on how each object looks when the object is viewed from the position of each camera 20, and depends on the position of the object on each tracker. It is fixed.

For example, consider a case where an object is in front of another object (on the camera 20 side) when a certain object and another object are viewed from a certain camera 20. In this case, the object on the back side (another object) is hidden by the object on the front side (a certain object), and there is a high possibility that the object cannot be seen from the camera 20. At this time, the integrated tracking unit 200 includes information indicating the overlap between such objects as appearance information in a tracker (tracking result) related to other objects, and outputs the tracking result.

Next, the specific operation of each part of the object tracking device 10 according to the present embodiment will be described.

The integrated tracking unit 200 stores information regarding the arrangement of each camera 20 in, for example, the storage unit 220 shown in FIG. The information regarding the arrangement of the cameras 20 is, for example, information indicating at which position each camera 20 is arranged, in which direction the camera 20 is being photographed, and the like. Further, the integrated tracking unit 200 may include information on the position and direction of lighting in the shooting space, information on lighting characteristics, and information on lighting conditions such as which position in the shooting space is bright or dark as information on the arrangement of the camera 20. .. Further, the integrated tracking unit 200 may also hold direction information of the shooting space as information regarding the arrangement of the camera 20.

Similar to the integrated tracking unit 200 according to each of the above-described embodiments, the integrated tracking unit 200 predicts the movement of the object indicated by each tracker on the current frame, and associates the target with the tracker to position the tracker. Ask for.

Then, the integrated tracking unit 200 predicts the position of the object indicated by each tracker on the captured image taken by each camera 20 from the obtained position of the tracker and the information on the movement of each tracker.

Then, the integrated tracking unit 200 refers to the information regarding the arrangement of each camera 20 and predicts how each object at the predicted position will look (appearance) when viewed from each camera 20. That is, the integrated tracking unit 200 predicts for each of the plurality of cameras 20 whether or not the objects indicated by the trackers overlap each other at the timing of the next shooting.

Then, when the integrated tracking unit 200 determines that the objects overlap each other on the captured image of a certain camera 20 based on the predicted appearance, the objects may overlap and cannot be seen. Generates the information (appearance information) to be shown.

For example, the integrated tracking unit 200 determines whether or not there is another predicted object between the line segments connecting a certain camera 20 and a predicted object. If another predicted object is between the line segments, it is likely that this object will overlap with the other object. Therefore, the integrated tracking unit 200 obtains information on hidden (overlapping) objects and the degree of overlap (likelihood) as appearance information for the above-mentioned object.

Then, the integrated tracking unit 200 may include this determination result as appearance information in the tracking result of this certain object.

Then, the integrated tracking unit 200 includes the generated appearance information in the tracking information of the object that is likely to disappear from the captured image captured by a certain camera 20. At this time, it is preferable that the appearance information includes information indicating the camera 20 in which the object is likely to disappear.

Then, the integrated tracking unit 200 outputs tracking information including appearance information to each detection unit 100. The integrated tracking unit 200 may output tracking information including appearance information to a detection unit 100 associated with a camera 20 (a certain camera 20) in which objects are likely to overlap and cannot be seen. Then, the integrated tracking unit 200 may output tracking information that does not include appearance information to the object tracking device 10 associated with the other camera 20.

Further, when it is known that the lighting changes according to the position of the object and the hue and brightness of the object change, the integrated tracking unit 200 changes the appearance according to the position of the object. The information describing the above may be included in the tracking information.

For example, when the lighting is determined by the position of the object, the integrated tracking unit 200 predicts the lighting from that position and provides information such as brightening, darkening, and color change for each tracker. May be included in the tracking information.

Further, when the position of the lighting in the space where the object is placed is known, the integrated tracking unit 200 may display the shadow of the object or another object placed in this environment due to the relationship between the lighting and the object. Judge whether or not it overlaps with the object of. Then, when there is a possibility that the shadows overlap, the integrated tracking unit 200 calculates the possibility (likelihood) of the shadows overlapping for the object where the shadows overlap, and includes it in the tracking information.

Further, even when moving like the sun, the integrated tracking unit 200 obtains the position of the sun from the time and the information of the direction of the site, predicts the direction in which the shadow is formed, and gives it to the appearance of the object. The impact may be taken into account. For example, the integrated tracking unit 200 obtains the current position of the sun from the time information and predicts in which direction the shadow will be formed together with the direction information. Then, the integrated tracking unit 200 may calculate the possibility (likelihood) of overlapping shadows when there is a possibility that shadows of other objects may be cast, and include them in the tracking information.

Next, the operation of the detection unit 100 will be described. The detection unit 100 detects an object based on the tracking information, as in each of the above-described embodiments. At this time, the detection unit 100 of the object tracking device 10 according to the present embodiment controls the detection of the object by using the information regarding the appearance of each object included in the tracking information. Specifically, the detection unit 100 does not detect an object that is likely to overlap with another object and cannot be seen. For example, the detection unit 100 does not set the search range for this object that is likely to be invisible.

When the tracking information includes information on which the brightness and the hue change, the detection unit 100 may correct the effect of the illumination and perform detection when searching. For example, in a dark region, the detection unit 100 may correct the pixel value in the region to be bright and then perform detection.

Further, when the hue changes, the detection unit 100 considers the change in the hue when updating the matching parameter used in the template matching (that is, the detection parameter described above), and considers the change in the hue of the parameter. The information may be corrected. Further, when the change in color tone is large, the detection unit 100 may not use the color information. Further, the detection unit 100 may reduce the weight of the color information and increase the weight of other features such as edges in the features of the template to perform matching.

Further, when the detection unit 100 updates the detection parameters used for the object detection process, if it is determined that there is a high possibility that the objects overlap, the detection parameters may not be updated. ..

As described above, the detection unit 100 controls the detection of the object based on the appearance information included in the tracking information. As a result, the detection unit 100 can reduce the processing of detecting an invisible object and the processing of updating parameters such as template matching. As a result, the detection unit 100 can reduce the possibility of erroneous detection and update of erroneous parameters.

Similarly, when there is a high possibility that the lighting conditions have changed, the detection unit 100 may control to correct the effect and update the parameters, or control not to update the parameters. You may.

Further, when a plurality of detection algorithms can be switched, the object tracking device 10 may use a detector having a stronger overlap as the detection unit 100. Specifically, the object tracking device 10 normally uses a detector that detects the entire head, but when it overlaps, it uses a detector that detects only a part of the head, not the entire head. You may do so. As a result, the object tracking device 10 can usually use a simple detector, and if there is a possibility of overlapping, a more detailed detector can be used. This enables the object tracking device 10 to perform highly accurate detection while maintaining efficiency. Similarly, when the lighting condition changes, the object tracking device 10 may control the detection by using a detector (feature amount) having high robustness to the lighting condition.

As described above, in the object tracking device 10 according to the present embodiment, the integrated tracking unit 200 determines the appearance of the object by using the information of other cameras. Therefore, the integrated tracking unit 200 can determine the appearance with high accuracy even when it is difficult for a certain camera 20 to make a determination by the camera 20 alone, such as when objects are overlapped with each other. Then, the integrated tracking unit 200 can feed back this result to the detection unit 100. As a result, the detection unit 100 can reduce the false detection of the object. Then, since the integrated tracking unit 200 performs object tracking using this detection result, the tracking accuracy of the object can be improved.

<Hardware configuration example>
Here, a configuration example of hardware that can realize the object tracking device (10, 50) according to each of the above-described embodiments will be described. The object tracking device (10, 50) described above may be realized as a dedicated device, or may be realized by using a computer (information processing device).

FIG. 19 is a diagram illustrating a hardware configuration of a computer (information processing apparatus) capable of realizing each embodiment of the present invention.

The hardware of the information processing device (computer) 700 shown in FIG. 19 includes a CPU (Central Processing Unit) 11, a communication interface (I / F) 12, an input / output user interface 13, a ROM (Read Only Memory) 14, and a RAM ( It comprises a Random Access Memory) 15, a storage device 17, and a drive device 18 of a computer-readable storage medium 19, which are connected via a bus 16. The input / output user interface 13 is a man-machine interface such as a keyboard as an example of an input device and a display as an output device. The communication interface 12 is a general communication means for the device (FIGS. 1 and 9) according to each of the above-described embodiments to communicate with the external device via the communication network 600. In the hardware configuration, the CPU 11 controls the entire operation of the information processing device 700 that realizes the object tracking device (10, 50) according to each embodiment.

The present invention described by taking each of the above-described embodiments as an example supplies, for example, a program (computer program) capable of realizing the processes described in each of the above-described embodiments to the information processing apparatus 700 shown in FIG. After that, the program is read out to the CPU 11 and executed. The program may be described in, for example, various processes described in the flowchart (FIG. 8) referred to in the description of each of the above embodiments, or in FIGS. 1, 4 to 6, 9, 9, 15 to 17. In the block diagram shown, each part (each block) shown in the apparatus may be a feasible program.

Further, the program supplied in the information processing apparatus 700 may be stored in a readable / writable temporary storage memory (15) or a non-volatile storage device (17) such as a hard disk drive. That is, in the storage device 17, the program group 17A is, for example, a program capable of realizing the functions of each part shown in the object tracking device (10, 50) in each of the above-described embodiments. Further, the various stored information 17B is, for example, an object tracking result, a camera image, a camera parameter, a range of coordinate values of a common coordinate system visible by each camera 20, and the like in each of the above-described embodiments. However, when the program is mounted on the information processing apparatus 700, the constituent units of the individual program modules are the blocks shown in the block diagrams (FIGS. 1, FIG. 4 to FIG. 6, FIG. 9, and FIGS. 15 to 17). It is not limited to the classification, and a person skilled in the art may appropriately select it at the time of implementation.

Further, in the above case, the method of supplying the program into the device is installed in the device via various computer-readable recording media (19) such as a CD (Compact Disk) -ROM and a flash memory. Nowadays, general procedures can be adopted, such as a method and a method of downloading from the outside via a communication line (600) such as the Internet. In such a case, the present invention can be regarded as being composed of a code (program group 17A) constituting the computer program or a storage medium (19) in which the code is stored.

The present invention has been described above as an example in which the present invention is applied to the above-mentioned exemplary embodiment. However, the technical scope of the present invention is not limited to the scope described in each of the above-described embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to such embodiments. In such cases, new embodiments with such modifications or improvements may also be included in the technical scope of the invention. And this is clear from the matters described in the claims.

1 Object tracking system 2 Object tracking system 10 Object tracking device 100 Detection unit 110 Object detection unit 111 Recognition type object detection unit 112 Search range setting unit 120 Common coordinate conversion unit 130 Individual coordinate conversion unit 140 Object detection unit 141 Recognition type object detection unit 142 Non-recognizable object detection unit 143 Detection parameter update unit 144 Detection result integration unit 150 Display control unit 160 Storage unit 200 Integration tracking unit 210 Prediction unit 220 Storage unit 230 Correspondence unit 240 Update unit 300 Display control unit 400 Detection unit 500 Integration unit Tracking unit 510 Buffer unit 530 Correspondence unit 20 Camera 30 Display device 40 Network 50 Object tracking device

Claims (15)

A plurality of objects are detected from an image captured by one camera, and one including individual coordinate system position information which is position information expressed in an individual coordinate system unique to the one camera for each of the plurality of objects. One detection means that outputs the detection result,
With other detection means that detect a plurality of objects from an image captured by another camera and output other detection results including the individual coordinate system position information peculiar to the other camera for each of the plurality of objects. ,
Based on the one detection result and the other detection result, the common coordinate system position information, which is the position information expressed in the common coordinate system, is calculated for each of the plurality of objects, and the past position of each object is calculated. Predicts the current position of each object from, and predicts whether or not each object overlaps and the context of each other in the next image captured by each camera, and one object in the image is with another object. It is equipped with an integrated tracking means that generates appearance information to indicate that it is expected to overlap and be in front .
The integrated tracking means outputs common coordinate system position information and appearance information for each of the plurality of objects.
Each detection means is converted from the common coordinate system position information when it is predicted that the one object overlaps with the other object and is in front of the image captured by each corresponding camera next . The one object is detected based on the tracking information including the individual coordinate system position information peculiar to each corresponding camera and the appearance information, and the detection of the other object is stopped .
Object tracking system. Each detection means is a search range for searching each of the plurality of objects for the other objects in the image predicted to be in front of the other objects when the detection of the other objects is stopped, and the tracking information. Stop setting the search range represented by the individual coordinate system specific to each corresponding camera, which is specified based on.
The object tracking system according to claim 1. The one detection means is used in detecting each of a plurality of objects based on the tracking information.
Among the images to be detected by the plurality of objects, the search range for searching each of the plurality of objects is represented by the individual coordinate system peculiar to the one camera, which is specified based on the tracking information. Within the search range, each of the plurality of objects is detected.
The object tracking system according to claim 1 or 2 . The object tracking system according to any one of claims 1 to 3, wherein each detection of a plurality of objects is performed by using a discriminator trained in the image characteristics of the objects. The one detection result of each detection of the plurality of objects is the result of integrating the detection result by the discriminator and the detection result by template matching with the object region detected in the previous frame.
The object tracking system according to claim 4 . A plurality of objects are detected from an image captured by one camera, and one including individual coordinate system position information which is position information expressed in an individual coordinate system unique to the one camera for each of the plurality of objects. Output the detection result and
Multiple objects are detected from the images captured by other cameras, and other detection results including the individual coordinate system position information peculiar to the other cameras are output for each of the plurality of objects.
Based on the one detection result and the other detection result, the common coordinate system position information, which is the position information expressed in the common coordinate system, is calculated for each of the plurality of objects, and the past position of each object is calculated. Predicts the current position of each object from, and predicts whether or not each object overlaps and the context of each other in the next image captured by each camera, and one object in the image is with another object. When it overlaps and predicts that it is in front, it generates appearance information to that effect and generates appearance information to that effect.
The common coordinate system position information and the appearance information are output for each of the plurality of objects, and the common coordinate system position information is output.
When the one object overlaps with the other object and is predicted to be in front of the image captured by each camera next, the image predicted to be in front is converted from the common coordinate system position information. In addition, the one object is detected based on the tracking information including the individual coordinate system position information peculiar to the camera that captures the image predicted to be in front of the object and the appearance information, and the other object is detected. Stop detection ,
Object tracking method. When the detection of the other object is stopped, the search range for searching each of the plurality of objects for the other object in the image predicted to be in front of the object is specified based on the tracking information. , Stops setting the search range represented by the camera-specific individual coordinate system that captures the image predicted to be in front of it.
The object tracking method according to claim 6. In each detection of a plurality of objects based on the tracking information,
Among the images to be detected by the plurality of objects, the search range for searching each of the plurality of objects is represented by the individual coordinate system peculiar to the one camera, which is specified based on the tracking information. Within the search range, each of the plurality of objects is detected.
The object tracking method according to claim 6 or 7 . The object tracking method according to any one of claims 6 to 8, wherein each of the plurality of objects is detected by using a discriminator trained in the image characteristics of the objects. The one detection result of each detection of the plurality of objects is the result of integrating the detection result by the discriminator and the detection result by template matching with the object region detected in the previous frame.
The object tracking method according to claim 9 . Processing to detect multiple objects from images captured by one camera,
A process of outputting one detection result including the individual coordinate system position information which is the position information expressed in the individual coordinate system unique to the one camera for each of the plurality of objects.
A process of detecting a plurality of objects from images captured by another camera and outputting other detection results including the individual coordinate system position information peculiar to the other camera for each of the plurality of objects.
Based on the one detection result and the other detection result, the common coordinate system position information, which is the position information expressed in the common coordinate system, is calculated for each of the plurality of objects, and the past position of each object is calculated. Predicts the current position of each object from, and predicts whether or not each object overlaps and the context of each other in the next image captured by each camera, and one object in the image is with another object. A process to generate appearance information indicating that when it is predicted that it overlaps and is in front ,
Processing to output common coordinate system position information and appearance information for each of the plurality of objects,
When the one object overlaps with the other object and is predicted to be in front of the image captured by each camera next, the image predicted to be in front is converted from the common coordinate system position information. In addition, the one object is detected based on the tracking information including the individual coordinate system position information peculiar to the camera that captures the image predicted to be in front of the object and the appearance information, and the other object is detected. Processing to stop the detection of
A program that causes a computer to run . When the detection of the other object is stopped, the search range for searching each of the plurality of objects for the other object in the image predicted to be in front of the object is specified based on the tracking information. The process of stopping the setting of the search range expressed by the camera-specific individual coordinate system that captures the image predicted to be in front of the above.
The program according to claim 11. In the process of detecting each of a plurality of objects based on the tracking information.
Among the images to be detected by the plurality of objects, the search range for searching each of the plurality of objects is represented by the individual coordinate system peculiar to the one camera, which is specified based on the tracking information. Processing to detect each of the plurality of objects within the search range
The program according to claim 11 or 12 . The program according to any one of claims 11 to 13, wherein the process of detecting each of a plurality of objects is performed by using a discriminator trained in the image characteristics of the objects. The one detection result of the process of detecting each of the plurality of objects is the result of integrating the detection result by the discriminator and the detection result by template matching with the object region detected in the previous frame. ,
The program according to claim 14 .

Images