JP7272417B2 - Object tracking system, object tracking device, and object tracking method - Google Patents

Description

The present invention relates to an object tracking device, an object tracking system, an object tracking method, a display control device, an object detection device, a program and a recording medium.

In recent years, systems have been developed that use a plurality of cameras or the like to track a person. For example, the moving object tracking system described in Patent Document 1 uses a plurality of in-camera tracking means that track a person in a distributed manner for each camera. The system then tracks the moving object in cooperation with a plurality of in-camera tracking means. Further, Patent Document 2 describes a method of tracking the same object imaged by a plurality of imaging units based on the tracking results of each individual object.

Further, as a related technique, Patent Document 3 describes a method for early removing an object that does not need to be tracked from the tracked object.

Further, Patent Document 4 describes a device for detecting a moving object in an image captured by one image capturing means.

JP-A-2004-72628 Japanese Patent Publication No. 2009-510541 WO2013/012091 Japanese Patent Application Laid-Open No. 2006-202047

However, with the technology described in Patent Literature 1 or 2, for example, if the camera exists at a position far away from the object, there is a possibility that the tracking accuracy of the object (moving object) in the captured image of this camera will be low. be. In this case, the technique of Patent Documents 1 and 2 cannot integrate the tracking results of the object due to the influence of the tracking accuracy of the camera. Accuracy was sometimes low.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique capable of tracking an object with higher accuracy.

An object tracking device according to an aspect of the present invention includes a plurality of detection means for detecting an object from output information of a sensor and outputting a detection result, and a plurality of the detection results output by each of the plurality of detection means. and an integrated tracking means for generating tracking information of the object expressed in a common coordinate system, wherein the integrated tracking means applies the generated tracking information to the plurality of detection means and the detection means detects the object based on the tracking information.

An object tracking system according to an aspect of the present invention includes a sensor and an object tracking device that receives output information made up of information acquired by the sensor, and the object tracking device detects the object from the output information. a plurality of detection means for outputting detection results; and tracking the object based on the plurality of detection results output by each of the plurality of detection means, and tracking the object expressed in a common coordinate system. and integrated tracking means for generating information, wherein the integrated tracking means outputs the generated tracking information to each of the plurality of detection means, and the detection means detects the object based on the tracking information. to detect

An object tracking method according to an aspect of the present invention detects an object from output information of a sensor, outputs a detection result, tracks the object based on the plurality of output detection results, and detects the object in a common coordinate system. Generating tracking information of the represented object, outputting the generated tracking information, and detecting the object detects the object based on the tracking information.

A display control device according to an aspect of the present invention is a display control device that causes a display device to display display data, wherein the display data searches for an object based on tracking information of an object among output information of a sensor. The search range represents a search range expressed in an individual coordinate system unique to the sensor that outputs the output information, and the tracking information of the object is the search range in the output information of each of a plurality of sensors Information indicating the result of tracking the object based on the detection result of the object detected from within.

An object detection device according to an aspect of the present invention provides tracking information indicating a tracking result of an object, which is tracked based on a plurality of detection results output from each of a plurality of object detection devices, and which is a common coordinate system. The object is detected from the output information of the sensor based on the tracking information represented by .

A computer program for realizing each of the above devices, object tracking system, or object tracking method by a computer, and a computer-readable storage medium storing the computer program are also included in the scope of the present invention.

According to the present invention, an object can be tracked with higher accuracy.

1 is a functional block diagram showing an example of functional configuration of an object tracking device according to a first embodiment of the present invention; FIG. BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows an example of an outline of the whole structure of the object tracking system which concerns on the 1st Embodiment of this invention. FIG. 4 is a diagram for explaining processing for associating targets and trackers; FIG. It is a functional block diagram showing an example of functional composition of a primary detecting element of an object tracking device concerning a 1st embodiment of the present invention. 3 is a functional block diagram showing an example of the functional configuration of an object detection section in the detection section of the object tracking device according to the first embodiment of the present invention; FIG. 3 is a functional block diagram showing an example of functional configuration of an integrated tracking unit of the object tracking device according to the first embodiment of the present invention; FIG. FIG. 5 is a diagram for explaining sequential object tracking processing performed by the integrated tracking unit according to the first embodiment of the present invention; 4 is a flow chart showing an example of the flow of object tracking processing of the object tracking device according to the first embodiment of the present invention; FIG. 5 is a functional block diagram showing an example of functional configuration of an object tracking device according to a second embodiment of the present invention; FIG. 7 is a functional block diagram showing an example of functional configuration of a detection unit of the object tracking device according to the second embodiment of the present invention; It is a figure for demonstrating the application example of the object tracking device which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating the application example of the object tracking device which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating the application example of the object tracking device which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating the application example of the object tracking device which concerns on the 2nd Embodiment of this invention. FIG. 11 is a functional block diagram showing an example of functional configuration of a detection unit of an object tracking device according to a third embodiment of the present invention; FIG. 10 is a functional block diagram showing an example of the functional configuration of an object detection section in the detection section of the object tracking device according to the third embodiment of the present invention; FIG. 11 is a functional block diagram showing an example of functional configuration of an integrated tracking unit of an object tracking device according to a fourth embodiment of the present invention; FIG. 14 is a diagram for explaining batch tracking processing of objects performed by the integrated tracking unit according to the fourth embodiment of the present invention; 1 is a diagram illustrating an exemplary hardware configuration of a computer (information processing device) that can implement each embodiment of the present invention; FIG.

<First embodiment>
A first embodiment of the present invention will be described in detail with reference to the drawings. First, referring to FIG. 2, the overall configuration of the object tracking system (also simply called the system) of the present invention will be described. FIG. 2 is a diagram showing an example of a schematic overall configuration of the object tracking system 1 according to this embodiment. As shown in FIG. 2, the object tracking system 1 according to the present embodiment includes an object tracking device 10, a plurality of cameras (20-1 to 20-N (N is a natural number)), and one or more display devices 30. It has In this embodiment, the plurality of cameras (20-1 to 20-N) will be referred to as cameras 20 when they are not distinguished from each other or when they are collectively referred to.

Object tracking device 10 , camera 20 and display device 30 are communicably connected to each other via network 40 . Note that the display device 30 may not be included in the object tracking system 1 . Also, the display device 30 may be configured to be directly connected to the object tracking device 10 without going through the network 40 .

Camera 20 functions as a sensor that detects an object. In this embodiment, the case where the camera 20 is used as a sensor for detecting an object will be described as an example, but the present invention is not limited to this. The sensor is not limited to a camera, and may be any device capable of positioning, such as a radio wave sensor. Also, a sensor in which a plurality of sensors are mixed, such as a sensor in which a radio wave sensor and a camera are integrated, may be used. By using the camera 20 as a sensor, the object tracking device 10 can more preferably acquire visual information such as color.

Further, in the present embodiment, the information acquired by the sensor is assumed to be the image captured by the camera. is a radio wave.

The object tracking device 10 is a device that tracks an object included in images captured by each of a plurality of cameras 20 . Note that the functional configuration of the object tracking device 10 will be described with different drawings.

The display device 30 displays the tracking result of the object by the object tracking device 10 . Note that the display device 30 may display an image captured by the camera 20 . The display device 30 may also display other information such as flow line information.

(Object tracking device 10)
Next, functions of the object tracking device 10 will be described. FIG. 1 is a functional block diagram showing an example of the functional configuration of an object tracking device 10 according to this embodiment. As shown in FIG. 1, the object tracking device 10 includes a plurality of detection units (100-1 to 100-N) and an integrated tracking unit 200. FIG. It should be noted that, in the present embodiment, the plurality of detection units (100-1 to 100-N) are referred to as the detection unit 100 when they are not distinguished from each other or when they are collectively referred to.

(Detector 100)
The detection unit 100 detects the object from the output information of the camera 20 based on the tracking information output from the integrated tracking unit 200, which will be described later, and the tracking information of the object in the previous frame of the frame in which the object is to be detected. to detect Here, in the present embodiment, the output information of the camera 20 indicates video data representing the video captured by the camera 20 .

In the present embodiment, it is assumed that the plurality of detection units 100 and the plurality of cameras 20 are associated on a one-to-one basis. For example, the detection unit 100-1 detects an object from an image captured by the camera 20-1, and the detection unit 100-2 detects an object from an image captured by the camera 20-2. Note that the present embodiment is not limited to this, and for example, the detection unit 100-1 may detect an object from an image captured by the camera 20-N.

Also, the cameras 20 and the detection units 100 do not have to be associated one-to-one. For example, the detection unit 100-1 may detect objects from images captured by each of the plurality of cameras 20. FIG.

The operation of the detection unit 100 will be described below. The detection unit 100 receives, from the camera 20, image data representing an image captured by the camera 20 (hereinafter referred to as camera image). In FIG. 1, the image data representing the image captured by the camera 20-n (n is 1 to N) is described as camera image (n). Here, the camera image may be an image captured by the camera 20 such as a surveillance camera and obtained in real time, or the image captured by the camera 20 may be temporarily stored in a storage unit (not shown) and stored later. may be decoded (or reproduced) by This video data includes time information indicating the time when the image was taken.

The detection unit 100 also receives object tracking information in the previous frame from the integrated tracking unit 200 . Note that the previous frame may be the frame immediately before the frame (current frame) in which the object is to be detected, or may be the frame a predetermined number before the current frame. Also, the number of previous frames may be one, or may be plural. When the detection unit 100 detects an object in the first frame, the detection unit 100 does not receive (do not use) the object tracking information in the previous frame because there is no object tracking information in the previous frame.

The detection unit 100 uses the received camera image and object tracking information in the previous frame to detect an object from the camera image (referred to as object detection). As described above, when performing object detection on the first frame, the detection unit 100 performs detection without using object tracking information in the previous frame. In addition, below, the detected object is called a target. That is, the object detection result (also referred to as object detection result or simply detection result) is a set of targets.

The object detection result includes, for each target, information representing the position of the target, information representing the size of the target, and the like. Specifically, the object detection result includes, for example, information on the circumscribed rectangle of the area occupied by the target (target area) in the frame in the video where the object was detected, information indicating the coordinates of the center of gravity of the target area, and information indicating the width of the target. , and information indicating the height of the target. Note that the object detection result is not limited to this. For example, the object detection result may include, in place of or in addition to the coordinate values of the center of gravity of the target area, the coordinate values of the uppermost edge and the lowermost edge of the target area. The object detection result may include information representing the position, size, etc. of the target for each target.

In this embodiment, the object detection result includes, for each target, the coordinate value of the lowest point of the target and information indicating the circumscribing rectangle of the target. Note that the coordinate value of the bottom end of the target indicates the coordinate value of the point where the object touches the floor (ground) and/or the coordinate value of the midpoint of the lower side of the circumscribing rectangle of the object. Also, the coordinate values of the bottom end of the target may be the coordinate values of the feet when the object is a person.

Then, the detection unit 100 converts the coordinate values included in the object detection result into coordinate values in a common coordinate system defined in a space (capturing space) photographed by the plurality of cameras 20, and converts the converted coordinates Let the value be the object detection result.

Also, the object detection result may include information representing the shape of the target in addition to the information described above. That is, the object detection result may include silhouette information representing the target area. Here, the silhouette information is information for distinguishing pixels inside and outside the target area. It is a value obtained by extracting a shape descriptor (shape feature amount) standardized by -7 from a silhouette shape. Also, the object detection result may include the feature amount of the appearance of the object. For example, the object detection result may also include feature amounts such as the color, pattern, and shape of the object.

Furthermore, the object detection result may include, for each target, information describing likelihood (target likelihood information) representing the likelihood of object detection. Target likelihood information is the information necessary to calculate the likelihood of a target, and is related to the accuracy of object detection, such as the score value at the time of object detection, the distance from the camera of the detected object, and the size of the detected object. Information. Alternatively, the detection unit 100 may calculate the target likelihood itself, and use the calculated likelihood as the target likelihood information.

The detection unit 100 then outputs the object detection result to the integrated tracking unit 200 .

(Integrated tracking unit 200)
The integrated tracking unit 200 receives detection results output from each of the detection units 100 . Then, the integrated tracking unit 200 tracks the object based on each detection result. Specifically, the integrated tracking unit 200 uses object detection results for one or a plurality of objects detected by the detection units 100 from images captured by the cameras 20 linked to the detection units 100, Track objects (do object tracking). The integrated tracking unit 200 then generates an object tracking result (object tracking result) expressed in the common coordinate system. In this way, the integrated tracking unit 200 performs object tracking by integrating the object detection results detected by the detection units 100 from the images captured by the cameras 20 linked to the detection units 100 . Therefore, object tracking performed by the integrated tracking unit 200 is also called object integrated tracking.

Information for each object generated as a result of object tracking is hereinafter referred to as a tracker. In other words, the tracker includes information indicating the position of the tracked object, a motion model of the object, etc. as information of the tracked object (object tracking result), but the present invention is limited to this. isn't it. Note that the position of the tracked object is also called the past position of the object because it is the position before the current time (past) of the object.

In other words, object tracking can be regarded as a process of associating objects between frames by associating a target detected by object detection with a tracker generated before detection of this target. This object tracking will be described with reference to FIG. FIG. 3 is a diagram for explaining the process of matching targets and trackers by the integrated tracking unit 200. As shown in FIG. As shown in FIG. 3, the number of targets is M and the number of trackers is K (M and K are integers of 0 or more). The integrated tracking unit 200 associates the M targets with the K trackers. When associating the target with the tracker, the integrated tracking unit 200 first predicts the current position of the object from the past position of the object indicated by the information included in the tracker, and expresses the relationship between the target and the tracker. Correlation is performed using indices.

That is, the integrated tracking unit 200 predicts the position of the object in the current frame based on the position of the object detected in the previous frame and the motion model of the object calculated and held for each tracker. As this method, various existing methods such as a method using a Kalman filter or a method using a particle filter can be used.

Then, the integrated tracking unit 200 associates the object tracking result (tracker) in the previous frame with the object (target) included in the detection result based on the following information (1) to (3), for example.
(1) The closeness of the distance between the position of the object on the current frame predicted using the tracker and the position of the target (2) The similarity of appearance features between the target and the object whose tracking result is indicated by the tracker (3) Likelihoods of Targets and Trackers, respectively This process of matching can be reduced to a cost minimization problem for a bipartite graph as shown in FIG. Therefore, the integrated tracking unit 200 can solve this problem with an algorithm such as the Hungarian method.

In FIG. 3, the arrows indicate the case where the target and the tracker are associated with each other. That is, the topmost target is associated with the topmost tracker.

Then, if there is a target that does not correspond to the tracker, the integrated tracking unit 200 determines whether the target can be regarded as a newly appearing object. When the integrated tracking unit 200 determines that there is a high possibility that the target has newly appeared, the integrated tracking unit 200 adds a new tracker related to the target. In FIG. 3, it is assumed that a target denoted by symbol m (referred to as target m) is a target that does not correspond to a tracker. At this time, the integrated tracking unit 200 determines whether or not the target m can be regarded as a newly appearing object.

On the other hand, if there is a tracker that does not correspond to the target, the integrated tracking unit 200 determines whether or not the tracker is information about an object that has disappeared from the shooting space. Then, the integrated tracking unit 200 deletes the tracker when there is a high possibility that the tracker is information about an object that has disappeared from the shooting space. In FIG. 3, it is assumed that the tracker indicated by symbol k (called tracker k) is a tracker that does not correspond to any target. At this time, the integrated tracking unit 200 determines whether or not the tracker k is information related to the disappeared object, and deletes the tracker k if the tracker k is information related to the disappeared object.

The integrated tracking unit 200 performs object tracking by repeating these processes on a frame-by-frame basis. The integrated tracking unit 200 gives the tracker a unique ID (identifier) common to all the cameras 20, and manages the tracking result (tracker) by this ID. In addition, the integrated tracking unit 200 includes a value obtained by evaluating the likelihood of the tracking result (hereinafter referred to as tracker likelihood (weight)) as a tracker parameter in the tracker. The position of the most recent object indicated by information indicating the position of the tracked object included in the tracker is called the position of the tracker. The size of the object at this time is also called the size of the tracker.

In addition, the integrated tracking unit 200 updates information indicating the position of each tracker, information on the likelihood of the tracker, and the like, based on the result of the association. Information on the position of the tracker is information expressed in a common coordinate system defined within an imaging space photographed by a plurality of cameras 20 . The information expressed in this common coordinate system is coordinate values in the common coordinate system. The coordinate values in this common coordinate system are, for example, a coordinate system indicating the position of the floor in the real world when the cameras 20 are installed in a certain store. On the other hand, the coordinate system unique to each camera 20 is called the individual coordinate system of the camera 20 . This individual coordinate system is a coordinate system on the captured image of the camera 20 . Hereinafter, position information expressed in the common coordinate system will be described as coordinate values in the common coordinate system. Also, position information expressed in the individual coordinate system of the camera 20 will be described as coordinate values in the individual coordinate system of the camera 20 .

Then, the integrated tracking unit 200 generates a tracker whose information has been updated based on the association result as a new object tracking result. Then, the integrated tracking unit 200 extracts the information indicating the position and/or size of the tracker, the likelihood information of the tracker, etc. from among the generated object tracking results (trackers) as information indicating the tracking result of the object tracking. Output as (tracking information).

This tracking information includes coordinate values of the common coordinate system of each tracker as information indicating the position of each tracker. This tracking information is fed back to the detector 100 . That is, the detection unit 100 receives this tracking information and uses the tracking information for object detection for subsequent frames.

Note that the integrated tracking unit 200 may be configured to output to the detection unit 100 an object tracking result including the tracking information and other information of the tracker.

As described above, the object tracking apparatus 10 according to the present embodiment integrates the object detection results for the images captured by each of the plurality of cameras 20 using the images captured by each of the plurality of cameras. , to do object tracking. Then, the object tracking device 10 feeds back the obtained tracking information to object detection in the next frame.

Thus, the object tracking device 10 uses the tracking result for the previous frame to detect the object from the video. For example, if there is an object that cannot be seen from a certain camera 20 but is visible from another camera 20, the object tracking device 10 can also track the object that cannot be seen from a certain camera 20 and the object in the image of this certain camera 20. Used for detection. Thereby, when the same object appears in the range visible from the certain camera 20, the detection unit 100 can suitably detect this object. Therefore, the object tracking device 10 can accurately track the object.

Therefore, the object tracking device 10 can improve the object detection accuracy as compared with the case where the tracking result for the previous frame is not used. In addition, since the object tracking device 10 performs object tracking using all detection results with high detection accuracy, the accuracy of the tracking results obtained as a whole is also improved.

(Details of detection unit 100)
Next, the function of each part of the object tracking device 10 will be described in more detail with reference to FIGS. 4 to 8. FIG. FIG. 4 is a functional block diagram showing an example of a more detailed functional configuration of the detection unit 100 of the object tracking device 10 according to this embodiment. As shown in FIG. 4 , the detection unit 100 includes an object detection unit 110 , a common coordinate transformation unit (second transformation unit) 120 and an individual coordinate transformation unit (first transformation unit) 130 . In FIG. 4, camera images (n) (n is 1 to N) received by the detection unit 100 are simply referred to as camera images.

Individual coordinate transformation section 130 receives tracking information output from integrated tracking section 200 from integrated tracking section 200 . Then, the individual coordinate transformation unit 130 transforms the coordinate values of the common coordinate system of each tracker included in this tracking information into the coordinate values on the frame captured by each camera 20 (that is, in the individual coordinate system unique to each camera 20). expressed coordinate values). Let the coordinate values of the common coordinate system be (X, Y, Z) and the coordinate values of the individual coordinate system of the camera 20 be (x, y). From the values (X, Y, Z), the coordinate values (x, y) of the individual coordinate system of the camera 20 linked to the detection unit 100 including the individual coordinate transformation unit 130 are obtained. At this time, the individual coordinate transformation unit 130 preferably obtains at least camera parameters representing the camera position, orientation, etc. of the camera 20 linked to the detection unit 100 by calibration. Thereby, the individual coordinate transformation unit 130 transforms the coordinate values of the common coordinate system into the coordinate values of the individual coordinate system of the camera 20 using the obtained camera parameters.

Note that the camera parameters may be stored in a storage unit (not shown) in the detection unit 100 or may be stored in a storage area in the individual coordinate transformation unit 130 . In the latter case, the individual coordinate transformation unit 130 may be configured to supply camera parameters to the common coordinate transformation unit 120 .

For example, assume that the object is a person, and the information indicating the position of the tracker is the coordinate value indicating the foot position and the coordinate value indicating the top of the head position of the person. Then, let the coordinate values of the foot position and the coordinate value of the top of the head be (X0, Y0, 0) and (X0, Y0, H) (H represents the height of the person), respectively. It is also assumed that the camera 20-1 is the camera 20 linked to the detection unit 100 including the individual coordinate transformation unit 130. FIG.

At this time, the individual coordinate transformation unit 130 converts the foot position (x0, y0) and the top of the head position (x1, y1) on the frame photographed by the camera 20-1 using camera parameters related to the camera 20-1. demand. If the information indicating the position of the tracker includes the information indicating the bounding rectangle, the value indicating the width of the bounding rectangle is previously obtained by converting to the coordinate values of the common coordinate system using the camera parameters. ing. Therefore, the individual coordinate transformation unit 130 may use a value transformed again using the camera parameters as the width of the circumscribing rectangle.

Also, not all objects whose tracking results are indicated by the tracker are visible from one camera 20 and may exist outside the field of view of this camera 20 . Therefore, when the information indicating the position of the tracker included in the tracking information is information about an object that cannot be seen from the camera 20 linked to the detection unit 100, the individual coordinate transformation unit 130 transforms the object into the individual coordinate system of the camera 20. cannot find the coordinates of Therefore, the individual coordinate transformation unit 130 may not transform the coordinate values of the tracker's common coordinate system for objects that cannot be seen outside the field angle of the camera 20 into the coordinate values of the individual coordinate system. At this time, the individual coordinate transformation unit 130 registers in advance the range of coordinate values of the common coordinate system that can be seen by each camera 20 in a storage unit (not shown) or the like for each camera 20, and each object falls within this range. You may make it determine whether it exists. In addition, the individual coordinate conversion unit 130 actually converts the coordinate values into the coordinate values of the individual coordinate system, resulting in an abnormal value indicating the outside of the area monitored by the linked camera 20, or an unobtainable value. It may be determined that the object whose coordinate values have been transformed is an object that cannot be seen from the camera 20 when the object does not exist.

Then, the individual coordinate transformation unit 130 transforms the coordinate values of the common coordinate system of the tracker included in the tracking information output from the integrated tracking unit 200 into the coordinate values of the individual coordinate system of the camera 20 linked to the detection unit 100. The converted result (tracking information) is output to the object detection unit 110 . That is, the individual coordinate transformation unit 130 transforms the tracking information expressed in the common coordinate system into the tracking information expressed in the individual coordinate system, and outputs the converted tracking information to the object detection unit 110 . Hereinafter, when simply described as “coordinate values of the individual coordinate system”, it indicates the coordinate values of the individual coordinate system of the camera 20 linked to the detection unit 100 .

The object detection unit 110 receives camera video from the camera 20 linked to the detection unit 100 including the object detection unit 110 . Also, the object detection unit 110 receives the tracking information transformed into the coordinate values of the individual coordinate system from the individual coordinate transformation unit 130 . Then, the object detection unit 110 detects an object from the received camera image based on the tracking information.

The object detection unit 110 then generates a detection result. The object detection section 110 outputs the generated detection result to the common coordinate transformation section 120 . Note that this detection result is a detection result expressed in an individual coordinate system.

The configuration of the object detection unit 110 will be described in more detail with reference to FIG. FIG. 5 is a functional block diagram showing an example of the functional configuration of object detection section 110 according to the present embodiment. As shown in FIG. 5 , the object detection unit 110 includes a recognition type object detection unit (first object detection unit) 111 and a search range setting unit 112 .

The search range setting unit 112 receives the tracking information transformed into the coordinate values of the individual coordinate system from the individual coordinate transformation unit 130 . Then, the search range setting unit 112 uses the tracking information converted into the coordinate values of the individual coordinate system to obtain an area (search range) for object detection in the current frame. In other words, the search range setting unit 112 predicts the position of the object in the current frame based on the tracking information that is the tracking result of the previous frame converted into the coordinate values of the individual coordinate system. Then, the search range setting unit 112 obtains a detection range for searching for the object from the predicted position. This search range is also called an object detection range.

Here, the tracking information received by the search range setting unit 112 is the result of object tracking in the past frame (also referred to as the past tracking result) when viewed from the time of the frame to be processed at present. Therefore, the search range setting unit 112 predicts the motion of each object and predicts the position of each object in the current frame. Hereinafter, this predicted position of the object will be referred to as a predicted position. Then, the search range setting unit 112 sets the vicinity of this predicted position as the search range of the object.

The search range setting unit 112 preferably predicts the motion of each object using a motion model of each object calculated from past tracking results. For example, the search range setting unit 112 determines that the object is stationary when the position of the object has not changed in the tracking results of the past several frames (may be two frames), and determines that the object is stationary. The position of is used as the predicted position as it is. In addition, if the object is moving in the tracking results of the past few frames, the search range setting unit 112 assumes that the object is moving at a constant speed, and determines the predicted position by considering the time difference from the past frame. may be asked for.

Tracking results of the past few frames used by the search range setting unit 112 to predict the motion of each object may be included in the tracking information. Further, the motion model of the object obtained from the tracking results of the past several frames may also be included in the tracking information.

Also, this predicted position may be included in the tracking information. That is, the integrated tracking unit 200 may include the value obtained by the Kalman filter or the particle filter during object tracking as the predicted position in the tracking information.

Also, for example, a new object may appear in the angle of view of the camera 20 at the outer edge of the range that the camera 20 can capture. Also, if the location captured by the camera 20 includes a location such as a doorway, there is a possibility that a new object will appear in the angle of view of the camera 20 . Therefore, it is preferable that the search range setting unit 112 also include these areas (the outer edge of the frame and/or the entrance/exit portion) on the frame included in the image captured by the camera 20 in the object search range. .

The search range setting unit 112 outputs information (search range information) indicating the set search range of the object to the recognition type object detection unit 111 .

The recognition type object detection unit 111 receives search range information from the search range setting unit 112 . The recognition type object detection unit 111 detects an object from the camera image input to the recognition type object detection unit 111 based on the received search range information. Recognition-type object detection unit 111 temporarily stores frames of an input camera image in storage means such as a buffer in recognition-type object detection unit 111, and when search range information is received, it applies this information. Perform object detection processing. Specifically, the recognition-type object detection unit 111 performs object detection on the area (search range) indicated by the search range information using a classifier that has learned the image features of the object.

For example, when the object is a person, the recognition-type object detection unit 111 detects the person by applying a classifier that has learned a characteristic part of the person (for example, the head or upper body). Further, the recognition type object detection unit 111 may use a classifier that has learned the entire person as the classifier. The recognition type object detection unit 111 can use various types of classifiers. For example, the recognition-type object detection unit 111 can use a classifier obtained by learning images of the head, upper body, whole body, etc. of a person using a CNN (Convolutional Neural Network). In addition, the recognition type object detection unit 111 performs feature extraction such as HOG (Histogram Of Gaussian), SVM (Support Vector Machine), or GLVQ (Generalized Learning Vector Quantization). You may make it use discriminators, such as. In addition to the above, the recognition-type object detection unit 111 can use various existing recognition-based detection techniques.

In this manner, the recognition-type object detection unit 111 according to this embodiment detects objects within the search range set by the search range setting unit 112 . That is, the recognition-type object detection unit 111 performs object detection within the search range narrowed down by the search range setting unit 112 using the tracking result in the previous frame. Therefore, the recognition-type object detection unit 111 does not perform object detection in a range in which there is a low probability that an object exists in the frame, so that extra false detection can be reduced. In addition, the recognition type object detection unit 111 can speed up the processing of object detection.

Further, the search range in which the recognition-type object detection unit 111 detects objects may include not only the area indicated by the search range information, but also the area determined by the silhouette information obtained by the background difference or the like. Further, the recognition-type object detection unit 111 may set a common portion of the area determined by the silhouette information and the area specified by the object search range information as the area (search range) for executing object detection.

Then, the recognition-type object detection unit 111 generates a result (detection result) of object detection, and outputs the detection result to the common coordinate transformation unit 120 . At this time, the coordinate values of the object included in the detection result are the coordinate values of the individual coordinate system.

Returning to FIG. 4, the function of the common coordinate transformation unit 120 of the detection unit 100 will be described. The common coordinate transformation unit 120 receives object detection results expressed in the individual coordinate system from the object detection unit 110 . Then, the individual coordinate transformation unit 130 transforms the coordinate values of the individual coordinate system included in the received detection result into the coordinate values of the common coordinate system. As a result, the common coordinate transformation unit 120 can generate information for integrating the object detection positions for the individual cameras 20 .

Specifically, the common coordinate transformation unit 120 uses the camera parameters of the camera 20 linked to the detection unit 100 including the common coordinate transformation unit 120 to convert the coordinate values of the individual coordinate system included in the object detection results into a common coordinate system. Convert to the coordinate value of the coordinate system. For example, when the coordinates of the lower end of the object on the frame photographed by the camera 20 are (x0, y0), the common coordinate transformation unit 120 transforms them into the coordinates of the common coordinate system (X0, Y0, 0). do. Here, since the ground is a plane of Z=0, the component in the Z-axis direction is 0. Also, when the coordinates of the top end of the object are (x1, y1) (here, it is assumed that the top end of the object is directly above the bottom end of the object (vertically upward)), the height of the object is set to H. At this time, the common coordinate transformation unit 120 transforms the coordinates (x1, y1) of the upper end of this object into the coordinates (X0, Y0, H) of the common coordinate system. Note that the common coordinate transformation unit 120 obtains the height of the object by searching for H that satisfies this condition. In this manner, the common coordinate transformation unit 120 obtains coordinate values (X, Y, Z) in the common coordinate system for each detected object.

Note that when H is known, the common coordinate transformation unit 120 may use the known value as it is.

The common coordinate transformation unit 120 outputs the detection result including the coordinate values after the coordinate transformation (coordinate values of the common coordinate system) to the integrated tracking unit 200 . In other words, the common coordinate transformation unit 120 outputs the detection result expressed in the common coordinate system to the integrated tracking unit 200 . Note that the common coordinate transformation unit 120 converts information such as silhouette information and feature amounts of appearance features (color, pattern, shape, etc.) of each of one or more targets (objects) included in the detection result. It may be included as information about the object. Then, the common coordinate transforming section 120 may output the detection result including these information to the integrated tracking section 200 .

(Details of Integrated Tracking Unit 200)
Next, with reference to FIG. 6, the functional configuration of integrated tracking section 200 will be described in more detail. FIG. 6 is a functional block diagram showing an example of a more detailed functional configuration of integrated tracking section 200 of object tracking device 10 according to the present embodiment. As shown in FIG. 6 , integrated tracking section 200 includes prediction section 210 , storage section 220 , association section 230 and update section 240 . Note that the integrated tracking unit 200 according to the present embodiment is also called a sequential tracking unit because it sequentially tracks the video from each camera 20 on a camera-by-camera basis.

Sequential object tracking (also referred to as sequential object tracking or sequential integrated tracking) performed by the integrated tracking unit 200 will be described with reference to FIG. FIG. 7 is a diagram for explaining sequential object tracking processing performed by the integrated tracking unit 200 according to the present embodiment. FIG. 7 shows an example of the timing at which images are acquired by each of camera A, camera B, and camera C when the number of cameras is three. In FIG. 7, the horizontal axis indicates the time axis, and the further to the right, the later in time. As shown in FIG. 7, camera A acquires images at times t1, t5 and t8. Similarly, camera B has acquired images at times t2, t4, t6 and t9, and camera C has acquired images at times t3 and t7.

As shown in FIG. 7, the times (timestamps) of the frames (images) acquired by each camera 20 are not necessarily the same for all cameras 20, and are usually different. Also, the frame interval may vary from camera to camera 20 and may be non-uniform even for the same camera 20 .

Then, the detection unit 100 performs detection in chronological order from the camera images output asynchronously between the cameras 20 and outputs the detection result to the integrated tracking unit 200 .

Integrated tracking section 200 according to the present embodiment sequentially performs tracking processing in order from the earliest image. That is, in the case of FIG. 7, the integrated tracking unit 200 first uses the object detection result for the image of camera A at time t1 to integrate a plurality of cameras to track the object. After that, the integrated tracking unit 200 uses the object detection results in order of time t2 for camera B, time t3 for camera C, time t4 for camera B, . . . and perform object tracking. At this time, since not all objects are visible from any camera 20 , the integrated tracking unit 200 performs object tracking on objects that are highly likely to be visible from each camera 20 .

Returning to FIG. 6, each part of the integrated tracking unit 200 will be described.

The storage unit 220 stores tracker information associated with an object (target) included in the detection result received by the integrated tracking unit 200 . The tracker information stored in the storage unit 220 is managed by the updating unit 240 using the tracker ID. The tracker information is, for example, information about an object whose tracking result is included in the tracker, parameters including the likelihood of the tracker, and the like, but the present invention is not limited to this. The information about the object includes information indicating the past position of the object, the motion model of the object, etc., but the present invention is not limited to this. may include information that is

Note that FIG. 6 illustrates an example in which the storage unit 220 is built in the integrated tracking unit 200, but the present invention is not limited to this. The storage unit 220 may be provided in the object tracking device 10 separately from the integrated tracking unit 200 . Also, the storage unit 220 may be realized by a storage device or the like separate from the object tracking device 10 .

If the storage unit 220 is not built in the integrated tracking unit 200 , the storage unit 220 may be configured to store data and the like used in the object tracking device 10 . For example, the storage unit 220 may store camera images captured by the cameras 20, camera parameters of each camera 20, the range of coordinate values of the common coordinate system visible by each camera 20, and the like.

The prediction unit 210 refers to the storage unit 220 and predicts the position of the object on the current frame. Specifically, the prediction unit 210 predicts the current position of the object based on the motion model of the object using the tracking result (tracker) of the object in the previous frame. Here, information indicating the position of an object is expressed in a common coordinate system.

The motion model of the object used by the prediction unit 210 to predict the position may be stored in the storage unit 220, or the prediction unit 210 may use the tracking result to predict the position of the object. It may be calculated before performing.

Prediction processing such as a Kalman filter or a particle filter can be applied to the prediction of the position of the object by the prediction unit 210, for example. In addition, the prediction unit 210 simply calculates the velocity of the object from the tracking results of the past several times, assumes uniform linear motion, predicts the amount of movement from the position in the previous frame from the velocity, and The current position may be predicted by adding to the position in .

The prediction section 210 then outputs the prediction result to the associating section 230 .

The associating unit 230 receives detection results output from each of the detecting units 100 . In FIG. 6, detection results (n) (n is 1 to N) indicate detection results output from the detection section 100-n. The associating unit 230 also receives prediction results from the prediction unit 210 . Then, the associating unit 230 refers to the storage unit 220 and uses the prediction result to associate the target included in the detection result with the tracker.

The associating unit 230 obtains a combination with the highest accuracy as a whole associating. The likelihood that a certain target m and a certain tracker k correspond is the product of the likelihoods Pm and ηk of the target m and the tracker k, respectively, and the likelihood qkm representing the possibility that both are the same object. become. Therefore, the associating unit 230 calculates this value for each pair of target and tracker, and finds the maximum combination as a whole.

Here, the likelihood of the target (first likelihood) is a value representing the certainty (probability) of object detection. The accuracy of object detection depends on the size of the object to be detected (called a detection object) on the screen (on the frame), the distance from the camera 20 to the detection position of the object, the appearance of the object from the camera 20, etc. do.

For example, if the detected object is small and the size of the detected object is close to the detectable size limit, the accuracy of object detection will be low. Also, if the size of the detected object deviates from the apparent size of the object assumed by the camera parameters, the accuracy of object detection will be low. Also, if the detection position of the object is far from the camera 20, or if the lighting conditions for the area where the object exists are poor and the object is difficult to detect, the accuracy of object detection is low. In addition, when the data used for learning the classifier and the actual appearance are significantly different (for example, the angle is different), the accuracy of object detection is lowered.

The association unit 230 calculates the likelihood of the target by reflecting such characteristics. Specifically, the associating section 230 calculates the likelihood of the target using the target likelihood information included in the detection result received from the detecting section 100 . Note that when the detection unit 100 calculates the likelihood of the target and includes the calculated likelihood in the detection result as target likelihood information, the associating unit 230 calculates the likelihood included in the target likelihood information. may be used as is. It should be noted that the target likelihood does not need to reflect all the items described here, and may reflect only major factors.

The likelihood of the tracker (second likelihood) is a value representing the certainty (probability) of object tracking. The accuracy of object tracking changes depending on the tracking result of object tracking in the previous frame. For example, in the tracking results up to the previous (past) frame of the current frame, it can be said that the accuracy of object tracking is high for a tracker that is reliably associated with a target. can be said to be low. Therefore, the association unit 230 may change the likelihood based on the result of whether or not the target and the tracker are associated in each frame. It is sufficient to lower the likelihood of the tracker when it is not attached.

Also, at this time, if the tracker is positioned far from the camera 20, the error in the position of the tracker is considered to increase. As a result, such trackers are less likely to correspond to objects (targets) included in detection results. Therefore, the associating unit 230 may change the ratio of changing the likelihood of the tracker according to the distance between the position of the tracker and the camera 20 . Further, the associating unit 230 determines the tracking result of the tracker according to the angle (depression angle or elevation angle) formed between the horizontal plane including the camera 20 and the line-of-sight direction when the camera 20 sees the object whose tracking result is indicated by the tracker. You may change the ratio which changes a likelihood.

For example, when an object whose tracking result is indicated by the tracker (hereinafter referred to as a tracker object) is close to the camera 20 and the depression angle of the camera 20 with respect to the object is greater than a predetermined angle, the accuracy of the object position is high. Therefore, it is easy to associate the tracker with the target. Therefore, in such a case, the associating unit 230 increases the ratio of changing the likelihood of the tracker.

Also, for example, when the object of the tracker is far from the camera 20 and the depression angle of the camera 20 with respect to the object is shallower than a predetermined angle, the size of the object on the frame captured by the camera 20 becomes small. Also, a slight positional shift on the image results in a large shift on the real space. Therefore, there is a high possibility that the accuracy of the detected position of the object will be low. Therefore, in such a case, the associating unit 230 reduces the ratio of changing the likelihood of the tracker. Thus, the association unit 230 calculates the likelihood of the tracker.

As described above, the associating unit 230 can preferentially reflect the detection result detected by the camera 20 close to the tracker's object in the likelihood of the tracker, so that the tracking accuracy can be improved as a whole. Note that it is not necessary to reflect all the items described here in the likelihood of the tracker, and only the major factors may be reflected.

Also, the likelihood qkm representing the identity between the target m and the tracker k represents the probability that both are the same. If the object indicated by the target and the object of the tracker are the same object, there is a high possibility that the positions of the target and the object of the tracker will be close to each other. Therefore, the associating unit 230 changes the likelihood according to the distance between the target and the tracker object. That is, the association unit 230 increases the value of the likelihood qkm when the distance between the target m and the object of the tracker k is short, and decreases the value of the likelihood qkm when the distance is long. .

At this time, if the target m is far from the camera 20 or the depression angle of the camera 20 with respect to the target m is shallower than a predetermined angle, the accuracy of the position of the target m is likely to be low. Therefore, the associating unit 230 calculates the distance between the target and the tracker object using the Mahalanobis distance, which takes into consideration the error (ambiguity) included in the detected position of the target, instead of calculating the distance using a simple Euclidean distance. You can find the distance between In addition to the above method, the associating unit 230 may control the degree of change of the likelihood qkm according to the distance in consideration of ambiguity. That is, when the ambiguity is greater, the associating unit 230 reduces the change in the likelihood qkm according to the distance between the target m and the object of the tracker k. As a result, the associating unit 230 reduces the influence of the positional deviation of the target on the associating.

Additionally, the matcher 230 may also consider the similarity in appearance of the target and tracker. That is, the association unit 230 may extract features such as color, pattern, and shape of the target and tracker objects, evaluate their similarity, and obtain the likelihood qkm.

For example, the associating unit 230 may calculate the color histograms of the objects for both the target and tracker objects, evaluate the degree of similarity between them by overlapping the color histograms, etc., and reflect them in the likelihood qkm. . Like the tracker likelihood ηk and the target likelihood Pm, the likelihood qkm representing the identity of the target and the tracker does not need to reflect all the items described above, but only major factors. can be

Further, the associating unit 230 may calculate each of the above likelihoods in consideration of a case where an object is not detected because it is out of the angle of view of the camera 20 or is blocked by another object. As a result, the integrated tracking unit 200 can track the object with high accuracy even if the object is not detected or is out of the angle of view.

As described above, the problem of calculating each likelihood and determining the correspondence between the target and the tracker that maximizes each likelihood as a whole can be solved by converting each likelihood into a cost using a monotonically non-increasing function. , it can be reduced to the minimum-cost allocation problem (which target is mapped to which tracker). This assignment problem can be efficiently calculated by, for example, the Hungarian method.

The association unit 230 then outputs the association result to the update unit 240 . The result of this association includes information indicating which target and tracker are associated, and each likelihood including at least the likelihood of the tracker.

In the present embodiment, associating section 230 performs association using both the likelihood of the target and the likelihood of the tracker. Correspondence may be made.

The updating unit 240 updates tracker information. The updating unit 240 then generates this tracker as a new object tracking result. Specifically, the update unit 240 receives the association result from the association unit 230 . Based on this result, the updating unit 240 calculates the current position of the tracker object. Then, the updating unit 240 updates the tracker information stored in the storage unit 220 . The information to be updated is, for example, the position and/or size of the object whose tracking results are included in the tracker, the motion model of the object, and parameters such as likelihood of the tracker, but the present invention is limited to this. not a thing The updating unit 240 may update the updated information among the information stored in the storage unit 220 .

First, calculation of the current position of the tracker object by the updating unit 240 will be described. The updating unit 240 calculates the current position of the tracker object, taking into account the accuracy of the target's position. For example, suppose that the updating unit 240 weights the predicted position of the object using the tracker and the detected position of the target associated with the tracker, and calculates the current position of the object of the tracker. . In this case, the updating unit 240 may control the weight according to the accuracy of the target's position.

For example, if the target is located away from the camera 20 and the depression angle of the camera 20 with respect to the target is shallow, the accuracy of the target position is likely to be low. In such a case, the updating unit 240 assigns a smaller weight to the position of this target.

On the other hand, if the target is located near the camera 20 and the depression angle of the camera 20 with respect to the target is greater than the predetermined value, it is assumed that the accuracy of the target position is high. In such a case, the updating unit 240 increases the weight for the position of this target.

The updating unit 240 calculates the current position of the tracker object using the predicted position and the weighted position.

In this way, the update unit 240 determines the weight for the target position, so that the prediction result of the detected position of the object by the camera 20 near the target is reflected more strongly. Therefore, the object tracking device 10 can improve the prediction accuracy of the position of the object.

Then, the updating unit 240 updates the latest position of the object included in the information about the object stored in the storage unit 220 to the calculated current position of the object.

Next, the update of the tracker likelihood performed by the update unit 240 will be described.

When the tracker object is located far from the camera 20, the size of the object is reduced. Therefore, it becomes difficult for the detection unit 100 to detect such an object.

A case will be described in which the recognition-type object detection unit 111 of the detection unit 100 performs recognition-type object detection when an object included in a frame has a different appearance from the object used for learning. When an object included in a frame has a different appearance from the object used for learning, for example, the depression angle of the camera 20 with respect to the object, which is assumed from the position of the tracker object, and the angle of depression used for learning. This is the case where the angle of depression of the camera 20 with respect to the object is significantly different. In such a case, it becomes difficult for the recognition-type object detection unit 111 of the detection unit 100 to detect an object included in the frame.

In such a situation where it is difficult to detect an object, there is a possibility that the object included in the frame will not be detected. In this case, there may be no target associated with the tracker associated with this object.

Therefore, the updating unit 240 suppresses the change in the likelihood of the object tracker, among the trackers not associated with the target, that is in a situation where the object is difficult to be detected.

In this way, the updating unit 240 can reduce the influence on tracking when an object is difficult to detect, and can largely reflect the detection result of the camera, which is easy to detect, in the likelihood of the tracker. Then, during object tracking for the next frame, the matching unit 230 performs matching based on the likelihood of this tracker, so the integrated tracking unit 200 can further improve the accuracy of object tracking. .

Then, among the tracker likelihoods stored in the storage unit 220, the tracker likelihood calculated by the associating unit 230 and the tracker likelihood with the change suppressed to a small value are updated.

Next, updating of the motion model of the tracker object by the updating unit 240 will be described. For example, a case where the integrated tracking unit 200 performs object tracking by predicting the position of the object using a Kalman filter will be described. In this case, the update unit 240 substitutes the position coordinates of the object of the tracker associated with the target as the detected position coordinates into the update formula for the state variables of the Kalman filter stored in the storage unit 220, and calculates the state of the Kalman filter. to update.

In addition to the above, the update unit 240 updates other parameters of the tracker stored in the storage unit 220 .

For example, the object itself may change pose. For example, when the object is a person, the apparent height of the object changes when the person squats or bends down. In this way, when there is a change in the size of the object in addition to the motion model, the updating unit 240 updates the changed information among the information stored in the storage unit 220 .

Furthermore, if the tracker parameters include the probability that the tracker object exists, the weight representing the reliability of the tracking result, and the like, the weight changes according to the result of association by the association unit 230 . Therefore, the updating unit 240 updates parameters such as weights.

The update unit 240 also creates and deletes trackers as tracker updates. First, the update unit 240 determines whether or not there is a target that is not associated with the tracker after the association processing by the association unit 230 . A target that does not correspond to this tracker may be an object newly appearing within the range captured by the camera 20 . Therefore, when there is a target that does not correspond to the tracker, the updating unit 240 determines whether or not this target can be regarded as an object newly appearing within the range.

That is, when there is a target that does not correspond to any tracker, the updating unit 240 evaluates the probability that this target exists. Then, the update unit 240 determines whether or not this probability is equal to or greater than a predetermined value. Then, if this probability is equal to or greater than a predetermined value, the update unit 240 determines that the object indicated by this target is an object that has newly appeared within the range. Then, the update unit 240 newly creates a tracker related to the object (target) determined to be the object newly appearing within the range.

Also, the updating unit 240 determines whether or not there is a tracker that is not associated with the target. A tracker that does not correspond to this target may be a tracker for an object that has disappeared from the range captured by the camera 20 (moved from within the range to out of range). Therefore, when there is a tracker that does not correspond to a target, the update unit 240 determines whether this target can be regarded as a tracker related to an object that has disappeared from the range.

In other words, when there is a tracker that does not correspond to a target, the updating unit 240 evaluates the probability of existence of an object related to this tracker. The updating unit 240 then determines whether this probability is below a predetermined value. Then, when this probability is less than a predetermined value, the update unit 240 determines that the object related to this tracker is an object that has disappeared from the range. Then, the update unit 240 deletes the tracker associated with the object determined to have disappeared from the range.

The probability that an object exists is determined by the likelihood of the tracker. In other words, when there is a tracker that does not correspond to the target, the updating unit 240 reduces the likelihood of this tracker. Then, when the likelihood value of the tracker falls below a predetermined threshold, the updating unit 240 deletes the tracker.

Then, the update unit 240 generates the final remaining tracker as the tracking result of the object in this frame. Then, the updating unit 240 outputs the information indicating the position of the tracker, the information about the size of the object of the tracker, and the like among them as the information indicating the tracking result of the object tracking (tracking information).

As described above, according to the object tracking device 10 according to the present embodiment, object tracking is performed in order from the oldest time information included in the camera images output from each camera 20 . Objects detected from the video data of each of the cameras 20 are detected by integrating detection results with high accuracy for each camera 20 and using tracking results that reflect these detection results and past tracking results. Therefore, the tracking accuracy of object tracking performed by the integrated tracking unit 200 of the object tracking device 10 is improved.

Next, the flow of object tracking processing of the object tracking device 10 according to the present embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing an example of the flow of object tracking processing by the object tracking device 10 according to this embodiment.

As shown in FIG. 8, first, the recognition-type object detection unit 111 of the detection unit 100 receives a camera image from the camera 20 linked to the detection unit 100 including the object detection unit 110 (step S81).

The detection unit 100 checks whether the received frame of the camera image is the first frame output from the camera 20 (step S82). Proceed to S85.

If the received frame of the camera image is not the first frame (NO in step S82), individual coordinate transformation section 130 converts the tracking information for the previous frame, which is output from integrated tracking section 200, into an individual coordinate system. tracking information (step S83).

Then, the search range setting unit 112 of the object detection unit 110 uses the tracking information converted in step S83 to set the search range of the object for the current frame (step S84).

Then, the recognition type object detection unit 111 detects an object from the received camera image (step S85).

Next, the individual coordinate transformation unit 130 transforms the detection result by the recognition type object detection unit 111 into the detection result represented by the common coordinate system (step S86).

Next, the prediction unit 210 of the integrated tracking unit 200 predicts the position of the object on the current frame using tracker information (step S87).

Then, the associating unit 230 associates the object (target) included in the detection result with the tracker (step S88).

Next, the update unit 240 updates tracker information such as the position of the tracker object and the motion model of the object (step S89).

Further, the update unit 240 creates and/or deletes trackers (step S90). The object tracking device 10 repeats this process until no more frames are input to the detection unit 100 .

(effect)
As described above, object tracking device 10 according to the present embodiment can track an object with higher accuracy. This is because the detection unit 100 detects an object from the output information of the camera 20 based on the tracking information for the output information preceding the output information (video frame). This is because the integrated tracking unit 200 tracks the object based on the plurality of detection results output by the detection units 100 and generates object tracking information expressed in a common coordinate system.

For example, if there is an object that cannot be seen from a certain camera 20 but is visible from another camera 20, the object tracking device 10 can also track the object that cannot be seen from a certain camera 20 and the object in the image of this certain camera 20. Used for detection. Thereby, when the same object appears in the range visible from the certain camera 20, the detection unit 100 can suitably detect this object. Therefore, the object tracking device 10 can accurately track the object.

Therefore, the object tracking device 10 can improve the object detection accuracy as compared with the case where the tracking result for the previous frame is not used. In addition, since the object tracking device 10 performs object tracking using all detection results with high detection accuracy, the accuracy of the tracking results obtained as a whole is also improved.

As described above, the object tracking device 10 according to the present embodiment can extract the line of flow of a person or the like moving across the areas photographed by the plurality of cameras 20 . As a result, the results of tracking by the object tracking device 10 can be used as basic information for marketing and changing the layout of the store, for example, by analyzing the behavior of customers walking around the store. The tracking results can also be used to detect people loitering between areas for security purposes.

In addition, since the search range setting unit 112 uses the tracking result to set the search range for object detection in the camera image, the recognition type object detection unit 111 can reduce unnecessary false detection. In addition, the recognition type object detection unit 111 can speed up the processing of object detection.

In addition, the integrated tracking unit 200 performs object tracking using the likelihood of the target and/or the likelihood of the tracker, so that the object tracking device 10 can obtain more reliable object tracking results. Further, the object tracking device 10 uses the object tracking result thus obtained to search for an object, so that the accuracy of object tracking can be further improved. As a result, the object tracking device 10 can improve the accuracy of object tracking as a whole.

<Second Embodiment>
Next, a second embodiment of the invention will be described with reference to the drawings. For convenience of explanation, members having the same functions as members included in the drawings described in the first embodiment are denoted by the same reference numerals, and descriptions thereof are omitted.

An object tracking system 2 according to the present embodiment includes an object tracking device 50 instead of the object tracking device 10 of the object tracking system 1 according to the first embodiment described using FIG. Other system configurations of the object tracking system 2 are the same as those of the object tracking system 1 shown in FIG. 2, so description thereof is omitted.

(Object tracking device 50)
The function of the object tracking device 50 will be described with reference to FIG. FIG. 9 is a functional block diagram showing an example of the functional configuration of object tracking device 50 according to the present embodiment. As shown in FIG. 9, the object tracking device 50 includes a plurality of detection units (100-1 to 100-N), an integrated tracking unit 200, and a display control unit 300. In this embodiment, as in the first embodiment described above, when the plurality of object detection units (100-1 to 100-N) are not distinguished from each other, or when they are collectively referred to, they are It is called a detection unit 100 .

The display control unit 300 controls an image (video) displayed on the display device 30 . Specifically, the display control unit 300 generates display data by converting the tracking information output by the integrated tracking unit 200 into data that can be displayed on the display device 30 , and transmits the display data to the display device 30 . The tracking information output by the integrated tracking unit 200 is expressed in a common coordinate system. Therefore, the display control unit 300 generates display data that can be displayed on the display device 30 in the common coordinate system.

At this time, the integrated tracking unit 200 uses information necessary for displaying the flow line of the object on the display device 30 (for example, information indicating the past position of the object of the tracker) as tracking information, and the display control unit 300 It is preferable to output to This tracking information may be the same as the information fed back to the detection unit 100, or may be different.

Then, the display device 30 displays the received display data on the screen. Thereby, the object tracking device 50 can present the tracking result to the user.

Further, the detection unit 100 of the object tracking device 50 according to the present embodiment converts the search range set by the search range setting unit 112 of the object detection unit 110 into data that can be displayed on the display device 30 to generate display data. and transmitted to the display device 30 . The search range information output by the search range setting unit 112 is expressed in an individual coordinate system. Therefore, the display control unit 300 uses the camera parameters of the camera 20 associated with the detection unit 100 that has output the search range information to generate display data that can be displayed on the display device 30 in the individual coordinate system.

Then, the display device 30 displays the received display data on the screen. Accordingly, the object tracking device 50 can use the object search range information output from each detection unit 100 to present the search range to the user.

A plurality of display devices 30 may be provided. For example, the display device 30 receives display data displayed in the common coordinate system and display data displayed in the individual coordinate system by different display devices 30, and displays the received display data on the screen respectively. It may be a configuration. Further, the display device 30 may be configured to divide the display area of one screen and display a plurality of display data on the screen. Thus, the display data display method in display device 30 according to the present embodiment is not particularly limited.

Moreover, the display control unit 300 may be configured to be provided in the detection unit 100 . FIG. 10 is a functional block diagram showing an example of the functional configuration of the detection unit 100 of the object tracking device 50 according to this embodiment. As shown in FIG. 10 , the detection section 100 includes an object detection section 110 , a common coordinate transformation section 120 , an individual coordinate transformation section 130 and a display control section 150 . Further, the object detection unit 110 includes a recognition type object detection unit 111 and a search range setting unit 112, like the object detection unit 110 shown in FIG.

The search range setting unit 112 shown in FIG. 10 outputs search range information indicating the set search range to the display control unit 150 . The display control unit 150 receives the search range information output from the search range setting unit 112 and, like the display control unit 300, generates display data converted into data that can be displayed on the display device 30. FIG. The search range information output by the search range setting unit 112 is expressed in an individual coordinate system. Therefore, the display control unit 150 uses camera parameters of the camera 20 linked to the detection unit 100 including the display control unit 150 to generate display data that can be displayed on the display device 30 in the individual coordinate system.

The display control unit 150 then transmits the generated display data to the display device 30 . Then, the display device 30 displays the received display data on the screen.

(Application example)
An application example of the object tracking device 50 according to the present embodiment will be described with reference to FIGS. 11 to 14. FIG. 11 to 14 are diagrams for explaining application examples of the object tracking device 50 according to the present embodiment.

First, FIG. 11 is a diagram showing an example of a room in which a shelf R1, a shelf R2, and a plurality of cameras (A to F) are installed, viewed from a direction opposite to the direction of gravity. As shown in FIG. 11, the horizontal direction of FIG. 11 is the X-axis in the common coordinate system, and the vertical direction is the Y-axis. The shelf R1 and the shelf R2 are installed side by side on the X-axis so that the longitudinal direction is parallel to the Y-axis direction.

Camera A is installed at a position close to the doorway of this room. In this embodiment, it is assumed that the entire interior of the room is captured by cameras A to F. In other words, as shown in FIG. 11, an indoor space in which a plurality of cameras (A to F) are installed becomes a shooting space. In addition, the cameras A to F photograph locations common to each other.

12A and 12B are diagrams showing examples of images captured by the camera A and the camera B, respectively. The upper diagram in FIG. 12 is a diagram showing a frame with an image taken by camera A, and the lower diagram is a diagram showing a frame with an image taken by camera B. As shown in FIG. Coordinate values in these frames are represented by coordinate values of the individual coordinate system for each camera.

Note that object tracking device 50 in the present embodiment may be configured to display an image captured by camera 20 on display device 30 .

As shown in FIG. 12, the image captured by camera A includes person C1. Also, since the camera A is installed in the vicinity of the entrance, the entrance is included in this image. Also, the video captured by camera B includes person C1 and person C2.

When viewed from the camera A, the person C2 is hidden behind the shelf R1. Therefore, the person C2 is an object that cannot be seen from the camera A at the time of the image shown in FIG. If these video frames are the first frames of the video shot by camera A and camera B, since there is no tracking information in the previous frame, the object tracking device 50 detects and tracks the object from these frames. Generate information.

Then, the search range setting unit 112 sets the object search range for the next frame of the image captured by the camera A using the tracking information expressed in the individual coordinate system. Similarly, the search range setting unit 112 sets an object search range for the next frame of the image captured by camera B using the tracking information expressed in the individual coordinate system.

FIG. 13 is a diagram showing an example of the search range displayed on the display device 30. As shown in FIG. The upper diagram in FIG. 13 is a diagram showing an example of an object search range for frames output from camera A, and the lower diagram is an example of an object search range for frames output from camera B. It is a figure which shows.

As shown in the upper diagram of FIG. 13, the search range setting unit 112 obtains the search range A1 from the position of the person C1 in the upper diagram of FIG. Further, the search range setting unit 112 obtains the region of the doorway to the room as the search range N1. The search range setting unit 112 also obtains the outer edge of the frame as search ranges N2 and N3. The search range setting unit 112 then outputs information summarizing the obtained search ranges A1, N1 to N3 to the recognition type object detection unit 111 and the display control unit 150 or the display control unit 300 as search range information. Then, the display control unit 150 or the display control unit 300 converts the search range indicated by the search range information on the display device 30 into display data that can be displayed on the screen, and transmits the display data to the display device 30 .

The display device 30 that receives the display data from the display control unit 150 or the display control unit 300 displays the search range on the screen as shown in the upper diagram of FIG.

Next, the diagram on the lower side of FIG. 13 will be described. As shown in the lower diagram of FIG. 13, the search range setting unit 112 obtains the search range B1 and the search range B2 from the positions of the person C1 and the person C2 in the lower diagram of FIG. 12, respectively. Further, the search range setting unit 112 obtains the outer edge of the frame as search ranges N4 to N7. The search range setting unit 112 then outputs information summarizing the obtained search ranges B1, B2, N4 to N7 to the recognition type object detection unit 111 and the display control unit 150 or the display control unit 300 as search range information. Then, the display control unit 150 or the display control unit 300 converts the search range indicated by the search range information on the display device 30 into display data that can be displayed on the screen, and transmits the display data to the display device 30 .

The display device 30 that has received the display data from the display control unit 150 or the display control unit 300 displays the search range on the screen as shown in the lower diagram of FIG.

Note that the display device 30 may display the search range in a different manner for each area. For example, the display device 30 may display the search range for the already detected object and the search range for the outer edge of the frame in different colors.

The integrated tracking unit 200 then generates trackers for persons C1 and C2 detected in subsequent frames. Then, the integrated tracking unit 200 outputs to the display control unit 300 information necessary for displaying the flow lines of the person C1 and the person C2 as tracking information.

The display control unit 300 that receives the tracking information from the integrated tracking unit 200 converts the tracking information into display data that can be displayed on the display device 30 and transmits the display data to the display device 30 .

Then, the display device 30 displays the display data received from the display control section 300 on the screen. FIG. 14 is a diagram showing an example when the display device 30 displays on the screen (display screen) flow lines indicating the tracking results of the person C1 and the person C2. As shown in FIG. 14, in this application example, the display screen displays the tracking result of the object on the XY plane in the common coordinate system. In FIG. 14, the line of flow of the person C1 is displayed as a solid line, and the line of flow of the person C2 is displayed as a chain line. Thus, the display device 30 can display the tracking result of the object on the screen.

<Third Embodiment>
Next, a third embodiment of the invention will be described with reference to the drawings. For convenience of explanation, members having the same functions as members included in the drawings described in the first and second embodiments are denoted by the same reference numerals, and descriptions thereof are omitted.

Object tracking device 10 according to the present embodiment is configured to include detection unit 400 instead of detection unit 100 of object tracking device 10 shown in FIG. The configuration of this detection unit 400 will be described with reference to FIG. FIG. 15 is a functional block diagram showing an example of the functional configuration of the detection unit 400 of the object tracking device 10 according to this embodiment.

The detection unit 400 includes an object detection unit 140 instead of the object detection unit 110 of the detection unit 100 shown in FIGS. 4 and 5. FIG. Moreover, the detection unit 400 is configured to further include a storage unit 160 . That is, the detection unit 400 according to the present embodiment includes an object detection unit 140, a common coordinate transformation unit 120, an individual coordinate transformation unit 130, and a storage unit 160, as shown in FIG.

In the present embodiment, a configuration including an object detection unit 140 instead of the object detection unit 110 of the detection unit 100 of the object tracking device 10 according to the first embodiment will be described as an example. Note that the present invention is not limited to this, and an object detection unit 140 may be provided instead of the object detection unit 110 of the detection unit 100 of the object tracking device 50 according to the second embodiment. In other words, detection unit 100 according to the present embodiment may be configured to output data to be displayed to display control unit 150 or display control unit 300 .

The storage unit 160 stores camera parameters for each camera 20, which are used when transforming the coordinate system. Furthermore, in the storage unit 160, the individual coordinate transformation unit 130 is used when confirming whether or not the coordinate values of the common coordinate system included in the tracking information are included in the range photographed by the associated camera 20. Information indicating the range of coordinate values in the common coordinate system is stored. Also, the storage unit 160 may store the video captured by the camera 20 . Note that this video may be temporarily stored.

Note that FIG. 15 illustrates an example in which the storage unit 160 is built in the detection unit 400, but the present invention is not limited to this. The storage unit 160 may be provided in the object tracking device 10 separately from the detection unit 400 . Also, the storage unit 160 may be realized by a storage device or the like separate from the object tracking device 10 .

Next, a detailed functional configuration of the object detection section 140 of the detection section 400 will be described with reference to FIG. FIG. 16 is a functional block diagram showing an example of the functional configuration of object detection section 140 of detection section 400 according to the present embodiment. As shown in FIG. 16, the object detection unit 140 includes a recognition type object detection unit (first object detection means) 141, a non-recognition type object detection unit (second object detection means) 142, and a detection parameter update unit 143. and a detection result integration unit 144 .

In the present embodiment, object detection using a dictionary (identifier) or the like is called "recognition-type object detection". On the other hand, object detection that does not use a discriminator or the like is called “non-recognition type object detection”.

The recognition type object detection unit 141 detects an object from the camera image input to the recognition type object detection unit 141 . The recognition type object detection unit 141 detects an object for the entire frame. Note that the recognition-type object detection unit 141 may perform object detection based on search range information output from the detection parameter update unit 143, which will be described later, as indicated by the dashed line in FIG. At this time, the recognition type object detection unit 141 performs object detection in the same manner as the recognition type object detection unit 111 described in the first embodiment.

Further, when the search range information is not output from the detection parameter update unit 143, the recognition type object detection unit 141 may perform object detection using another criterion instead of performing object detection on the entire frame. good. For example, the recognition-type object detection unit 141 may use silhouette information to detect an object only in an area with a silhouette and its surrounding area.

The recognition type object detection unit 141 outputs the detection result of the object detection to the detection result integration unit 144 as the first detection result.

At this point, the recognizable object detection unit 141 may extract the appearance features of the object in preparation for object detection by the non-recognizable object detection unit 142, which will be described later. Appearance features of an object include information such as the color, pattern, and shape of the object, but the present invention is not limited to this. The recognition type object detection unit 141 extracts these feature amounts as appearance features of the object. At this time, the area used for object detection by the recognition type object detection unit 141 and the area used for object detection by the non-recognition type object detection unit 142 may not be the same. For example, when the object is a person, the object detection by the recognition type object detection unit 141 detects the head, and the object detection by the non-recognition type object detection unit 142 detects up to the clothing area. At this time, the recognition-type object detection unit 141 extracts the feature amount of the appearance features of the object so as to include the area of the clothes. Then, the recognition-type object detection unit 141 may output the extracted feature quantity as template information together with information (extraction region information) indicating the region used for extraction. Further, the recognition type object detection unit 141 may hold the feature amount itself inside the recognition type object detection unit 141 and output only the information for identifying the feature amount.

The detection parameter updating unit 143 receives the tracking information expressed in the individual coordinate system from the individual coordinate transforming unit 130 . Then, the detection parameter updating unit 143 uses this tracking information to obtain parameters (called detection parameters) necessary for object detection. The detection parameters are parameters required for object detection processing. The detection parameters include, for example, the predicted position of the object in the current frame (prediction region), the search range to which object detection is applied, the size of the template used for template matching, and the features of the target template previously associated with the tracker (template information), etc. Note that the detection parameters may not include all of these pieces of information, as long as they include parameters necessary for object detection by the non-recognition type object detection unit 142 . Also, the detection parameters may include information on the target associated with the tracker in the tracking result of the object in the previous frame.

For example, the detection parameter updating unit 143 uses the position of the object in the current frame as the predicted position for the target (object) detected in the previous frame and associated with the tracker based on the tracking information of the object. demand. This prediction processing is the same as the prediction processing of the predicted position in the search range setting unit 112 according to the first embodiment. Note that the detection parameter updating unit 143 may obtain an area including this predicted position as the predicted area.

Also, for example, the detection parameter updating unit 143 may obtain a range to which object detection by template matching is applied, centering on the prediction region, and include this range as the search range for objects included in the detection parameters.

The detection parameter updating unit 143 obtains the detection parameter for each object included in the tracking information. Then, the detection parameter update unit 143 updates the obtained detection parameter as the detection parameter used in the object detection process. The detection parameter updating unit 143 then outputs the detection parameters to the unrecognized object detection unit 142 .

Note that the detection parameter updating unit 143 uses the tracking information converted into the coordinate values of the individual coordinate system, similarly to the search range setting unit 112 of the detecting unit 100 according to the first embodiment. A search range may be obtained. Then, the detection parameter update unit 143 may output search range information indicating the search range of the obtained object to the recognition type object detection unit 141 .

The unrecognized object detection unit 142 receives detection parameters from the detection parameter update unit 143 . Then, the non-recognition type object detection unit 142 detects an object from the camera image input to the non-recognition type object detection unit 142 based on the received detection parameters. Unlike the recognition type object detection section 141, the non-recognition type object detection section 142 detects an object based on the similarity in appearance of the object detected in the previous frame.

That is, when an object is detected in the previous frame, the non-recognition type object detection unit 142 stores the image feature of the area (or the partial image itself of the detection area may be used) as a template. Then, the non-recognition type object detection unit 142 performs object detection by examining whether or not an area similar to the stored template exists in the current frame by template matching. As the features of the image used at this time, for example, information about color patterns and distributions, edge and luminance gradient distribution information, or features obtained by combining these can be used.

The detection parameters used for object detection in the non-recognition type object detection unit 142 are controlled by the detection parameters output from the detection parameter update unit 143 . Specifically, the unrecognized object detection unit 142 performs object detection by template matching on the predicted position (prediction region) of the object predicted by the detection parameter updating unit 143 and its vicinity. That is, the non-recognition type object detection unit 142 sets a search range for template matching centering on the predicted object existence range (prediction region), and performs template matching on the surrounding area. At this time, the non-recognition type object detection unit 142 may also consider that the apparent size of the object changes due to the movement of the position of the object. This change can be calculated by using the camera parameters. Therefore, the non-recognition type object detection unit 142 may calculate the change in the size of the object, reflect it in the template, and then perform template matching.

Further, the template information for template matching performed by the non-recognition type object detection unit 142 may be the feature amount extracted by the recognition type object detection unit 141 in the object detection processing in the previous frame.

In this way, the non-recognition type object detection unit 142 performs object detection based on the tracking result for the object tracked by the integrated tracking unit 200, so detection accuracy is improved compared to when the tracking result is not used. can be made

Then, the non-recognition type object detection unit 142 outputs the detection result of the object detection to the detection result integration unit 144 as the second detection result.

The detection result integration unit 144 receives the first detection result from the recognition type object detection unit 141 . Also, the detection result integration unit 144 receives the second detection result from the unrecognized object detection unit 142 . Then, the detection result integration unit 144 integrates the first detection result and the second detection result. Then, the detection result integrating section 144 outputs the integrated result to the common coordinate transformation section 120 as the detection result of object detection in the object detecting section 140 .

There may be objects that are included in both the first detection result and the second detection result, and objects that are included in only one of them. Therefore, the detection result integration unit 144 associates and integrates objects included in each of the first detection result and the second detection result. For this association, for example, the degree of overlapping of object regions can be used.

That is, the detection result integration unit 144 calculates the overlap ratio between object regions (for example, the overlap ratio of object circumscribing rectangles), and if this is greater than a predetermined value, the object included in the first detection result and the object included in the first detection result. 2 are associated with the objects included in the detection results.

Further, the detection result integration unit 144 may formulate a graph problem in which the overlap ratio of regions between objects is used as a weight, and associate the objects with each other. For example, the detection result integration unit 144 converts the overlapping ratio into a cost using a monotonically non-increasing function, and then calculates the optimum correspondence using the Hungarian method or the like to associate objects.

The detection result integration unit 144 may merge at this point if the value used in the association (for example, the overlap ratio or the cost) is greater than a predetermined value as a result of the association. Further, the detection result integration unit 144 may generate information indicating that the data are not merged at this point, but are associated with each other. Then, the detection result integration unit 144 outputs the detection result obtained by combining the first detection result and the second detection result with the information indicating the correspondence as the detection result of the object detection unit 140, At the time of integrated tracking, tracking may be performed using the information of correspondence.

Also, the unrecognized object detection unit 142 outputs a second detection result based on the object tracking result for the previous frame. Therefore, the second detection result may be generated later than the first detection result. In such a case, the detection result integration unit 144 temporarily stores the first detection result in storage means such as a buffer in the detection result integration unit 144 or the storage unit 160 . Then, the detection result integration unit 144 may integrate both results upon receiving the second detection result for the frame corresponding to the frame for which the first detection result is to be generated.

As described above, the detection unit 400 of the object tracking device 10 according to the present embodiment detects the object detection result (first detection result) by the recognition type object detection unit 141 and the object detection result by the non-recognition type object detection unit 142. A result obtained by integrating the detection result (second detection result) is output as a detection result. At this time, the non-recognition type object detection unit 142 detects the object by performing template matching based on the tracking result of the object tracked by the integrated tracking unit 200 . As a result, the detection unit 400 can further improve the accuracy of object detection compared to the case where only object detection (recognition-type object detection) is performed by identifying the object.

Therefore, the object tracking device 10 can track the object with higher accuracy.

<Fourth Embodiment>
Next, a fourth embodiment of the present invention will be described with reference to the drawings. For convenience of explanation, members having the same functions as members included in the drawings described in each of the above-described embodiments are denoted by the same reference numerals, and descriptions thereof are omitted.

The object tracking device 10 according to the present embodiment is configured to include an integrated tracking unit 500 instead of the integrated tracking unit 200 of the object tracking device 10 shown in FIG. The configuration of this integrated tracking unit 500 will be described with reference to FIG. FIG. 17 is a functional block diagram showing an example of the functional configuration of integrated tracking section 500 of object tracking device 10 according to the present embodiment. As shown in FIG. 17 , integrated tracking section 500 includes buffer section 510 , prediction section 210 , storage section 220 , association section 530 and update section 240 .

In the present embodiment, a configuration including an integrated tracking section 500 instead of the integrated tracking section 200 of the object tracking device 10 according to the first embodiment will be described as an example. It should be noted that the present invention is not limited to this, and an integrated tracking unit 500 may be provided instead of the integrated tracking unit 200 of the object tracking device 50 according to the second embodiment. In other words, integrated tracking section 500 according to the present embodiment may be configured to output data to be displayed to display control section 300 .

Also, the detection unit that outputs the detection result to the integrated tracking unit 500 according to the present embodiment may be the detection unit 400 described in the third embodiment.

The buffer unit 510 is means for temporarily storing the detection results expressed in the common coordinate system output from the detection unit 100 . Among the data (detection results) buffered in the buffer unit 510 , the association unit 530 acquires data whose time information included in the detected camera video is within a predetermined period. This predetermined period is a periodic period. Then, the associating unit 530 performs object tracking using one or a plurality of detection results acquired in a certain cycle. As described above, the integrated tracking unit 500 according to the present embodiment is also called a collective tracking unit because it performs object tracking using images from a plurality of cameras out of the images from each camera 20 .

Object tracking (also called batch integrated tracking) performed by the integrated tracking unit 500 will be described with reference to FIG. FIG. 18 is a diagram for explaining batch tracking processing of objects performed by the integrated tracking unit 500 according to the present embodiment. Similar to FIG. 7, FIG. 18 shows an example of the timing at which images are acquired by each of camera A, camera B, and camera C when the number of cameras is three. In FIG. 18, the horizontal axis indicates the time axis, and the further to the right, the later in time. As shown in FIG. 18, camera A acquires images at times t1, t5 and t8. Similarly, camera B has acquired images at times t2, t4, t6 and t9, and camera C has acquired images at times t3 and t7.

Then, object detection is sequentially performed on the images acquired at each of these timings. For convenience of explanation, the time shown in FIG. 18 is assumed to be substantially the same as the time when the object detection result is output. In other words, the time t1 is the time when the detection result for the frame of the video imaged by the camera A is output from the detection unit 100 and buffered in the buffer unit 510 .

Also, the time axis at the bottom of FIG. 18 shows an example of a periodic period.

The associating unit 530 acquires, from among one or a plurality of detection results buffered in the buffer unit 510, detection results for which the time information included in the camera image used for object detection is within a predetermined period. In addition, as described above, this predetermined period is a periodic period. Also, in this embodiment, the buffered time and the time of the camera image are considered to be the same. Therefore, it can be said that the associating unit 530 acquires one or a plurality of detection results buffered in the buffer unit 510 at predetermined intervals.

Specifically, the associating unit 530 first acquires the detection result buffered in the first period T1. That is, the associating unit 530 acquires the detection results buffered at times t1, t2, and t3. The detection result buffered at time t1 is the detection result for the frame of the video captured by camera A. FIG. Further, the detection result buffered at time t2 is the detection result for the video frame shot by camera B, and the detection result buffered at time t3 is the detection result for the video frame shot by camera C. This is the result. Therefore, the associating unit 530 stores the detection results buffered in the buffer unit 510 within a predetermined period (in this case, T1), and the plurality of detection results corresponding to the frames of the video captured by each of the plurality of cameras 20. is acquired from the buffer unit 510 . Then, the associating unit 530 performs object tracking using the acquired detection result.

Similarly, the correlating unit 530 also acquires the detection results buffered within these periodic periods during periods T2, T3, and T4, and performs object tracking.

Note that, in the present embodiment, a configuration will be described in which the associating unit 530 acquires the data (a plurality of detection results) buffered in the buffer unit 510 from the buffer unit 510 at a predetermined cycle. may be configured to receive this data from the buffer section 510 at a predetermined cycle. That is, buffer section 510 may have a function of outputting this data to association section 530 at a predetermined cycle.

Returning to FIG. 17, the associating unit 530 of the integrated tracking unit 500 will be further described.

The associating unit 530 obtains the distance between the targets from the positions of the targets included in each acquired detection result, and associates the targets that are close to each other. At this time, the associating unit 530 performs the associating by a method such as the Hungarian method using the distance between the targets. Further, the associating unit 530 may simultaneously use the similarity of the appearance features of the targets in addition to the distance between the targets. For example, targets that are close to each other and have similar colors are likely to be the same object. Therefore, the matching unit 530 may perform matching using such characteristics. Note that the feature for determining the similarity of the appearance features is not limited to the color, and may be, for example, the pattern of the target.

Then, the associating unit 530 integrates the detection results for the mutually associated targets. In other words, the associating unit 530 obtains the position of the object using the detection result of each of the associated targets after associating the targets. At this time, the associating unit 530 may evaluate the likelihood of each target and/or the accuracy of the predicted position, and set the position with the maximum accuracy as the position of the object.

The association unit 530 also weights the position of each target based on the accuracy of the predicted position determined by the angle (depression angle or elevation angle) of the camera 20 with respect to each target and the distance from the camera 20 to the target. may Then, the associating unit 530 may calculate a statistic such as an average value from the weighted positions, and use the position indicated by the calculated statistic as the position of the object.

Then, the associating unit 530 sets the obtained position of the object as the position of the target with respect to the period of obtaining the detection result. The associating unit 530 performs association using the positions of the targets in the same manner as the associating unit 230 according to the first embodiment. Also, the processing of association by the integrated tracking unit 500 and the subsequent processing are the same as the processing in the integrated tracking unit 200 described in the first embodiment, so description thereof will be omitted.

Also, the associating unit 530 may associate each target with a tracker and integrate them before associating targets. That is, when there are multiple targets associated with the same tracker, the associating unit 530 performs merging processing between these targets. In this case, the associating unit 530 may perform merging by prioritizing targets with higher likelihoods and/or higher accuracy of predicted positions. In this way, the association unit 530 may evaluate detection results and integrate targets associated with the same tracker based on these pieces of information.

As described above, the object tracking apparatus 10 according to the present embodiment performs object tracking using all detection results for camera images captured by each camera 20 within a predetermined period. This makes it easier for the object tracking device 10 to prioritize the object search results among the plurality of cameras 20 and apply the process of tracking the object.

Further, for example, when the frame rate of all cameras 20 is stably the same, by setting a predetermined period according to the frame interval, the frames of all cameras 20 are included in this predetermined period. Therefore, the object tracking device 10 according to the present embodiment can perform object tracking on the frames for all the cameras 20. FIG. As a result, the object tracking device 10 can simultaneously evaluate the detection results of objects that are simultaneously visible from a plurality of cameras 20, so that the reliability of the detection results can be directly reflected by tracking.

<Fifth Embodiment>
A fifth embodiment of the present invention will be described. In this embodiment, the minimum configuration for solving the problems of the present invention will be described.

Since the object tracking device 10 according to the present embodiment has the same configuration as the object tracking device 10 shown in FIG. 1 described in the first embodiment, it will be described with reference to FIG.

As shown in FIG. 1, the object tracking device 10 according to this embodiment includes a plurality of detection units (100-1 to 100-N) (N is a natural number) and an integrated tracking unit 200. FIG. It should be noted that, in the present embodiment, the plurality of detection units (100-1 to 100-N) are referred to as the detection unit 100 when they are not distinguished from each other or when they are collectively referred to.

Each of the plurality of detection units 100 detects an object from output information output from the sensor. In FIG. 1, the sensor is a camera and the output information of the sensor is described as a camera image (image data), but the sensor is not limited to the camera. Specifically, the detection unit 100 detects an object based on tracking information output from the integrated tracking unit 200 . The detection unit 100 outputs detection results to the integrated tracking unit 200 .

Integrated tracking unit 200 tracks each of one or more objects indicated by the detection results based on the plurality of detection results output by each of the plurality of detection units (100-1 to 100-N). . Then, the integrated tracking unit 200 generates tracking information of the object expressed in the common coordinate system as the tracking result. The integrated tracking unit 200 then outputs to each of the plurality of detection units (100-1 to 100-N).

Thus, the detection unit 100 of the object tracking device 10 according to the present embodiment detects the object from the output information output from the sensor based on the tracking result of the object tracked by the integrated tracking unit 200.

In this way, the object tracking device 10 uses the tracking result for the previous frame to detect the object from the video, so that the object detection accuracy can be improved compared to when the tracking result is not used. In addition, since object tracking is performed using all object detection results for images captured by each camera 20, the object tracking device 10 can improve tracking accuracy compared to the case where object tracking is performed independently for each camera. . Further, the object tracking device 10 performs object tracking using all detection results with high detection accuracy, so that the object can be tracked with higher accuracy.

In each of the above-described embodiments, the object tracking device 10 includes the detection unit (100, 400) and the integrated tracking unit (200, 500). and may be implemented by separate devices. That is, the detection units (100, 400) may be implemented as object detection devices, and the integrated tracking units (200, 500) may be implemented as separate devices as integrated tracking devices. Also, the display control unit 300 may be realized by a display control device separate from the object tracking device 50 . This display control device may be built in the display device 30 .
<Sixth Embodiment>
A sixth embodiment of the present invention will be described. Since the object tracking device 10 according to the present embodiment has the same configuration as the object tracking device 10 shown in FIG. 1 described in the first embodiment, it will be described with reference to FIG. Although the object tracking device 10 according to the present embodiment has a configuration in which the following functions are added to the object tracking device 10 according to the first embodiment, the present invention is limited to this. not to be The object tracking device 10 according to this embodiment can also be applied to the object tracking devices according to the second to fifth embodiments described above.

In this embodiment, the integrated tracking unit 200 further acquires information about how the object looks, and includes the acquired information in the tracking information. Then, the detection unit 100 controls the detection of objects using the information regarding the appearance of each object included in the tracking information output from the integrated tracking unit 200 .

This information about how an object looks (hereafter, how it looks) is information about how each object looks when viewed from the position of each camera 20. Depending on the position of each tracker object, It is determined.

For example, consider a case where an object is in front of another object (on the side of the camera 20) when an object and another object are viewed from a camera 20. FIG. In this case, the rear object (another object) is hidden by the front object (certain object), and there is a high possibility that the object cannot be seen from the camera 20 . At this time, the integrated tracking unit 200 includes information representing such overlapping objects as appearance information in a tracker (tracking result) relating to other objects, and outputs the tracking result.

Next, specific operations of each part of the object tracking device 10 according to the present embodiment will be described.

The integrated tracking unit 200 stores information regarding the placement of each camera 20 in, for example, the storage unit 220 shown in FIG. The information about the arrangement of the cameras 20 is, for example, information indicating where each camera 20 is arranged, in which direction the camera 20 is photographed, and the like. In addition, the integrated tracking unit 200 may include, as information on the arrangement of the camera 20, information on the position and direction of lighting in the shooting space, information on lighting characteristics, and information on lighting conditions such as which position in the shooting space is bright or dark. . In addition, the integrated tracking unit 200 may also hold direction information of the shooting space as information regarding the arrangement of the camera 20 .

Similar to the integrated tracking unit 200 according to each embodiment described above, the integrated tracking unit 200 predicts the movement of the object indicated by each tracker on the current frame, associates the target with the tracker, and determines the position of the tracker. Ask for

Then, the integrated tracking unit 200 predicts the position of the object indicated by each tracker on the captured image captured by each camera 20 from the obtained positions of the trackers and the movement information of each tracker.

Then, the integrated tracking unit 200 refers to the information regarding the placement of each camera 20 and predicts how each object at the predicted position will look when viewed from each camera 20 (appearance). That is, the integrated tracking unit 200 predicts whether or not the objects indicated by the respective trackers will overlap each other at the next photographing timing for each of the plurality of cameras 20 .

Then, when the integrated tracking unit 200 determines that the objects overlap each other on the image captured by the camera 20 based on the predicted appearance, the integrated tracking unit 200 recognizes that there is a possibility that the objects overlap and cannot be seen. information (appearance information) is generated.

For example, the integrated tracking unit 200 determines whether there is another predicted object between a line segment connecting a camera 20 and a predicted object. If the predicted other object is between the line segments, there is a high possibility that this certain object overlaps with the other object. Therefore, the integrated tracking unit 200 obtains information on hidden (overlapping) objects and the degree of overlap (likelihood) as appearance information for the certain object.

Then, the integrated tracking unit 200 may include this determination result as appearance information in the tracking result of this certain object.

Then, the integrated tracking unit 200 includes the generated appearance information in the tracking information of an object that is highly likely to disappear from the captured image captured by a certain camera 20 . At this time, the appearance information preferably includes information indicating the camera 20 with a high possibility that the object cannot be seen.

The integrated tracking unit 200 then outputs tracking information including the appearance information to each detection unit 100 . Note that the integrated tracking unit 200 may output the tracking information including the appearance information to the detection unit 100 associated with the camera 20 (certain camera 20) with a high possibility that the objects overlap and cannot be seen. The integrated tracking unit 200 may then output the tracking information that does not include the appearance information to the object tracking device 10 associated with another camera 20 .

In addition, if it is known that the lighting changes according to the position of the object, and the color tone and brightness of the object change, the integrated tracking unit 200 detects the change in appearance according to the position of the object. may be included in the tracking information.

For example, if the lighting is determined by the position of the object, the integrated tracking unit 200 predicts the lighting from the position, and obtains information such as brightening, darkening, and color change for each tracker. may be included in the tracking information.

Also, when the position of the lighting in the space where the object is placed is known, the integrated tracking unit 200 determines that the shadow of the object or other objects placed in this environment is different from the other, based on the relationship between the lighting and the object. It is determined whether or not it overlaps with the object of . Then, if there is a possibility that the shadows overlap, the integrated tracking unit 200 calculates the possibility (likelihood) that the shadows overlap with the object on which the shadows overlap, and includes it in the tracking information.

In addition, even if the sun moves, the integrated tracking unit 200 obtains the position of the sun from the information on the time and the direction of the site, predicts the direction in which a shadow will be cast, and affects the appearance of the object. You may make it consider an influence. For example, the integrated tracking unit 200 obtains the current position of the sun from the time information and, together with the direction information, predicts in which direction a shadow will form. Then, the integrated tracking unit 200 may calculate the possibility (likelihood) of overlapping shadows when there is a possibility of being shadowed by another object, and include it in the tracking information.

Next, operation of the detection unit 100 will be described. The detection unit 100 detects an object based on the tracking information, as in each embodiment described above. At this time, the detection unit 100 of the object tracking device 10 according to the present embodiment controls object detection using information about how each object appears, which is included in the tracking information. Specifically, the detection unit 100 does not detect an object that is highly likely to overlap other objects and not be visible. For example, the detection unit 100 does not set a search range for this object that is highly likely to be invisible.

Note that if the tracking information includes information that changes the brightness or color tone, the detection unit 100 may correct the effect of the lighting when performing detection. For example, in a dark area, the detection unit 100 may perform detection after correcting the pixel values of the area to be brighter.

In addition, when the hue changes, the detection unit 100 considers the change in hue when updating the matching parameters used in template matching (that is, the detection parameters described above). Information may be corrected. Further, when the change in color tone is large, the detection unit 100 may not use color information. Further, the detection unit 100 may perform matching by lowering the weight of color information among the features of the template and increasing the weight of other features such as edges.

Further, when the detecting unit 100 updates the detection parameters used in the object detection process, if it is determined that there is a high possibility that the objects overlap, the detection parameters may not be updated. .

As described above, the detection unit 100 controls object detection based on the appearance information included in the tracking information. As a result, the detection unit 100 can reduce processing for detecting invisible objects and processing for updating parameters such as template matching. Thereby, the detection unit 100 can reduce the possibility of erroneous detection and erroneous parameter update.

Similarly, when there is a high possibility that the lighting conditions have changed, the detection unit 100 may perform control so as to correct the effect and update the parameters, or control so as not to update the parameters. may

Moreover, when a plurality of detection algorithms are switchable, the object tracking device 10 may use a detector that is more resistant to overlap as the detection unit 100 . Specifically, the object tracking device 10 normally uses a detector that detects the entire head, but if there is an overlap, it uses a detector that detects only a portion of the head instead of the entire head. You may do so. This allows the object tracking device 10 to normally use a simple detector and use a more detailed detector when there is a possibility of overlap. This allows the object tracking device 10 to perform highly accurate detection while maintaining efficiency. Similarly, when the lighting conditions change, the object tracking device 10 may control detection using detectors (feature quantities) that are highly robust to the lighting conditions.

As described above, in the object tracking device 10 according to the present embodiment, the integrated tracking unit 200 uses information from other cameras to determine how the object looks. Therefore, the integrated tracking unit 200 can accurately determine how an object looks from a certain camera 20 even when it is difficult to determine with only that camera 20, such as when objects overlap each other. The integrated tracking unit 200 can then feed this result back to the detection unit 100 . Thereby, the detection unit 100 can reduce erroneous detection of objects. Since the integrated tracking unit 200 tracks the object using this detection result, it is possible to improve the tracking accuracy of the object.

<Hardware configuration example>
Here, a configuration example of hardware capable of realizing the object tracking device (10, 50) according to each of the embodiments described above will be described. The object tracking devices (10, 50) described above may be implemented as dedicated devices, or may be implemented using a computer (information processing device).

FIG. 19 is a diagram illustrating a hardware configuration of a computer (information processing device) that can implement each embodiment of the present invention.

The hardware of the information processing apparatus (computer) 700 shown in FIG. Random Access Memory) 15 , a storage device 17 , and a drive device 18 for computer-readable storage media 19 , which are connected via a bus 16 . The input/output user interface 13 is a man-machine interface such as a keyboard as an example of an input device and a display as an output device. The communication interface 12 is general communication means for the devices (FIGS. 1 and 9) according to the above-described embodiments to communicate with external devices via the communication network 600. FIG. In such a hardware configuration, the CPU 11 controls the overall operation of the information processing device 700 that implements the object tracking device (10, 50) according to each embodiment.

The present invention, which has been described with the above embodiments as examples, supplies a program (computer program) capable of implementing the processing described in each of the above embodiments to the information processing apparatus 700 shown in FIG. After that, the program is read out to the CPU 11 and executed. Note that the program may be, for example, the various processes described in the flowchart (FIG. 8) referred to in the description of the above embodiments, or the processes shown in FIGS. It may be a program capable of realizing each part (each block) shown in the apparatus in the block diagram shown.

Also, the program supplied to the information processing apparatus 700 may be stored in a readable/writable temporary memory (15) or a non-volatile storage device (17) such as a hard disk drive. That is, in the storage device 17, the program group 17A is, for example, a program capable of realizing the function of each part shown in the object tracking device (10, 50) in each embodiment described above. Various types of stored information 17B are, for example, object tracking results, camera images, camera parameters, range of coordinate values of the common coordinate system visible by each camera 20, and the like in each of the above-described embodiments. However, when the program is installed in the information processing apparatus 700, the constituent units of the individual program modules are the blocks shown in the block diagrams (FIGS. 1, 4 to 6, 9, and 15 to 17). The division is not limited, and a person skilled in the art may appropriately select the division upon implementation.

In the above case, the method of supplying the program into the device is to install it in the device via various computer-readable recording media (19) such as CD (Compact Disk)-ROM, flash memory, etc. At present, general procedures can be adopted, such as a method, a method of downloading from the outside via a communication line (600) such as the Internet, and the like. In such a case, the present invention can be considered to be constituted by the code (program group 17A) constituting such a computer program or the storage medium (19) storing such code.

The present invention has been described above as an example applied to the exemplary embodiment described above. However, the technical scope of the present invention is not limited to the scope described in each embodiment described above. It is obvious to those skilled in the art that various modifications or improvements can be made to such embodiments. In such cases, new embodiments with such changes or improvements may also be included in the technical scope of the present invention. And this is clear from the matters described in the claims.

1 object tracking system 2 object tracking system 10 object tracking device 100 detection unit 110 object detection unit 111 recognition type object detection unit 112 search range setting unit 120 common coordinate transformation unit 130 individual coordinate transformation unit 140 object detection unit 141 recognition type object detection unit 142 non-recognition type object detection unit 143 detection parameter update unit 144 detection result integration unit 150 display control unit 160 storage unit 200 integrated tracking unit 210 prediction unit 220 storage unit 230 association unit 240 update unit 300 display control unit 400 detection unit 500 integration Tracking unit 510 Buffer unit 530 Association unit 20 Camera 30 Display device 40 Network 50 Object tracking device

Claims (12)

an object detection means for detecting an object from an image obtained by the photographing means;
prediction means for predicting position information of the object in a predetermined space based on outputs from the plurality of object detection means ;
individual coordinate transformation means for transforming position information of the object predicted in the predetermined space into two-dimensional position information;
The object tracking system, wherein the object detection means detects the object from the position information of the object converted into two-dimensional position information and the image obtained from the photographing means. 2. The object tracking system according to claim 1, wherein said object detection means detects the position of said object. 3. The object tracking system according to claim 2, wherein the position of said object is represented by a unique coordinate system in said photographing means. The object tracking system according to any one of claims 1 to 3, wherein the position information of the object predicted by the prediction means is three-dimensional position information. further comprising display control means;
5. The object tracking system according to any one of claims 1 to 4, wherein said display control means transmits position information of said object within said predetermined space to a display device. 6. The object tracking system according to claim 5, wherein said display control means transmits information for displaying a flow line of said object based on past positional information of said object to said display device. The object detection means includes individual coordinate transformation means for transforming the position information of the object predicted by the prediction means into a unique coordinate system in the photographing means, and the output of the individual coordinate transformation means and the 7. An object tracking system according to any one of claims 1 to 6, wherein said object is detected from an image obtained. 8. The object tracking system according to any one of claims 1 to 7, wherein said prediction means predicts the position information of said object based on an image captured prior to the image obtained by said imaging means. an object detection means for detecting an object from an image obtained by the photographing means;
prediction means for predicting position information of the object in a predetermined space based on outputs from the plurality of object detection means ;
individual coordinate transformation means for transforming position information of the object predicted in the predetermined space into two-dimensional position information;
The object tracking device, wherein the object detection means detects the object from the position information of the object converted into two-dimensional position information and the image obtained from the photographing means. Detecting an object from an image obtained by a photographing means,
Predicting position information of the object in a predetermined space based on outputs from a plurality of the detections;
converting position information of the object predicted in the predetermined space into two-dimensional position information;
The object tracking method, wherein in the detection, the object is detected from the position information of the object converted into two-dimensional position information and the image obtained from the photographing means. Obtaining first coordinates of the object based on a first image captured by a first camera and a first predicted position of the object;
Obtaining second coordinates of the object based on a second image captured by a second camera and the first predicted position;
determining a second predicted position of the object within a predetermined space based on the first coordinates and the second coordinates;
A process of converting the second predicted position into two-dimensional position information,
The first coordinates are represented by two-dimensional coordinates on the image captured by the first camera,
The second coordinates are represented by two-dimensional coordinates on the image captured by the second camera,
the predetermined space is a three-dimensional space;
The second predicted position is represented by three-dimensional position information,
Object tracking method. The two-dimensional position information based on the second predicted position is used when obtaining the first coordinates of the object from an image acquired at a time later than the first image. used as
The object tracking method according to claim 11.

Images