JP2005346425A - Automatic tracking system and automatic tracking method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make a stable tracking possible without the influence of similar patterns in a background similar with a template image when an object such as an invader in an image is tracked. <P>SOLUTION: A template matching part 102 searches an area having a high correlation with a preliminarily registered template image from input images inputted from an image input part 101 and obtains a correlation map in which correlation values of positions in the searched area are two-dimensionally arranged. An object area extraction part 104 cancels the motion of a background by using a background vector predicted by a background vector prediction part 103 and extracts an object area based on a difference between frames. A template position correction part 105 generates a movement map indicating an object-movement occurrence density in the searched area, predicts the movement destination of the template by using the correlation map and the movement map, and corrects the position of the template so that the template may not deviate from the object area. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an automatic tracking device and an automatic tracking method for tracking a specific object in a captured image in a monitoring system for an intruding object or the like.

  Remote monitoring camera systems have become widespread, and it has become possible to monitor an image taken at a specific point in a remote place and track an intruder or the like. Cameras used in surveillance camera systems are generally equipped with a wide-angle lens as a photographic lens to monitor a wide area, and although it is possible to determine the presence or absence of an intruding object, details such as the intruder's human phase and facial expression, etc. Has a drawback that cannot be grasped.

  Therefore, in addition to cameras equipped with wide-angle lenses, cameras equipped with pan, tilt, and zoom (hereinafter collectively referred to as PTZ) functions are installed, and the details of the intruder's human phase, etc. are controlled by the PTZ control of the observer. There is a monitoring system that can acquire a simple image (Non-Patent Document 1). However, manual operation of a camera having a PTZ function using an operation device such as a joystick to keep track of an intruder requires advanced technology, and many observers feel difficulty. Therefore, an automatic tracking device that can automatically track a tracking object such as an intruder in a captured image using an image recognition technique is desired.

  As an apparatus capable of such automatic tracking, an object tracking method and an object tracking apparatus using a template matching method have been proposed (Patent Document 1). Template matching is a method in which a partial image obtained by cutting out an image area of a tracking object at a certain time is used as a template image, and an area having the highest correlation with the template image is obtained by correlation calculation or the like in images at different times, and the tracking object is tracked. is there.

  FIG. 6 is a flowchart showing the operation at the time of object tracking in the conventional example of Patent Document 1. First, object detection processing is performed on an input image at a certain time t1 (step 901), and an image of the detected object region is registered as an initial template image (step 902). Next, an input image at a new time t2 is input (step 903), template matching is performed to detect a region having the highest correlation with the template image from the input image, and the object is tracked (step 904).

  Then, it is determined whether template matching processing has been performed for all stored template images (step 905). If all template matching processing has been completed, maximum matching determination processing is performed (step 906). ). If all the template matching processes are not completed, the template matching process in step 904 is repeated until the template matching process is completed.

  In step 906, if the maximum matching degree is equal to or smaller than the predetermined value, it is determined that there is no tracking object in the input image, and the process returns to step 901 to perform object detection processing again. On the other hand, when the maximum matching degree is a predetermined value or more, the position of the template image is corrected according to the result of template matching (step 907), and the corrected plurality of template images are stored (step 908). The plurality of template images are template images related to the same object although the acquisition time or update time is different and the appearance is slightly different. Subsequently, the camera platform is controlled so that the optical axis direction of the camera faces the tracking object (step 909). Then, for example, when the tracked object is present within a predetermined alarm range, an alarm is sounded or an image of the tracked object is displayed on the monitoring monitor (step 910).

JP 2001-60269 A Ushita, Fujiwara, Tadami, “Swinging, Tracking Intrusion Monitoring System Using Zoom Camera”, IEICE Technical Report PRMU 99-67 (P23-30), September 1999

  However, in the conventional automatic tracking method as described above, if a region having a high correlation with the template image exists in the background, the template image is likely to be erroneously attached to the background, resulting in failure to track the object. There was a problem that it was easy. In addition, when a pan / tilt operation is performed by moving the camera platform, and a captured image whose pan / tilt angle is being controlled is input, movement occurs not only in the object but also in the background, so only the object region is extracted using the movement. There is a problem that the destination of the template image cannot be corrected using the object region.

  The present invention has been made in view of the above circumstances, and when tracking an object such as an intruder in an image, stable tracking is possible without being affected by a similar pattern in the background similar to the template image. It is an object to provide an automatic tracking device and an automatic tracking method.

  An automatic tracking device according to the present invention is an automatic tracking device that inputs an image captured by an imaging unit and tracks a target object in the image, and correlates a template image indicating the characteristics of the target object with an input image. Template matching means for performing calculation, background vector estimation means for estimating a motion vector of the background indicating the motion of the entire image of the input image, canceling the motion of the entire image using the motion vector of the background, and the image Object region extraction means for extracting a motion different from the overall motion as an object region, and template position correction means for estimating the destination of the template image based on the extraction result of the object region and correcting the position of the template image Are provided.

  As a result, for example, even if the image is moving in the background due to the shooting range control by panning, tilting, zooming, etc., the object area that moves differently from the background is extracted, and the template image is moved. The position of the tip can be corrected near the object region. Therefore, the object can be tracked more stably without being affected by the similar pattern in the background similar to the template image.

Further, as one aspect of the present invention, in the automatic tracking device described above, based on a position correction result of the template image, shooting that controls a shooting range of the imaging unit so that the target object is positioned in the input image Range control means shall be provided.
Thus, by controlling the shooting range of the image pickup unit in accordance with the position correction of the template image, panning, tilting, zooming operation, and the like can be controlled so that the tracking target object does not fall out of the image.

Also, as one aspect of the present invention, in the automatic tracking device described above, the template position correction unit generates a motion map indicating the density of pixels that have become the object region around a target pixel, and the motion map It is assumed that the destination of the template image is estimated using.
This makes it possible to accurately estimate the destination of the template image using the motion map.

Further, as one aspect of the present invention, in the above-described automatic tracking device, the template position correction unit cancels background motion by an amount corresponding to a time difference in imaging time based on the background motion vector, By logically calculating the object area extraction results obtained from multiple frames of input images with different imaging times, the object area that occurs only in the current frame is distinguished from the object area that occurs elsewhere. In addition, the motion map is generated.
As a result, when generating a motion map, for example, an object generated only in the latest frame is distinguished by adding a weight by distinguishing between an object area occurring only in the current frame and an object area occurring elsewhere. By increasing the weight for the region, it is possible to more accurately estimate the destination of the template image.

  The automatic tracking method of the present invention is an automatic tracking method for inputting an image taken by an imaging means and tracking a target object in the image, and correlating a template image indicating the characteristics of the target object with an input image. A template matching step for performing an operation; a background vector estimation step for estimating a motion vector of a background indicating the motion of the entire image of the input image; and canceling the motion of the entire image using the motion vector of the background; An object region extraction step for extracting a motion different from the overall motion as an object region; and a template position correction step for estimating the destination of the template image based on the extraction result of the object region and correcting the position of the template image It has.

  As a result, for example, even if the image is moving in the background due to the shooting range control by panning, tilting, zooming, etc., the object area that moves differently from the background is extracted, and the template image is moved. The position of the tip can be corrected near the object region. Therefore, the object can be tracked more stably without being affected by the similar pattern in the background similar to the template image.

  The present invention also provides a program for causing a computer to execute each procedure of the automatic tracking method. With this program, the automatic tracking method according to the present invention can be implemented by a computer. This program can also be stored in a recording medium such as a memory, a hard disk, or a CD-ROM.

  According to the present invention, when tracking an object such as an intruder in an image, an automatic tracking device and an automatic tracking method capable of stable tracking without being affected by a similar pattern in the background similar to the template image. Can provide.

  Hereinafter, as an embodiment of the present invention, in a remote monitoring system for monitoring an intruding object or the like, an object in a captured image is tracked using a template, and panning, tilting, and zooming operations (PTZ operation) of the camera are controlled in detail. An example of an automatic tracking device such as an automatic tracking camera for monitoring an intruding object and an automatic tracking method will be described.

FIG. 1 is a diagram showing a functional configuration of an automatic tracking device according to an embodiment of the present invention.
The automatic tracking device 100 has a correlation between an image input unit 101 that inputs an image captured by an imaging device 108 (corresponding to an example of an imaging unit) 108 such as a camera, and a template image registered in advance from the input image. A template matching unit (corresponding to an example of a template matching unit) 102 that searches for a high area, a background vector estimation unit (corresponding to an example of a background vector estimation unit) 103 that estimates a motion vector of a background other than an object, and a background An object region extracting unit (corresponding to an example of an object region extracting means) 104 that cancels the motion of the entire image using the motion vector of the image and extracts an object region different from the motion of the entire screen due to the difference between frames, and a template Template position correction unit that corrects the position so that the image does not deviate from the object area (template position correction 105 (corresponding to an example of a stage), an imaging device control unit (corresponding to an example of imaging range control means) 106 that causes the imaging device to perform a PTZ operation so that the tracking object is located at the center of the image, and an image The image output unit 107 outputs to the image display device 109.

  In the present embodiment, a region having a high correlation with a pre-registered template image is searched from input images photographed by the imaging device 108 and input from the image input unit 101, and the correlation value of each position in the search region. Is obtained as a correlation map. Then, the object region extraction unit 104 cancels the background motion using the background vector estimated by the background vector estimation unit 103, and extracts the object region based on the inter-frame difference. Then, the template position correcting unit 105 corrects the position using the object region and the correlation map so that the template does not deviate from the object region.

  Next, the operation of the main part of the automatic tracking device 100 will be described. In this embodiment, a rectangular area including pixels of an object area to be tracked from an input image (an image taken by the imaging device 108) is set and registered as a template image in advance.

  FIG. 2 is a diagram for explaining the operation of the template matching unit 102. The template matching unit 102 obtains a correlation between a new input image input from the image input unit 101 and a registered template image, and searches for a highly correlated area. In FIG. 2, the template image of the immediately preceding frame set by the template setting unit 102 as shown in FIG. 2A is set as a template image T, and the newly input input image as shown in FIG. To do. The template matching unit 102 searches the corresponding area 201 having the highest correlation with the template image T from the input image I. Here, the search for the corresponding region 201 can be repeated every tens to hundreds of milliseconds to track the object position. When there are a plurality of template images T, the search is performed using all target template images.

  At this time, a search range 202 of about several tens of pixels in the horizontal and vertical directions is set in the latest input image I with reference to the region having the highest correlation in the processing of the immediately preceding frame. When obtaining the correlation, the correlation with the template image T is sequentially obtained while scanning the search range 202. Although it is conceivable that the search range 202 is the entire input image, the search range is generally limited because the amount of calculation is too large or miscorrespondence is likely to occur. The width of the search range 202 is about several to several tens of pixels in the horizontal and vertical directions, depending on the movement of the object.

  An example in which normalized correlation is used as a method for calculating a correlation value in an image will be described. The size of the template image T is set to K (H) * L (V) pixels, and the template image T (p, q) is moved so that the rectangular area K (H) in the current frame image I (x, y) is moved. * L (V) pixels are assumed. The normalized correlation value NRML (xv, yv) between the template image T (p, q) and the rectangular area K (H) * L (V) pixel in the current frame image I (x, y) is expressed by the following formula ( It can be expressed as 1).

  The normalized correlation value NRML is closer to 1.0 as the correlation between both images is higher, and is 0 when there is no correlation. When an image having exactly the same pixel value as that of the template image T exists in the search range 202, the normalized correlation value NRML is 1.0. A position on the current frame having the highest correlation is defined as a corresponding area 201. The movement amount of the corresponding area 201 from the position of the template image extracted in the immediately preceding frame is (xv, yv), which indicates the motion vector of the template.

  In the above example, the method of obtaining a correlation value between a new input image and a template image by normalized correlation has been described. However, as a correlation value calculation method, a difference method or a difference absolute value sum method with a smaller amount of calculation is used. This method can also be used. Further, although an example using a luminance value has been described, it is also conceivable to use a color, an edge, or the like depending on the case in the correlation value calculation.

  In the template matching unit 102, a correlation map in which the correlation values of the respective positions in the search area by the search range 202 are arranged in a two-dimensional array is generated. This correlation map is used in the template position correction unit 105 described later. At this stage, the template destination has not been determined.

  As a setting method of the template image, the operator may manually set and operate the object region while looking at the input image, or the region where the luminance change has occurred in the input image is regarded as the object region, An area including pixels assigned with the same label after the labeling process may be automatically set as a template image. Alternatively, when an image region is recognized as an object to be tracked using a pattern recognition technique, this object region may be set as a template.

  To create an initial template at the start of tracking, the operator may specify the object to be tracked with a pointing device such as a mouse, or recognize an area where there is a high probability that a desired object exists in the input image. It may be created after extraction, or a template relating to an object to be tracked may be stored in advance. As described above, any method may be used for creating the template.

  The size of the template is generally set to a size including the entire object region, but a part of the object region such as the periphery of the head may be set as a template image. The template image is cut out from the input image and is a part of the input image. However, the template image may be used without any particular processing, or may be used as a template image by performing edge extraction or various filtering processes. .

  It is assumed that a plurality of template images are held for the same object with different acquisition times and update times, but a single template image may be used. When the correlation between the template image and the input image becomes low, the maximum number of templates that can be stored is added. When the maximum number is reached, delete some templates and add new ones. Examples of the template deletion rule include deletion from an old one, deletion from a lower correlation, and the like.

  The background vector estimation unit 103 estimates a background motion vector from images with different imaging times. When a camera capable of pan / tilt operation is used in the imaging device 108, movement occurs in the background other than the tracking object during camera control. In order to extract only the object region, first the background motion is estimated.

  FIG. 3 is a diagram illustrating an example of each process of background motion vector estimation, object region extraction, and template position correction. FIG. 3A illustrates the previous frame, previous frame, and current frame in the operation of the background vector estimation unit. FIG. 3B is a diagram illustrating background vector estimation, FIG. 3B is a diagram illustrating moving object region extraction in the operation of the object region extraction unit, and FIG. 3C is a diagram illustrating a binarized image that distinguishes the object region from the background region. FIG. 3D is a diagram showing an object region generated only in the current frame obtained by logical operation in the template position correction unit and an object region generated in either the current frame or the previous frame.

  As shown in FIG. 3A, the background vector estimation unit 103 estimates the background vector (bgx1, bgy1) from the previous frame and the previous frame and the background vector (bgx2, bgy2) from the current frame and the previous frame. ). As shown in FIG. 3A, the background vector estimation method divides the current frame into four regions, and estimates a motion vector that is a motion amount from the previous frame for each region. A representative motion vector is obtained by taking an intermediate value for each horizontal and vertical direction from the motion vectors obtained in each region, and this is used as a background vector (bgx, bgy) indicating the motion of the entire screen. As a background vector estimation method, a generally known method such as a block matching method may be used. A more detailed explanation here is omitted.

  Here, an example has been described in which the current frame is divided into four regions and the motion vector that is the amount of motion from the previous frame is estimated for each region, but the background vector is estimated from the entire image without dividing the image. Alternatively, it may be estimated by dividing into four regions.

  The object region extraction unit 104 extracts a moving object region in the image based on a difference between frames of the current frame and the previous frame having different imaging times, using the background vector obtained by the background vector estimation unit 103.

  There is a possibility that the entire screen of the video acquired by the imaging device 108 performing the pan / tilt operation and controlling the pan / tilt angle is moving. In such a case, the simple inter-frame difference method may extract the entire screen as a movement has occurred.

  Therefore, in the present embodiment, as shown in FIG. 3B, the background frame (bgx, bgy) estimated by the background vector estimation unit 103 is used to translate the previous frame to cancel the background motion. The difference between the current frame and the interframe difference SA (x, y) is obtained using equation (2). As a result, when there is an object that moves differently from the background in the screen, only the object part that moves can be extracted as the object region.

SA (x, y) = | I _t (x, y) −I _t−1 (x + bgx, y + bgy) | (2)

  By binarizing the difference result between frames acquired by the above equation (2) with a threshold value, a binarized image in which the object region and the background region shown in FIG. 3C are distinguished is obtained.

  In the above description, an example in which an object region is extracted based on a difference between frames has been described. However, a change in some feature between frames may be detected to distinguish between the object region and the background region.

  The template position correction unit 105 uses the correlation map acquired by the template matching unit 102 and the object region extracted by the object region extraction unit 104 to correct the template position so that the template does not deviate from the object region.

  When the template position correction unit 105 corrects the position of the template, first, among the inter-frame difference results obtained by Expression (2), the same region as the search region to be searched for using the template by the template matching unit 102 for each pixel. The pixels that are the surrounding object regions are counted. That is, for each pixel in the search area, the density of the pixels that have become object areas in the vicinity is obtained, and a two-dimensional array of the density at each pixel position is generated as a motion map.

  Further, by performing a logical operation using a binarized image for two processing cycles as shown in FIG. 3C, an object region generated only in the current frame as shown in FIG. The object area generated in either the frame or the previous frame can be distinguished. Since there is a movement of the screen corresponding to the background vector (bgx2, bgy2) between the binarized images for two processing cycles shown in FIG. 3C, a logical operation is performed after canceling this by parallel movement. . At this time, it is determined that the object area that occurs only in the latest inter-frame difference is the object area of the current frame, and the object area that occurs in any frame is the object area that occurs in both. To do.

  When creating a motion map, in the case of an object region that occurs only in the current frame, weighting is performed by calculating the density, assuming that the weight is greater than the object region occurring in both the current frame and the previous frame. Also good.

  FIG. 4 is a diagram illustrating an example of processing when an integrated map is created using a correlation map and a motion map, FIG. 4A is a diagram illustrating a correlation map, and FIG. 4B is a diagram illustrating a motion map. FIG. 4C shows an integrated map.

  In the present embodiment, the correlation map as shown in FIG. 4A and the motion map as shown in FIG. 4B are added after performing level correction so that the respective dependencies are appropriate, and FIG. Generate an integrated map like In order to enhance the effect of template movement destination correction by movement, the movement map is integrated with the correlation map at a relatively high level. In the integrated map, a position where movement occurs and the correlation with the template is high appears as a peak. Each map shows that the smaller the value is, the higher the possibility that it is the destination of the template. The position corresponding to the peak in the integrated map is determined as the template destination.

  As described above, using the correlation map indicating the correlation with the template and the motion map indicating the motion occurrence density of the object, the level correction is performed so that the degree of dependence of each map becomes appropriate, and the integrated map is added. And the destination of the template is estimated using this integrated map. Thereby, the movement destination of the template can be corrected so that the template does not deviate from the object region.

  Next, the overall operation of the automatic tracking apparatus of the present embodiment configured as described above will be described with reference to FIG. FIG. 5 is a flowchart showing an operation procedure of the automatic tracking device according to the present embodiment.

  First, the image input unit 101 inputs an image captured by the imaging device 108 (step 501). Then, the template matching unit 102 compares the input image with a pre-registered template image to calculate a correlation value, and selects a region having a high correlation with the pre-registered template image from the new input images. Search (step 502). A correlation map is generated by this template matching process.

  Next, the background vector estimation unit 103 estimates a background motion vector (step 503). Then, the object region extraction unit 104 extracts a moving object region in the image based on the difference between the current frame and the previous frame having different imaging times using the background vector obtained in step 503 (step 504).

  Subsequently, the template position correction unit 105 corrects the position of the template so that the template does not deviate from the object region, using the correlation map acquired by the template matching unit 102 and the object region extracted by the object region extraction unit 104. (Step 505).

  Thereafter, the imaging device control unit 106 controls the pan, tilt, and zoom of the imaging device 108 based on the position correction processing result in the template position correction unit 105, and images the tracking object so that it is positioned at the center of the image. The apparatus 108 is PTZ-operated (step 506), and the monitoring area including the tracking object is imaged by the imaging apparatus 108 (step 507). Then, the image output unit 107 outputs the image captured by the imaging device 108 to the external image display device 109 (step 508). Thereafter, the process returns to the new image input process of step 501 and the same process is repeated.

  As described above, in the automatic tracking device according to the present embodiment, the movement destination of the template is estimated using the correlation map generated by the template matching process and the motion map indicating the motion occurrence density of the object, and the position of the template is determined. to correct. In this way, instead of searching only for image areas that have a high correlation with the template, by finding a moving object area that moves differently from the background due to interframe differences, the correlation with the template is high and movement occurs. The area to be moved can be extracted, and the template movement destination can be corrected in this area. As a result, it is possible to eliminate tracking errors caused by incorrect template correspondence and pasting to similar patterns in the background, and it becomes possible to track an object more stably.

  The present invention has an effect of enabling stable tracking without being affected by a similar pattern in the background similar to the template image when tracking an object such as an intruder in the image. This is useful for an automatic tracking device, an automatic tracking method, and the like that track a specific object in a captured image.

The figure which shows the function structure of the automatic tracking apparatus which concerns on embodiment of this invention. The figure explaining operation | movement of the template matching part which concerns on embodiment of this invention. The figure which shows an example of each process of the motion vector estimation of a background, object area extraction, and template position correction in this embodiment The figure which shows an example of the process at the time of creating an integrated map by the correlation map and motion map in this embodiment The flowchart which shows the operation | movement procedure of the automatic tracking apparatus in this embodiment. Flow chart showing the operation at the time of object tracking in the conventional example

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Automatic tracking apparatus 101 Image input part 102 Template matching part 103 Background vector estimation part 104 Object area extraction part 105 Template position correction part 106 Imaging apparatus control part 107 Image output part 108 Imaging apparatus 109 Image display apparatus

Claims (6)

An automatic tracking device that inputs an image captured by an imaging means and tracks a target object in the image,
Template matching means for performing a correlation operation between a template image indicating the characteristics of the target object and an input image;
Background vector estimation means for estimating a motion vector of a background indicating the motion of the entire image of the input image;
Object region extraction means for canceling the movement of the entire image using the motion vector of the background and extracting a movement different from the movement of the entire image as an object region;
An automatic tracking device comprising: template position correcting means for estimating a destination of the template image based on the extraction result of the object region and correcting the position of the template image. The automatic tracking device according to claim 1,
An automatic tracking apparatus comprising: an imaging range control unit that controls an imaging range of the imaging unit so that the target object is positioned in an input image based on a position correction result of the template image. The automatic tracking device according to claim 1 or 2,
The template position correcting means generates a motion map indicating the density of pixels that have become the object region around a pixel of interest, and estimates the destination of the template image using the motion map. Tracking device. An automatic tracking device according to claim 3,
The template position correcting means cancels the background motion corresponding to the time difference of the imaging time based on the background motion vector, and then extracts the object region extraction result obtained from the input images of a plurality of frames having different imaging times. An automatic tracking device that generates a motion map by adding a weight by distinguishing between an object region that occurs only in the current frame and an object region that occurs elsewhere by performing a logical operation. An automatic tracking method for inputting an image taken by an imaging means and tracking a target object in the image,
A template matching step for performing a correlation operation between a template image indicating the characteristics of the target object and an input image;
A background vector estimation step for estimating a motion vector of a background indicating the movement of the entire image of the input image;
An object region extraction step of canceling the movement of the entire image using the motion vector of the background and extracting a movement different from the movement of the entire image as an object region;
A template position correcting step of estimating a destination of the template image based on the extraction result of the object region and correcting the position of the template image.   The program for making a computer perform each procedure of the automatic tracking method of Claim 5.

Images