JP5116605B2 - Automatic tracking device - Google Patents

Description

  The present invention relates to an automatic tracking device that automatically tracks and images a tracking target using a camera capable of controlling pan, tilt, and zoom.

Conventionally, as such an automatic tracking device, for example, an automatic tracking device disclosed in Patent Document 1 below has been proposed. The automatic tracking device includes an image input / output unit that inputs a video signal from a television camera and outputs the video signal to a display unit, an image processing unit that performs preprocessing of the video signal from the image input / output unit, and an image A correlation calculation unit that obtains a correlation value between the reference image data and the search image data input via the processing unit and detects a position having the highest correlation on the search image, and the television camera based on a signal from the correlation calculation unit And a central processing unit for controlling the above-mentioned units as well as driving and controlling the swivel device and the zoom lens.
JP-A-11-187387

  However, in the conventional automatic tracking device, since the position of the tracking target is obtained using the correlation calculation, it is weak against changes in the size and shape of the tracking target, the influence of occlusion (concealment) and the influence of image fluctuation due to camera control. Therefore, the tracking target cannot be tracked with high accuracy.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an automatic tracking device that can track and image a tracking target with higher accuracy.

  The following aspects are presented as means for solving the problems. An automatic tracking device according to a first aspect includes a camera capable of controlling pan, tilt, and zoom, tracking processing means for performing tracking processing for tracking a tracking target based on an image captured by the camera, and the tracking Control means for controlling pan, tilt and zoom of the camera so that the camera tracks and tracks the tracking target according to the result of the tracking processing by the processing means. The tracking processing means includes position estimating means for estimating the position of the tracking target as a part of the tracking result by a particle filter using a plurality of particles whose pixel positions are based on an image captured by the camera. Including. The particle filter, for each particle, is a response of an AdaBoost discriminator constructed so as to identify whether the pixel is a tracking target pixel or a background pixel based on one or more feature quantities related to the pixel. A likelihood calculated from a response value based on the one or more feature amounts related to the pixel at the position of the particle.

  According to the first aspect, since the position of the tracking target is estimated by the particle filter, unlike the correlation calculation, since there are a plurality of solution candidates (a plurality of particles), the possibility of recovery from a tracking failure is high. Thus, it is strong against occlusion and complex backgrounds, and can perform tracking processing with higher accuracy. As a result, the tracking target can be tracked with higher accuracy and imaged. A deterministic method (for example, template matching) cannot recover from a tracking failure because the solution is uniquely determined. According to the first aspect, the tracking target pixel and the background pixel are identified by the AdaBoost classifier, and the particle filter uses the likelihood calculated from the response value of the AdaBoost classifier. It is less affected by background changes such as background shape, size, color, lightness and darkness, etc. From this point, it is possible to perform tracking processing with even higher accuracy, and in turn, the tracking target is more accurately controlled. It can be tracked and imaged.

  In the automatic tracking device according to a second aspect, in the first aspect, the position estimation unit uses the likelihood of the position of the tracking target as a pixel position that is the state of the plurality of particles using the likelihood. It is estimated as a value. The second mode is an example of a specific method for estimating the position of the tracking target.

  In the automatic tracking device according to the third aspect, in the first or second aspect, the tracking processing unit responds to a pixel identified as a tracking target pixel by the AdaBoost classifier among the plurality of particles. Based on the particles, a size estimation means for estimating the size of the tracking target as another part of the tracking result is included.

  According to the third aspect, since the size of the tracking target is estimated based on the particles corresponding to the pixel identified as the tracking target pixel by the AdaBoost discriminator, the size of the tracking target is also occluded. It is strong against a complicated background or the like, and can perform a tracking process with higher accuracy. As a result, the tracking target can be tracked with higher accuracy and imaged.

  The automatic tracking device according to a fourth aspect is the automatic tracking device according to any one of the first to third aspects, wherein the tracking processing unit is a pixel identified as a tracking target pixel by the AdaBoost classifier among the plurality of particles. Distribution state of particles having a Mahalanobis distance equal to or less than a predetermined value using a covariance matrix weighted by the likelihood of the particle with respect to the tracking target position estimated by the position estimation unit The size estimation means for estimating the size of the tracking target is included. The fourth mode is an example of a specific method for estimating the size of the tracking target.

The automatic tracking device according to a fifth aspect is the automatic tracking device according to any one of the first to fourth aspects, wherein the one or more feature amounts are (i) a first predetermined color space of a pixel in a local region including the pixel. Thru | or the average value of said 1st value among 3rd values, (ii) The dispersion value of said 1st value of the pixel of the local region containing the said pixel, (iii) The pixel of the local region containing the said pixel An average value of the second values, (iv) a variance value of the second values of the pixels in the local area including the pixel, and (v) an average value of the third values of the pixels in the local area including the pixel. (Vi) a variance value of the third value of the pixel in the local area including the pixel; (vii) an edge direction histogram in the local area including the pixel; and (viii) a local binary in the local area including the pixel. Including at least one of the histograms of the pattern. The predetermined color space may be a CIE1976L ^* u ^* v ^* color space, but is not necessarily limited thereto.

  In the fifth aspect, a specific example of the feature amount is given. For example, although only (i) to (vi) may be adopted as the one or more feature quantities, compared to that case, only (i) to (vii) are adopted, or (i) The cases where only (vi) and (viii) are adopted and the cases where (i) to (viii) are adopted are preferable because the tracking performance is enhanced. In particular, it has been experimentally confirmed that, in those cases, when (i) to (viii) are adopted as the one or more feature quantities, the tracking performance is most enhanced.

  The automatic tracking device according to a sixth aspect is the automatic tracking device according to any one of the first to fifth aspects, wherein a predetermined particle of the plurality of particles is used as a learning sample for a tracking target pixel, Update means for updating the AdaBoost discriminator is provided using other predetermined particles as learning samples for background pixels.

  According to the sixth aspect, since the AdaBoost discriminator is updated, it is possible to cope with a change in the appearance of the tracking target and a change in the environment (such as lighting and sunshine conditions), and thereby the tracking by the AdaBoost discriminator. The accuracy of identifying whether the pixel is the target pixel or the background pixel is increased. Therefore, according to the sixth aspect, it is possible to perform the tracking process with higher accuracy, and as a result, it is possible to track and image the tracking target with higher accuracy.

  In the automatic tracking device according to a seventh aspect, in any one of the first to sixth aspects, a calculation unit that calculates a spatially weighted average value of the likelihoods of the plurality of particles, and the spatial weighting Determining means for determining whether or not the average value is equal to or greater than a predetermined value.

  The spatial weighted average value may indicate the reliability of the tracking result. If the spatial weighted average value is large, the reliability of the tracking result is high, while the reliability of the tracking result is low. According to the seventh aspect, since the spatial weighted average value is calculated and it is determined whether or not the value is equal to or greater than a predetermined value, it is determined whether the tracking has succeeded or failed in the end. Can be determined. Therefore, by using the determination result, it is possible to avoid a situation in which the control of the camera is continued based on the erroneous tracking result although the tracking has failed.

  The automatic tracking device according to an eighth aspect is the automatic tracking device according to any one of the first to seventh aspects, wherein the control means is a tracking target after a predetermined time has elapsed from the present based on the result of the tracking process by the tracking processing means. Predicting means for predicting the position and size of the camera, and the control means corrects the current pan, tilt and zoom control states for the camera according to the prediction result by the predicting means, It controls tilt and zoom.

  According to the eighth aspect, since predictive control is introduced, for example, the operation time until the camera responds to the control command and enters the command state is longer than the image processing time. However, it is possible to cope with a sudden change in the tracking target, and the tracking target can be tracked with higher accuracy and imaged. Note that if the camera pan, tilt, and zoom control speeds are too fast, the camera pan, tilt, and zoom will change too rapidly when the observer follows the tracking target. Although it may be uncomfortable and unsuitable for monitoring, a camera with a relatively slow control speed can be used, so the camera's pan, tilt and zoom changes are smoothed for tracking that is more suitable for monitoring. Can be realized.

  In the automatic tracking device according to a ninth aspect, in the eighth aspect, the prediction means predicts the position and size of the tracking target after a predetermined time has elapsed from the present time, using a Kalman filter. In the ninth aspect, since the Kalman filter is used, the position and size of the tracking target can be predicted with high accuracy, and as a result, the tracking target can be tracked with higher accuracy and imaged. But in the said 8th aspect, a prediction means is not limited to the thing using a Kalman filter.

  ADVANTAGE OF THE INVENTION According to this invention, the automatic tracking apparatus which can track and image a tracking object more accurately can be provided.

  Hereinafter, an automatic tracking device according to the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically showing an automatic tracking device according to an embodiment of the present invention. As shown in FIG. 1, the automatic tracking device according to the present embodiment includes a camera 1 that can control pan, tilt, and zoom, a processing unit 2, a distributor 3, a display / recording control unit 4, and a liquid crystal display. A display unit 5 such as a panel and a recording unit 6 are provided.

  The camera 1 includes a camera body 1 a, a zoom lens 1 b that is mounted on the camera body 1 a and sets a magnification according to a control signal that controls zoom from the processing unit 2, and a pan from the processing unit 2. And a turntable 1c for setting pan and tilt of the camera body 1a according to a control signal for controlling the tilt.

  The distributor 3 distributes and supplies the image signal from the camera 1 to the processing unit 2 and the display / recording control unit 4. The image processing unit 2 processes an image captured by the camera 1 based on the image signal from the camera supplied via the distributor 3, and the camera 1 tracks the tracking target 10 (such as an intruder or an intruding object). The pan, tilt, and zoom of the camera 1 are controlled so as to automatically track and capture an image (see FIG. 2 described later). The display / recording control unit 4 displays the image indicated by the image signal from the camera supplied via the distributor 3 on the display unit 5 or records the image on the recording unit 6. The monitor can monitor the image displayed on the display unit 5. Note that when the monitor does not monitor the image, the image signal from the camera 1 may be directly input to the processing unit 2 without providing the distributor 3.

  FIG. 2 is a diagram schematically illustrating an example of the tracking state of the tracking target 10 by the camera 1. FIG. 2 shows a state in which the visual field of the camera 1 is changed by tracking the tracking target 10 such as an intruder. Actually, as the tracking target 10 moves, the pan and tilt of the turntable 1c change to change the direction of the field of view of the camera 1, and the zoom lens 1b operates to enlarge the field of view of the camera 1. Although reduced, in FIG. 2, illustration of each part of the camera 1 is omitted, and only the field of view of the camera 1 is schematically shown.

  Next, an example of the operation of the processing unit 2 of the automatic tracking device according to the present embodiment will be described with reference to FIGS. FIG. 3 is a schematic flowchart illustrating an example of the operation of the processing unit 2. FIG. 4 is a flowchart showing in detail the tracking target detection process (step S2) in FIG. FIG. 8 is a flowchart showing in detail the tracking process (step S5) in FIG. FIG. 11 is a flowchart showing in detail the update process (step S218) of the discriminator in FIG. 12 to 15 are flowcharts showing in detail the camera control process (step 7) in FIG. 5 to 7, 9 and 10 will be described later.

  As shown in FIG. 3, when the processing unit 2 starts operation, first, the processing unit 2 sets the camera 1 in a preset state (step S1). That is, the processing unit 2 changes the pan, tilt, and zoom of the camera 1 to predetermined pan, tilt, and zoom.

  Next, the processing unit 2 performs a tracking target detection process for detecting a tracking target (moving object detection) in a preset state, that is, in a state where the pan, tilt, and zoom of the camera 1 are fixed (step S2). In addition to applying a commonly used method (S101 to S112 in FIG. 4), this detection may be performed by another sensing device such as a laser radar, or a person displayed on the screen may be detected. The monitoring may be performed by specifying with a pointing device such as a mouse as a specifying means.

  Here, an example of the tracking target detection process (step S2) will be described with reference to FIG. Note that the tracking target detection process (step S2) in FIG. 3 is not limited to the example shown in FIG.

  When the tracking target detection process (step S2) is started, as illustrated in FIG. 4, the processing unit 2 first samples two consecutive images captured by the camera 1 (steps S101 and S102), and these images. The difference image (inter-frame difference image) is generated (step S103).

  Next, the processing unit 2 binarizes the difference image generated in step S103 (step S104). The threshold value used for the binarization may be a fixed threshold value or a variable threshold value represented by discriminant analysis method.

  Subsequently, the processing unit 2 labels the image binarized in step S104 (step S105). Then, the processing unit 2 determines whether there is a labeled item (step S106). If there is no labeled item, the processing unit 2 proceeds to step S112. If there is a labeled item, the processing unit 2 proceeds to step S107. To do.

  In step S107, the processing unit 2 acquires feature amounts for all the labeled items (steps S107 and S108). The feature amount referred to here is necessary for accurately detecting the tracking target 10 such as an area or a circularity.

  Thereafter, the processing unit 2 determines whether there is a candidate for the tracking target 10 from the feature values of all the labels acquired in step S107 (step S109). If it does not exist, the process proceeds to step S112. If it exists, the process proceeds to step S110.

  In step S <b> 110, the processing unit 2 determines the tracking target 10 from the tracking target 10 candidates. At this time, if there is one candidate for the tracking target 10, it is determined as the tracking target 10, and if there are a plurality of candidates for the tracking target 10, it is narrowed down to one according to a predetermined judgment criterion, and is selected as the tracking target 10. Determine as.

  After step S111, the processing unit 2 sets a tracking target detection flag indicating whether the tracking target 10 has been detected to 1 (1 indicates that the tracking target 10 has been detected) (step S111). The tracking target detection process (step S2) is terminated, and the process proceeds to step S3 in FIG.

  In step S112, the processing unit 2 sets the tracking target detection flag to 0 (0 indicates that the tracking target 10 has not been detected). Thereafter, the tracking target detection process (step S2) is terminated, and the process proceeds to step S3 in FIG.

  Referring to FIG. 3 again, in step S3, the processing unit 2 determines whether or not the tracking target 10 is detected in step S2. This determination is made depending on whether the tracking target detection flag is 1 or 0. When the tracking target 10 is detected (when the tracking target detection flag is 1), the process proceeds to step S4. On the other hand, when the tracking target 10 is not detected (when the tracking target detection flag is 0), step S2 is performed. Returning to Fig. 4, the tracking target detection process (step S2) is repeated.

  FIG. 5 shows an image captured by the camera 1, a tracking target region (region of the tracking target 10) detected from the image by the tracking target detection process (step S 2) (or designated by the monitoring device with the pointing device). FIG. 4 is a diagram illustrating an example of a background region set in accordance with the tracking target region. In the example shown in FIG. 5, the inner rectangular area that just surrounds the whole body of the person is the tracking target area. A region between the inner rectangle and the outer rectangle is a background region to be noted in step S4 described later. The outer shape (outer rectangle) of the background region is, for example, concentric with the outer shape (inner rectangle) of the tracking target region and the outer shape (inner rectangle) of the tracking target region in the horizontal direction (x direction) and the vertical direction (y direction). ) Are respectively set to be enlarged at a predetermined magnification.

In step S4, the processing unit 2 performs processing (initial learning) for constructing an AdaBoost discriminator as initialization processing. In this initialization process, the processing unit 2 converts the AdaBoost discriminator into one or more pixels related to the pixel based on the tracking target area and the background area as shown in FIG. 5 detected by the tracking target detection process (step S2). Based on the feature vector v as the feature quantity, processing is performed to identify whether the pixel is a tracking target pixel or a background pixel. In the following description, the code of the tracking target area is R _target and the code of the background area is R _back .

In this initialization process (step S4), a positive sample representing the tracking target and a negative sample representing the background are acquired from the tracking _target region R _target and the background region R _back as follows.

However, in Equation 1, x ₁ ⁽ⁱ⁾ is a position (x, y), and l ₁ ⁽ⁱ⁾ is a class label. An initial AdaBoost classifier is constructed using initial learning samples (v (x ₁ ⁽ⁱ⁾ ), l ₁ ⁽ⁱ⁾ ) composed of the class label l ₁ ⁽ⁱ⁾ and the feature vector v. Here, the feature vector v is assumed to be a feature vector using all of the average values and variance values of Luv pixel values, which will be described later, and histograms of EOH and LBP. In this embodiment, the learning algorithm of the AdaBoost classifier uses the Gentle AdaBoost algorithm. However, in the present invention, the learning algorithm of the AdaBoost classifier is not limited to the Gentle AdaBoost algorithm.

  The Gentle AdaBoost algorithm is described in the literature “J. Friedman, T. Hastie, and R. Tibshirani,“ Additive Logistic Regression: a Statistical View of Boosting ”, technical report, Department of Statistics, Stanford University, 1998”. And is as follows. N is the number of samples, and M is the number of weak classifiers.

(1) Initialization of learning sample weight w _i by the following equation 2 and initialization of strong classifier F (x)
(2) Repetitive processing (m = 1, 2,..., M)
(A) learning using the weighted learning sample weak discriminators f _{m (x)} (weighted least squares method)
(B) Value setting of weak classifier f _m (x) by the following equation (3)
In number 3, the P _w marked with ^ show a weighted identification rate using the learning sample weight.
(C) Updating the classifier according to the following equation (4)
(D) Update of learning sample weight by the following equation 5
(3) Construction of discriminator by the following formula 6

  Here, the feature vector v used for the AdaBoost discriminator in the present embodiment will be described. It is possible to input other types of feature quantities into the AdaBoost classifier. In the present embodiment, Luv average values and variance values, edge direction histograms (EOH, Edge Orientation Histogram), and local binary pattern (LBP) histograms are used as feature quantities (elements of feature vector v) relating to pixels. Is used.

First, the average value and variance value of Luv will be described. Here, L, u, and v, which are three values of the CIE 1976 L ^* u ^* v ^* color space of the pixel, are used. The average value of L, which is one of the feature amounts of the target pixel, is the average value of L of the local region W (for example, 5 × 5 pixels) including the target pixel as the center. The L variance value, which is one of the feature amounts of the pixel of interest, is the L variance value of the local region W (for example, 5 × 5 pixels) including the pixel of interest at the center. The average value of u, which is one of the feature amounts of the target pixel, is the average value of u in the local region W (for example, 5 × 5 pixels) including the target pixel as the center. The variance value of u, which is one of the feature amounts of the target pixel, is the variance value of u in the local region W (for example, 5 × 5 pixels) including the target pixel as the center. The average value of v, which is one of the feature amounts of the target pixel, is the average value of v of the local region W (for example, 5 × 5 pixels) including the target pixel as the center. The variance value of v, which is one of the feature amounts of the target pixel, is the variance value of v in the local region W (for example, 5 × 5 pixels) including the target pixel as the center. When these are expressed using mathematical expressions, the following equations 7 and 8 are obtained. In Equations 7 and 8, the Luv pixel value in the local region W is I = (L, u, v), the average value is ￣I = (￣L, ￣u, ￣v), and the variance value is σ. _f = (σ _l , σ _u , σ _v ), and A is the number of pixels in the local region W. Here, ￣I indicates I with an upper bar, and ￣L, ￣u, and ￣v similarly indicate L, u, and v with an upper bar, respectively.

Next, the edge direction histogram (EOH) will be described. The edge direction histogram, which is one of the feature amounts of the target pixel, is a histogram obtained by weighting the edge direction in the local region (for example, 5 × 5 pixels) including the target pixel as the center by the edge strength. Assuming that the grayscale image in the local region is I, the gradient (edge) images G _x and G _y at the coordinates (x, y) can be expressed as in the following _equations (9) and (10).

In Equations 9 and 10, Sobel _x and Sobel _y are operators for calculating edges in the x and y directions, respectively. From Equation 9 and Equation 10, the gradient (edge) strength G (x, y) and the edge direction angle θ (x, y) at the coordinate (x, y) of interest are expressed by Equations 11 and 12 below. As shown in.

  Using this edge strength G and the direction angle θ, a histogram of K classes is created. This histogram is an edge direction histogram (EOH). FIG. 6 is a schematic diagram showing an edge direction histogram. FIG. 6A shows the distribution of edge vectors (edge strength and direction angle) in the local region (5 × 5 pixels). FIG. 6B shows an 8-level edge direction histogram created from the distribution of FIG.

  Next, a local binary pattern (LBP) histogram will be described. The local binary pattern is a binary pattern in which the magnitude relationship between pixel values in the vicinity of a target pixel is changed. As the grayscale image I, the LBP near 4 of the target pixel (x, y) as shown in FIG. FIG. 7 is an explanatory diagram of a local binary pattern.

  This LBP is calculated within the local region to obtain a histogram. Since LBP near 4 takes 16 values from 0 to 15, a 16-level histogram is created. In the present embodiment, the local binary pattern histogram, which is one of the feature quantities of the pixel of interest, is a local binary pattern histogram calculated within a local region centered on the pixel of interest.

  The feature amount (element of the feature vector v) related to the pixel used in the AdaBoost discriminator in the present embodiment has been described above. However, the feature quantity used in the present invention is not limited to the example of the feature vector v described above.

  Referring to FIG. 3 again, when the AdaBoost discriminator is constructed and the initialization process (step S4) ends, the process proceeds to step S5. In step S <b> 5, the processing unit 2 performs tracking processing for tracking the tracking target based on the image captured by the camera 1. In the present embodiment, in the tracking process (step S5), the processing unit 2 uses the particle filter with a plurality of particles whose pixel positions are based on the image captured by the camera 1 to partly track the result. The position of the tracking target is estimated as follows. The AdaBoost is constructed such that, for each particle, the AdaBoost is configured to identify whether the pixel is a tracking target pixel or a background pixel based on one or more feature amounts (feature vector v) regarding the pixel. It is a response value of the discriminator, and uses a likelihood calculated from a response value based on the one or more feature amounts (feature vector v) regarding the pixel at the position of the particle. The processing unit 2 estimates the size of the tracking target as another part of the tracking result based on the particles corresponding to the pixel identified as the tracking target pixel by the AdaBoost classifier among the plurality of particles. To do. Note that the particles may be referred to as particles.

  Here, an example of the tracking process (step S5) will be described with reference to FIG. The tracking process (step S5) in FIG. 3 is not limited to the example shown in FIG.

  When the tracking process (step S5) is started, as shown in FIG. 8, the processing unit 2 first samples the current image (step S201).

Next, the processing unit 2 performs particle filter selection / prediction (particle group selection / prediction) (step S202). Here, it is assumed that N particle groups (particle groups) having a state x at a certain time t are {s _t ⁽ⁱ⁾ } _{i = 1,.} However, the state x of each particle is a position (x, y) (that is, a pixel position). Also, let the likelihood of each particle be {π _t ⁽ⁱ⁾ } _{i = 1,.}

In step S202, the processing unit 2 performs the particle filter selection process based on the resampling technique, based on the resampling method, the particle group {s _t-1 ⁽ⁱ⁾ } _{i =} at the previous time t−1. _{1,..., N} likelihood {π _t−1 ⁽ⁱ⁾ } _{i = 1,..., N} , and a new particle group {s ′ _t−1 ⁽ⁱ⁾ } _{i = 1,.} select. By increasing the number of particles with high likelihood (split) and decreasing the number of particles with low likelihood (disappear), it is possible to approximate the position probability distribution of the tracking target with a limited number of particles. If the number of particles is less than the preset number N due to the disappearance of the particles, new particles are added to the target area and the background area. If this step is immediately after the start of the tracking process (when moving from step S4 to step S5), the likelihood of particles is not obtained, so this process is omitted.

In step S202, the processing unit 2 predicts the particle s _t ⁽ⁱ⁾ at the time t by causing the state transition of each particle s ′ _t-1 ⁽ⁱ⁾ at the time t−1 according to the motion model as the prediction process of the particle filter. . In this embodiment, a random walk model is used as the motion model. This is because many environments are assumed here, and in particular, when tracking a tracking target with the camera 1 capable of controlling pan, tilt, and zoom, the movement of the tracking target is approximated by a uniform linear motion or the like. This is because it is difficult. Equation 14 below shows the formula of the random walk model.

However, w _t−1 is Gaussian noise. w _t−1 may be a random image according to a Gaussian image. That is, the position at time t is determined by adding a random number to the position at time t-1.

  In the case immediately after the start of the tracking process (when the process proceeds from step S4 to step S5), the data used for learning in step S4 is treated as the data at time t-1 in Formula 14, and the state is changed.

  After step S202, the processing unit 2 sets the value of the particle number n to 1 (step S203).

  Next, the processing unit 2 sets the pixel at the position of the n-th particle as the target pixel, and calculates an average value and a variance value of the Luv pixel values related to the target pixel (step S204). The calculation method is as described above.

  Subsequently, the processing unit 2 uses the pixel at the position of the nth particle as the pixel of interest, and acquires an edge direction histogram (EOH) regarding the pixel of interest (step S205). The acquisition method is as described above.

  Thereafter, the processing unit 2 uses the pixel at the position of the n-th particle as a target pixel, and acquires a histogram of a local binary pattern (LBP) related to the target pixel (step S206). The acquisition method is as described above.

  Next, the processing unit 2 calculates the response value F of the AdaBoost discriminator by the feature vector v having the feature amount acquired in steps S204 to S206 as the element for the n-th particle according to the following formula 15 (step S207). ). If the response value F is positive, it means that the pixel at the position of the nth particle is identified as the tracking target pixel. If the response value F is negative, the pixel at the position of the nth particle is This means that the pixel is identified as a background pixel.

Next, the processing unit 2 calculates the likelihood π _t ⁽ⁱ⁾ of the nth particle from the response value F calculated in step S207 according to the following _equation 16 (step S208). In Equation 16, a is a constant. This process corresponds to the observation of the particle filter.

Note that the equation for calculating the likelihood π _t ⁽ⁱ⁾ is not limited to Equation 16. As the calculation formula, the response value F takes a value from −1 to +1.

Subsequently, the processing unit 2 determines whether or not the likelihood π _t ⁽ⁱ⁾ has been calculated by performing steps S204 to S208 for all particles (step S209). If the calculation has been completed for all the particle filters, the process proceeds to S211. On the other hand, if not completed, the value of the particle number n is incremented by 1 (step S210), and then the process returns to step S204.

In step S211, the processing unit 2 calculates the weighted average value E [S _t ] using the likelihood π _t ⁽ⁱ⁾ of each particle calculated in step S208 at the pixel position that is the state of each particle by the following number. 17 is calculated as an estimation result (part of the tracking result) of the tracking target position (here, the center of gravity position), which is the state estimation result.

Then, the processing unit 2, the each particle _(particle) for ^{s t (i),} the center of gravity of the tracking target, obtained as the estimation result in step S211 (¯x, ¯y) with respect to the particle _s ^{t (i)} The Mahalanobis distance D _t ⁽ⁱ⁾ using the covariance matrix weighted by the likelihood π _t ⁽ⁱ⁾ is calculated according to the following _equation 22 (step S212). Note that ￣x indicates x with an upper bar, and ￣y indicates y with an upper bar.

The following _equations 18 to 21 show the weighted covariance matrix Σ _t ′ of each particle s _t ⁽ⁱ⁾ = x _t ⁽ⁱ⁾ = (x _t ⁽ⁱ⁾ , y _t ⁽ⁱ⁾ ).

The Mahalanobis distance D _t ^{(i) with} respect to the center-of-gravity position (￣x, ￣y) of the tracking target of each particle s _t ⁽ⁱ⁾ is expressed by Equation 22 below. In Equation 22, ￣s _t = ￣x _t = (￣x _t , ￣y _t ), and Σ _t ' ^-1 is an inverse matrix of Σ _t '.

  An ellipse is drawn by connecting points with equal Mahalanobis distance. Assuming that a sample follows a normal distribution, the Mahalanobis distance can be thought of as representing the confidence interval for that sample. In the present embodiment, for example, the Mahalanobis distance D≈1.40 is set so as to empirically include about 80% of the sample, that is, the weighted particle group. An ellipse (inner ellipse) drawn using this Mahalanobis distance is shown as an object ellipse in FIG. In FIG. 9, the outer ellipse is drawn as the Mahalanobis distance D = 10 so as to include all the particle part groups, and in FIG. 9, the white point is the response value of the AdaBoost discriminator. F is a positive particle (that is, identified as a tracking target), and a black dot represents a particle whose response value F is negative (that is, identified as a background).

After calculating the Mahalanobis distance D _t ⁽ⁱ⁾ for each particle s _t ⁽ⁱ⁾ in step S212, the processing unit 2 is identified as a tracking target pixel by the AdaBoost classifier among all the particles s _t ^(i). The size of the tracking target is estimated based on the particles corresponding to the pixels (that is, particles with a positive response value F) (step S213). In the present embodiment, the processing unit 2 includes particles having a positive response value F and having a Mahalanobis distance D _t ⁽ⁱ⁾ calculated in step S212 of a predetermined value (eg, 1.40) or less. Based on the distribution situation, the size of the tracking target is estimated.

In the present embodiment, specifically, in step S213, the processing unit 2 narrows down to particles whose response value F is positive and whose Mahalanobis distance D _t ⁽ⁱ⁾ is a predetermined value (eg, 1.40) or less. Then, the processing unit obtains a maximum value and a minimum value for the x-coordinate and the y-coordinate of the narrowed-down particle position (pixel position in the state of the particle), respectively. At this time, the minimum value of the x coordinate is x _s , the maximum value of the x coordinate is x _e , the minimum value of the y coordinate is y _s , and the maximum value of the y coordinate is y _e . As a result, the processing unit 2 sets the rectangular area of the tracking target area as a rectangular area having a start point (x _s , y _s ) and an end point (x _e , y _e ) with the size of this area. Assuming there is a size of the tracking target area. When the zoom of the camera 1 is controlled in the camera control process (step S7) in FIG. 3, data indicating this rectangular area is used.

Next, the processing unit 2 determines the reliability ct of the tracking result (the tracking target position estimated in step S211 and the tracking target size estimated in step S213) according to the following _equations 23 and 24. A spatially weighted average value of the likelihood of all particles is calculated (step S214). However, Equation 25 holds. In Equations 23 to 25, N is the number of particles, w _t ⁽ⁱ⁾ is the spatial weight of each particle, and D _t ⁽ⁱ⁾ is the Mahalanobis distance of Equation 22 for each particle.

In the particle filter described above, the position (x, y) on the image is treated as a state, and the likelihood indicates the probability that each particle is a tracking target or the background. Since the tracking target is a set of points, it is possible to treat the average value of the likelihood of each position as the reliability of the tracking target. However, the reliability c shown in Equation 23 is used in order to treat it as a more reliable value. _t is calculated.

Here, it is assumed that the tracking target particles are concentrated, and instead of simply calculating an average value for the likelihood, a spatial weighted average value is calculated as shown in Equation 23. It is the reliability _{c t.} FIG. 10 is an explanatory diagram thereof. FIG. 10A shows the particle distribution with the upper left corner of the image as the origin. FIG. 10B shows the likelihood of each particle. FIG. 10C shows the above-described spatial weighting, which corresponds to w _t ⁽ⁱ⁾ in Expression 24. When the likelihood of FIG. 10 (b) is multiplied by the spatial weight of each particle of FIG. 10 (c), the result is as shown in FIG. 10 (d). The result of this product-sum operation is the reliability c used this time. _t .

After step S214, the processing unit 2 determines the calculated reliability _{c t} is whether higher than a preset threshold at step S214 (step S215). If it is higher, the process proceeds to step S216, and if not higher, the process proceeds to step S217.

In step S216, the processing unit 2, if the reliability c _t is higher than the threshold value, it is determined that the tracking is successful, and 1 tracking result flag. Thereafter, the process proceeds to step S218.

In step S217, the processing unit 2, if the reliability c _t is not higher than the threshold value, it determines that tracking is unsuccessful, the tracking result flag to 0. Thereafter, the process proceeds to step S218.

In step S218, the processing unit 2 performs an update process for updating the AdaBoost discriminator. In this update process, the processing unit 2 uses predetermined particles among the particles s _t ⁽ⁱ⁾ at the time t whose state has been changed and predicted in step S202 as learning samples for the tracking target pixel, and at the time t. The AdaBoost discriminator is updated using another predetermined particle of the particles s _t ⁽ⁱ⁾ as a learning sample for the background pixel.

  Here, an example of the update process (step S218) of the AdaBoost discriminator will be described with reference to FIG. The AdaBoost discriminator update process (step S218) in FIG. 8 is not limited to the example shown in FIG.

When the AdaBoost discriminator update process (step S218) is started, as shown in FIG. 11, the processing unit 2 first classifies the learning sample to be tracked and the background learning sample (step S301). Specifically, the processing unit 2 sets the region in the target ellipse (inner ellipse, Mahalanobis distance D≈1.40) shown in FIG. 9 as the tracking target region R _target, and the background shown in FIG. A region between an ellipse (an outer ellipse, an ellipse with Mahalanobis distance D = 10) and the target ellipse is defined as a background region R _back . Then, the processing unit 2 learns the background of all particles in the background region R _back , a positive sample that uses particles having a positive response value F of the AdaBoost discriminator for learning in the target region R _target. Negative sample used for If this is expressed by a mathematical expression, the upper part of the following equation 26 is a positive sample, and the lower part is a negative sample.

  Next, the processing unit 2 calculates the weight of the learning sample for each sample classified in step S301 (step S302). At this time, the weight of the positive sample is calculated by the upper part of the following expression 27, and the weight of the negative sample is calculated by the lower part of the following expression 27.

In Equation 27, F (v (s _t ⁽ⁱ⁾ )) is the response value of the AdaBoost discriminator for the particle s _t ⁽ⁱ⁾ , and D _t ⁽ⁱ⁾ is the Mahalanobis distance of Equation 22 for each particle.

  Subsequently, the processing unit 2 determines whether or not weights have been calculated for all samples (step S303). If it is calculated, the process proceeds to step S304, and if not calculated, the process returns to S302.

  In step S304, the processing unit 2 performs learning (relearning) similar to step S4 using the weight newly calculated in step S302, and creates a discriminator.

Next, the processing unit 2 updates the classifier by integrating the newly learned classifier and the existing classifier (step S305). In order to prevent excessive updating, in the present embodiment, the strong classifier F ₁ (v) learned at initialization time t = 1, and the classifier F _t (v) learned by sequential updating at t> 1 To integrate. Equations 28 and 29 below show formulas for F ₁ (v) and F _t (v).

Where f _{1, m} (v) is the weak classifier at t = 1, M is the number of weak classifiers, f _t (v) is the weak classifier at t> 1, T is the number of history of the updated weak classifiers, K is the number of weak classifiers to be updated, and _ct is the reliability of the tracking target. The discriminator F ₁ (v) at the time of initialization and the discriminator F _t (v) obtained by the additional learning are integrated as shown in Equation 30 below, and the discriminator F (v) is updated.

Here, β ₁ and β _t are weights based on the misclassification rate in the learning samples of the initial discriminator F ₁ (v) and the additionally learned discriminator F _t (v), respectively. Define as follows.

Here, e ₁ and e _t are the weighted misidentification rates for the learning samples of the discriminator F ₁ (v) at t = 1 at initialization and the discriminator F _t (v) additionally learned at t> 1, respectively. It is. That, _{e t} is the calculated value in step S305.

  When step S305 ends, the discriminator update process (step S218) in FIG. 8 ends, and the process proceeds to step S6 in FIG.

  Referring to FIG. 3 again, in step S6, the processing unit 2 determines whether the tracking by the tracking process in step S5 is successful by determining whether the current tracking result flag is 1 or not. . If the tracking is successful (if the tracking result flag is 1), the process proceeds to step S7. On the other hand, if tracking fails (if the tracking result flag is 0), the process proceeds to step S8. In step S8, the processing unit 2 determines whether or not the follow-up failure state continues for a certain period of time. If it does not continue for a certain time, the process returns to the tracking process (step S5) and the process is repeated. If it continues for a certain period of time, it is considered that the tracking process is unlikely to succeed, and the process returns to step S1 (preset state).

If it is determined in step S6 that the tracking is successful (the tracking result flag is 1), the processing unit 2 estimates the tracking result obtained in the tracking process (step S5), that is, the estimation in step S211 in FIG. Camera control for controlling panning, tilting and zooming of the camera 1 so that the camera 1 tracks the tracking target according to the position of the tracking target and the size of the tracking target estimated in step S213 in FIG. Process. In this camera control process, the processing unit 2 predicts the position and size of the tracking target after a predetermined time has elapsed from the present (after n _f frames) based on the tracking result, and the camera 1 according to the prediction result. The current pan, tilt and zoom control states of the camera 1 are corrected to control the pan, tilt and zoom of the camera 1. Here, in the present embodiment, the position and size of the tracking target are predicted by the Kalman filter.

  Here, an example of the camera control process (step S7) will be described with reference to FIGS. Note that the camera control process (step S7) in FIG. 3 is not limited to the examples shown in FIGS.

  When the camera control process (step S7) is started, as shown in FIG. 12, the processing unit 2 first determines that the processing unit 2 uses the tracking result (step S211 in FIG. 8 as information used in the camera control process). The position of the tracking target and the size of the tracking target estimated in step S213 in FIG. 8 are acquired (step S501). Since the tracking result is originally possessed by the processing unit 2, the acquisition operation is originally unnecessary, but step S 501 is inserted here for easy understanding.

  Next, the processing unit 2 acquires a pan control flag, a tilt control flag, and a zoom control flag as information indicating the respective pan, tilt, and zoom control states of the camera 1 (step S502). In the present embodiment, the camera 1 performs control operations after receiving control commands from the processing unit 2 for each of pan, tilt, and zoom, and returns a control completion signal to the processing unit 2 when the control operations are completed. It has become. The processing unit 2 sets a pan control flag to 1 when a pan control command is given to the camera 1, and resets the pan control flag to 0 by an interrupt process when a pan control completion signal is received from the camera 1. Further, the processing unit 2 sets the tilt control flag to 1 when giving a tilt control command to the camera 1, and resets the tilt control flag to 0 by interruption processing when receiving a tilt control completion signal from the camera 1. Further, the processing unit 2 sets the zoom control flag to 1 when giving a zoom control command to the camera 1, and resets the zoom control flag to 0 by interrupt processing when receiving a zoom control completion signal from the camera 1. As described above, when each control flag of pan, tilt, and zoom is 1, it indicates that the corresponding operation is being controlled, and when it is 0, it indicates that the corresponding operation is being stopped. As can be understood from the above description, since the processing unit 2 originally has the pan control flag, the tilt control flag, and the zoom control flag, the acquisition operation is originally unnecessary, but here, for easy understanding. This step S502 is inserted.

  Next, the processing unit 2 determines whether all control flags (pan control flag, tilt control flag, zoom control flag) are 0 (step S503). The process proceeds to S519, and if any one or more control flags is 1, the process proceeds to Step S504.

  In step S504, the processing unit 2 compares the tracking result (particularly the tracking target position) acquired in step S501 from the center of the image compared to the tracking result (particularly the tracking target position) acquired in the previous frame. Determine if you are away. If it is away, the process proceeds to step S506, and if it is approaching, the process proceeds to step S505.

  If it is away from the center of the image, it is determined that the control from the previous frame is not suitable, and the processing unit 2 resets the pan, tilt, and zoom control flags to 0 in step S506, and further performs camera control. Is stopped (step S507).

  In step S505, based on the tracking result acquired in step S501, the processing unit 2 changes the direction of movement and the size of the tracking target up to that point (whether it gradually increases or decreases) Determine if has changed. If there is no change in both the traveling direction and the size, the process proceeds to step S508, and if either has changed, the process proceeds to step S506.

  In step S508, the processing unit 2 determines whether or not the zoom control flag is 1, thereby determining whether or not zoom control is currently in progress. If zoom control is being performed, the process proceeds to step S509. If zoom control is not being performed, the process proceeds to step S511.

  In step S509, the processing unit 2 determines whether the tracking result acquired in step S501 has reached a target size range that has already been set in advance. If it has reached, the process proceeds to S510, and if not, the process proceeds to Step S511.

  In step S510, the processing unit 2 sets the zoom control flag to 0. This is because the zoom control is preferably stopped at that time because the target size range set in advance has been reached.

  In step S511, the processing unit 2 determines whether or not the pan control flag is 1, thereby determining whether or not the pan control is currently in progress. If pan control is being performed, the process proceeds to step S512, and if pan control is not being performed, the process proceeds to step S514.

  In step S512, the processing unit 2 determines whether or not the tracking result acquired in step S501 has reached the target horizontal position range that has been set in advance. If it has reached, the process proceeds to step S513, and if not, the process proceeds to S514.

  In step S513, the processing unit 2 sets the pan control flag to 0. This is because it is preferable to stop the pan control at that time point because the target horizontal position range set in advance has been reached.

  In step S514, it is determined whether or not the tilt control flag is 1, thereby determining whether or not the tilt control is currently in progress. If the tilt control is being performed, the process proceeds to step S515, and if the tilt control is not being performed, the process proceeds to S517.

  In step S515, the processing unit 2 determines whether or not the tracking result acquired in S501 has reached the target vertical position range that has been set in advance. If it has reached, the process proceeds to S516, and if not, the process proceeds to S517.

  In step S516, the processing unit 2 sets the tilt control flag to 0. This is because the tilt control is preferably stopped at that time because the target vertical position range has been set in advance.

  In step S517, the processing unit 2 determines whether any of the pan control flag, the tilt control flag, and the zoom control flag has been changed. If any one of steps S510, S513, and S516 is performed, the process proceeds to S518. If none is performed, the process proceeds to S519 in order to continue the control.

  In step S518, the processing unit 2 changes the control. This is because reaching step S518 indicates that there is a difference between the control of the pan, tilt and zoom of the camera 1 based on the previous prediction and the actual movement of the tracking target.

  In step S <b> 519, the processing unit 2 acquires the current posture of the camera 1 (pan, tilt, zoom position) from the camera 1.

Next, the processing unit 2 predicts the position and size of the tracking target after n _f frames from the tracking result acquired in step S501 and the result of the previous tracking process (step S520). For example, in the case of an NTSC signal, “after n _f frames” corresponds to (n _f / 30) seconds later.

Here, in order to predict the tracking position and size of the target after n _f frames, using a Kalman filter.

  Here, the Kalman filter is configured on the assumption that the change in the position and size of the tracking target is constant and the change is smooth. In the Kalman filter, even if the change in the state of the tracking target is not strictly applied to the set model, there are many cases where it can be applied approximately because there is an error term.

A state variable vector x _k at time k is defined as in the following Expression 34.

Here, y _k is the vertical coordinate and velocity, s _k and top to dot x _k marked with dots on the x _k is denoted by the dot target rectangle center horizontal coordinates and velocity of the upper and y _k in the image Sk with a _represents a size (product of a horizontal width and a vertical width of a rectangle) and its change.

  A system equation in consideration of this state vector, control of the camera 1, and error is defined by the following equation (35).

  In Equation 35, A is a constant matrix shown in Equation 36 below.

The horizontal width of the input image is W _src , the vertical width is H _src , the horizontal angle of view at time k is θ _k , the vertical angle of view is φ _k , the pan angle speed of the camera is P _k , and the tilt angular speed is up When T _k marked with dots, horizontal and vertical variation of the pixel by the pan-tilt control at each time it is represented by the respective by the number 37 and number 38 below.

  If the change in the angle of view due to the zoom operation is λ, λ can be expressed by the following equation (39).

From the above elements, the control vector u _k is given by the following numbers 40. u _k is a control vector which represents the variation of the image by the camera control.

w _k is a system error, and follows a normal white process with a covariance matrix Q _k and an average of 0, as shown in Equation 41 below.

  Here, the position and size of the rectangle surrounding the tracking target obtained from the tracking process described above are taken as observation values, and the observation vector at time k is defined as in the following equation (42).

  The observation equation is expressed by the following Equation 43.

Here, H is a constant matrix shown in the following equation 44. Further, as shown in Expression 45, the observation error v _k follows a normal white process having a covariance matrix R _k and an average of 0.

  In the Kalman filter, the estimated amount of the current time is estimated using the observed amount of the current time and the state amount of the previous period. The state of the current time k of the system is represented by two variables represented by the following formula 46. In addition, in this specification, the code | symbol ^ attached | subjected on the symbol means the estimated value.

  The Kalman filter performs two procedures, prediction and update, to advance one time step. In the prediction procedure, the estimated state of the current time is calculated from the estimated state of the previous time. In the update, a more accurate state is estimated by using the observation at the current time and correcting the estimated value.

  Regarding the prediction, the estimated value of the current time is expressed by the following formula 47, and the covariance matrix of the current time error is expressed by the following formula 48.

  Regarding the update, the Kalman filter calculates a Kalman gain that minimizes the estimated value of the error after the update by the following formulas 49 to 53, and updates the state.

  With the above calculation, it is possible to estimate the state quantity at the current time in consideration of the error.

Here, setting of initial conditions of the Kalman filter will be described. Assuming that the center coordinates of the tracking target rectangle at the start of tracking are (x ₀ , y ₀ ) and the size is s ₀ , the initial value of the state is as shown in Equation 54 below. Here, the speed is zero.

  If there is an error in the initial condition, a covariance matrix of the error is given as in Equation 55 below.

  The characteristic of the filter is the variance ratio between the system error and the observation error. The larger the variance ratio, the more accurate the estimated value after filtering is, but the more sensitive the error is. Follow-up performance against is reduced. In the present embodiment, it is possible to quickly cope with a change in the moving direction of the target person, a stoppage, and the like, but a dispersion ratio that is not easily affected by an error included in the tracking result is empirically used.

  The Kalman filter used in the process of step S520 in FIG. 13 has been described above.

After step S520, the processing unit 2 calculates the horizontal angle of view after n _f frames from the size of the tracking target after n _f frames obtained by the prediction at step S520 and the size of the tracking target to be targeted s _i. A vertical angle of view is calculated (step S521). However, the following formula 56 calculates the horizontal angle of view.

  Next, the processing unit 2 determines whether or not the horizontal angle of view calculated in step S521 has reached the zoom limit (step S522). If the zoom limit has been reached, the process proceeds to step S549. However, when the limit is reached in zooming out, the process proceeds to step S523. If neither zoom-in nor zoom-out has reached the limit, the process proceeds to step S523.

  In step S523, the processing unit 2 compares the current horizontal angle of view from the result obtained in S521, and determines whether or not the zoom control amount is smaller than a predetermined value. If it is smaller, the process proceeds to S524, and if it is larger, the process proceeds to S526.

  In step S524, the processing unit 2 sets the zoom control speed to zero. Subsequently, the processing unit 2 sets the zoom control flag to 0 (step S525), and proceeds to step S530. Thus, zoom control is not performed. Thus, when the control amount of zoom is small, zoom control is not performed. This is because when the zoom control amount is small, if the zoom control is performed, a fine movement is caused, which may cause discomfort to the monitor who monitors the display unit 5.

In step S526, the zoom control speed Z _speed is calculated by the following _equation 57. Here, f _r indicates a frame rate (in the case of NTSC, f _r = 30). Z _pt is the zoom position when the horizontal angle of view is time t.

Thereafter, the processing unit 2 determines whether or not the difference between the zoom control speed Z _speed calculated in step S526 and the _speed currently being controlled is greater than a predetermined value (step S527). If the difference is large, the process proceeds to step S528, and if the difference is small, the process proceeds to step S529. Again, if the difference is not greater than a certain value, the zoom control speed is not changed because the frequent change of the zoom control speed may cause discomfort for the observer who visually monitors the display unit 5. This is to avoid this. In addition, the tracking target only needs to be near the center of the image, and the purpose is not to maintain a state in which there is no deviation from the center of the image.

  In step S528, the processing unit 2 changes the zoom control speed value calculated in step S526. Thereafter, the process proceeds to step S529.

  In step S529, the processing unit 2 sets the zoom control flag to 1. Thereafter, the process proceeds to step S530.

  In step S530, the processing unit 2 determines whether or not the pan control amount is smaller than a predetermined value from the result obtained by the prediction in step S520. If so, the process proceeds to step S531, and if greater, the process proceeds to step S533.

  In step S531, the processing unit 2 sets the pan control speed to zero. Subsequently, the processing unit 2 sets the pan control flag to 0 (step S532), and proceeds to step S538. Thus, pan control is not performed. Thus, when the pan control amount is small, pan control is not performed. This is because if the pan control amount is small, if the pan control is performed, a fine movement occurs, which may cause discomfort to the monitor who monitors the display unit 5.

In step S <b> 533, the processing unit 2 calculates the pan control speed P _speed by the following equation 58. Here, W _src is the horizontal width of the input image.

Thereafter, the processing unit 2 assumes that the pan is controlled at the pan control speed P _speed calculated in step S533, and when the time corresponding to n _f frames has elapsed (after n _f / f _r seconds), The position is calculated, and it is determined whether or not the value reaches the panning limit or exceeds the panning limit (step S534). If the value reaches the panning limit or exceeds the panning limit, the process proceeds to step S549, and if not, the process proceeds to step S535.

In step S535, the processing unit 2 determines whether or not the difference between the pan control speed P _speed calculated in step S533 and the _speed currently being controlled is greater than a predetermined value. If the difference is large, the process proceeds to step S536, and if the difference is small, the process proceeds to step S537.

  In step S536, the processing unit 2 changes the pan control speed value calculated in step S536. Thereafter, the process proceeds to step S537.

  In step S537, the processing unit 2 sets the pan control flag to 1. Thereafter, the process proceeds to step S538.

  In step S538, the processing unit 2 determines whether the tilt control amount is smaller than a predetermined value from the result obtained in the prediction in step S520. If it is smaller, the process proceeds to step S539, and if it is larger, the process proceeds to step S541.

  In step S539, the processing unit 2 sets the tilt control speed to zero. Subsequently, the processing unit 2 sets the tilt control flag to 0 (step S540), and proceeds to step S546. Thus, tilt control is not performed. Thus, when the amount of tilt control is small, tilt control is not performed. This is because when the tilt control amount is small, if the tilt control is performed, a fine movement is caused, which may cause discomfort to the monitor who monitors the display unit 5.

In step S <b> 541, the processing unit 2 calculates the tilt control speed T _speed by the following equation 59. Here, H _src is the height (length in the vertical direction) of the input image.

Thereafter, the processing unit 2 assumes that the tilt is controlled at the tilt control speed T _speed calculated in S541, and when the time corresponding to n _f frames has elapsed (after n _f / f _r seconds), The position is calculated, and it is determined whether or not the value reaches the tilt limit or exceeds the tilt limit (step S542). If the value reaches the tilt limit or exceeds the tilt limit, the process proceeds to step S549, and if not, the process proceeds to step S543.

In step S543, the processing unit 2 determines whether or not the difference between the tilt control speed T _speed calculated in step S541 and the _speed currently being controlled is greater than a predetermined value. If the difference is large, the process proceeds to S544, and if the difference is small, the process proceeds to S545.

  In step S544, the processing unit 2 changes the value of the tilt control speed calculated in step S541. Thereafter, the process proceeds to step S545.

  In step S545, the processing unit 2 sets the tilt control flag to 1. Thereafter, the process proceeds to step S546.

  In step S546, the processing unit 2 determines whether or not all control flags (pan control flag, tilt control flag, zoom control flag) are zero. If all the control flags are 0, control is not performed, and the process proceeds to step S548 without passing through step S547. If any one or more control flags are 1, it will transfer to step S547.

In step S547, the processing unit 2 gives a control command to the camera 1 so as to perform control according to the control flag that is 1 and the control speed corresponding thereto. Its control is performed n _{f /} f _r seconds depending on the speed, may no wait for the control to complete the process proceeds to step S548 exits this loop. In the present embodiment, a control command is not given to the camera 1 by the control flag itself, but a control command according to the control flag and the control speed is given to the camera 1 by the operation as in step S547. ing.

  In step S548, the processing unit 2 sets the tracking limit flag to 0 (0 indicates that none of pan, tilt, and zoom can reach the limit). This is because step S548 is reached when none of pan, tilt, and zoom can reach the limit. After step S548, the camera control process (step S7) is terminated, and the process proceeds to step S9 in FIG.

  In step S549, the processing unit 2 sets the tracking limit flag to 1 (1 indicates that any of pan, tilt, and zoom may reach the limit). This is because step S549 is reached when there is a possibility that at least one of pan, tilt, and zoom may reach the limit. After step S549, in step S550, the processing unit 2 stops the control if the posture control of the camera 1 is being performed. After step S550, the camera control process (step S7) is terminated, and the process proceeds to step S9 in FIG.

  Referring to FIG. 3 again, in step S9, the processing unit 2 determines whether or not the tracking limit flag is 1. If the tracking limit flag is 1, it is determined that it is difficult to continue tracking of the tracking target, and the process returns to step S1 (preset state). If the tracking limit flag is 0, it is possible to continue tracking the tracking target, so the process returns to step S5 to continue tracking the tracking target.

  According to the present embodiment, the tracking target position is estimated by the particle filter (step S211 in FIG. 8). Therefore, unlike the correlation calculation, a plurality of solution candidates (a plurality of particles) are included, so tracking failure occurs. Therefore, the tracking process can be performed with higher accuracy and more accurately with respect to occlusion and complicated backgrounds. As a result, the tracking target can be tracked with higher accuracy and imaged. According to the present embodiment, the tracking target pixel and the background pixel are identified by the AdaBoost classifier, and the particle filter uses the likelihood calculated from the response value of the AdaBoost classifier. This makes it less susceptible to changes in the background, such as the shape, size, color, and light / dark changes in the background. From this point, it is possible to perform tracking processing with even higher accuracy, and, in turn, to track the tracking target with higher accuracy. Image.

  Further, according to the present embodiment, the size of the tracking target is estimated based on the particles corresponding to the pixel identified as the tracking target pixel by the AdaBoost discriminator (step S213 in FIG. 8). The size of the tracking target is also strong against occlusion and a complicated background, so that tracking processing can be performed with higher accuracy. As a result, the tracking target can be tracked with higher accuracy and imaged.

  Furthermore, according to the present embodiment, since the AdaBoost discriminator is updated, it is possible to cope with changes in the appearance of the tracking target and changes in the environment (such as lighting and sunshine conditions). The accuracy of identifying whether the pixel is a tracking target pixel or a background pixel is increased. Therefore, according to the present embodiment, it is possible to perform the tracking process with higher accuracy, and as a result, it is possible to track and image the tracking target with higher accuracy.

According to this embodiment, as the reliability reliability c _t of the tracking target, and calculates the spatial weighted mean value of the likelihood of all particles (step S214), whether the value is a predetermined value or more (Steps S111, S112, S6), it can be determined whether the tracking has succeeded or failed. Therefore, in this embodiment, if NO in step S6 and YES in step S8, the process returns to step S1, so that control of the camera is continued based on the erroneous tracking result even though tracking has failed. Can be avoided.

  Further, according to the present embodiment, since predictive control is introduced in the camera control process, for example, the operation time until the camera 1 responds to the control command and enters the command state is the image processing time. Even if it is longer than this, it is possible to cope with a sudden change in the tracking target, and it is possible to track and image the tracking target with higher accuracy. Note that if the camera pan, tilt, and zoom control speeds are too fast, the camera pan, tilt, and zoom will change too rapidly when the observer follows the tracking target. Although it is uncomfortable and not suitable for monitoring, a camera with a relatively slow control speed can be used as the camera 1, so that the panning, tilting and zooming changes of the camera 1 are smoothed and more suitable for monitoring. Tracking can be realized.

  Furthermore, according to the present embodiment, since the Kalman filter is used for such prediction, the position and size of the tracking target region can be predicted with high accuracy, and thus the tracking target can be tracked with higher accuracy. Can be taken.

  Although one embodiment of the present invention has been described above, the present invention is not limited to this embodiment.

1 is a block diagram schematically showing an automatic tracking device according to an embodiment of the present invention. It is a figure which shows typically the example of the mode of the tracking of the tracking target by a camera. It is a schematic flowchart which shows an example of operation | movement of the process part in FIG. It is a flowchart which shows the tracking object detection process (step S2) in FIG. 3 in detail. It is a figure which shows the example of the image imaged with the camera, a tracking object area | region, and a background area | region. It is the schematic which shows an edge direction histogram. It is explanatory drawing of a local binary pattern. It is a flowchart which shows the tracking process (step S5) in FIG. 3 in detail. It is a figure which shows the example of a target ellipse and a background ellipse. It is explanatory drawing of reliability. It is a flowchart which shows the update process (step S218) of the identification device in FIG. 8 in detail. It is a flowchart which shows the camera control process (step 7) in FIG. 3 in detail. It is a flowchart following FIG. It is a flowchart following FIG. It is a flowchart following FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera 1a Camera body 1b Zoom lens 1c Turntable 2 Processing part

Claims (14)

A camera capable of controlling pan, tilt and zoom;
Tracking processing means for performing tracking processing for tracking a tracking target based on an image captured by the camera;
Control means for controlling pan, tilt, and zoom of the camera so that the camera tracks and captures the tracking target according to the result of the tracking processing by the tracking processing means;
With
The tracking processing means includes position estimating means for estimating the position of the tracking target as a part of the tracking result by a particle filter using a plurality of particles whose pixel positions are based on an image captured by the camera. Including
The particle filter is a response value of an AdaBoost discriminator constructed to identify whether the pixel is a tracking target pixel or a background pixel based on one or more feature amounts related to the pixel for each particle. there are, from the response value by the one or more feature amount relating to the pixel position of the particle state, and are not used the calculated likelihood,
The tracking processing unit includes the particle corresponding to the pixel identified as the tracking target pixel by the AdaBoost classifier among the plurality of particles, the tracking target position estimated by the position estimation unit. Size estimation means for estimating the size of the tracking target based on a particle distribution state in which a Mahalanobis distance using a covariance matrix weighted by the likelihood of the particles is a predetermined value or less,
An automatic tracking device characterized by that. The one or more feature amounts are an average value of the first value of the first to third values of a predetermined color space of a pixel in the local region including the pixel, and a pixel in the local region including the pixel. The variance value of the first value, the average value of the second values of the pixels in the local area including the pixel, the variance value of the second value of the pixels in the local area including the pixel, and the local value including the pixel The average value of the third values of the pixels in the region, the variance value of the third value of the pixels in the local region including the pixel, the edge direction histogram in the local region including the pixel, and the local region including the pixel The automatic tracking device according to claim 1, comprising at least one of a histogram of a local binary pattern at. A predetermined particle of the plurality of particles is used as a learning sample for a tracking target pixel, and the AdaBoost discriminator is updated using another predetermined particle of the plurality of particles as a learning sample for a background pixel. The automatic tracking device according to claim 1, further comprising an updating unit. A calculation unit that calculates a spatial weighted average value of the likelihoods of the plurality of particles, and a determination unit that determines whether or not the spatial weighted average value is equal to or greater than a predetermined value. The automatic tracking device according to any one of claims 1 to 3. A camera capable of controlling pan, tilt and zoom;
Tracking processing means for performing tracking processing for tracking a tracking target based on an image captured by the camera;
Control means for controlling pan, tilt, and zoom of the camera so that the camera tracks and captures the tracking target according to the result of the tracking processing by the tracking processing means;
With
The tracking processing means includes position estimating means for estimating the position of the tracking target as a part of the tracking result by a particle filter using a plurality of particles whose pixel positions are based on an image captured by the camera. Including
The particle filter is a response value of an AdaBoost discriminator constructed to identify whether the pixel is a tracking target pixel or a background pixel based on one or more feature amounts related to the pixel for each particle. Then, the likelihood calculated from the response value by the one or more feature amounts related to the pixel at the position of the particle is used,
The one or more feature amounts are an average value of the first value of the first to third values of a predetermined color space of a pixel in the local region including the pixel, and a pixel in the local region including the pixel. The variance value of the first value, the average value of the second values of the pixels in the local area including the pixel, the variance value of the second value of the pixels in the local area including the pixel, and the local value including the pixel The average value of the third values of the pixels in the region, the variance value of the third value of the pixels in the local region including the pixel, the edge direction histogram in the local region including the pixel, and the local region including the pixel comprising at least one of the histogram of local binary pattern, in,
An automatic tracking device characterized by that. A predetermined particle of the plurality of particles is used as a learning sample for a tracking target pixel, and the AdaBoost discriminator is updated using another predetermined particle of the plurality of particles as a learning sample for a background pixel. 6. The automatic tracking device according to claim 5, further comprising updating means. A calculation unit that calculates a spatial weighted average value of the likelihoods of the plurality of particles, and a determination unit that determines whether or not the spatial weighted average value is equal to or greater than a predetermined value. The automatic tracking device according to claim 5 or 6 . A camera capable of controlling pan, tilt and zoom;
Tracking processing means for performing tracking processing for tracking a tracking target based on an image captured by the camera;
Control means for controlling pan, tilt, and zoom of the camera so that the camera tracks and captures the tracking target according to the result of the tracking processing by the tracking processing means;
With
The tracking processing means includes position estimating means for estimating the position of the tracking target as a part of the tracking result by a particle filter using a plurality of particles whose pixel positions are based on an image captured by the camera. Including
The particle filter is a response value of an AdaBoost discriminator constructed to identify whether the pixel is a tracking target pixel or a background pixel based on one or more feature amounts related to the pixel for each particle. Then, the likelihood calculated from the response value by the one or more feature quantities related to the pixel at the position of the particle is used,
A predetermined particle of the plurality of particles is used as a learning sample for a tracking target pixel, and the AdaBoost discriminator is updated using another predetermined particle of the plurality of particles as a learning sample for a background pixel. With updating means,
An automatic tracking device characterized by that. A calculation unit that calculates a spatial weighted average value of the likelihoods of the plurality of particles, and a determination unit that determines whether or not the spatial weighted average value is equal to or greater than a predetermined value. The automatic tracking device according to claim 8. A camera capable of controlling pan, tilt and zoom;
Tracking processing means for performing tracking processing for tracking a tracking target based on an image captured by the camera;
Control means for controlling pan, tilt, and zoom of the camera so that the camera tracks and captures the tracking target according to the result of the tracking processing by the tracking processing means;
With
The tracking processing means includes position estimating means for estimating the position of the tracking target as a part of the tracking result by a particle filter using a plurality of particles whose pixel positions are based on an image captured by the camera. Including
The particle filter is a response value of an AdaBoost discriminator constructed to identify whether the pixel is a tracking target pixel or a background pixel based on one or more feature amounts related to the pixel for each particle. Then, the likelihood calculated from the response value by the one or more feature quantities related to the pixel at the position of the particle is used,
A calculation unit that calculates a spatial weighted average value of the likelihoods of the plurality of particles, and a determination unit that determines whether the spatial weighted average value is a predetermined value or more.
An automatic tracking device characterized by that. The position estimation unit estimates the position of the tracking target as a weighted average value using the likelihood of pixel positions that are states of the plurality of particles . automatic tracking device according to. The tracking processing means determines the size of the tracking target as another part of the tracking result based on the particles corresponding to the pixel identified as the tracking target pixel by the AdaBoost classifier among the plurality of particles. The automatic tracking device according to any one of claims 1 to 11 , further comprising a size estimation means for estimation. The control means includes a prediction means for predicting the position and size of the tracking target after a predetermined time has elapsed from the current time based on the result of the tracking processing by the tracking processing means,
The control means controls the pan, tilt and zoom of the camera by correcting a current pan, tilt and zoom control state for the camera according to a prediction result by the prediction means.
The automatic tracking device according to any one of claims 1 to 12 , 14. The automatic tracking device according to claim 13 , wherein the prediction means predicts the position and size of the tracking target after a predetermined time has elapsed from the present time by using a Kalman filter.