JP2006318345A - Object tracking method, program for object tracking method, recording medium where program for object tracking method is recorded, and object tracking device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent deterioration in detection precision due to noise by applying an object tracking method, a program for the object tracking method, a recording medium where the program for the object tracking method is recorded, and an object tracking device to, for example, a monitor device. <P>SOLUTION: Reliability indicating likelihood of a feature point on an object to be tracked is detected and a moving vector is processed based upon the reliability to be reflected on a processing result more as the reliability is higher, thereby detecting a movement vector of the moving body. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an object tracking method, a program for the object tracking method, a recording medium on which the program for the object tracking method is recorded, and an object tracking apparatus, and can be applied to, for example, a monitoring apparatus. The present invention detects a reliability indicating the probability of a feature point existing on a tracking target object, and processes a motion vector so that a motion vector having low reliability is not reflected in a processing result by this reliability. By detecting this movement vector, the deterioration of detection accuracy due to noise is prevented.

  Conventionally, a method for tracking a moving object by comparing a processing target frame and a next frame has been proposed for a monitoring device or the like, and in this method, by a comparison method between the processing target frame and the next frame, It is roughly divided into a pattern matching method and a feature point motion vector detection method.

  Here, tracking of a moving object by the pattern matching method is performed by calculating a correlation value using a pattern corresponding to a tracking target in each frame of a moving image, as proposed in, for example, Japanese Patent Laid-Open No. 5-298915. This is a method for detecting a location having the highest correlation, and sets a range in which such a correlation value is calculated by predicting motion. However, in the actual moving image, the shape of the moving body to be tracked changes from moment to moment, which makes it necessary to update the pattern used for detection of the moving body at high speed. Further, the calculation of the correlation value itself needs to execute complicated processing, and there is a drawback that it cannot be processed at high speed.

  On the other hand, tracking of a moving body by motion vector detection of a feature point detects a feature point that is a characteristic part in each frame of a moving image, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2003-44860. The moving object is tracked by detecting the motion vector of the feature point.

  In this method, a tracking target area is set, and a plurality of feature points are detected in the tracking target area. In addition, a motion vector is detected at each of the plurality of feature points, the tracking target area is moved by the average value of the detected motion vectors, and the moving body is tracked by these.

  However, in this method, the background may be included in the tracking target region, and the motion vector of the feature point detected in this background deteriorates the detection accuracy. When the tracking target is a person, the movement of the limb may be different from the movement of the body, and in this case, the movement of the limb deteriorates the detection accuracy. Hereinafter, the feature point that degrades the detection accuracy and the motion vector related to the feature point are appropriately referred to as noise.

Accordingly, it is required to prevent the detection accuracy from being deteriorated due to such noise.
Japanese Patent Laid-Open No. 5-298591 Japanese Patent Laid-Open No. 2003-44860

  The present invention has been made in consideration of the above points. An object tracking method capable of preventing deterioration in detection accuracy due to noise, a program for the object tracking method, a recording medium on which the program for the object tracking method is recorded, and object tracking The device is to be proposed.

  In order to solve such a problem, the invention of claim 1 is applied to an object tracking method for detecting a feature point in each frame of a moving image and tracking a moving object by detecting a motion vector of the feature point. A feature point setting step for setting a predetermined number of the feature points in the tracking target region set in the region where the moving body is imaged, and each frame based on the feature points set by the feature point setting step A repetitive processing step of sequentially processing the image data, and the repetitive processing step detects a reliability indicating the probability of the feature point existing on the moving body for each feature point. A motion vector detecting step for detecting the motion vector to the next frame of the feature point, and the next frame of the moving body based on the motion vector. A movement vector detecting step for detecting a movement vector to a frame, and the tracking target area is moved by the movement vector detected in the movement vector detecting step to set the tracking target area in the next frame. A tracking target region moving step for setting the processing target frame in the subsequent repeated processing of the next frame, and the moving vector detecting step processes the motion vector having low reliability according to the reliability. The motion vector is detected so as not to be reflected in the result, and the movement vector is detected.

  The invention of claim 11 is applied to a program of an object tracking method for detecting a feature point in each frame of a moving image and tracking the moving body by detecting a motion vector of the feature point. A feature point setting step for setting a predetermined number of the feature points in the tracking target region set in the imaged region, and image data of each frame based on the feature points set by the feature point setting step Repetitive processing step for sequentially processing, and the repetitive processing step detects a reliability indicating the probability of the feature point existing on the moving body for each feature point, and A motion vector detecting step for detecting the motion vector to the next frame of the feature point, and the next frame of the moving body based on the motion vector; A movement vector detection step for detecting a movement vector of the second movement vector, and the movement vector detected in the movement vector detection step to move the tracking target area to set the tracking target area in the next frame. A tracking target region moving step for setting the frame as the processing target frame in the subsequent repetitive processing, and the moving vector detection step reflects the motion vector having low reliability in the processing result according to the reliability. The motion vector is processed so as not to detect the movement vector.

  According to the invention of claim 12, the object tracking is applied to a recording medium on which a program of an object tracking method for detecting a feature point in each frame of a moving image and tracking a moving object by detecting a motion vector of the feature point is recorded. The program is set by a feature point setting step for setting a predetermined number of feature points in a tracking target region set in a region where the moving object of the processing target frame is imaged, and the feature point setting step. A repetitive processing step for sequentially processing the image data of each frame based on the feature points, and the repetitive processing step has a reliability indicating the probability of the feature points existing on the moving object. A reliability detection step for detecting point by point, a motion vector detection step for detecting the motion vector to the next frame of the feature point, and the motion A movement vector detection step for detecting a movement vector to the next frame of the moving body based on the vector, and the movement vector detected by the movement vector detection step moves the tracking target area, and The tracking target area is set in a frame, and the tracking target area moving step is set in the processing target frame in the repetitive processing of continuing the next frame, and the movement vector detecting step is performed according to the reliability. Then, the motion vector is processed so that the motion vector with low reliability is not reflected in the processing result, and the movement vector is detected.

  The invention of claim 13 is applied to object tracking in which a feature point is detected in each frame of a moving image by repeating a predetermined repetition process, and a moving object is tracked by detecting a motion vector of the feature point. Feature point setting means for setting a predetermined number of the feature points in the tracking target region set in the region where the moving body of the frame is imaged, and reliability indicating the probability of the feature points existing on the moving body Reliability detection means for detecting the degree of each feature point; motion vector detection means for detecting the motion vector to the next frame of the feature point; and the next frame of the moving body based on the motion vector The movement vector detecting means for detecting the movement vector to the position and the movement vector detected by the movement vector detecting means move the tracking target area, and The tracking target area is set in the next frame, and the tracking target area moving means is set in the processing target frame in the subsequent repetitive processing of the next frame, and the movement vector detecting means corresponds to the reliability. Then, the motion vector is processed so that the motion vector with low reliability is not reflected in the processing result, and the movement vector is detected.

  According to the configuration of claim 1, a feature point is detected in each frame of a moving image, and the moving body of the processing target frame is imaged by applying to an object tracking method of tracking the moving body by detecting a motion vector of the feature point. A feature point setting step for setting a predetermined number of the feature points in the tracking target region set in the region, and sequentially processing the image data of each frame based on the feature points set by the feature point setting step A repetitive processing step, wherein the repetitive processing step detects a reliability indicating the probability of the feature point existing on the moving body for each feature point, and A motion vector detecting step for detecting each of the motion vectors to the next frame, and a moving vector of the moving body to the next frame based on the motion vectors; The movement vector detection step for detecting the movement vector and the movement vector detected in the movement vector detection step move the tracking target area, set the tracking target area in the next frame, and continue the next frame A tracking target region moving step set in the processing target frame in the repetitive processing, and the moving vector detecting step does not reflect the motion vector having low reliability in the processing result according to the reliability. If the motion vector is processed to detect the movement vector, noise due to the background or the like can be prevented from being reflected in the movement vector. Thereby, deterioration of detection accuracy due to noise can be prevented.

  According to the configurations of claims 11, 12, and 13, the object tracking method program capable of preventing the deterioration of detection accuracy due to noise, the recording medium storing the object tracking method program, and the object A tracking device can be provided.

  According to the present invention, it is possible to prevent deterioration in detection accuracy due to noise.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate.

(1) Configuration of Embodiment (1-1) Overall Configuration FIG. 1 is a block diagram illustrating an object tracking system according to Embodiment 1 of the present invention. In this object tracking system 1, the signal source 2 is various devices that output image data SV to be processed. For example, an imaging device that outputs an imaging result as image data SV, an imaging result recorded by the imaging device, and image data The recording / reproducing apparatus etc. which output by SV are comprised.

  The object tracking device 3 is configured by, for example, a computer, a dedicated device having a hardware configuration, etc., processes the image data SV from the signal source 2 to track the moving body, and outputs a tracking result S1. That is, when the object tracking device 3 is configured by a computer, the work tracking device 3 starts up the operation by securing a work area in the random access memory (RAM) 5 according to the recording in the read only memory (ROM) 4 and the central processing unit (CPU) 6. Thus, a processing program related to object tracking recorded in the hard disk device (HDD) 7 is executed. Thus, the object tracking device 3 processes the image data SV input via the interface (I / F) 8 in real time, or once records the data in the hard disk device 7, and then processes the image data SV to process the object tracking. The tracking result S1 is output. Further, an image or the like of the process is displayed on the monitor 9 as necessary. In this embodiment, the processing program related to the object tracking is provided by being installed in advance in the object tracking device 3, but instead of such a prior installation, it can be downloaded through a network such as the Internet. You may make it provide, You may make it provide by recording on various recording media, such as an optical disk, a magnetic disc, and a memory card.

  The processing device 10 executes various processes related to the object tracking system 1 based on the tracking result S1. Note that this process is a process according to the system to which the object tracking system 1 is applied. For example, a process of panning, tilting, and zooming the imaging device so as to track the moving body based on the tracking result S1. The focus control is performed so that the area where the tracking target moving body is imaged is just-focused based on the tracking result, and the area where the tracking target moving body is imaged is extracted based on the tracking result.

  FIG. 2 is a flowchart showing the processing procedure of the central processing unit 6 by the processing program related to object tracking in the object tracking device 3. When the central processing unit 6 is instructed to execute this processing program by an operator's instruction or the like, the central processing unit 6 starts this processing procedure and proceeds from step SP1 to step SP2.

  Here, the central processing unit 6 acquires the image data SV of the current frame and the previous frame, and detects the moving object that is the tracking target by comparing the image data SV of the current frame with the image data SV of the previous frame. Here, the previous frame is a frame to be processed in this embodiment, and the current frame is a frame following the previous frame. Various methods can be applied to this moving object detection method, and the background subtraction method is applied in this embodiment. Here, the background difference method is performed by calculating the inter-frame difference value between the current frame and the previous frame for each pixel, for each predetermined pixel, and for each block of the predetermined pixel. This is a process for detecting a moving pixel or block and detecting a region where the moving body is imaged.

  When the mobile unit is detected in this way, the central processing unit 6 moves to step SP3, and executes the tracking target area setting process and the feature point setting process by the initialization process. The central processing unit 6 sets the tracking target area for tracking the moving object detected in step SP2 in the previous frame by the tracking target area setting process. In this embodiment, as shown in FIG. 3, by setting the rectangular frame W so as to surround the moving body based on the position information of the region detected in step SP2, the region surrounded by the frame W is tracked. Set to the target area. In this process, the central processing unit 6 sets a frame W for each moving object detected in step SP2, and sets a tracking target area for each moving object. Note that the setting of the tracking target area is not limited to the rectangular shape, but according to the shape of the moving body to be tracked, such as an elliptical shape or a triangular shape, the arithmetic processing capability of the central processing unit 6, etc. It can be set according to various shapes. Instead of setting the tracking target area by automatic detection of such a moving body, the tracking target area may be set by an input by an operator.

  Further, the central processing unit 6 detects a predetermined number of feature points in each tracking target region by the feature point setting process in step SP3. Here, various methods such as a KLT (Kanade-Lucas Track) method, a SUSAN method, a Harris operator method, and the like can be applied to the feature point setting process. In this embodiment, feature points are set using the KLT method, and the feature point tracking process in the subsequent step SP4 is also executed. Incidentally, the KLT method is a method of detecting the motion of the feature point detected by the gradient method after detecting and setting the feature point.

  Subsequently, the central processing unit 6 moves to step SP4, selects one of the tracking target areas set in step SP3, and for each feature point included in the selected tracking target area, from the position coordinates of the previous frame, the current frame. The movement of each feature point is detected by detecting the corresponding position coordinates. Specifically, the central processing unit 6 calculates the k-th feature point corresponding to the current frame n + 1 represented by the equation (2) with respect to the position coordinates of the k-th feature point of the previous frame n represented by the equation (1). Detect position coordinates.

Subsequently, as shown by the following equation, the central processing unit 6 calculates a difference value between the position coordinates of the corresponding feature points detected in the previous frame and the current frame, thereby obtaining a motion vector VFP _k of each feature point. calculate. Note that various methods such as a block matching method can be widely applied to detect the motion vector VFP _k .

  In the subsequent step SP5, the central processing unit 6 detects a movement vector, which is a vector indicating the movement of the moving body, from the motion vectors of the feature points detected in this way. At this time, the central processing unit 6 executes noise countermeasure processing, thereby preventing a reduction in detection accuracy due to noise.

  That is, as shown in FIG. 4, the tracking target area set in step SP3 in this way includes not only the tracking target moving body but also the foreground such as a stationary object and the background of a swaying tree. As a result, the motion vector detected in step SP4 includes not only the feature point P of the moving object but also the motion vector of the feature point P in the foreground and background. As a result, the motion vectors due to the foreground and background feature points become noise when calculating the motion of the moving object. When the tracking target is a person, the movement of the limb may differ from the movement of the body. In this case, the motion vector of the feature point set for the limb is used to calculate the movement of the moving body. Noise.

  For this reason, the central processing unit 6 detects the possibility of such noise by the reliability indicating the probability of the feature point existing on the tracking target object, and processes a motion vector having low reliability by this reliability. A motion vector is detected by processing the motion vector of each feature point so as not to be reflected in the result. For motion vectors with low reliability, feature points are excluded from processing in subsequent frames. In this embodiment, the probability of a feature point existing on such an object to be tracked includes the probability that the feature point exists on this object and the probability of the feature point itself as will be described later. To detect.

  Subsequently, the central processing unit 6 moves to step SP6 and executes shielding countermeasure processing. That is, when a person or the like is actually tracked, the whole or a part of the tracking target may be shielded by the front structure or person. In this case, depending on the conventional tracking method, the movement of the tracking target cannot be detected correctly. Therefore, the central processing unit 6 detects the degree of occlusion indicating the degree to which the tracking target is occluded in this step SP6, and the movement vector obtained in step SP5 based on this detection result is detected in the past frame. Correction is made with reference, thereby preventing the influence of shielding and detecting the movement of the moving object.

  In the subsequent step SP7, the central processing unit 6 corrects the shift of the movement vector with respect to the moving body. In other words, the motion vector detection based on feature points can accurately track the motion of a local motion, but does not necessarily reflect the motion of the tracking target. As a result, when the moving object is tracked by statistically processing motion vectors detected at a plurality of feature points and moving the tracking target area as in this embodiment, the tracking target area is detected with respect to the movement of the moving object. There is a case in which the tracking target area gradually shifts with respect to the moving object due to the accumulation of the errors.

  As a result, the central processing unit 6 corrects the displacement with respect to the moving body with respect to the moving vector obtained from the plurality of feature points in this step SP7, so that the moving body can be traced reliably.

  In the subsequent step SP8, the central processing unit 6 moves the frame W set as the previous frame by the shift corrected movement vector, thereby setting the tracking target region in the current frame. Further, the position information of the tracking target area set in the current frame is output to the subsequent processing device 10 as the tracking result S1. At this time, feature points that are not included in the tracking target area are deleted.

  In the subsequent step SP9, the central processing unit 6 additionally sets feature points for the tracking target region set in step SP8 by the amount of the feature points deleted in step SP5 and step SP8, thereby setting this tracking target region. The number of feature points thus set is set to the initial number set in step SP3.

  The central processing unit 6 thereby completes the processing of the previous frame for one tracking target area, and determines whether or not the processing has been completed for all the tracking target areas set in the previous frame in the subsequent step SP10. . If a negative result is obtained here, the central processing unit 6 moves from step SP10 to step SP11, switches the processing target to another tracking target region set in the previous frame, and then returns to step SP4. Thereby, the central processing unit 6 repeats the processing procedure of steps SP4-SP5-SP6-SP7-SP8-SP9-SP10-SP11-SP4 for each tracking target area set in the previous frame, and is set in the previous frame. When this series of processing is completed for all the tracking target areas, an affirmative result is obtained in step SP10, and the process proceeds from step SP10 to step SP12.

  In step SP12, the central processing unit 6 determines the presence or absence of the next frame. If the next frame exists in the image data SV to be processed, the process proceeds from step SP12 to step SP13, and the current frame up to that point is changed to the previous frame. In addition to setting the frame, the image data SV of the next frame is acquired and set to the current frame, and the process returns to step SP4. Thereby, the central processing unit 6 repeats this processing procedure for each frame of the image data SV, and when the processing is completed for all the frames, a negative result is obtained in step SP12, so that the process proceeds from step SP12 to step SP14. This processing procedure is completed.

(1-2) Movement vector calculation processing (noise countermeasure processing)
FIG. 5 is a flowchart showing in detail the movement vector calculation process according to step SP5 of FIG. When starting this processing procedure, the central processing unit 6 proceeds from step SP21 to step SP22, and calculates the reliability Ck of each feature point.

  Here, the reliability Ck is a value indicating the probability of the feature point k existing on the tracking target moving body, and is represented by the following equation. Accordingly, the closer the value Ck is to 1, the higher the possibility that the moving object to be tracked is set. When the value C is 0, this feature point is a feature point set in the background or the like. Is set to indicate that

  In this step SP22, the central processing unit 6 calculates a plurality of types of reliability for each feature point based on different detection principles, and in the subsequent step SP23, deletes feature points having low reliability, thereby obtaining this reliability. Are excluded from the calculation target of the movement vector and the processing of the subsequent frames. Thus, the central processing unit 6 simplifies the processing by reducing the number of feature points used for the processing. In the subsequent step SP24, the central processing unit 6 calculates the movement vector of the moving object by processing the motion vectors of the remaining feature points using a plurality of types of reliability, and then proceeds to step SP25 to return to the original processing vector. Return to the procedure. Thus, the central processing unit 6 prevents the influence of noise and prevents the detection accuracy from deteriorating.

  In the processing at step SP22, the central processing unit 6 first calculates the reliability Ck based on the number of tracking times at step SP22-1. Here, in this embodiment, the feature point is deleted by the deletion process of step SP23 of the movement vector calculation process, and further by the process of step SP8, and the feature point is added in step SP9 as much as the feature point is deleted. By repeating a series of processing procedures for each frame while setting, the feature points that are correctly set on the moving object and that are not subject to occlusion etc. are consecutive in consecutive frames. Thus, the motion vector is detected and used repeatedly to detect the motion vector of the moving body. In other words, such feature points are continuously tracked in successive frames.

  Thus, it can be said that there is a high probability that the feature point that has been continuously tracked many times in this way is present on the moving body. Thereby, in this embodiment, the central processing unit 6 sets a count value indicating the number of times of repeated tracking in this series of processing for each feature point, and additional setting in step SP9 by setting the feature point in step SP2. Thus, the count value of the corresponding feature point is set to the count value (value 0) by the initial setting. For feature points that have not been deleted even after the tracking target area is moved in step SP9, the count value is incremented by one.

  In this way, the central processing unit 6 initializes and increments the count value, and in the processing of step SP22-1, the central processing unit 6 increases the value according to the count value as shown in FIG. The reliability ωc according to the number of times of tracking is set so that increases.

  In this embodiment, when the count value is near the value 0 and smaller than the predetermined threshold value t1, the reliability ωc according to the number of times of tracking is set to the value 0, and the count value is a sufficiently large value. When it is larger than the threshold value t2, the reliability ωc according to the number of times of tracking is set to 1. In the range from the threshold value t1 to t2, the reliability ωc according to the number of tracking is set by linear interpolation using a linear function so that the value increases as the count value increases.

  Subsequently, the central processing unit 6 moves to step SP22-2 and calculates the temporal reliability. Here, the temporal reliability indicates the probability that the feature point exists on the moving object by determining the difference vector between the moving vector of the moving object detected in the past frame and the motion vector detected in step SP4. Is. In this embodiment, the movement vector detected in the immediately preceding frame is applied to the movement vector of the moving body detected in the past frame.

  That is, when the x-direction component Vx and the y-direction component Vy of the motion vector are set to a two-dimensional coordinate axis, and the motion vector detected at each feature point is represented in the coordinate space by this coordinate axis, as shown in FIG. The motion vectors detected at the points will form a group. Here, in the feature point group G1 in which both the x-direction component Vx and the y-direction component Vy are close to the value 0, the feature points set as a stationary object such as a background (corresponding to an obstacle in FIG. 4). It is determined to be a motion vector. On the other hand, in this way, the x-direction component Vx and the y-direction component Vy are not near the value 0, and the group G2 having a larger number than the others is a motion vector of feature points set in the moving body. Similarly, the group G3 in which both the x-direction component Vx and the y-direction component Vy are not close to the value 0 and is smaller than the group G2 is another mobile unit included in the tracking target region (FIG. 4). In this case, it is determined that the motion vector is a feature point motion vector set to “swaying tree”.

  In such a distribution, when the moving object is actually imaged, the moving object continuously moves in successive frames, so that the motion vector existing on the moving object has a time axis. It can be determined that the change in direction is continuously changing. As a result, it can be determined that a motion vector that is significantly different from the motion vector of the moving body one frame before is less likely to exist on the moving body.

That is, when the motion vector of each feature point obtained in the previous frame n is VFR _k [n] and the moving body inertial movement vector obtained in the previous frame n−1 is V _obj [n−1], When this feature point is set in the moving body, the motion vector VFR _k [n] should have a similar value to the inertial movement vector V _obj [n−1]. Here, the moving body inertial movement vector, V _obj, is a movement vector obtained by a shielding process, which will be described later. If the moving body is not shielded, the movement vector V _pres [n -1].

As a result, as indicated by the group G3 in FIG. 7, the motion vector VFR _k [n] of the feature point set in the moving body is based on the moving body inertial movement vector V _obj [n−1] detected in the immediately preceding frame. It can be said that the points P1 are distributed about the center.

As a result, as shown in FIG. 8, the central processing unit 6 uses the inertial movement vector V _obj [n−1] detected in the immediately preceding frame as a reference and uses the x-direction component Vx and the y-direction component Vy as 2 In the dimensional space, the temporal reliability ωt is set according to the distance r from the point P1 by the moving body inertial movement vector V _obj [n−1].

In this embodiment, this distance r is determined by three threshold values th1, th2, and th3, and when the distance r is smaller than the smallest threshold value th1, that is, the moving body inertial movement detected one frame before. If the following relational expression is established between the vector V _obj [n−1] and the motion vector VFP _k detected at the feature point, the temporal reliability ωt is set to a value 1.

When the distance r is greater than the threshold value th3 having the largest value, that is, the moving body inertial motion vector V _obj [n−1] detected one frame before and the motion vector VFP _k detected at the feature point. If the following relational expression holds, the feature point is set as a deletion target, the feature point is not used for the calculation of the movement vector, and is excluded from the subsequent frame processing.

In contrast, when the distance r is larger than the threshold value th2 having a large value and the distance r is smaller than the threshold value th3 having a large value, that is, the moving body inertial movement vector V _obj [ n−1] and the motion vector VFP _k detected at the feature point, the temporal reliability ωt is set to 0 when the following relational expression holds.

On the other hand, when the distance r is larger than the threshold value th1 having the smallest value, and subsequently the distance r is smaller than the threshold value th2 having the smallest value, that is, the moving body inertial movement vector V _obj [ n−1] and the motion vector VFP _k detected at the feature point, when the following relational expression is established, the value gradually increases from the value 1 by increasing the distance r by linear interpolation using a linear function. The temporal reliability ωt is set so as to decrease to zero.

Thus, when setting the temporal reliability ωt for the feature point when the y-direction component Vy is 0, the central processing unit 6 sets the temporal reliability ωt according to the characteristics shown in FIG. become. In the determination of the motion vector based on the distance r, instead of the determination using a circle as shown in FIG. You may make it perform by determining the distribution with respect to the moving body inertial movement vector _Vobj [n-1] of the moving body detected before the flame | frame. Specifically, for example, in the case of tracking a person, the movement in the horizontal direction is almost the case. In this case, the horizontal direction is determined by an ellipse having a long axis, and the horizontal direction is determined. In contrast, the determination criteria in the vertical direction are stricter, and the reliability setting accuracy can be improved.

  In addition, when the average value of the motion vectors detected in the immediately preceding multiple frames is applied instead of the motion vector detected in the immediately preceding frame, or when the motion vector detected two frames before is applied, the difference value is used. In the movement vector used for generating the above, various movement vectors detected in the past can be applied as necessary.

  FIG. 10 is a diagram showing a series of processing procedures of the central processing unit 6 relating to the setting of the temporal reliability ωt. In this embodiment, the central processing unit 6 accepts the settings of the threshold values th1 to th3 by the operator's prior settings. In accepting such a setting, for example, a series of processing is executed in advance by setting a standard threshold value to display a change in the tracking target area (frame w), and the operator can change the setting. The case where it accepts etc. can be considered.

  When the temporal reliability ωt is calculated in this way, the central processing unit 6 determines the change in the distance to other feature points in subsequent step SP22-3, and totals the determination results to determine the space. The reliability ωs is calculated.

  Here, as shown in FIGS. 11A1 and 11A2, when the feature points FP1 to FP5 detected in the n frame are moved to the position of the subsequent n + 1 frame by the motion vectors V1 to V5, respectively, When the feature point FP1 shows a different movement from the remaining other feature points FP2 to FP5, there is a high possibility that this one feature point FP1 is not set as the tracking target moving body.

  On the other hand, as shown in FIGS. 11B1 and 11B2, even if one feature point FP1 shows a different movement from the other feature points FP2 to FP5 as described above, When one feature point FP1 is a movement related to the remaining other feature points FP2 to FP5, it is highly likely that these feature points FP1 to FP5 are set to the same moving body. . Incidentally, FIGS. 11B1 and 11B2 show a case where the mutual distance between the feature points FP1 to FP5 is widened, and one feature point compared to the distance between the other feature points FP2 to FP5. This is a case where these five feature points FP1 to FP5 are moved in the same direction while the gap between the FP1 and the FP1 is widened. For example, when the moving object to be tracked is a person, the person is stretched while moving. This is the case.

  On the other hand, as shown in FIGS. 11 (C1) and (C2), when one feature point FP1 moves differently from the remaining other feature points FP2 to FP5, When the relative positional relationship between the feature points FP2 to FP5 is not maintained, there is a high possibility that this one feature point FP1 is not set as the tracking target moving body.

  However, in these examples of FIGS. 11A1 and 11C2, the other feature points FP2 to FP5 are set as tracking target moving bodies because they move while maintaining their mutual positional relationship. There is a high possibility.

  As a result, as shown by the following equation, the central processing unit 6 determines a change in distance to another feature point for each feature point based on a predetermined threshold thΔ, and exceeds the threshold thΔ. The number of feature points that change greatly is counted at each feature point. In addition, the subtracted value obtained by subtracting the number of feature points to be processed (value 1) from all the feature points is divided and normalized by dividing the count value thus calculated, thereby calculating the elasticity index k of the kth feature point. To do. In equation (9), FP1 and FPk are the feature points to be processed and the coordinate values of other feature points, respectively, and n−1 and n indicated by subscripts indicate the previous frame and the current frame, respectively. In addition, # indicates a count value.

  Although this threshold thΔ is a constant in this embodiment, a value is set for each feature point depending on the magnitude of the motion vector at the feature point to be processed, the processing results in steps SP22-1 and SP22-2, and the like. It may be changed.

Specifically, the relationship between the first feature point FP1 and the fifth feature point FP5 in FIGS. 11B1 and 11B2 will be described. The fifth feature point from the first feature point FP1 in the nth frame. Between the distance δ _{n, 1} (5) to FP5 and the distance δ _{n, 5} (1) from the fifth feature point FP5 to the first feature point FP1 in the nth frame, δ _{n, 1} ( 5) = δ _{n, 5} The relational expression (1) holds, and the distance δ _{n + 1,1} (5) from the first feature point FP1 to the fifth feature point FP5 in the subsequent n + 1 frame, and these Comparing the distance δ _{n, 1} (5) or δ _{n, 5} (1), it can be seen that the distance δ _{n + 1,1} (5) is extended in this case.

Thus, the central processing unit 6 in this case, when the difference value between these distances δ _{n + 1,1} (5) and δ _{n, 1} (5) is larger than the threshold thΔ, as shown by the following equation: The count value counterFP1 related to the feature point FP1 is up-counted.

  In this way, the difference value of the distance is similarly determined from the threshold value thΔ between the remaining feature points FP2 to FP4, and the count value is updated. In the example of FIG. The elastic index k is calculated by dividing the count value by the value 4 obtained by subtracting 1 from the value. Further, as shown by the following equation, the elastic index k is subtracted from the value 1, thereby calculating the spatial reliability ωs.

  FIG. 12 is a diagram showing a series of processing procedures of the central processing unit 6 relating to the setting of the spatial reliability ωs. Thus, the central processing unit 6 records and holds information on the distance between each feature point detected in the immediately preceding frame in a memory. The central processing unit 6 is elastic by recording the memory and information on the distance detected in the previous frame. After calculating the index, the spatial reliability ωs is calculated for each feature point.

  When the spatial reliability ωs is calculated in this way, the central processing unit 6 executes a stationary point detection process in subsequent step SP22-4. Here, FIG. 13 is a flowchart showing a processing procedure of the central processing unit 6 in the stationary point detection processing. When starting the processing procedure, the central processing unit 6 proceeds from step SP31 to step SP32. Here, the central processing unit 6 determines whether or not the moving body is stationary. If it is determined that the moving body is stationary, the central processing unit 6 proceeds from step SP32 to step SP33 and ends this processing procedure. Here, in determining whether or not the moving body is stationary, the movement vector is calculated based on various reliability levels detected in Step SP22-1 to Step SP22-4 or a combination of these reliability levels. For example, various determination methods can be widely applied. When the central processing unit 6 moves from step SP32 to step SP33 and completes the processing in this way, the count value is set to 0 for the still image counter (described later) related to each feature point of the tracking target area. Reset.

  On the other hand, if the determination result that the moving body is not stationary is obtained, the central processing unit 6 proceeds from step SP32 to step SP34. Here, the central processing unit 6 selects one feature point set in the tracking target region to be processed, and determines whether or not this feature point is located at the same position as one frame before. Here, in this determination, the difference value between the position (xk, yk) n detected one frame before and the position (xk, yk) n + 1 detected in the processing target frame which is the previous frame is calculated. Judgment is made based on a predetermined threshold value, and when the difference value is smaller than this threshold value, the same position is determined.

  If the determination result that the central processing unit 6 is at the same position cannot be obtained, the central processing unit 6 proceeds from step SP34 to step SP35. Here, the central processing unit 6 resets the stillness counter k indicating the number of frames that have been still to 0, and then proceeds to step SP36. Here, the central processing unit 6 determines whether or not the processing has been completed for all the feature points in the tracking target area. If a negative result is obtained here, the process proceeds from step SP36 to step SP37, and the processing target is set to the next processing target. Switching to the feature point returns to step SP34.

  On the other hand, if a positive result is obtained in step SP34, the central processing unit 6 proceeds from step SP34 to step SP38. Here, the central processing unit 6 increments the stationary counter k by the value 1, and then determines whether or not the count value of the stationary counter k coincides with a predetermined threshold value th5 in the following step SP39.

  If an affirmative result is obtained here, in this case, the feature point to be processed is the same in consecutive frames corresponding to the threshold th5, regardless of whether the moving body is moving. In this case, the central processing unit 6 moves from step SP39 to step SP40, and sets this feature point as a stationary point. Subsequently, the process proceeds to step SP36, where it is determined whether or not the processing has been completed for all the feature points in the tracking target region. If a negative result is obtained here, the process proceeds from step SP36 to step SP37, where the processing target is changed. Switch to the next feature point and return to step SP34.

  On the other hand, if a negative result is obtained in step SP39, the central processing unit 6 proceeds from step SP39 to step SP41. Here, the central processing unit 6 determines whether or not the spatial reliability ωs related to this feature point detected in step SP22-3 is smaller than a predetermined threshold th6, that is, for many other feature points. It is determined whether the degree of exhibiting different movements is large.

  If a positive result is obtained here, the central processing unit 6 moves from step SP41 to step SP40, sets this feature point as a stationary point, and then moves to step SP36. On the other hand, if a negative result is obtained in step SP41, the process directly proceeds from step SP41 to step SP36.

  Thus, the central processing unit 6 detects the reliability of the stationary point indicating the probability that the characteristic point exists on the moving object by the number of times it has been repeatedly stationary, by determining the position coordinates of the characteristic point that is repeatedly detected. To do.

  When the central processing unit 6 detects the still point as described above, the central processing unit 6 executes smooth block processing in the following step SP22-5. Here, the central processing unit 6 calculates, for each feature point, a difference value between the sampling values with the surrounding sampling points. Here, when the difference value is small, the feature point has low reliability as the feature point. Thereby, the central processing unit 6 detects a feature point whose difference value is equal to or smaller than a certain value. In this process, for example, the difference absolute value sum of the sampling values is calculated between the sampling points related to the feature points and the sampling points adjacent to the sampling points in the horizontal and vertical directions. The sum is determined by a threshold and executed. It should be noted that, instead of such processing between sampling points adjacent in the horizontal direction and vertical direction, when processing between eight sampling points in the vicinity, etc. Various settings can be made.

  As a result, the central processing unit 6 determines the probability of the feature points existing on the moving object by determining the difference value of the sampling values with respect to the surrounding sampling points for each feature point. The smoothness reliability shown by comparison with is detected.

  Thus, the central processing unit 6 deletes feature points with low reliability by using a part of the plurality of types of reliability detected by the different detection principles in the following step SP23. The low feature points are not used for the calculation of the movement vector, and are excluded from the processing target of subsequent frames. Further, the feature points are additionally set in the above-described step SP9 because the feature points are deleted in this way. As a result, the central processing unit 6 detects the motion vector so as not to reflect the motion vector with low reliability in the processing result by the feature point deletion processing in addition to the weighted average processing described later. The process related to the detection of the movement vector is simplified.

  Specifically, the central processing unit 6 determines, from the feature points of the tracking target area that is the processing target, the feature points set as deletion targets in step SP22, the still points detected in step SP22-4, and the step SP22-5. The detected feature point with low smoothness reliability is deleted.

  The deletion of the feature points is not limited to the case of deleting by these still points and smoothness reliability, but may be deleted by other reliability instead of or in addition to these. In addition, it is also possible to delete using all of a plurality of types of reliability detected by different detection principles. Further, for each reliability used for the deletion process, the corresponding feature point is deleted immediately after each reliability is calculated, so that the subsequent reliability detection process can be simplified.

  Further, the central processing unit 6 calculates the movement vector by weighting processing using a part of the plurality of types of reliability detected by the different detection principles in the subsequent calculation of the movement vector in step SP24. In this embodiment, the tracking reliability ωc, the temporal reliability ωt, and the spatial reliability ωs are applied to the partial reliability. In addition, a weighted average process is applied to this weighting process.

  As a result, in this step SP24, the central processing unit 6 calculates the total reliability ω by multiplying the tracking number reliability ωc, the temporal reliability ωt, and the spatial reliability ωs as shown by the following equation.

  Further, a weighted average value using the calculated total reliability ω is calculated, thereby calculating a motion vector of the moving object in the tracking target region which is the processing target. As a result, in this embodiment, the influence of noise is prevented, and deterioration of detection accuracy is prevented.

  The central processing unit 6 performs a shielding countermeasure process on the movement vector obtained in this way at step SP6, performs a shift correction process at step SP7, and then moves the frame W at step SP8 to set the tracking target area in the current frame. To do. At this time, feature points not included in the tracking target area are further deleted. Further, the feature points are additionally set in step SP9 for the portion deleted in step SP8.

  Note that the determination in step SP32 described above is based on the movement vector detected by repeating this series of processes, and if the movement vector has been continuously below the predetermined value, it is determined that the moving body is stationary.

(1-3) Shielding Process FIG. 14 is a schematic diagram for explaining the shielding. The shielding occurs when the moving body M crosses behind the shielding object N, and in this example of FIG. 14, feature points (indicated by black circles) set in the moving body from the successive n frames to n + 5 frames. Are sequentially changed depending on the degree of overlap between the shield N and the moving body M.

  As a result, as shown in FIG. 15A, the number of feature points set in the moving body gradually decreases as the shielding starts, and then gradually increases as the shielding ends. It will return to the value before shielding. That is, in this case, in the above-described processing for detecting the reliability ωc of the number of times of tracking, feature points with a low number of times of tracking ωc having a tracking number of less than or equal to the threshold value t1 increase. Feature points with a high tracking number reliability ωc that are equal to or greater than the value t2 decrease.

  Accordingly, the degree of shielding indicating the degree of shielding can be detected by obtaining the ratio of the number of feature points that have been successfully tracked for a predetermined number of times or more with respect to the number of feature points in the tracking target region. In this embodiment, feature points with low reliability are deleted by repeated frame processing, and feature points are additionally set for the deleted amount, so that tracking has succeeded more than a predetermined number of times in this way. The feature points that have been added are feature points that have not been additionally set in the repeated processing more than the predetermined number of times.

  As a result, the central processing unit 6 calculates the movement detected in step SP5 by calculating the movement vector detected in step SP5 and the movement vector detected in the past repetition process according to the degree of shielding. Correct the vector and track moving objects that are occluded.

  That is, FIG. 16 is a flowchart showing the processing procedure of the central processing unit 6 according to this shielding measure. When starting this processing procedure, the central processing unit 6 moves from step SP51 to step SP52 and detects the degree of shielding. Here, as shown by the following equation, the central processing unit 6 calculates the number of feature points whose number of tracking is a predetermined number or more, and subtracts the calculated number of feature points from a predetermined reference value th7 (FIG. 15B), The subtraction value obtained as a result is divided by the reference value th7 and normalized (FIG. 15C), and the shielding degree is detected. Here, V4.0 is the number of feature points whose tracking number is a predetermined number or more. It should be noted that instead of the number of feature points having a tracking number of more than a predetermined number, the degree of occlusion may be detected by detecting the number of feature points having a tracking number reliability ωc of a predetermined value or more. In this embodiment, the reference value th7 is set to a value that is ½ of the number of feature points set in the tracking target area.

  As a result, as shown in FIG. 15C, the central processing unit 6 has the number of feature points whose number of tracking is equal to or more than a predetermined number of times or more than half of the number of feature points held without being deleted in the tracking target region. In the range, the tracking is performed when the number of feature points of which the number of tracking is equal to or greater than the predetermined number of times is less than ½ of the number of feature points that are not deleted in the tracking target area, so that the value is maintained at a value of 0. The degree of occlusion is calculated so that the value gradually increases from the value 0 as the number of feature points decreases by a predetermined number of times or more, and the value of 1 that causes the number of tracking points to be zero when the number of tracking times is a predetermined number or more. .

  Subsequently, in step SP53, the central processing unit 6 calculates a moving body inertial movement vector using this shielding degree. Here, the moving body inertial movement vector is a motion vector indicating the movement of the moving body when the moving body is moving due to inertia, and is calculated by the following arithmetic processing. Here, the movement vector is the movement vector calculated in step SP5. The past movement vector is a movement vector detected in the past frame of the moving body. For example, a motion vector of the moving body detected one frame before, a motion vector of the moving body detected ten frames ago, and the like. Applied. Α is a parameter for suppressing the feedback rate, and in this embodiment, the degree of shielding is substituted. As a result, the central processing unit 6 changes the weighting coefficient in accordance with the degree of occlusion, and detects in step SP5 by the weighted average process of the movement vector detected in step SP5 and the movement vector detected in the past iteration process. Correct the movement vector.

  As is clear from the equation (14), in this embodiment, the moving object motion vector based on the feature points and the moving function motion vector detected in the past one frame are used for shielding by a linear function calculation process. Although the case of calculating the motion of the moving body described above has been described, the motion of the moving body may be calculated from motion vectors calculated in a plurality of past frames by a calculation process such as a quadratic function.

  As a result, the central processing unit 6 tracks the moving body by the movement vector detected by the motion vector of the feature point until the moving body is shielded by a certain value or more. The movement vector detected by the motion vector is corrected using the movement vector detected in the past by the feedback rate of α). Therefore, the central processing unit 6 sets the moving body inertial movement vector as the motion vector of the moving body, and then moves to step SP54 to end the processing procedure.

(1-4) Deviation Correction Processing By the way, the motion vector detection of each feature point can accurately track the movement of a local movement, but does not necessarily reflect the movement of the tracking target correctly. Thus, when the tracking target area is moved by statistically processing motion vectors detected at a plurality of feature points as in this embodiment, the movement of the tracking target area has an error with respect to the movement of the moving object. In FIG. 17, as shown in FIG. 17, the tracking target region may gradually shift with respect to the moving object due to the accumulation of errors, as shown in the relationship between the moving object in the consecutive N frames and the tracking target region by the frame W. There will be a gradual decrease in tracking accuracy.

  Therefore, the central processing unit 6 corrects such a shift by a shift correction process shown in FIG. That is, when starting this processing procedure, the central processing unit 6 proceeds from step SP61 to step SP62. Here, the central processing unit 6 determines whether or not the processing target frame is being occluded by determining the occlusion degree in the occlusion processing described above. If a positive result is obtained here, the central processing unit 6 moves from step SP62 to step SP63, and in this case, returns to the original processing procedure without performing any correction processing. As a result, the central processing unit 6 ends this processing procedure without correcting any shift of the movement vector in the case of shielding. It should be noted that here, in the determination of whether or not the shielding is in this embodiment, the flag is set by rising from the shielding degree value 0, and is executed by the determination of this flag.

  On the other hand, if a negative result is obtained in step SP62, the central processing unit 6 moves from step SP62 to step SP64, and determines whether or not the frame to be processed is a frame immediately after the occlusion. It should be noted that the frame immediately after the end of occlusion is determined until the predetermined number of frames N have elapsed after the occlusion ends. The determination of whether or not the shielding is in progress is not limited to the above-described determination of the shielding degree, but may be determined separately from the shielding process.

  If a negative result is obtained here, the central processing unit 6 proceeds from step SP64 to step SP65. Here, the central processing unit 6 creates a fixed pattern from the image data SV of the frame in which the movement vector has already been detected and the moving body is not shielded (FIG. 17A). Specifically, in this embodiment, this fixed pattern is created using the image data SV processed N frames before, so that the predetermined number N of frames is processed from the processing target frame indicated by time t1 in FIG. A fixed pattern is created from the image data SV of the same tracking target region in the frame indicated by the time point t1-N when the time axis is reversed (FIG. 17A). Thereby, the central processing unit 6 is not used for the shift correction process in the frame that is being shielded.

  Here, this fixed pattern is a template for correlation detection, and is created by cutting out image data of the tracking target area. Note that the fixed pattern is not limited to the case where the fixed pattern is generated by the image data SV N frames before in this way, but the case where the fixed pattern is generated by the average value of the image data SV of K frames that are further reversed from N frames before, An image of a frame in which a moving vector has already been detected and the moving object is not shielded, such as a case where the image is created using a weighted average value of image data of these K frames, or a case where the image is created by a feedback filter technique. It can be created in various ways from data.

  The central processing unit 6 further creates a weight pattern in which a weight coefficient is set for the fixed pattern created in this way. Here, the weight pattern is obtained by using the reliability value of each feature point at each of the sampling points of the feature points and the sampling points in the vicinity of the sampling points for the feature points set in the tracking target region related to the fixed pattern. A weighting factor is set, and a weighting factor of value 0 is set for the remaining sampling points. As for the reliability, either the overall reliability related to the above-described noise removal processing or each reliability provided for calculation of the overall reliability can be applied. In addition, the sampling points in the vicinity of the feature points that set the weighting coefficient according to the reliability value can be set variously as necessary, and in the horizontal direction and the vertical direction depending on the moving object. The number of sampling points may be varied.

  The central processing unit 6 then proceeds to step SP66 and detects the position with the highest correlation by block matching using this fixed pattern and weight pattern. Specifically, the central processing unit 6 sets a range of ± S pixels in the vertical direction and the horizontal direction as search ranges around the position where the fixed pattern is superimposed on the tracking target region, and sets the frame to be processed. The fixed pattern is sequentially moved in the search range with respect to the image related to. In addition, for each moving position, a difference value of pixel values is calculated between overlapping sampling points between the fixed pattern and the image of the processing target frame, and each difference value is multiplied by a corresponding weighting coefficient based on the weight pattern. The central processing unit 6 calculates the sum of absolute values of the multiplication values and thereby calculates the correlation value. Further, the position of the correlation value having the smallest value than the correlation value detected at each position is detected. Accordingly, the search range shown in FIGS. 17 and 19 is a search range for one sampling point of the fixed pattern. In addition, the detection of the correlation value is not limited to the case where it is executed by a fixed pattern using all sampling points in the tracking target area, but various pattern matching methods such as a case where it is executed using a low resolution fixed pattern obtained by thinning sampling points. Can be widely applied.

  As a result, the central processing unit 6 uses the pattern matching between the tracking target region set in the past frame and the next frame and the tracking target region set in the past frame and the next frame. A position with high correlation is detected, and thereby the position of the moving object in the processing target frame is detected by the pattern matching method.

  Subsequently, the central processing unit 6 moves to step SP67, and detects a vector from the center position of the corresponding tracking target area of the processing target frame to the position with the highest correlation detected in this way. The shift of the tracking target area is detected. The central processing unit 6 uses the detected vector as a vector for correcting the deviation, and in the subsequent step SP68, corrects the movement vector by adding the correction vector to the movement vector detected in the processing target frame. After correcting the shift of the tracking target area with respect to the moving object, the process proceeds to step SP63 and the processing procedure is terminated.

  On the other hand, if a positive result is obtained in step SP64, the central processing unit 6 proceeds from step SP64 to step SP69. Here, the central processing unit 6 creates a fixed pattern with the frames that are further raised up by the amount of the shielded frames. Thus, in the example shown in FIG. 19, in the frame at time t3, a fixed pattern is created using the frame at time t2-N that is back by N frames from time t2 at which occlusion is started, A weight pattern is created so that the occluded frame is not used for processing.

  In the subsequent step SP65, the central processing unit 6 calculates a weight correlation value from the fixed pattern and the weight pattern in the same manner as described above. In this processing, the central processing unit 6 calculates the correlation value by expanding the search range immediately after the end of shielding as compared to the case where nothing is shielded. As a result, the central processing unit 6 reliably corrects the deviation even immediately after the end of shielding. Note that the expansion of the search range is set according to the number of frames for which the displacement correction is stopped due to shielding, and for example, the search range is set to expand in proportion to the number of frames.

  FIG. 20 is a functional block diagram of the object tracking device 3 configured by the series of processing of the central processing unit 6. In this object tracking device 3, the feature point processing unit 11 sets a tracking target region in the previous frame to set a feature point, and additionally sets a feature point by the amount deleted by a series of processes. 12, and a feature point tracking unit 13 that detects a motion vector of the feature point to the current frame by the feature point selection unit 12 and tracks the feature point. On the other hand, the motion analysis unit 14 detects a movement vector from the motion vector detected by the feature point processing unit 11. That is, in the motion analysis unit 14, the noise reduction / deletion unit 15 detects the reliability and deletes the feature points to reduce the influence of noise, and the motion calculation unit 16 of the current frame calculates the remaining feature points. The movement vector is calculated from the motion vector. On the other hand, the shielding countermeasure unit 17 performs a shielding countermeasure process on the movement vector, and the shielding detection unit 18 detects the shielding degree of the moving object. Further, based on the detected shielding degree, the inertial unit 20 detects the moving body inertial movement vector using the past motion information 19 and outputs the movement vector of the moving body based on the moving body inertial movement vector. On the other hand, the deviation countermeasure unit 21 corrects the shift of the movement vector with respect to the moving body. That is, in the deviation countermeasure unit 21, the immediately after shielding unit 22 detects the frame immediately after shielding, and the deviation correction unit 23 refers to the detection result of the immediately after shielding unit 22 and sets the tracking target area to be set to the next frame. The position information of the tracking target area is output as the object tracking result S1.

(2) Operation of Embodiment In the above configuration, the object tracking device 3 (FIGS. 1 and 2) is a tracking target that indicates a region of a moving object in the processing target frame by processing the processing target frame with the image data SV. An area is set, and the movement vector of the moving object is detected by comparing the tracking target area with the image data SV of the next frame. Further, the tracking target area is set in the next frame based on the moving vector of the moving body, and the same processing is repeated by setting the tracking target region as the processing target frame in the subsequent frame. As a result, the object tracking device 3 tracks the moving object by moving the tracking target region so as to follow the movement of the moving body sequentially in successive frames.

  In the object tracking device 3, the process of detecting the movement vector by comparing the tracking target area with the image data SV of the next frame is executed by the process of the motion vector detected at the feature point. In other words, in the object tracking device 3, a region having motion is detected from the processing target frame of the previous frame, a rectangular frame W is set so as to surround this region, and a tracking target region is set. A predetermined number of points are detected (FIG. 3). In the object tracking device 3, a motion vector of the feature point to the next frame is detected, and a motion vector indicating a motion of the moving object to the next frame is detected by statistical processing of the motion vector.

  However, in the tracking target area detected in this way (FIG. 4), a feature point may be set for a background, a premise, etc. other than the moving object. Therefore, the movement vector detection noise is not correctly reflected, and this becomes a noise of movement vector detection. Also, for example, there are feature points that are part of a moving body, such as the limbs of a walking person, but exhibit a different motion from the moving body, and the motion vector of such a feature point also becomes noise for motion vector detection. . In the presence of such noise, the detection accuracy of the movement vector is deteriorated.

  Thereby, in this embodiment, the reliability indicating the certainty of the feature points existing on the moving object is detected at each feature point, and the motion vector detected at the feature point with low reliability is processed by this reliability. The motion vector is processed so as not to be reflected in the motion vector and the movement vector is detected. Thus, in this embodiment, the feature points of the limbs and the like whose movement greatly differs with respect to the movement of the background, the foreground, and the movement of the moving body can be prevented from being reflected in the movement vector. Deterioration of detection accuracy is prevented.

  In addition, by detecting the movement vector in this way, the feature point with low reliability is deleted, and the motion vector of the feature point is set not to be used for detection of the movement vector, and continues for the deleted amount. Feature points are set in the region to be tracked in the iterative process, and this reduces the number of motion vectors used for the motion vector calculation process, so that motion vectors with low reliability are not reflected in the motion vector. Therefore, deterioration of detection accuracy due to noise can be prevented by simple processing.

  Also, the motion vector is detected by the motion vector weighting process using the reliability, and this is also set so that the motion vector with low reliability is not reflected in the motion vector, and this also reliably degrades the detection accuracy due to noise. Is prevented.

  More specifically, in this embodiment, multiple types of reliability are detected by different detection methods (FIG. 5). That is, feature points are deleted and set with such reliability, and the process is repeated, and the reliability of the number of tracking times is detected for each feature point by the number of times of tracking in the iterative processing (FIG. 6). For example, the feature points set on the shielding object immediately after disclosing the shielding of the moving object, the feature points of the moving object that had been hidden until then, etc. are detected with low reliability. It is set so that the motion vector due to the feature point is not reflected in the movement vector, and deterioration of detection accuracy due to noise is prevented.

  In addition, a difference vector between the motion vector and the motion vector detected immediately before is determined, and the temporal reliability is detected (FIGS. 7 to 10). The feature points such as the background and the foreground to be presented are set so that the motion vector is not reflected in the movement vector, and this also prevents deterioration in detection accuracy due to noise.

  In addition, a change in the distance from another feature point is determined using the elastic index, and the spatial reliability is detected by counting the determination results (FIGS. 11 and 12). As a result, for example, feature points set on the extremities of the moving human body, which have different movements with respect to the movement of the human body, and for feature points such as the background and foreground, the motion vector is not reflected in the movement vector. This also prevents deterioration in detection accuracy due to noise.

  Also, on the condition that the moving body is moving, the determination of the position coordinates of the feature point, the stationary point reliability indicating the reliability is detected based on the number of times of stationary until then, the stationary background, The motion vector of the feature point due to the foreground or the like is set so as not to be reflected in the movement vector, and this also prevents the detection accuracy from deteriorating due to noise (FIG. 5).

  In addition, for each feature point, the difference value of the sampling value is determined with respect to the surrounding sampling points, and thereby the smoothness reliability indicating the reliability is detected by comparison with the surrounding sampling points. Those having low reliability are set so that the motion vector is not reflected in the movement vector, and this also prevents deterioration in detection accuracy due to noise (FIG. 5).

  In this embodiment, the total reliability is obtained by multiplying the reliability of the number of times of tracking, the temporal reliability, and the spatial reliability, which are some of the multiple types of reliability detected in this way. Detected. Further, the motion vector is detected by a weighted average which is one of the motion vector weighting processes using the total reliability. Thereby, in this embodiment, even if the influence of noise cannot be removed by processing using the reliability detected by one technique, the influence of noise can be removed by the reliability detected by another technique. It is possible to prevent the deterioration of detection accuracy due to noise more reliably.

  Similarly, feature points are deleted based on temporal reliability, static point reliability, and smoothness reliability, which are some of the reliability types detected in this way. Depending on the process using the reliability detected by one method, the feature points related to noise that is difficult to delete can be deleted by the reliability detected by other methods, and this also makes it more reliable due to noise. Deterioration of detection accuracy is prevented.

  However, even if the influence is eliminated by noise as described above, it is difficult to track the moving body when the moving body is blocked by the shielding object. In addition, when tracking is performed over a plurality of frames, the tracking target frame gradually shifts with respect to the moving object, thereby degrading the detection accuracy.

  For this reason, in this embodiment, the degree of occlusion indicating the degree of occlusion of the moving object is detected, and a calculation according to the degree of occlusion based on the movement vector detected from the motion vector and the movement vector detected in the past iteration process. The movement vector detected by the motion vector is corrected by the processing (FIGS. 14 to 16). Thus, in this embodiment, it is possible to track the moving body even when it is shielded.

  Further, the block matching using the tracking target area in the frame past the processing target frame corrects the shift of the tracking target area in the next frame with respect to the moving object (FIGS. 17 to 19). As a result, in this embodiment, misalignment is prevented and detection accuracy is improved.

  In this embodiment, among these processes, the process related to shielding is a weighted average process between a motion vector based on a motion vector and a motion vector detected in a past iteration process, in which the weighting coefficient is varied according to the degree of shielding. It is executed by. Accordingly, the movement vector can be processed without switching the process between the case where the moving body is not shielded and the case where the moving body is shielded, and the processing is simplified correspondingly. If the degree of occlusion is small, the movement vector based on the motion vector is preferentially used to obtain the movement vector, and the correction amount based on the movement vector detected in the past iterative process is increased as the degree of occlusion increases. Thus, it is possible to detect the movement vector, thereby preventing a tracking failure in the case where, for example, the moving object is blocked in the process of changing the moving speed.

  In addition, for the motion vector detected by the iterative process used for correction of such a motion vector, the motion vector detected in the previous frame by the predetermined number of frames of the processing target frame is applied. By appropriately setting the predetermined number of frames between a moving body that moves at the speed of the moving object and a moving body that moves rapidly, it is possible to correctly track the shielded moving body. Further, a movement vector based on an average value of movement vectors detected in a plurality of frames before the processing target frame may be applied to this movement vector. In this case, a moving body that moves at a substantially constant speed is used. , The influence of the detected movement vector variation can be reduced, and the shielded moving object can be correctly tracked.

  On the other hand, in the calculation of the degree of occlusion, the number of feature points that are continuously tracked according to the detection of the tracking number of reliability described above, that is, the number of feature points that are not additionally set in the repeated processing more than a predetermined number of times. Is detected, and the degree of occlusion is detected by the ratio of the number of feature points to the number of feature points in the tracking target region. Accordingly, the shielding rate is calculated by effectively using the configuration for tracking the moving object using the motion vector based on the feature points, and the entire processing is simplified accordingly. Further, in this embodiment, the shielding rate can be calculated using the tracking number reliability detection process, which also simplifies the overall process.

  On the other hand, in the case of misalignment, a fixed pattern is created based on the sampling value of each sampling point in the tracking target area set in the past frame, and each sampling point corresponding to this fixed pattern has a reliability level. A corresponding weighting coefficient is set to generate a weight pattern. Further, the correlation value at each scanning position is detected by scanning the fixed pattern and summing up the difference values of the sampling values between the sampling points overlapping at each scanning position by weighting with the corresponding weighting coefficient of the weight pattern. Further, the correlation value is determined, the position with the highest correlation is detected, and the positional deviation is corrected.

  As a result, in this embodiment, the position with low reliability is not used for the detection of the correlation value by the fixed pattern, so that the detection accuracy is prevented from deteriorating and the position shift is prevented correctly.

  Further, such a correlation value is detected by searching in a predetermined search range based on the center of the tracking target area of the next frame, thereby simplifying the process by omitting unnecessary correlation value detection processing. .

  Further, during the shielding, such processing is stopped, thereby preventing erroneous misalignment prevention processing.

  In addition, the processing target frame from the start of shielding to the end of shielding is set not to be set as a past frame, and this also prevents erroneous misalignment prevention processing.

  Further, immediately after the end of the shielding, the search range is expanded compared to immediately before the beginning of the shielding, so that the positional displacement is reliably prevented even when the positional displacement is accumulated by a plurality of frames during the shielding period.

(3) Effects of the embodiment According to the above configuration, the reliability indicating the certainty of the feature point existing on the tracking target object is detected, and the motion vector having low reliability is not reflected in the processing result by this reliability. Thus, by detecting the motion vector of the moving body by processing the motion vector, it is possible to prevent deterioration in detection accuracy due to noise.

  In other words, feature points with low reliability are deleted so that they are not used for calculation of movement vectors, and are not used for processing of subsequent frames, so that motion vectors with low reliability are not reflected in the processing results. Thus, the motion vector can be processed to detect the moving vector of the moving body, and the deterioration of detection accuracy due to noise can be prevented.

  Also, by detecting a motion vector by motion vector weighting processing using such reliability, the motion vector is processed so that the motion vector with low reliability is not reflected in the processing result, and the motion vector of the moving object is determined. Detection can be performed, and deterioration in detection accuracy due to noise can be prevented.

  In addition, since the reliability is the tracking frequency reliability based on the tracking frequency of the feature point represented by the continuous processing frequency, for example, the feature set in the shielding object related to the shielding immediately after disclosing the shielding of the moving object On the other hand, it is possible to prevent the detection accuracy from being deteriorated due to noise by not reflecting the motion vector due to the feature point of the moving object which has been hidden until then.

  In addition, since the reliability is the temporal reliability based on the determination of the difference vector between the motion vector detected so far and the motion vector, the motion appears different from that of the moving object as observed by the change on the time axis. For feature points such as the background and foreground, the motion vector can be prevented from being reflected in the movement vector, and this can also prevent deterioration in detection accuracy due to noise.

  In addition, the reliability is a feature point set on the extremity of a moving human body, for example, by determining a change in distance between other feature points and summing up the determination results. For feature points that vary greatly in movement with respect to the movement of the human body, and for feature points such as the background and foreground, the motion vector can be set so that it is not reflected in the motion vector. Deterioration can be prevented.

  In addition, since the reliability is the reliability of the stationary point based on the number of times that the feature point position coordinates are repeatedly stationary, the motion vector of the feature point based on the stationary background, foreground, etc. is set not to be reflected in the movement vector. This can also prevent deterioration in detection accuracy due to noise.

  In addition, since the reliability is the smoothness reliability detected by determining the difference between the sampling value of the feature point and the sampling points around the feature point, the reliability of the feature point itself Those having a low value can be set so as not to be reflected in the movement vector, and this can also prevent deterioration in detection accuracy due to noise.

  In addition, the total reliability is calculated based on all or part of the tracking number reliability, temporal reliability, spatial reliability, stationary point reliability, and smoothness reliability, and the movement vector is calculated based on the total reliability. By obtaining the above, even if the influence of noise cannot be removed by the process using the reliability detected by one technique, the influence of noise can be removed by the reliability detected by the other technique. Deterioration of detection accuracy due to noise can be prevented more reliably.

  Further, on the premise of the configuration that prevents the detection accuracy from being deteriorated due to noise in this way, the movement vector detected from the motion vector and the movement vector detected in the past by the detection of the shielding degree are arithmetically processed to obtain the motion vector. By detecting and correcting the shift of the tracking target area in the next frame by the pattern matching using the tracking target area of the previous frame from the processing target frame, by correcting the detected movement vector, The moving body can be traced more reliably.

  In the above-described embodiment, feature points with low reliability are deleted by a part of the tracking number reliability, temporal reliability, spatial reliability, stationary point reliability, and smoothness reliability, and the overall reliability Although the case where the motion vector is weighted and averaged by calculating the degree is described, the present invention is not limited to this, and for example, feature points having low reliability are deleted based on all of these reliability levels, and the overall reliability level is calculated. Various settings can be made in applying each reliability to each process, such as when weighted averaging of motion vectors. Further, reliability other than these may be used in addition to or instead of these reliability.

  In the above-described embodiment, a case has been described in which a motion vector with low reliability is not reflected in the processing result by processing for deleting feature points based on reliability and processing for weighted averaging of motion vectors. However, the present invention is not limited to this, and if a practically sufficient characteristic can be secured, only one of these processes may be employed to obtain the movement vector.

  In the above-described embodiment, the case where the shielding degree is calculated from the number of times of tracking has been described. However, the present invention is not limited thereto, and the shielding ratio is calculated when the shielding ratio is calculated based on the correlation value described above for the positional deviation correction. Various methods can be widely applied to the rate calculation method.

  The present invention relates to an object tracking method, a program for the object tracking method, a recording medium on which the program for the object tracking method is recorded, and an object tracking apparatus, and can be applied to, for example, a monitoring apparatus.

1 is a block diagram illustrating an object tracking system according to an embodiment of the present invention. It is a flowchart which shows the process sequence of the central processing unit in the object tracking apparatus of FIG. It is a basic diagram which shows a tracking object area | region. It is an approximate line figure used for explanation of noise. It is a flowchart which shows the movement vector calculation process of FIG. 2 in detail. It is a characteristic curve figure which shows tracking frequency reliability. It is a characteristic curve figure which shows distribution of a motion vector. It is a characteristic curve figure which shows distribution of a motion vector by the relationship with the movement vector detected in the past. It is a characteristic curve figure with which it uses for description of temporal reliability. It is a flowchart with which it uses for the process of temporal reliability. It is a characteristic curve figure with which it uses for description of the change of a feature point. It is a flowchart with which it uses for the process of spatial reliability. It is a flowchart which shows a still point detection process. It is a basic diagram with which it uses for description of shielding. It is a characteristic curve figure with which it uses for description of a shielding degree. It is a flowchart which shows a shielding countermeasure process. It is a basic diagram with which it uses for description of position shift. It is a flowchart which shows misalignment correction processing. It is an approximate line figure used for explanation of gap correction processing. It is a functional block diagram of an object tracking device.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Object tracking system, 2 ... Signal source, 3 ... Object tracking apparatus, 6 ... Central processing unit, 10 ... Processing apparatus

Claims (13)

In an object tracking method for detecting a feature point in each frame of a moving image and tracking a moving object by detecting a motion vector of the feature point,
A feature point setting step of setting a predetermined number of the feature points in a tracking target region set in a region where the moving object of the processing target frame is imaged;
Repetitive processing step of sequentially processing the image data of each frame based on the feature points set in the feature point setting step,
The iterative processing step includes
A reliability detection step of detecting, for each feature point, a reliability level indicating the certainty of the feature point existing on the moving body;
A motion vector detection step of detecting each of the motion vectors to the next frame of the feature points;
A moving vector detection step of detecting a moving vector of the moving body to the next frame based on the motion vector;
The tracking target region is moved by the moving vector detected in the moving vector detection step to set the tracking target region in the next frame, and the processing target frame in the iterative processing step that continues the next frame A tracking target area moving step set to
The moving vector detection step includes:
According to the reliability, the motion vector is detected by processing the motion vector so that the motion vector having low reliability is not reflected in the processing result. The moving vector detection step includes:
A feature point deleting step of deleting the feature point with low reliability and setting the motion vector of the feature point not to be used for detection of the movement vector;
A feature point addition setting step for setting the feature points in the tracking target region of the processing target frame in the subsequent iterative processing step by the number of feature points deleted by the feature point deleting step. The object tracking method according to claim 1. The moving vector detection step includes:
The object tracking method according to claim 1, wherein the movement vector is detected by weighting processing of the motion vector using the reliability. The moving vector detection step includes:
A feature point deleting step of deleting the feature point with low reliability and setting the motion vector of the feature point not to be used for detection of the movement vector;
A feature point addition setting step for setting the feature points in the tracking target region of the processing target frame in the subsequent iterative processing step by the number of feature points deleted by the feature point deleting step,
The reliability is
The object tracking method according to claim 1, wherein the object tracking method is a tracking frequency reliability based on a tracking frequency of a feature point represented by a continuous processing frequency in the repetitive processing step. The reliability is
2. The object tracking method according to claim 1, wherein the object tracking method is a temporal reliability based on a determination of a difference vector between the motion vector detected in the previous iteration processing step and the motion vector. The reliability is
The object tracking method according to claim 1, wherein the object tracking method is a spatial reliability obtained by determining a change in a distance from another feature point and totaling the determination results. The reliability is
The object tracking method according to claim 1, wherein the object tracking method is a reliability of a stationary point based on the number of times the stationary point has been repeatedly stationary by determining the position coordinates of the feature point. The reliability is
The object according to claim 1, wherein the smoothness reliability is detected by determining a difference value of sampling values between a sampling point of the feature point and a sampling point in the vicinity of the feature point. Tracking method. The moving vector detection step includes:
A feature point deleting step of deleting the feature point with low reliability and setting the motion vector of the feature point not to be used for detection of the movement vector;
A feature point addition setting step for setting the feature points in the tracking target region of the processing target frame in the subsequent iterative processing step by the number of feature points deleted by the feature point deleting step,
The reliability detection step includes:
The tracking number reliability by the tracking number of feature points represented by the continuous processing number in the iterative processing step,
A temporal reliability based on a determination of a difference vector between the motion vector detected in the previous iteration processing step and the motion vector;
The change in distance between other feature points is determined, and the spatial reliability obtained by tabulating the determination results;
By determining the position coordinates of the feature point, the stationary point reliability based on the number of times of stationary until then,
The total reliability is obtained by processing all or part of the smoothness reliability detected by determining the difference value of the sampling values between the sampling points of the feature points and the sampling points around the feature points. Calculate
The object tracking method according to claim 1, wherein a reliability used for detecting the movement vector in the movement vector detection step is the total reliability. The moving vector detection step includes:
A shielding degree detecting step of detecting a shielding degree indicating a degree of shielding of the moving body;
A shielding process for correcting the movement vector obtained from the motion vector by an arithmetic process using the movement vector obtained from the motion vector according to the degree of shielding and the movement vector detected in the past iterative processing step. And having steps
The iterative processing step includes:
By pattern matching using the tracking target area in a frame that is reversed by a plurality of frames from the processing target frame, the tracking target area in the next frame set in the tracking target area moving step is shifted from the moving object. A correction vector detection step of detecting a correction vector to be indicated;
The object tracking method according to claim 1, further comprising: a correction step of correcting the shift by the correction vector. In a program of an object tracking method for detecting a feature point in each frame of a moving image and tracking a moving object by detecting a motion vector of the feature point,
A feature point setting step of setting a predetermined number of the feature points in a tracking target region set in a region where the moving object of the processing target frame is imaged;
Repetitive processing step of sequentially processing the image data of each frame based on the feature points set in the feature point setting step,
The iterative processing step includes
A reliability detection step of detecting, for each feature point, a reliability level indicating the certainty of the feature point existing on the moving body;
A motion vector detection step of detecting each of the motion vectors to the next frame of the feature points;
A moving vector detection step of detecting a moving vector of the moving body to the next frame based on the motion vector;
The tracking target area is moved by the movement vector detected in the movement vector detecting step, the tracking target area is set in the next frame, and the processing target in the repetitive processing step following the next frame A tracking target area moving step set in the frame,
The moving vector detection step includes:
According to the reliability, the motion vector is detected by processing the motion vector so that the motion vector having low reliability is not reflected in the processing result. In a recording medium recording a program of an object tracking method for detecting a feature point in each frame of a moving image and tracking a moving object by detecting a motion vector of the feature point,
The object tracking program includes:
A feature point setting step of setting a predetermined number of the feature points in a tracking target region set in a region where the moving object of the processing target frame is imaged;
Repetitive processing step of sequentially processing the image data of each frame based on the feature points set in the feature point setting step,
The iterative processing step includes
A reliability detection step of detecting, for each feature point, a reliability level indicating the certainty of the feature point existing on the moving body;
A motion vector detection step of detecting each of the motion vectors to the next frame of the feature points;
A moving vector detection step of detecting a moving vector of the moving body to the next frame based on the motion vector;
The tracking target area is moved by the movement vector detected in the movement vector detecting step, the tracking target area is set in the next frame, and the processing target in the repetitive processing step following the next frame A tracking target area moving step set in the frame,
The moving vector detection step includes:
According to the reliability, the motion vector is processed so that the motion vector having low reliability is not reflected in the processing result, and the movement vector is detected. Medium. In an object tracking device that detects a feature point that is a characteristic part in each frame of a moving image by repeating a predetermined repetition process, and tracks a moving object by detecting a motion vector of the feature point,
Feature point setting means for setting a predetermined number of the feature points in a tracking target region set in a region where the moving body of the processing target frame is imaged;
Reliability detection means for detecting the reliability indicating the certainty of the feature points existing on the moving body for each feature point;
Motion vector detection means for detecting each of the motion vectors to the next frame of the feature points;
Movement vector detection means for detecting a movement vector of the moving body to the next frame based on the motion vector;
The tracking target area is moved by the movement vector detected by the moving vector detection means, the tracking target area is set in the next frame, and the next frame is set as the processing target frame in the repeated processing. A tracking target area moving means to set,
The movement vector detection means includes
An object tracking device, wherein the motion vector is detected by processing the motion vector so that the motion vector having low reliability is not reflected in a processing result according to the reliability.

Images