JP2022140105A - Object tracking device and object tracking method - Google Patents

Abstract

To accurately update the feature of an object to be tracked to perform accurate object tracking.SOLUTION: An object tracking device tracks an object included in an image composed of a plurality of frames, and comprises: an image acquisition unit that acquires the image; a storage unit that stores the feature of an object to be detected and the feature of an object to be tracked; an update unit that updates the feature of the object to be tracked in the storage unit; a detection unit that, based on the feature of the object to be detected stored in the storage unit, detects the object to be detected from the acquired image; and a tracking unit that, based on the feature of the object to be tracked stored in the storage unit, tracks the object to be tracked from the acquired image. The object tracking device performs both detection of the object to be detected performed by the detection unit and tracking of the object to be tracked performed by the tracking unit for the current frame. The update unit updates the feature of the object to be tracked based on a result of tracking and a result of detection. The tracking unit performs the tracking for the next frame by using the updated feature of the object to be tracked.SELECTED DRAWING: Figure 1

Description

The present invention relates to an object tracking device and an object tracking method.

Object tracking includes detection processing for searching for an object to be detected (detection target) from an input image and tracking processing for searching each frame for the same object as the detected object. In object tracking, a template obtained by extracting the characteristics of an object to be tracked (tracking object) is used to search an input image for an area having characteristics similar to those of the template. Here, the characteristics of the tracked object may change as the object moves or the background changes. Therefore, by performing tracking while sequentially updating the template, it is possible to perform tracking with high accuracy.

One of the conventional methods used for template updating is a sequential learning method in which the results obtained by tracking are partially reflected in the template. In this method, since the template is updated considering the past state of the tracked object, it is possible to perform robust tracking against changes in the object. There is In addition, there is a risk that tracking errors will accumulate and tracking will fail.

Another conventional technique is a reset type technique that replaces the template each time it is updated. In this method, sticking to the background is less likely to occur, but since past information about the object to be tracked is not learned, tracking is likely to fail if the condition of the object changes abruptly from the previous frame.

As described above, each method has a good situation and a bad situation. However, since all situations are assumed for the object, it is difficult to determine whether the sequential learning type or the reset type should be used in template updating. On the other hand, Patent Document 1 proposes a method of calculating the reliability of the tracking result, and updating the template and determining the end of the tracking based on the reliability. The method of Patent Document 1 detects only the first frame, updates the template based on the reliability of the tracking result for subsequent frames, and performs tracking using the updated template. However, in the case of a target object whose shape is constantly changing, the reliability is calculated to be low in each frame even if the tracking is performed correctly, so there is a risk of erroneously ending the tracking, which is a problem in that the tracking is likely to fail. be. Therefore, with the method as disclosed in Japanese Patent Application Laid-Open No. 2002-200012, the template cannot be updated with high accuracy, and tracking may fail.

JP 2010-147950 A

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and an object of the present invention is to perform object tracking with higher accuracy than in the past.

In order to achieve the above objects, the present invention employs the following configurations.

A first aspect of the present invention is an object tracking device for tracking an object contained in an image consisting of a plurality of frames, comprising: an image acquisition unit for acquiring the image; features of a detection object; a storage unit that stores features; an update unit that updates the features of the tracked object stored in the storage unit; and a tracking unit that tracks the tracked object from an acquired image based on the characteristics of the tracked object stored in the storage unit. detection of the detection target by the detection unit and tracking of the tracking target by the tracking unit, and the updating unit updates the characteristics of the tracking target based on the tracking result and the detection result. and the tracking unit performs the tracking using the updated features of the tracked object for the next frame.

For the features of the detection target and the tracking target, for example, feature amounts related to color and shape may be used. Detection and tracking results may include information about the detected or tracked area.

The object tracking device according to this aspect performs both detection and tracking for each frame, and updates the template (features of the tracked object) used for tracking using the tracking result or the detection result. Tracking is performed using the updated template for the next frame. In this way, by performing tracking using a template updated based on the tracking result or detection result, it is possible to use a template for tracking according to changes in the tracked object or background, and to perform accurate tracking. can.

The update unit in this aspect may switch the template update method according to the degree of matching between the tracking result and the detection result. By switching the template update method according to the degree of matching, it is possible to prevent tracking errors from occurring in the next frame.

The template update method may be switched as follows, for example, depending on whether the degree of matching is higher than the first threshold. Here, the first threshold value is set to a value at which the tracking result can be considered correct if the degree of matching is equal to or higher than that value. If the degree of matching is higher than the first threshold, the template may be updated using features of either or both of the detection result and tracking result (first update process). On the other hand, when the degree of matching is equal to or less than the first threshold, it is preferable to update the template (second update processing) so that the detection result is affected more than when the degree of matching is higher than the first threshold. If the detection result and the tracking result do not match, it is assumed that there is an error in either. Although the tracking process rarely captures an object that is significantly different from the target, the detection process rarely captures an object that is significantly different from the target. Therefore, by giving priority to the detection result when the detection result and the tracking result do not match as described above, the possibility of erroneous tracking in the next frame can be suppressed.

The degree of matching in this aspect is an index representing the degree of matching between the object obtained by detection and the object obtained by tracking. The degree of matching can be calculated, for example, based on the difference in center position or size between the tracked object included in the tracking result and the detected object included in the detection result.

The update unit in this aspect updates the template using the tracking result of the tracking unit for the current frame in the first update process, and updates the template using the detection result of the current frame by the detection unit in the second update process. may Alternatively, the update unit may replace the detection result of the current frame by the detection unit as a template in the second update process. Also, the update unit may set the learning rate in template update larger in the second update process than in the first update process. In this way, the template can be updated such that when the degree of matching is equal to or less than the first threshold, the detection result is affected more than when the degree of matching is higher than the first threshold.

The update unit in this aspect may switch the template update method according to the reliability of the tracking result or the detection result. The reliability of the tracking result or detection result may be calculated using, for example, the difference between the tracking result or detection result of the current frame and the background image. Note that the background image is an image when the detection target does not exist in the detection area. Also, the reliability of the tracking result or detection result may be calculated using the difference in tracking position or detection position between frames. The reliability of the tracking result may be calculated using a tracking score that indicates the certainty of the tracking result, and the reliability of the detection result may be calculated using a detection score that indicates the certainty of the detection result. good too. In this way, the template can be updated more appropriately by switching using the reliability and the detection score.

The template update method may be switched as follows depending on the reliability of the tracking result or detection result. For example, the updating unit may not update the template in the current frame when the reliability of both the tracking result and the detection result is equal to or lower than the second threshold. Alternatively, if the reliability of the tracking result is higher than the second threshold and the reliability of the detection result is lower than or equal to the second threshold, the updating unit performs the first update process regardless of the degree of matching between the tracking result and the detection result. may be performed. Further, when the reliability of the tracking result is equal to or less than the second threshold and the reliability of the detection result is higher than the second threshold, the update unit performs the second update process regardless of the degree of matching between the tracking result and the detection result. may be performed. Furthermore, when the reliability of both the tracking result and the detection result is higher than the second threshold, the updating unit preferably performs the first updating process or the second updating process according to the degree of matching. Furthermore, the update unit may switch the template update method in consideration of only one of the tracking result and the detection result. For example, if the reliability of the detection result is higher than the third threshold, the first update process or the second update process is performed according to the degree of matching between the tracking result and the detection result. Alternatively, the template may not be updated in the current frame. The same can be applied to the case of switching only considering the reliability of the tracking result.

When a plurality of tracking results and detection results are obtained from the same frame, the object tracking device according to this aspect preferably associates (pairs) the tracking results and the detection results. By pairing the tracking result and the detection result in this way, even when a plurality of tracking results and detection results are obtained, the template can be updated for each pair, and accurate tracking can be performed.

A second aspect of the present invention is a computer-performed object tracking method for tracking an object included in an image consisting of a plurality of frames, comprising: an image acquisition step of acquiring the image; A detection step of detecting the detection target from the acquired image based on the features of the detection target, and tracking the tracking target from the acquired image based on the features of the tracking target stored in the storage unit. a tracking step, wherein the object tracking method performs the detecting step, the tracking step, and the updating step of updating features of the tracked object for a current frame; is an object tracking method characterized in that the tracking step is performed using updated features of the tracked object.

The present invention may be regarded as an object tracking device having at least part of the above means. Moreover, the present invention may be regarded as an object tracking method including at least part of the above processing. Further, the present invention can also be regarded as a program for realizing such a method and a recording medium on which the program is non-temporarily recorded. It should be noted that each of the means and processes described above can be combined with each other as much as possible to constitute the present invention.

According to the present invention, object tracking can be performed with higher accuracy than in the past.

FIG. 1 is a diagram showing the functional configuration of an object tracking device. FIG. 2 is a diagram for explaining template update (learning) processing in the object tracking device. FIG. 3 is a diagram showing the hardware configuration of the object tracking device. FIG. 4 is a flowchart of overall processing performed by the object tracking device. FIG. 5 is a flow chart for using the reliability of detection results and tracking results. FIG. 6 is a flow chart when using the detection score.

<Application example>
An application example of an object tracking device according to the present invention will be described with reference to FIG. The object tracking device 1 is a device that analyzes an acquired image to detect and track an object. The image may be obtained from a camera, received from a communication device, or read from a recording medium. The target object can be any object, for example, human body, face, animal, vehicle, and parts. Here, as an example, it is assumed that a human body is tracked. This object tracking device 1 detects, recognizes, and tracks a target human body, for example, in an office, a factory, or the like. The tracking result of the object tracking device 1 is output to an external device and used, for example, for counting the number of people and controlling various devices such as lighting and air conditioning.

In this application example, both detection and tracking are performed every frame. Learning of the template (feature of the object to be tracked) used for tracking is basically performed by a sequential learning algorithm. More specifically, the template is sequentially updated using the results obtained by detection or tracking, and tracking is performed using the updated template in the next frame. However, if inaccurate tracking results are obtained, or if features other than the target of tracking, such as the background, are gradually introduced as a result of sequential learning, the tracking error will accumulate and the tracking may eventually fail. There is Therefore, this application example proposes a method for improving tracking performance by performing both detection and tracking for the current frame and switching the template update method.

The image acquisition unit 2 has a function of acquiring image data for object tracking. The detection unit 3 searches for the human body based on the characteristics of the human body, which is the detection target, from the acquired image. The tracking unit 4 searches for the same human body as the human body detected by the detection unit 3 up to the previous frame from the acquired image. The template used for tracking is updated by the updating unit 5 .

The update unit 5 updates the template using the tracking result or detection result. The tracking unit 4 performs tracking using the updated template from the next frame. In this manner, by performing tracking using a template that is updated based on the tracking result or detection result of the current frame, accurate tracking can be performed.

The template update method may be switched according to, for example, the degree of matching between the detection result and the tracking result. The matching degree represents the degree of matching between the human body indicated by the detection unit 3 and the human body indicated by the tracking unit 4 . For example, it may be calculated using the degree of matching between the position coordinates of the detection rectangle and the tracking rectangle, or the degree of matching between the images of the detection rectangle and the tracking rectangle.

The template may be updated so that the detection result has a greater influence when the degree of matching is higher than the first threshold compared to when it is not. For example, if the degree of matching is higher than the first threshold, the template may be updated using the tracking result, and if the degree of matching is equal to or less than the first threshold, the template may be updated using the detection result. Note that switching of the update method is not limited to this method.

By switching the template update method according to the degree of matching in this way, the template can be updated more accurately, and the tracking accuracy is improved.

Also, the switching of the features used for template updating may be performed by switching the result used for updating according to the reliability of the detection result or tracking result. The reliability may be calculated for both the tracking result and the detection result, or may be calculated for either one. The reliability of the tracking result and the detection result may be calculated, for example, based on the background image, or may be calculated based on the positional difference between the frames of the tracked or detected object. Also, the reliability of the detection result may be calculated based on a detection score representing the certainty of the detection result. In this way, by switching the features to be used for template update in consideration of reliability and detection score as well, the template can be updated more accurately, and the tracking accuracy can be improved.

<Embodiment 1>
The embodiments described below relate to an object tracking device 1 . As shown in FIG. 2, here, as an example, a device for tracking a human body 10 from an image captured by a camera is assumed, but this is merely an example, and the object to be detected and tracked from the image does not have to be a human body. can be an object of Examples include animals, vehicles, and parts. Also, the image acquisition source is not limited to a camera, and the object tracking device may be applied to images recorded in a hard disk, for example.

The object tracking device 1 according to this embodiment can be used, for example, for human body detection using a surveillance camera or an in-vehicle camera, autofocus (AF), and counting people.

<Hardware configuration>
FIG. 3 is a diagram showing the hardware configuration of the object tracking device 1 according to this embodiment. The object tracking device 1 includes a communication IF (interface) 20 , a communication device 21 , a storage device 22 , an arithmetic device 23 , an input device 24 and an output device 25 . The communication IF 20 receives image data directly from the camera in this embodiment, but may receive image data via the communication device 21 or a recording medium. The communication device 21 is a device for the object tracking device 1 to communicate with an external computer. The form of communication may be wired or wireless, and the communication standard may be arbitrary. The storage device 22 includes a main storage device and an auxiliary storage device, and stores programs executed by the arithmetic unit 23, image data, and temporary data during program execution. The input device 24 is a device for the user to input instructions to the object tracking device 1, and is composed of, for example, buttons and a keyboard. The output device 25 is a device for the object tracking device to output to the user, and is composed of, for example, a display device, a control device, a speaker, and the like.

FIG. 1 is a functional block diagram of an object tracking device 1 according to this embodiment. The object tracking device 1 of this embodiment has functional units such as an image acquiring unit 2 , a detecting unit 3 , a tracking unit 4 , an updating unit 5 , a storage unit 6 and an output unit 7 . These functional units are realized by the arithmetic device 23 executing programs stored in the storage device 22 . However, some or all of these functional units may be implemented by dedicated devices such as ASICs.

(Image acquisition unit 2)
The image acquisition unit 2 has a function of acquiring image data for object tracking. The image data to be acquired is an image consisting of a plurality of frames, and may be a still image or a moving image. Also, the image to be acquired may be acquired from the camera 26 or may be acquired from the communication device 21 or the storage device 22 . The captured image data is handed over to the detection unit 3 and the tracking unit 4 .

(Detector 3)
The detection unit 3 detects the detection target from the image acquired by the image acquisition unit 2 based on the features of the detection target stored in the storage unit 6 . In other words, the detection unit 3 searches for a region including a human body based on features common to people, who are detection targets. Any algorithm for detecting a human body can be adopted as the algorithm used for detection. The detection result is handed over to the updating unit 5 .

(Tracking unit 4)
The tracking unit 4 tracks the tracking object from the image acquired by the image acquisition unit 2 based on the characteristics of the tracking object stored in the storage unit 6 . That is, the tracking unit 4 searches for a region containing the same human body as the human body detected by the detection unit 3 in the previous frame, based on the characteristics that can identify a specific human body detected by the detection unit 3, which is the object to be tracked. do. The template used for tracking by the tracking unit 4 is updated by the updating unit 5, and the updated template is used for tracking for the next frame. The tracking result is handed over to the updating unit 5 .

Here, in the present embodiment, a color histogram, which is a color feature amount, is used as an example of features used by the detection unit 3 and the tracking unit 4 to search for an object. The color histogram has a constant color distribution even when the shape changes, such as when a person is walking, so it is suitable for tracking a person. Note that the feature amount to be used is not limited to color. For example, a feature amount related to brightness or shape may be used, or a combination of a plurality of feature amounts may be used.

(Update part 5)
The update unit 5 has a function of switching update (learning) methods according to the degree of matching between the detection result by the detection unit 3 and the tracking result by the tracking unit 4 . In this embodiment, the updating unit 5 switches the feature used for template update between the tracking result and the detection result according to the degree of matching, and changes the learning rate. The matching degree is calculated using the difference between the center positions of the tracking rectangle 12 indicated by the tracking unit 4 and the detection rectangle 11 indicated by the detection unit 3 . Note that the degree of coincidence may be anything as long as the degree of similarity between the tracking result and the detection result can be compared. For example, the resulting similarity may be calculated using the distance between centers, the difference in area size, IoU (Intersection over Union), or may be calculated from the image similarity. The image similarity may be calculated using SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), cross-correlation, etc. It is calculated using the similarity of the subject in the image such as face orientation and posture. may

Here, when both tracking and detection are performed correctly, similar regions are output for the tracking result and the detection result. On the other hand, if the tracking result and the detection result differ from each other, the tracking may not be performed correctly. Therefore, the first threshold value is set to a value at which it can be determined that tracking is being performed correctly if the degree of matching is equal to or higher than that value. By setting such a value and switching the method used for template update with the first threshold as a boundary, template update can be performed with higher accuracy. Note that the first threshold value may be an arbitrary value, and for example, it may be set to be larger by a predetermined percentage in the case of object tracking that requires strict accuracy than in other cases. The switching of the update method by the update unit 5 will be described in detail below with reference to the flowchart.

(storage unit 6)
The storage unit 6 has a function of storing the features (detection template) of the detection object used by the detection unit 3 and the features (tracking template) of the tracking object used by the tracking unit 4 . The storage unit 6 also stores various parameters such as coefficients used for updating, which are used for tracking processing and updating processing.

(Output unit 7)
The output unit 7 has a function of outputting information such as images and detection results/tracking results to an external device. For example, the output unit 7 may display information on a display as an external device, may transmit information and control signals to a lighting device or an air conditioner as an external device, or may be stored in a recording device. .

<Overall processing>
FIG. 4 is an overall flowchart of object tracking processing. The overall flow of object tracking processing will be described with reference to FIGS. 1 and 4. FIG.

In step S101, the detection unit 3 detects a human body from the current frame image based on the characteristics of the human body, which is the object to be detected.

In step S102, the tracking unit 4 tracks the tracked object based on the template from the current frame image. The tracked object is the human body that was tracked in the previous frame image when the track was performed up to the previous frame image. If there is only the detection result by the detection unit 3 without tracking up to the previous frame, it means the human body detected by the detection unit 3 .

In step S103, the updating unit 5 performs association (pairing) based on the center-to-center distance of the rectangles of the tracking result and the detection result. The pairing may be performed based on the distance between the centers of the rectangles, the shortest distance between the rectangles, the similarity of the size of the rectangles, or the like. The processing after the loop processing L1 of steps S104 to S106 respectively processes the paired tracking result and detection result.

In step S<b>104 , the update unit 5 determines the degree of matching from the center coordinates of the tracking rectangle 12 indicated by the tracking unit 4 and the center coordinates of the detection rectangle 11 indicated by the detection unit 3 . If the degree of matching is higher than the first threshold, the process proceeds to step S105, and if it is equal to or less than the first threshold, the process proceeds to step S106. In this embodiment, the process of step S105 corresponds to the first update process, and the process of step S106 corresponds to the second update process.

In this embodiment, the template is updated using the tracking result or detection result of the current frame. An example of the expression used for template update is described below.

_{Fn+1 =} α×Fn+(1−α)× _Fn−1
Here, F _n+1 : feature quantity used for tracking in the next frame F _n : feature quantity of the object obtained from the current frame F _n-1 : feature quantity stored up to the previous frame α: learning rate.

In step S105, the update unit 5 uses the "feature amount indicated by the tracking result of the current frame" as the "feature amount (Fn) of the object obtained from the current frame", and sets the learning rate (α) to "0.1 ” to update the template (sequential learning). Note that the learning rate is a parameter that indicates how much the template stored in the storage unit 6 is changed. That is, in S105, as shown in the above formula, the template is updated so that the feature amount indicated by the tracking result of the current frame and the feature amount stored up to the previous frame are combined at a ratio of 1:9. As a result, tracking can be performed using an appropriately updated template in the next frame. In step S105 of the present embodiment, the tracking result is used as the feature used for updating, but the detection result may be used, or both the tracking result and the detection result may be used in combination. Also, the value of the learning rate is not limited to "0.1" and may be any value.

In step S106, the updating unit 5 uses the "feature amount indicated by the detection result of the current frame" as the "feature amount (Fn) of the object obtained from the current frame", and sets the learning rate (α) to "1". That is, the template is replaced (reset) with the feature amount indicated by the detection result of the current frame. Here, if the match is low, the tracking results may indicate inaccurate results. Therefore, by immediately replacing the template with the detection result of the current frame, it is possible to quickly recover the tracking accuracy from the next frame. The learning rate may be higher than that in S105, and for example, a process of "increasing the learning rate" with the learning rate set to "0.5" may be employed. In this case, the template can be updated so as to approach the detection result of the current frame while also maintaining the information of the past template. For example, "reset" and "learning rate up" may be switched at a predetermined threshold smaller than the first threshold.

Note that in this embodiment, when the degree of matching is higher than the first threshold (step S105), the template is updated using the tracking result of the tracking unit 4 for the current frame, and when the degree of matching is equal to or less than the first threshold ( In step S106), the template is updated using the detection result of the detection unit 3 for the current frame, and the process of using different learning rates in steps S105 and S106 is described. However, if the template can be updated such that the detection result has a greater effect when the degree of matching is higher than the first threshold compared to when it is not, the switching of the update method is not limited to this method. For example, the updating unit 5 switches between the tracking result and the detection result according to the matching degree for the feature used for template updating, but the same value of the learning rate may be used regardless of the matching degree. In this case, the update unit 5 uses the tracking result when the degree of matching is higher than the first threshold, uses the detection result when the degree of matching is equal to or less than the first threshold, and sets the learning rate to an arbitrary value ( For example, "0.1") may update the template. Further, the update unit 5 may use the detection result for template update regardless of the degree of matching, and change the learning rate according to the degree of matching. In this case, the learning rate when the degree of matching is higher than the first threshold may be higher than the learning rate when the degree of matching is equal to or less than the first threshold. Further, the update unit 5 may use the result of synthesizing the tracking result and the detection result for template update regardless of the degree of matching, and may change the ratio of synthesis according to the degree of matching. In this case, when the degree of matching is higher than the first threshold, the updating unit 5 may combine the tracking results so that the weight for the tracking result is greater than when the degree of matching is equal to or lower than the first threshold. Furthermore, in this case, the learning rate may be changed according to the degree of matching, or the same learning rate may be used. Also, the template may be updated by combining the plurality of update methods described above. In this way, when the degree of matching is equal to or less than the first threshold, it is preferable to update the template so that the detection result has a greater influence than otherwise. As a result, even if the degree of matching is low and either the tracking result or the detection result may be inaccurate, it is possible to avoid a decrease in tracking accuracy due to inaccurate template updating.

The tracking unit 4 performs tracking using the updated template in the next frame, and the processing from step S101 to step S105 or step S106 is repeated for each frame.

In this embodiment, both detection by the detection unit and tracking by the tracking unit are performed on the current frame image, and the template update method is switched according to the degree of matching between the detection result and the tracking result. If there is a high degree of agreement between the detection result and the tracking result, either result can be used to correctly update the template. On the other hand, if the detection result and the tracking result are different, one of the results may be inaccurate. Although the tracking process rarely captures an object that is significantly different from the target, the detection process rarely captures an object that is significantly different from the target. Therefore, when the detection result and the tracking result do not match as described above, priority is given to the detection result. As a result, sticking to the background due to template update using inaccurate results can be reduced, and tracking accuracy can be improved.

<Second embodiment>
In the first embodiment, the update method is switched based on the degree of matching, but in the second embodiment, the update method is switched in consideration of the reliability of the tracking result/detection result. Since the configuration of the object tracking apparatus according to this embodiment is basically the same as that of the first embodiment, detailed description will be omitted, and mainly different points will be described.

Here, the reliability is an index indicating the certainty of the tracking result or detection result. In this embodiment, the reliability is calculated for both the tracking result and the detection result, but it may be calculated for either the tracking result or the detection result. In this way, by switching the template update method in consideration of the reliability, it is possible to update the template more appropriately and improve the tracking accuracy.

The reliability may be calculated using, for example, the difference from the background image. That is, the reliability of the tracking result by the tracking unit 4 is determined according to the difference between the tracking result for the current frame and the background image, and the reliability of the detection result by the detection unit 3 is determined by the detection result for the current frame and the background image. You may determine according to the difference with an image. Note that the background image is an image when the detection target does not exist in the detection area. In this embodiment, an image at a specific point in time when there is no object to be detected is used as a background image for background subtraction. As the background image, an image at a specific point in time may be used as the background image, or different images may be used as the background image. For example, as the background image, the frame image immediately before the detection target does not exist may be determined as the background image, or an image synthesized using the average of the most recent predetermined frames may be determined as the background image. The background image used for the background difference of the tracking result and the background difference of the detection result may be the same background image or different background images.

The degree of reliability based on the difference from the background image is evaluated by the degree of image dissimilarity between the area output by the tracking result or the detection result for the current frame and the corresponding area of the background image. Here, if the tracking result or detection result is similar to the background image, there is a possibility that the background is erroneously recognized as the object. Therefore, the degree of dissimilarity between the tracking result or detection result and the background image can be used as the degree of reliability, and the higher the degree of dissimilarity (the less similar), the higher the reliability is evaluated. Although the amount of image difference can be used as the degree of dissimilarity, other indices may be used.

With reference to FIG. 5, a method of switching between template update methods using a difference from a background image as an example of reliability will be described. The same reference numerals are assigned to the same processes as in FIG. 4, and detailed description thereof will be omitted. The flow of processing after step S201 will be described with reference to FIGS. 1 and 5. FIG.

In step S201, the reliability of the detection result by the detection unit 3 and the reliability of the tracking result by the tracking unit 4 are compared with a second threshold. Here, the second threshold value is set to a value at which detection results and tracking results can be judged to be reliable. Also, in the present embodiment, the same value is used as the second threshold for the detection result and the tracking result, but different values may be used.

If it is determined in step S201 that both the reliability of the detection result and the tracking result are higher than the second threshold, the process proceeds to step S104. That is, when both the detection result and the tracking result are reliable, the template update method is switched in consideration of the degree of matching as in the first embodiment.

In the determination in step S201, if the reliability of the detection result is equal to or lower than the second threshold and the reliability of the tracking result is higher than the second threshold, the detection result may be inaccurate, but the tracking result is It seems that the correct result is output. Therefore, the process proceeds to step S105 without judging the degree of matching, and the updating unit 5 uses the tracking result to update the template (sequential learning). This allows the template to be updated with more reliable results.

In the determination in step S201, if the reliability of the detection result is higher than the second threshold and the reliability of the tracking result is less than or equal to the second threshold, it is considered that the detection result is correct. Tracking results may be inaccurate. Therefore, the process proceeds to step S106 without judging the degree of matching, and the updating unit 5 replaces (resets) the template with the detection result of the current frame. Thereby, when the tracking result is inaccurate, the template is corrected so as to approach the detection result, and the tracking accuracy can be improved.

If it is determined in step S201 that both the reliability of the detection result and the tracking result are equal to or less than the second threshold, the process proceeds to step S107, and the updating unit 5 does not update the template ("no learning"). Note that "not updating the template" can be regarded as learning processing at a learning rate of 0%. This can avoid updating the template with potentially inaccurate features.

The tracking unit 4 performs tracking using the updated template in the next frame, and the processes after step S101 are repeated for each frame. In this way, by switching the template update method according to the reliability, the template can be updated by reflecting the result of the current frame more appropriately, and the tracking accuracy can be improved.

When using the reliability determination result, after the determination in step S201, all the processes of S104 to S107 may be adopted, or only one of the processes may be adopted. For example, if step S107 (“no learning”) is not adopted and both the reliability of the detection result and the reliability of the tracking result are equal to or less than the second threshold, the processing proceeds to the degree of matching in step S104. The template may be updated accordingly. Further, for example, if the degree of matching is not adopted in step S104 and the reliability of the tracking result is high, the template may be updated by "sequential learning" processing. Also, a template update method may adopt a switching method different from the above. For example, if either one of the reliability of the detection result and the reliability of the tracking result is lower than the second threshold, the template may not be updated (step S107). As in the first embodiment, the learning rate used in the second embodiment is merely an example, and any value may be used.

Also, the reliability is not limited to the degree of dissimilarity with the background image, and may be calculated based on other indexes. For example, the difference in position between frames may be employed as an indicator of reliability. In this case, if the difference in position between frames is within the range in which the object can move, it may be assumed that the result is reliable. The reliability of the tracking result by the tracking unit 4 is determined according to the difference in tracking position between frames, and the reliability of the detection result by the detecting unit 3 is determined according to the difference in detection position between frames. good too. The tracking position and detection position are coordinates of the center position of the tracked and detected area, for example. By calculating the reliability according to the positional difference between frames in this way, it is possible to reduce the possibility of updating the template using results that may have been erroneously tracked or detected.

Note that the criterion used for reliability is not limited to one condition. For example, the reliability may be calculated by combining the difference from the background image, the positional difference between frames, or other criteria.

In the second embodiment, the template update method is switched in consideration of the reliability of the detection result and the tracking result. According to this embodiment, since the template can be updated using reliable results, it is possible to perform more appropriate updating and further improve tracking accuracy.

<Modification>
In the second embodiment, the template update method is switched based on the reliability of the detection result and the tracking result, but the template update method may be switched based only on the reliability of the detection result. Since the configuration of the object tracking device according to this modification is basically the same as that of the second embodiment, detailed description is omitted, and mainly different points will be described.

FIG. 6 is a flow chart when the reliability is calculated only for the detection result. Here, a detection score representing the certainty of the detection result may be used as the reliability of the detection result. When the template update method is switched based only on the reliability of the detection result, it is not always necessary to use the detection score. For example, as described in the second embodiment, the difference from the background image or the difference in position between frames may be used as the reliability of the detection result, or the reliability calculated by other methods may be used as the detection result. may be used as the reliability of

The detection result score is an index indicating the probability that the detection target is included in the detection area, and is output by the detection unit together with the detection area. A higher value is calculated for the detection score as the probability that a human body is included in the detection area is higher. In this way, only the reliability of the detection result is calculated as the reliability, and the template update method is switched according to the reliability of the detection result, so that the tracking accuracy can be improved with simpler processing.

With reference to FIG. 6, a method of switching the template update method using the detection score will be described. Note that the processing up to step S103 is the same processing as in FIG. The flow of processing after step S301 will be described with reference to FIGS. 1 and 6. FIG.

In step S301, the detection score of the detection result is compared with the third threshold. Here, the third threshold value is set to a value at which it can be determined that the detection result includes the detection target and that the detection result is reliable if the detection score is equal to or higher than that value.

If the detection score is higher than the third threshold in the determination of step 301, the process proceeds to step S104. That is, when the detection result is reliable, the template update method is switched in consideration of the degree of matching as in the first embodiment.

In the judgment of step S301, when the detection score is equal to or less than the third threshold, the process proceeds to step S107, and the updating unit 5 does not update the template ("no learning"). Note that "not updating the template" can be regarded as learning processing at a learning rate of 0%. . If the detection score is less than or equal to the third threshold and there is a possibility that the detection result is inaccurate, it is unclear whether the tracking result indicates a correct result. So don't update the template, keep the current template. This can avoid updating the template with potentially inaccurate features.

The tracking unit 4 performs tracking using the updated template in the next frame, and the processes after step S101 are repeated for each frame. In this manner, by switching the template update method according to the detection score, the template can be updated by appropriately reflecting the result of the current frame, and the tracking accuracy can be improved.

In the modified example, as an example of switching the template update method using the reliability of either the tracking result or the detection result, the case where the reliability of the detection result is used has been described, but only the reliability of the tracking result is used. to switch the template update method.

<Others>
The above-described embodiment is merely an example of the configuration of the present invention. The present invention is not limited to the specific forms described above, and various modifications are possible within the technical scope of the present invention.

In the above description, a color histogram is used as a feature quantity representing a color feature as an example of a feature used by the detection unit 3 and the tracking unit 4 to search for the target object, but the present invention is not limited to this. For example, a luminance histogram may be used as the feature quantity representing the luminance feature, and HOG (Histogram of Gradient) feature quantity, SIFT (Scale-Invariant Feature Transform) feature quantity, SURF as the feature quantity representing the shape/contour of the object. (Speeded Up Robust Features) A feature amount or the like may be employed.

Any algorithm may be used as a method for detection. For example, a classifier that combines image features such as HOG or Haar-like with boosting may be used, or SVM
A pattern recognition model such as (Support Vector Machine) may be used, or image recognition by deep learning (eg, R-CNN, Fast R-CNN, YOLO, etc.) may be used.

In the "sequential learning" process (S105, S103, S303) of the above embodiment, an example of updating the template using the tracking result is shown. The combined results may be used to perform template updates. In addition, in FIGS. 5 and 6, in the processing when the degree of matching is low (S204, S304), an example of adopting the “reset” processing is shown, but in FIG. 5 and FIG. A process of "increase learning rate" having a higher learning rate than "learning" may be adopted. For example, the processing of “reset” and “increase learning rate” may be switched according to the value of the degree of matching or the number of consecutive frames with a degree of matching smaller than the first threshold. Note that the values of the learning rate are not limited to 10%, 50%, and 100%, and arbitrary values may be used.

In the modified example, when the detection score is low, neither the detection result nor the tracking result is used for updating, and the updating unit does not update the template (S305). good. For example, if the detection score is low but the tracking results are stable, tracking accuracy can be improved by updating the template using the tracking results.

In the above embodiment, a rectangular detection frame was exemplified (FIG. 2, etc.), but a detection frame in a form other than rectangular (polygon, ellipse, free figure, etc.) may be used.

<Appendix>
(1) An object tracking device (1) for tracking an object included in an image consisting of a plurality of frames,
an image acquisition unit (2) that acquires the image;
a storage unit (6) storing the features of the detection target and the features of the tracking target;
an updating unit (5) for updating the characteristics of the tracked object in the storage unit (6);
a detection unit (3) for detecting the detection object from an acquired image based on the characteristics of the detection object stored in the storage unit (6);
a tracking unit (4) for tracking the tracked object from an acquired image based on the characteristics of the tracked object stored in the storage unit (6);
with
The object tracking device (1) performs both detection of the detection target by the detection unit (3) and tracking of the tracking target by the tracking unit (4) for a current frame,
The updating unit (5) updates the characteristics of the tracked object based on the tracking result and the detection result,
The tracking unit (4) performs the tracking using the updated features of the tracked object for the next frame.
(2) A computer-performed object tracking method for tracking an object included in an image consisting of a plurality of frames, comprising: an image acquisition step of acquiring the image; A detection step (
S101) and a tracking step (S102) of tracking the tracked object based on the characteristics of the tracked object stored in a storage unit, wherein the object tracking method includes, for the current frame, the detection The step (S101), the tracking step (S102), and the updating steps (S105, S106) of updating the characteristics of the tracked object are performed, and for the next frame, the updated tracked object An object tracking method, characterized in that the tracking step (S102) is performed using features.

1: Object Tracking Device 2: Image Acquisition Unit 3: Detecting Unit 4: Tracking Unit 5: Update Unit 6: Tracking Unit 7: Output Unit

Claims (18)

An object tracking device for tracking an object included in an image consisting of a plurality of frames,
an image acquisition unit that acquires the image;
a storage unit that stores features of a detection target and features of a tracking target;
an updating unit that updates the characteristics of the tracked object in the storage unit;
a detection unit that detects the detection target from an acquired image based on the characteristics of the detection target stored in the storage unit;
a tracking unit that tracks the tracked object from an acquired image based on the characteristics of the tracked object stored in the storage unit;
with
The object tracking device performs both detection of the detection target by the detection unit and tracking of the tracking target by the tracking unit for the current frame,
The updating unit updates the characteristics of the tracked object based on the tracking result and the detection result,
The tracking unit performs the tracking using the updated features of the tracked object for the next frame.
An object tracking device characterized by: 2. The object tracking device according to claim 1, wherein the update unit switches the update method according to the degree of matching between the detection result by the detection unit and the tracking result by the tracking unit. The updating unit updates the characteristics of the tracked object using at least one of the tracking result of the tracking unit and the detection result of the detecting unit for the current frame when the degree of matching is higher than a first threshold. performing a first update process, and updating the detection result of the current frame by the detection unit so that, when the degree of matching is equal to or less than the first threshold value, the detection result of the detection unit has a greater influence than otherwise; to perform a second update process for updating the characteristics of the tracked object,
3. An object tracking device according to claim 2. The tracking result by the tracking unit and the detection result by the detection unit include information about the center position or size of the tracking object and the detection object, and the degree of matching is the center of the tracking result and the detection result. Calculated based on positional or magnitude differences,
4. An object tracking device according to claim 3. The updating unit updates the characteristics of the tracked object using the tracking result of the tracking unit for the current frame when the degree of matching is higher than the first threshold, and the degree of matching is equal to or less than the first threshold. , updating the feature of the tracked object using the detection result of the detection unit for the current frame;
An object tracking device according to any one of claims 3 to 4. When the degree of matching is equal to or less than the first threshold, the update unit replaces the detection result of the current frame by the detection unit as the feature of the tracked object.
An object tracking device according to any one of claims 3 to 5. The updating unit determines the reliability of at least one of the tracking result by the tracking unit and the detection result by the detecting unit, and switches the method of updating the characteristics of the tracked object according to the reliability.
An object tracking device according to any one of claims 3 to 6. The updating unit does not update the features of the tracked object in the current frame when the reliability of both the tracking result by the tracking unit and the detection result by the detecting unit is equal to or less than a second threshold.
The object tracking device according to claim 7. The update unit performs the first update process when the reliability of the tracking result by the tracking unit is higher than the second threshold and the reliability of the detection result by the detection unit is equal to or less than the second threshold.
The object tracking device according to claim 7 or 8. The update unit performs the second update process when the reliability of the tracking result by the tracking unit is less than or equal to the second threshold and the reliability of the detection result by the detection unit is higher than the second threshold.
Object tracking device according to any one of claims 7 to 9. When the reliability of both the tracking result by the tracking unit and the detection result by the detecting unit is higher than a second threshold, the updating unit performs the first updating process or the second updating process according to the degree of matching. The object tracking device according to any one of claims 7 to 10, wherein The update unit performs the first update process or the second update process according to the degree of matching when the reliability of the detection result by the detection unit is higher than a third threshold, and the reliability is increased to the third threshold. 12. An object tracking apparatus according to any one of claims 7 to 11, wherein if it is less than or equal to 3 thresholds, it does not update the characteristics of the tracked object in the current frame. The reliability of the tracking result by the tracking unit is determined according to the difference between the tracking result for the current frame and the background image,
The reliability of the detection result by the detection unit is determined according to the difference between the detection result for the current frame and the background image.
Object tracking device according to any one of claims 7 to 12. The reliability of the tracking result by the tracking unit is determined according to the difference in tracking position between frames,
The reliability of the detection result by the detection unit is determined according to the difference in detection position between frames,
Object tracking device according to any one of claims 7 to 12. The reliability of the detection result by the detection unit is determined according to a score indicating the likelihood of the detection result output by the detection unit.
Object tracking device according to any one of claims 7 to 12. When a plurality of the tracking results and the detection results are obtained from the same frame, the tracking results and the detection results are associated with each other, and the tracking target is determined using the corresponding tracking results and the detection results. 16. An object tracking device as claimed in any one of claims 1 to 15 for updating object characteristics. A computer-performed object tracking method for tracking an object included in an image consisting of a plurality of frames,
an image acquisition step of acquiring the image;
a detection step of detecting the detection target from an acquired image based on the characteristics of the detection target stored in a storage unit;
a tracking step of tracking the tracked object from the acquired image based on the characteristics of the tracked object stored in the storage unit;
The object tracking method performs the detecting step, the tracking step, and the updating step of updating the characteristics of the tracked object for a current frame, and performs the updated tracking for the next frame. performing the tracking step using features of the object;
An object tracking method characterized by: A program for causing a computer to perform the steps of the method of claim 17.

Images