JP3673578B2 - Correlation tracking system - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a correlation tracking system for tracking an object on an image.
[0002]
[Prior art]
Conventionally, the degree of coincidence with a template is calculated by correlation calculation on an image photographed by a camera, and a reference block (window) is moved to the most coincident place for tracking.
[0003]
[Problems to be solved by the invention]
Since the reference block is moved to the place where the degree of coincidence is the highest and the tracking is performed on the image taken with the conventional single template and the camera, the error is detected for the object whose shape has changed. Therefore, there is a problem that it is difficult to keep track of the image so as to minimize it, and that stable tracking is difficult.
[0004]
In order to solve this problem, as described in Japanese Patent Application Laid-Open No. 5-295591, when recognizing a running car from the image of a camera placed at the top, it is not the whole car but a characteristic feature. There is a technique for realizing stable tracking by tracking a plurality of such parts and evaluating the validity of the tracking from the speed relationship of the tracking block for each block. With this technology, if the shape changes, it is realized by performing a template update process, and there is a problem that it is not possible to track an object whose shape has already changed from the start of tracking or that has suddenly changed its shape. is there.
[0005]
In order to solve these problems, the present invention searches for a position having the highest degree of matching among a plurality of templates of the reference block in a search block provided around the reference block, and data corresponding to the movement vector and the matching template, The purpose is to output a command, and to track accurately and stably even if the shape of the object changes suddenly.
[0006]
[Means for Solving the Problems]
Means for solving the problem will be described with reference to FIG.
In FIG. 1, a template 1 is a plurality of templates that represent the shape of the object that changes or the different facial expressions of the object.
[0007]
The matching means 2 finds a template having the highest degree of matching by matching an object on the image with a plurality of templates 1.
Next, the operation will be described.
[0008]
The matching unit 2 sequentially matches a plurality of templates 1 on the image to find the template 1 having the highest degree of matching, and sequentially calculates and updates the movement vector to the position of the template 1 to track the object. ing.
[0009]
In addition, the matching unit 2 sequentially matches a plurality of templates 1 on the image to find the template 1 having the highest degree of matching, and causes the computer system to output data or commands corresponding to the found template 1. I have to.
[0010]
At this time, the plurality of templates 1 representing the shape of the object to be changed are set as the template 1 of the hand image associated with the meaning represented by the hand shape.
In addition, the plurality of templates 1 representing the shape of the object to be changed are templates 1 associated with different facial expressions such as human faces.
[0011]
Accordingly, the template 1 having the highest degree of matching among the plurality of templates 1 of the reference block in the search block provided around the reference block is calculated to obtain the movement vector 5 from the immediately preceding position to the current position, By outputting data, commands, or facial expressions corresponding to the template 1 having a high degree of coincidence, it becomes possible to track stably and accurately even if the shape of the object changes, and the shape of the object It is possible to recognize and output data, commands, facial expressions, etc. corresponding to changes in
[0012]
DETAILED DESCRIPTION OF THE INVENTION
Next, embodiments and operations of the present invention will be sequentially described in detail with reference to FIGS.
[0013]
FIG. 1 is a flowchart for explaining the operation of the present invention.
In FIG. 1, S1 captures an image. This captures an image obtained by photographing an object with a video camera.
[0014]
S2 designates the entire screen. This designates that the entire image displayed on the screen is searched for the image captured in S1.
In S3, one template is selected. In this case, one template is selected from a plurality of templates 1 created in advance representing the shape and expression of the object.
[0015]
In S4, matching is calculated. This performs pattern matching while sequentially shifting the template for the entire screen designated in S2, and calculates the degree of coincidence.
[0016]
In S5, the most matched position and the degree of coincidence are stored.
In S6, it is determined whether or not the end. In the case of YES, since matching has been completed for all templates, the process proceeds to S7. In the case of NO, the next template is taken out in S3 and S4 and subsequent steps are repeated.
[0017]
In S7, tracking is started from the position of the template having the highest degree of matching.
Through the processes from S1 to S7 described above, the position and the degree of coincidence that are most coincident for each of the plurality of templates 1 are calculated for the captured image, and the template 1 having the largest degree of coincidence among them can be found.
[0018]
In S8, an image is captured.
S9 sets a search block. As shown in FIG. 2, which will be described later, the position of the template 1 having the highest degree of matching found in S7 is set as a tracking start point, a search block of a predetermined size is set around the tracking start point, and matching is performed. Make the area to be smaller.
[0019]
In S10, one template is selected.
In step S11, a search is made for a position having the highest degree of matching in the search block. As shown in FIG. 2, which will be described later, pattern matching is performed while providing a reference block having the size of template 1 in the search block set in S9 and sequentially moving to search for a position with the highest degree of matching.
[0020]
S12 memorize | stores a coincidence degree and a position.
In S13, it is determined whether the process is over. In the case of YES, since matching has been completed for all templates 1, the process proceeds to S14. On the other hand, in the case of NO, one next template 1 is taken out in S10, and S11 and subsequent steps are repeated.
[0021]
In S14, the reference block is moved to the position indicated by the template having the highest degree of matching. As shown in FIG. 2, which will be described later, all the templates 1 are matched, the reference block is moved to the position of the template 1 having the highest degree of matching, and the movement vector at this time is obtained.
[0022]
In S15, it is determined whether there is an end instruction. If YES, the process ends. In the case of NO, the process returns to S8, the next image is captured, and S9 and subsequent steps are repeated.
Through S7 to S15, a search block having a predetermined size is set around the position of the template having the highest degree of coincidence on the image, and matching is performed by shifting a plurality of templates 1 with respect to the image taken in the search block. The position and the degree of coincidence having a high degree of coincidence are obtained, and the object 1 is tracked by repeatedly obtaining the movement vector to the template 1 by finding the template 1 having the largest degree of coincidence. As a result, even if the shape of the object changes, it matches any template 1 and can be tracked with good accuracy.
[0023]
FIG. 2 is a conceptual explanatory diagram of the present invention.
FIG. 2A shows a search block / reference block.
(A-1) in FIG. 2 shows the previous video frame, and (a-2) in FIG. 2 shows the current video frame. Here, the search block 3 in the immediately preceding video frame is a block of a predetermined size centering on the template 1 having the highest degree of matching, and represents a search area. The reference block 4 is a block having the size of the template 1, and for the template 1 of the reference block 4, the matching degree is obtained while sequentially shifting the search block 3, and the largest matching degree is calculated. When the reference block 4 having the best matching shown in the figure can be calculated, the movement vector 5 has the highest matching degree this time from the center position of the reference block 4 of the template 1 having the highest matching degree as shown in the figure. The movement vector 5 to the center position of the reference block 4 of the template 1 is obtained.
[0024]
As described above, the movement vector 5 from the center position of the template 1 having the highest matching degree in the immediately preceding video frame to the center position of the template 1 having the highest matching degree in the current video frame is obtained as shown in the figure. Even if the object changes, as long as the object matches one of the plurality of templates 1 created in advance, the object can be tracked with extremely high accuracy.
[0025]
FIG. 2B shows a state where the tracking object changes.
(B-1) in FIG. 2 shows an object existing in the reference block 4 of the immediately preceding video frame. The object at the time of the immediately preceding video frame is calculated with the highest degree of coincidence with the template {circle around (1)} of the plurality of templates 1 in FIG.
[0026]
FIG. 2B-2 shows an example template. Here, since the shape of the object is changed to any one of the four templates 1 of (1) to (4) shown in the figure, the four templates 1 are registered in advance as shown in the figure.
[0027]
(B-3) of FIG. 2 shows a state where the movement vector is calculated for the current video frame. Here, the lower right reference block (immediately before) 4 is the reference block 4 of the object immediately before (b-1) in FIG. 2, and has the highest degree of coincidence with the template (1). On the other hand, the reference block (current) 4 in the upper left is the reference block 4 of the current object, and the position and the degree of coincidence are obtained assuming that the degree of coincidence with the template (2) is the highest. Thus, the movement vector 5 is calculated as the movement vector 5 from the center position matched with the template (1) of the reference block (immediately) 4 to the center position matched with the template (2) of the reference block (current) 4. It becomes.
[0028]
As described above, even if the template 1 that most closely matches the object changes from the previous video frame to the current video frame, the template 1 is expected to change in advance even if the template 1 changes from the template 1 to the template 2. It is possible to register a template 1 having a shape to be traced, and to track the position and moving distance (moving vector) stably and with good accuracy as matching with any template 1 even if it changes rapidly.
[0029]
Next, according to the order shown in the flowchart of FIG. 3, the operation at the time of matching with a plurality of templates will be described in detail.
FIG. 3 shows a matching flowchart of the present invention.
[0030]
In FIG. 3, S21 is initially set to n = 1.
In step S22, the nth template is read.
In step S23, matching is performed with the template, and a matching degree and a movement vector are calculated. As described above, this is performed by matching the image while shifting the template, calculating the degree of coincidence to obtain the position of the largest degree of coincidence, and then calculating the movement vector from the immediately preceding position to the current position. calculate.
[0031]
In step S24, the matching degree and the movement vector corresponding to the template number are stored and stored.
S25 determines whether all registered templates have been processed. If YES, the process proceeds to S27. In the case of NO, n = n + 1 in S26, and S22 and subsequent steps are repeated for the next template.
[0032]
Through the processes from S21 to S26 described above, for each of a plurality of templates, the degree of coincidence that the current template most closely matches and the movement vector to this coincident position can be calculated from the position of the immediately preceding object.
[0033]
In S27, the template number and the movement vector having the highest degree of coincidence are extracted.
In S28, the position of the reference block is updated with the extracted movement vector.
S29 outputs the meaning of the matched template. This extracts information registered in advance in association with the template number having the highest degree of matching extracted in S27. .
[0034]
In S30, it is determined whether the information extracted in S29 is data or a command. In the case of a command, the corresponding process is activated in S31. In the case of data, the data is outputted in S32 (for example, in the case of (1) in FIG. “0” is output).
[0035]
Through the above S27 to S32, the template number and the movement vector of the template having the highest degree of coincidence are extracted from the plurality of templates, the data corresponding to the movement vector and the template number are output, and the processing according to the command is started. And so on.
[0036]
FIG. 4 shows an example of hand shape recognition according to the present invention.
4A shows the previous frame, and FIG. 4B shows the current frame. Here, the degree of coincidence with the template {circle around (6)} in FIG. 4C in which the object in the immediately preceding reference block 4 has opened its hand is the largest, and there are two objects in the current reference block 4. This is an example in which the degree of coincidence with the template (3) in FIG. In this case, the template (6) having the highest degree of matching of the reference block (immediately) 4 is calculated as the template (3) having the highest degree of matching of the reference block (current) 4. Therefore, the movement vector at this time 5 is obtained as shown and is obtained as the template number {circle over (3)} at that time, and the number “2” is output (S32 in FIG. 3).
[0037]
FIG. 4C shows an example template.
FIG. 4C-1 shows an example in which numbers are registered in association with templates. Here, numbers “0” to “5” are registered in correspondence with the templates (1) to (8) of the shape shown in the figure (note that a rotated image is also registered for the number “5”). Even if the rotated image is not registered, the template 1 may be rotated on the screen for matching.
[0038]
(C-2) in FIG. 4 shows an example in which meanings are registered in association with templates. Here, the meanings “ok”, “good”, “no good”, etc. are registered as shown in the figure in association with the templates {circle around (9)} to (16) shown in the figure. As a result, the meaning corresponding to the template number having the highest degree of matching in the illustrated template 1 is output.
[0039]
FIG. 5 shows an example of facial expression recognition according to the present invention.
FIG. 5A shows a reference block (immediately before) 4 set by detecting an eye position at the time when an image of a person's upper body is photographed by a camera. Here, the template 1 in the template 1 in FIG. 5B is matched with the highest degree of matching.
[0040]
FIG. 5B shows an example of a template. Here, templates (1) to (4) are registered in advance as changes in the shape of the human eye.
FIG. 5C shows the reference block (current) with the eyes closed. In this case, since the degree of coincidence is highest with the template (4) in FIG. 5B, the facial expression (for example, “eyes” registered in association with the template (4) at this time is calculated. It is possible to output the expression “closed” or any other expression).
[0041]
FIG. 5D shows the reference block (current) with the head tilted. In this case, since the degree of coincidence is the highest with the template (2) in FIG. 5B, the facial expression of the person registered in association with the template (2) at this time (for example, “neck” It is possible to output an expression “in a tilted state” or any other expression).
[0042]
【The invention's effect】
As described above, according to the present invention, the template 1 having the highest matching degree among the plurality of templates 1 of the reference block 4 within the search block 3 provided around the reference block 4 is calculated and the position immediately before is calculated. Since a configuration is employed in which the movement vector 5 to the current position is obtained, or data, commands or facial expressions corresponding to the template 1 having the highest degree of coincidence are output, the shape of the object changes. In addition to being able to track stably and accurately, it is possible to recognize and output data, commands, facial expressions, etc. corresponding to changes in the shape of the object.
[Brief description of the drawings]
FIG. 1 is a flowchart illustrating the operation of the present invention.
FIG. 2 is a conceptual explanatory diagram of the present invention.
FIG. 3 is a matching flowchart of the present invention.
FIG. 4 is an example of hand shape recognition according to the present invention.
FIG. 5 is an example of facial expression recognition according to the present invention.
[Explanation of symbols]
1: Template 2: Matching means 3: Search block 4: Reference block 5: Movement vector

Claims (1)

In a correlation tracking system that tracks objects on the screen,
A plurality of templates that represent changes in shape and expression including the rotation of the object, and predict the change in advance ,
Means for setting a reference block of the size of the template on the screen and a search block of a predetermined size around the reference block;
Means for sequentially matching a plurality of templates while shifting the reference block in the search block, calculating a movement vector up to a position where the degree of coincidence becomes the largest, and tracking the object while sequentially updating the position;
A correlation tracking system comprising: means for outputting data or commands associated with the found template.

Images