JP2000132691A - Method and device for tracking video area - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a method and a device that is capable of realizing the improvement of the tracking processing of an object which can act on a general computer at a high speed and also the improvement of the tracking ability of the object. SOLUTION: The video area tracking method and device are the ones where the tracking area of an initial picture is designated against a time-sequentially inputted picture group (S11), the area is divided into plural small areas, a travelling destination detecting processing at every small area is executed by utilizing the correlation of the picture group between the small areas and the picture including them, a largeness is selected at every small area, the small area where the processing for detecting the travelling destination of the small areas in the picture in another time is selected among the ones provided with an individually selected largeness.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for tracking an area of a specified object in an image. In particular, the present invention provides an object tracking device used when specifying an area in a moving image to be related when giving related information to an appearing object in the moving image like hypermedia. Also, an object tracking device for monitoring the position of an object of interest in a monitoring image in the image is provided.

[0002]

2. Description of the Related Art In a medium called hypermedia, related information is linked to characters, images, and objects in the image, and the related information can be easily accessed. However, at the stage of producing the hypermedia, a link operation of the related information is required, so that the load is so great as to be compared with the related art.

[0003] Characters and images to which related information is linked,
An object or the like in the image is called an anchor. The anchor setting work is not so large when using text and images as anchors, but when setting anchors on objects in moving images, especially moving objects in images, the load on the creator is very large . For example, according to the method described in Japanese Patent Application Laid-Open No. 4-163589, when an anchor is set to an object reflected in a moving image for 10 seconds, three times a second in an NTSC image.
Since there are 0 still images, it is necessary to specify an area where an object is reflected in 300 images.

On the other hand, the technique described in Japanese Patent Application Laid-Open No. 9-322155 is aimed at a video surveillance device. It can be used for labor saving of anchor setting work. That is, a plurality of templates are provided for the object to be anchored, and the destination of the template in another image is detected by template matching,
The destination of the object can be automatically set from the result. Since a plurality of templates are provided in the object, it is possible to cope with a change in size. By using such a method, the hypermedia creator only needs to specify the anchor area once at first, and the anchor is automatically set in the subsequent image, so that the work load is greatly reduced.

[0005]

Since the tracking process of an object generally takes a long time, the creator is made to wait during the tracking process, and there is a problem that the production takes a long time. In order to avoid this, hardware is required, and the apparatus becomes expensive. That is, it is necessary to perform a tracking process for an object that can operate at high speed on a general computer.

[0006] Further, if the tracking ability of the object is low, the region of the object must be re-designated many times, and the load on the creator is still large, so that the tracking performance needs to be improved. Was.

[0007]

In order to solve the above-mentioned problem, according to the present invention, a tracking area of an initial image is designated for an image sequence input in time series, and the designated area is divided into a plurality of areas. Dividing into small areas, performing a destination detection process for each of the small areas using the correlation of the image sequence between the small area and the image including the small area, and selecting a size for each of the small areas A video area tracking method and apparatus, wherein among the small areas having individually selected sizes, a small area for performing a process of detecting a movement destination of the small area in an image at another time is selected. provide.

Here, when the size is selected for each of the small areas, the reliability of the small area is determined from the distribution of correlation values calculated when performing the destination detection processing using the correlation of the image sequence. Calculating and calculating the reliability of the divided area when the small area is further divided into smaller areas from the distribution of correlation values calculated when performing the destination detection processing of the area in the same image. Alternatively, it may be determined whether to perform the division by comparing the reliability of the small area before the division and the reliability after the division.

When selecting a small area for performing the processing for detecting the destination, the reliability of the small area is calculated from a distribution of correlation values calculated when performing the destination detection processing. In accordance with the degree of reliability, it may be determined whether or not to select a small area for performing a process of detecting a destination of the small area in an image at another time.

According to the present invention, when a partial region in an image is tracked as a tracking region for an image sequence input in a time series, a movement / modification model of the tracking region is determined in advance and selected. Estimate the value of the parameter representing the moving / modified model from the destination of the small area detected as a template with a plurality of small areas in the tracking area, and calculate the value of the parameter in the tracking area in accordance with the estimated parameter of the moving / modified model. A video area tracking method and apparatus, wherein a plurality of small areas are moved and deformed, and the moved and deformed small areas are used as a new template.

Here, when estimating a value of a parameter representing a movement / modification model from a destination of a small area detected as a template using a plurality of small areas in a previously selected tracking area, the reliability of the estimated parameter is The degree may be calculated, and the template may not be changed when the calculated reliability is smaller than a predetermined value.

[0012]

Embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a flowchart showing a processing flow according to an embodiment of the present invention. An outline of the processing will be described according to this flowchart.

First, an initial image is input (step 10). The input of the initial image is specified by a user using a graphical user interface (GUI) in a case where an anchor area of related information is specified as in hypermedia authoring. In the case of monitoring or the like, a point in time when an intruding object is detected is set as an initial image. There are various methods for detecting an intruding object, and the simplest method is a method in which a point at which the sum of absolute value differences between pixels of a continuously input image becomes large is set as a detection point.

Next, a tracking area in the initial image is designated (step 11). In the hypermedia authoring, this process is also specified by the user himself using the GUI. As a specification method, there is a method of enclosing an object to be tracked in an image with a rectangle or specifying each point of a polygon in order. On the other hand, in the case of monitoring, the area of the intruding object is detected by repeating the process of detecting pixels with large luminance and color changes by threshold processing and integrating the detected pixels when they are adjacent. , A tracking area. At this time, it is possible to obtain a more accurate tracking area by removing a small area or taking in an adjacent pixel having similar luminance and color to the detection area into the tracking area.

Next, the tracking area designated in step 11 is divided into blocks, and only some of the blocks are selected as tracking blocks to be actually tracked (step 12). The block size of the tracking block at this time is not fixed, and blocks of several types are selected. FIG. 4 shows an example in which the tracking area is divided into blocks.
40 is an object to be tracked, 41 is a designated tracking area, and 42 is a block. FIG. 5 shows an example in which the block created in FIG. 4 is divided as necessary to obtain blocks of several sizes. 51 is a block that did not need to be divided, and 52 is a divided block. FIG. 6 illustrates an example of a tracking block selected as a tracking target among the plurality of blocks in FIG.
Reference numeral 60 denotes a tracking object, and reference numeral 61 denotes a tracking block.

Next, a process of selecting an appropriate size for each block so that block matching, which is a subsequent process, can be performed with high accuracy is performed (step 12). A detailed description will be given later.

Next is the input of the next image (step 1).
3). The next image after the image currently being processed is input. At this time, the image input may be appropriately thinned out according to the processing capacity and the target tracking accuracy. For example, when an image of 30 frames / sec is obtained, the processing cannot catch up, and therefore, it may be possible to thin out the image by half and to process only one out of two images. However, if the movement of the object to be tracked in the image is large, if the number of thinnings is too large, the tracking accuracy becomes extremely poor.

Next, for each tracking block, block matching is performed with the next image, and the destination of the tracking block is calculated (step 14). FIG. 7 shows an example in which block matching is performed. In the figure, 70 is the current image, 7
1 is a next image. Reference numeral 72 denotes the position of the original block, and reference numeral 73 denotes the destination of the same block in the next image calculated by block matching. Reference numeral 74 denotes a search range of a destination of a block in block matching. In block matching, it is estimated that the image has moved to the position with the lowest degree of dissimilarity using the degree of dissimilarity (correlation value) between two images as a measure. The sum of value differences, the sum of squares of differences in luminance and color for each pixel, and the like are used.

FIG. 8 is an example of a destination obtained as a result of performing block matching between the current image and the next image for each tracking block. Block matching is easy to implement, but has the disadvantage of outliers. Therefore, accurate tracking can be achieved by preparing a movement / deformation model of the tracking object in advance and estimating the movement / deformation of the model from the distribution of the movement destination of each tracking block.

Next, parameters describing the model are estimated from the result of block matching for each tracking block (step 15). Examples of the movement / deformation model of the tracking object include a simplest translation model having two unknown parameters, an affine transformation model having six unknown parameters, and a projective transformation model having eight unknown parameters.

In the translation model, the unknown parameter vector is (t1, t2), the position coordinate of block i is (xi, yi), and the result of the block matching is (x'i, y'i). Then, the error is ei = (x′i, y′i) − (xi, yi) − (t1,
t2). According to the least squares method that minimizes the sum of squares of the error, this solution can be easily obtained.

In the affine transformation model, unknown parameter vectors are represented by (a1, a2, a3, a4, a5,
a6), the error vector is ei = (x'ia1 * xi-a2 * yi-a3, y '
i−a4 * xi−a5 * yi−a6) As in the case of the translation model, a solution can be obtained by the least squares method that minimizes the sum of squares of the error.

Next, the unknown parameters of the model thus set are estimated (step 15). And
The position of each block is updated using the movement / deformation parameters of the tracking object estimated in step 15 (step 16). For example, in the translation model, the cell i is updated to (xi + t1, yi + t2), and in the affine transformation model, (a1 * xi + a2 * yi + a3, a4 * x
i + a5 * yi + a6). These positions are
This is the final position of the block in the next image.

FIG. 9 shows an example of the block position updated in step 16. 90 is the position of the tracked object in the current image, 91 is the position of the tracked object in the next image, 92
Is the updated block.

It is also possible to selectively update the block. That is, when estimating the movement / deformation parameters of the tracking object, the reliability of the parameters is calculated at the same time, and the template updating process is performed only when the reliability is high. When the selective update is performed in this manner, even if the tracked object is temporarily hidden by another object (occlusion), the tracking can be continued. When estimating the deformation / movement parameters using the least squares method, the variance of the parameters estimated from the estimation error can be used to calculate the reliability of the parameters.

Next, continuation or termination of the tracking process is determined (step 17). The determination is made based on whether the end of the tracking process is designated by the user, whether the next image can be obtained, whether the tracking result is reliable, whether the tracking object is sufficiently large, and the like. Done. When the tracking processing is continued, the processing from step 13 is repeated using the updated tracking block position as a new tracking initial value. The above is the description of the flow of the entire tracking process according to the embodiment of the present invention.

Next, step 12 in FIG. 1 will be described in more detail. FIG. 2 is a flowchart showing the flow of the process shown in step 12. In step 12, an appropriate block size is determined to improve the reliability of block matching, and a process of selecting only a highly reliable block as a tracking block to reduce the amount of calculation and enable high-speed processing is performed. .

In step 20, processing for arranging the blocks in the tracking area designated in step 11 is performed. The block size at this time is a predetermined maximum block size. FIG. 4 is an example in which blocks are arranged in the tracking area. In step 20, a dividable label is given to all the arranged blocks of the maximum size. A divisible label is a label indicating that a block may be divided into smaller sized blocks.

In the next step 21, one of the tracking blocks with a dividable label is taken out, and this is set as a block i. First, the reliability Li of the block i is calculated. The reliability calculation method used here will be described later. Then, four small blocks obtained by dividing the block i into four are defined as i1, i2, i3, and i4, and the reliability Li1, Li2, Li3, and Li4 are calculated for each of them. Then, find the maximum value of the reliability of the four small blocks,
This is taken as the reliability of the small block, Si = max (Li1, Li
2, Li3, Li4).

In step 22, the two reliability values Li and Si calculated in the process 21 are compared with each other.
If so, the process proceeds to step S23. That is,
The block i is divided into four parts. On the other hand, if Si is smaller than Li, the division cannot be improved because the division cannot improve the reliability. In step 24, the divisible label is removed from the block i. Since the block i is not further divided, the block size is determined.

If division of a block is selected in step 22 and division is performed in step 23, it is determined in next step 25 whether the small blocks i1 to i4 are blocks of a predetermined minimum size. I do. Only when the small block is not the minimum size, a dividable label is given to the small block in step 26.

Next, at step 27, it is determined whether the processing is to be continued or terminated. If a block with a dividable label remains, the processing from step 21 is performed again. If there are no more blocks with a divisible label, the sizes of all tracking blocks have been determined, and the process ends. The above is the detailed description of Step 12.

Finally, a method of calculating the reliability of each tracking block will be described. FIG. 3 is a flowchart showing the flow of processing for calculating the reliability of a block. First, step 3
0, the position of the block in the initial image is (u,
v), the block dissimilarity D (u,
Calculate v). As described above, D (u, v) uses the sum of absolute differences of luminance and color of each pixel, the sum of squares of luminance and color of each pixel, and the like. At this time, the dissimilarity D
The range in which (u, v) is calculated is as follows: −W ≦ u, v ≦ W by a value W that is predetermined according to the size of the block.
Is determined. FIG. 10 shows an example of the calculated D (u, v). In the figure, 100 is an initial image, 101 is a block whose reliability is to be calculated, and 102 is a block (u,
A position shifted by v), 103 is an image area determined by -W ≦ u, v ≦ W. Reference numeral 104 denotes the distribution of the dissimilarity calculated in the area 103.

In step 31, a matrix R of 2 rows and 2 columns is calculated using the distribution of D (u, v) calculated in step 30. R is calculated as the curvature of D (u, v) near (0,0). However, the curvature is calculated after D (u, v) is divided by the square root of the number M of pixels in the block so that blocks having different sizes can be compared. The curvature is obtained by a second derivative filter. D
(U, v) is calculated by performing block matching in one image. Therefore, since this processing can be executed immediately after the tracking area is specified, the time for the initial processing can be reduced.

In step 32, the reliability r is calculated from the matrix R calculated in the process 31. As this method, R
Various things calculated from are available. For example, R
And calculating two eigenvalues λ1 and λ2 of R, and among these, the minimum value r = min
(Λ1, λ2), sum of eigenvalues r = λ1 + λ2
And the like. The reliability of the block calculated as described above is one predicted value of the reliability of the movement amount obtained by performing the block matching.

[0036]

According to the video area tracking method and apparatus of the present invention, a process of dividing an area designated as an initial image tracking target into a plurality of small areas and performing block matching with an image at another time for each small area is performed. To improve the tracking ability by selecting the size for each small area, and to further shorten the processing time by selecting the small area to be subjected to block matching from the small areas having individually selected sizes. Can be. In addition, according to the moving image region tracking method and apparatus of the present invention, when selecting the size for each small region, block matching is performed between the small region and the image including the small region, and calculation is performed at the time of block matching. Calculate the reliability of the small area from the distribution of similarity or dissimilarity obtained,
In addition, when the small area is further divided into smaller areas, the reliability of the divided area is calculated from the similarity or dissimilarity distribution calculated by block matching in the same image, and the reliability of the small area and the division are calculated. By determining whether or not to perform division by comparing with later reliability, a small area can be selected before the tracking processing, so that the time required for the initial processing of the tracking processing can be reduced. Still further, according to the moving image region tracking method and apparatus of the present invention, the tracking region is divided into small regions, and the process of selecting a small region to be subjected to block matching processing in the subsequently input image is designated as the tracking region. By performing the process only on the initial image, the processing time at the time of tracking can be reduced.

[Brief description of the drawings]

FIG. 1 is a flowchart of a tracking process according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating details of step 12 in FIG. 1;

FIG. 3 is a flowchart showing a method of calculating the reliability of a tracking block.

FIG. 4 is a diagram showing an example in which a tracking area is divided into blocks.

FIG. 5 is a diagram showing an example of a block divided into several types of block sizes.

FIG. 6 is a diagram showing an example of selecting a tracking block.

FIG. 7 is a diagram showing an example of block matching of a tracking block.

FIG. 8 is a diagram showing an example of a destination of a tracking block obtained by block matching.

FIG. 9 is a diagram illustrating an example of updating a tracking block.

FIG. 10 is a diagram illustrating an example of a calculation result of a dissimilarity of a block in an initial image.

[Explanation of symbols]

 40, 50, 60 Tracking target object 41 Tracking area 42, 51, 52, 61, 92, 101 Block 70 Current image 71 Next image 90 Position of tracking object in current image 91 Position of tracking object in next image 100 Initial image 102 block 103 shifted by (u, v) 103-Image area determined by W ≦ u, v ≦ W 104 Distribution of dissimilarity calculated in area 103

Claims (10)

[Claims] A step of designating a tracking area of an initial image for an image sequence input in time series; a step of dividing the designated area into a plurality of small areas; A step of performing a destination detection process for each of the small areas using the correlation of the image sequence between the image and the image including; a step of selecting a size for each of the small areas; Selecting a small area for performing a process of detecting a movement destination of the small area in an image at another time from among the small areas included in the image area tracking method. 2. The method according to claim 2, wherein the step of selecting a size for each of the small areas comprises: determining a reliability of the small area from a distribution of correlation values calculated in a step of performing a destination detection process using correlation of the image sequence. Calculating the reliability of the divided area when the small area is further divided into smaller areas, and a distribution of correlation values calculated when performing the destination detection processing of the area in the same image. 2. The video area according to claim 1, further comprising: a step of calculating whether or not to perform the division by comparing the reliability of the small area before the division and the reliability after the division. Tracking method. 3. The step of selecting a small area in which the processing for detecting the destination is performed comprises calculating reliability of the small area from a distribution of correlation values calculated in the step of performing the destination detection processing. The method according to claim 1, further comprising: determining whether or not to select a small area for performing a process of detecting a movement destination of the small area in an image at another time according to the reliability. Image area tracking method as described. 4. When a partial region in an image is tracked as a tracking region for an image sequence input in a time series, a movement / modification model of the tracking region is determined in advance, and a tracking region selected in advance is selected. Estimating a value of a parameter representing a moving / modified model from a destination of the small region detected as a template using a plurality of small regions in the; and A video area tracking method, comprising: moving and deforming a small area; and using the moved and deformed small area as a new template. 5. A step of estimating a value of a parameter representing a movement / modification model from a destination of a small area detected using a plurality of small areas in a preselected tracking area as a template, comprising: 4. The video area tracking method according to claim 3, wherein the degree is calculated, and the template is not changed when the calculated reliability is smaller than a predetermined value. 6. A means for designating a tracking area of an initial image for an image sequence input in time series, means for dividing the designated area into a plurality of small areas, the small area and the small area Means for performing a destination detection process for each of the small areas using the correlation of the image sequence, and means for selecting a size for each of the small areas; and Means for selecting a small area for performing a process of detecting a movement destination of the small area in an image at another time, among the small areas having the image data. 7. A means for selecting a size for each of the small areas, comprising: calculating a reliability of the small area from a distribution of correlation values calculated when performing a destination detection process using correlation of the image sequence. Means for calculating the reliability of the divided area when the small area is further divided into smaller areas from the distribution of correlation values calculated when the destination detection processing of the area in the same image is performed. 7. The video area tracking apparatus according to claim 6, further comprising: means for comparing the reliability of the small area before the division and the reliability after the division to determine whether to perform the division. . 8. A means for selecting a small area for performing the processing for detecting the movement destination, comprising: means for calculating reliability of the small area from a distribution of correlation values calculated when performing the movement detection processing. 7. The apparatus according to claim 6, further comprising: means for determining whether or not to select a small area for performing a process of detecting a destination of the small area in an image at another time according to the reliability. Image area tracking device. 9. When a partial area in an image is tracked as a tracking area for an image sequence input in time series, a movement / modification model of the tracking area is determined in advance, and a tracking area selected in advance is selected. Means for estimating a value of a parameter representing a movement / variant model from a destination of the small area detected as a template using a plurality of small areas in the plurality of small areas, and a plurality of parameters in the tracking area according to the estimated parameters of the movement / variant model. An image area tracking apparatus comprising: means for moving and deforming a small area; and means for using the moved and deformed small area as a new template. 10. A method for estimating a value of a parameter representing a movement / modification model from a movement destination of a small area detected using a plurality of small areas in a tracking area selected in advance as a template, comprising the steps of: 10. The video area tracking apparatus according to claim 9, wherein the degree is calculated, and the template is not changed when the calculated reliability is smaller than a predetermined value.