JP2002525735A - Tracking semantic objects in vector image sequences - Google Patents

Abstract

(57) Abstract The semantic object tracking method tracks general semantic objects with multiple rigid motions, fragmentary components, and multiple colors used throughout a vector image sequence. The method accurately divides these general semantic objects by spatially dividing the image regions from the current frame, and then classifying those regions with respect to which semantic object of the previous frame as the source. To track. To classify each region, the method performs a region-based motion estimation between each spatially partitioned region and the previous frame, relative to the predicted position calculated in the previous frame. The method then classifies each region of the current frame as part of the semantic object based on which semantic object of the previous frame contains the most overlapping point of the predicted region. Using this method, each current region is tracked without gaps or overlaps to one semantic object from the previous frame. This method does not attempt to project and adjust the boundaries in frames where the boundaries of the object are unknown, but rather projects the region into the frame where the boundaries of the semantic objects were previously calculated, resulting in little or no error. Does not propagate.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001] The present invention relates to the analysis of video data, and more particularly to semantic objects, where meaningful entities move through a vector image sequence, such as a video sequence. At the time, it relates to a method of tracking them.

BACKGROUND OF THE INVENTION Semantic video objects represent significant entities of digital video clips, such as balls, cars, airplanes, buildings, cells, eyes, lips, hands, and heads. The term "meaning" in this context means that the viewer of the video clip associates a meaning with the object. For example, each object listed above represents certain real world entities, and the viewer associates the portion of the screen corresponding to these entities with the significant objects that it depicts. Semantic video objects include content-based communications, multimedia signal processing, digital video libraries,
It is very useful in a variety of new digital video applications, including digital cinema studios and computer screen and pattern recognition. In order to use semantic video objects in these applications, object segmentation and tracking methods need to identify the objects in each video frame.

[0003] The process of segmenting video objects generally refers to an automated or semi-automated method of extracting relevant objects in image data. Extracting semantic video objects from video clips remains a challenging task for many years.
In a typical video clip, semantic objects consist of fragmentary components, different colors,
Includes multiple rigid / non-rigid motions. Although semantic objects are easy for a viewer to recognize, it is difficult to automate this process on a computer due to the wide variety of shapes, colors, and movements of semantic objects. . Satisfactory results can be achieved by having the user draw the first outline of the semantic object in the first frame and then using that outline to calculate the pixels that are part of the object in that frame. . In each successive frame, motion estimation can be used to predict an initial boundary of the object based on the segmented object from the previous frame. This semi-automatic object division tracking method is based on Chuang Gu and Ming Ch.
Semantic Video Object Segm by ie Lee
It is described in co-pending US patent application Ser. No. 09 / 054,280, entitled "entation and Tracking," which is hereby incorporated by reference.

[0004] Tracking an object is the process of calculating the position of an object as it moves from frame to frame. To address more general semantic video objects, object tracking methods must be able to address objects that contain fragmentary components and multiple non-rigid motions. Although much of the research has been devoted to object tracking, existing methods still do not accurately track objects with multiple components that make non-rigid motions.

Some tracking techniques track an area using the same gray scale / color as a reference. May 1992, Santa Margherita, Italy, ECCV'9
2, pp. 476-484. Meyer and P.M. "R" by Bouthemy
egion-based tracking in an image seq
uence ", June 1995, Proceeding of the IEEE
E, Vol. 83, no. 6, PP. Ph. 84-857. Salembie
r, L.R. Torres, F.C. Meyer, C.I. "Region-ba" by Gu
sed video coding using material
morphology ", February 1997, San Jose, VCIP'97, Vo.
l. 3024, no. 1, pp. F. 190-199. Marques and Cri
"Object tracking for" by stina Molina
content-based functions ", and 19
October 1997, Santa Barbara, ICIP '97, Vol. I, pages 113-
116 C.I. Toklu, A .; Tekalp, A .; "Simu by Erdem
ltaneous alpha map generation and 2-
D mesh tracking for multimedia appli
See "sessions".

[0006] Some people use the same motion information to track a moving object. For example, 1
In September 994, IEEE Trans. on Image Processin
g, Vol. 3, No. 5. pp. J. 625-638. Wang and E.W. Ad
"Representing moving images by Elson
with layers ", September 1996, Lausanne, Switzerland, IC
IP'96, Vol. I, pp. N. 925-928. Brady and N.M. O'C
"Object detection and tracki"
ng using an em-based motion estimati
See "on and segmentation framework".

Others track objects using a combination of spatial and temporal references. 1
May 992, Santa Margherita, Italy, ECCV'92, pp. 485
M.-493. J. "Combining institution
and motion for incremental segmenta
Tion and tracking over long image se
quences ", 1995, New York, Plenum Press, M.
ultmedia Communication and Video Co
ding, pp. 233 to 240; Gu, T .; Ebrahim, M .; Ku
"Morphological moving object se
gmentation and tracking for content-
based video coding, "May 1996, GA, Atlanta, Proc. ICASSP '96, Vol. 4, pp. 1914-1917 F
. Moscheni, F.C. Dufaux, M .; "Object by Kunt
tracking based on temporal, and spati
al information ", and October 1997, Santa Barbara, ICIP'97, Vol. II, pages 514-517, C.I. Gu and M.M. C.
Lee's "Semantic video object segment"
ation and tracking using materialatica
l morphology and perspective motion
model ".

[0008] Most of these techniques use a forward tracking mechanism that projects a preceding region / object into the current frame and manages to assemble / adjust the projected region / object in the current frame. A major drawback of these forward techniques is that it is difficult to assemble / adjust the projection area in the current frame or to cope with multiple non-rigid motions. In many of these cases, indeterminate holes may appear or result in distorted boundaries.

FIGS. 1A-C provide a simple example of a semantic video object showing the difficulties associated with object tracking. FIG. 1A shows a semantic video object of a building 100 that includes a plurality of colors 102,104. Methods that assume that objects have the same color do not track these types of objects well. FIG. 1B shows the same building object as FIG. 1A, but with fragmented components 106, 108 due to trees 110 partially obstructing the building.
Is different. Methods that assume that objects are formed from connected groups of pixels do not track these types of fragmentary objects well.
Finally, FIG. 1C shows a simple semantic video object representing the person 112. Even for this simple object, a plurality of components 114, 116, 118, 12 with different movements
Has zero. Methods that assume that objects have the same motion do not track these types of objects well. In general, semantic video objects are fragmented components,
It is possible to have multiple colors, multiple movements, and arbitrary shapes.

[0010] In addition to dealing with these attributes of the general semantic video object, the tracking method must achieve an acceptable level of accuracy and avoid errors from propagating from frame to frame. No. Typically, the object tracking method delimits each frame based on the segmentation of the previous frame, so errors in the previous frame tend to propagate to the next frame. If the tracking method does not calculate pixel boundaries with pixel accuracy, significant errors can propagate to the next frame. as a result,
The object boundaries calculated for each frame are not precise, and after tracking several frames, objects may be lost.

SUMMARY OF THE INVENTION The present invention provides a method for tracking semantic objects in a vector image sequence. The invention is particularly suitable for tracking semantic video objects in digital video clips, but can also be used for various other vector image sequences. This method is realized by a module of a software program.
It can also be implemented in digital hardware logic, or a combination of hardware and software components.

The method tracks semantic objects in an image sequence by segmenting a region from a frame and then projecting the segmented region onto a target frame at which one or more semantic object boundaries are known. I do. The projection region is classified as a forming part of the semantic object by determining the extent to which it overlaps the semantic object of the target frame. For example, in a typical application, the tracking method repeats, for each frame, classifying the region by projecting the region into a previous frame where the semantic object boundaries have been previously calculated.

The tracking method assumes that semantic objects have already been identified in the first frame. To obtain the initial boundary of the semantic object, it is possible to identify the boundary of the semantic object in the first frame using the semantic object division method.

[0014] After the first frame, the tracking method operates based on the segmentation results of the preceding frame and the current and preceding image frames. For each frame of the sequence, a region extractor divides the same region from the frame. Next, a motion estimator (motion estimator)
Perform region-based matching on each of these regions to identify the region of the image value that is most closely matched in the previous frame. Using the motion parameters obtained in this step, the divided area is projected onto the preceding frame in which the division boundary has already been calculated. Next, an area classification process (region c)
classification classifies a region as a part of the semantic object of the current frame based on the degree to which the projected region overlaps with the semantic object of the preceding frame.

[0015] The above approach is particularly suitable when operating on an ordered sequence of frames. In these types of applications, the segmentation result of the previous frame is used to classify regions extracted from the next frame. However, it can also be used to track semantic objects between the input frame and any other target frame whose semantic object boundaries are known.

Some implementations of the method use a unique spatial partitioning method. In particular, this spatial partitioning method is a region generation process in which image points are added to a region as long as the difference between the minimum and maximum image values for the region points is less than a threshold. The method is implemented as a sequential partitioning method, starting with a first region at a certain starting point and forming successive regions one after another using the same test to identify the same group of image points.

The implementation of the method includes other features that improve the accuracy of the tracking method. For example, the tracking method preferably includes region-based pre-processing to remove image errors without blurring object boundaries and post-processing on calculated semantic object boundaries. The calculated boundaries of the object are formed from the individual regions classified as being associated with the same semantic object in the target frame. In some implementations, the post-processor uses a majority operator filter to smooth semantic object boundaries. This filter is
Examine the points in the image that are close to each point in the frame and determine the semantic object that contains the maximum number of these points. The point is then assigned to the semantic object containing the maximum number of points.

[0018] Other advantages and features of the present invention will become apparent from the following detailed description and the accompanying drawings.

Detailed Description Overview of Semantic Object Tracking System The following sections describe semantic object tracking methods. This method assumes that the semantic object for the first frame (I-frame) is known. The purpose of this method is based on the information from the preceding semantic segment image and the previous frame,
The purpose is to find a semantic segment image in the current frame.

A basic observation with semantic segmented images is that the boundaries of segmented images are located at the physical edges of significant entities. The physical edge is the location between two connected points, and the values of the image at these points (eg, three color intensities, grayscale values, motion vectors) are significantly different. . Tracking methods use this observation to solve semantic video object tracking systems using segmentation and overcoming strategies.

First, the tracking method finds a physical edge in the current frame. This is achieved using a partitioning method, in particular a spatial partitioning method. The purpose of this segmentation method is to use the same image values (eg, color intensity triplets, grayscale values) in the current frame.
Is to extract all connected regions having Second, the tracking method classifies each region extracted in the current frame and determines to which object in the previous frame it belongs. This classification analysis is a region-based classification problem. After the region-based classification problem has been solved, the semantic video objects of the current frame have been extracted and tracked.

FIG. 2 is a diagram illustrating a semantic video object tracking system. The tracking system comprises the following five modules. 1. 1. Area pre-processing 220 Region extraction 222 3. 3. Region based motion estimation 224 4. Region-based classification 226 Post-processing of area 228

In FIG. 2, the following notation is used. I _i - frame input to the i image S _i - result of the spatial division with respect to the frame i M _i - motion to the frame i parameter T _i - tracking results for frame i

The tracking method assumes that the semantic video object for the first frame I ₀ is already known. Starting from the first frame, the segmentation process determines the first partition that defines the semantic object boundaries of the frame. In FIG. 2, I-divide block 210 represents a program for dividing a semantic video object. This program is
Incorporating the first frame I _0, to calculate the boundaries of the sense object. Typically, this boundary is represented as a binary or alpha mask. Using various splitting techniques,
It is possible to find a semantic object for the first frame.

As described in co-pending US patent application Ser. No. 09 / 054,280 by Gu and Lee, one approach is to allow the user to move around the inside and outside boundaries of semantic video objects. The purpose is to provide a drawing tool capable of drawing a boundary. This user-drawn border then serves as a starting point for an automated method that snaps the calculated border to the edges of the semantic video object. For applications involving multiple video objects of interest, the I-segmentation process 210 computes a segmented image, such as a mask, for each object.

The post-processing block 212 used in the first frame smoothes the first segmented image
And remove the error. This process is used for subsequent framesI ₁,I ₂ Is the same as post-processing used to process the results of tracking semantic video objects in
Or similar.

The next frame (I ₁) Input for the tracking process starting with the preceding frame I ₀ And result of previous frame divisionT ₀including. Dashed line 216 indicates for each frame
Separate processing. A dashed line 214 indicates processing for the first frame and the next frame.
Separation, but dashed line 216, indicates that subsequent semantic video
Separate processing for frames.

The semantic video object tracking starts at frame I ₁ . In the first step, to simplify the input frame I _1. In Figure 2, a simplified block 220 represents the region preprocessing step used to simplify the input frame I ₁ before the other analysis. Often, the input data contains noise that can adversely affect the tracking results. Region pre-processing removes noise and ensures that other semantic object tracking is performed on clean input data.

The simplification block 220 provides clean results that allow the segmentation method to more accurately extract the connected pixel regions. In FIG. 2, divided block 2
Reference numeral 20 denotes a space division method for extracting connected regions having the same image value in an input frame.

For each region, the tracking system determines whether the connected region is from a previous semantic video object. When the tracking stage is complete for the current frame,
The boundaries of the semantic video object in the current frame consist of the boundaries of these connected regions. Therefore, spatial partitioning should provide reliable partitioning results for the current frame. That is, no region should be missing, and no region should include an area that does not belong to it.

The first step in determining whether a connected region belongs to a semantic video object is to match the connected region with the corresponding region of the previous frame. As shown in FIG. 2, the motion estimation block 226 takes as input the connected regions and the current and previous frames and finds the corresponding region of the previous frame that most closely matches each region in the current frame. For each region, motion estimation block 226 provides motion information and predicts where each region of the current frame comes from the previous frame. This motion information indicates the position of the ancestor of each area in the preceding frame. The location information is then used to determine whether the current region belongs to a semantic video object.

Next, the tracking system classifies each region as to whether each region is derived from a semantic video object. In FIG. 2, the classification block 226 identifies semantic objects in previous frames that each region may be a source of. The classification process uses the motion information for each region to predict where that region comes from the previous frame. By comparing the predicted region to the segmentation result of the previous frame, the classification process
Determine the extent to which the predicted region overlaps the semantic object or the object already calculated for the previous frame. The result of this classification process associates each region of the current frame with a semantic video object or background. The semantic video object tracked in the current frame comprises a union of all regions concatenated with the corresponding semantic video object in the previous frame.

Finally, the tracking system post-processes the connected area for each object. In FIG. 2, a post-processing block 228 refines the acquired boundaries of each semantic video object in the current frame. This process removes any errors introduced in the classification process,
Smooth borders to improve visual effects.

For each subsequent frame, the tracking system repeats the same steps in an automated fashion, using the previous frame, the tracking result of the previous frame, and the current frame as inputs. Figure 2 shows an example of the processing steps are repeated for the frame I _2. Blocks 240-248 represent the steps of the tracking system applied to the next frame.

Unlike other area and object tracking systems that use various forward tracking mechanisms, the tracking system shown in FIG. 2 performs backward tracking. The reverse domain-based classification approach has the advantage that as a result of the spatial division, the boundaries of the final semantic video object are always located at the physical edge of the significant entity. Also, since each region is treated individually, the tracking system can easily cope with fragmented semantic video objects or non-rigid motion.

Definitions Before describing the implementation of the tracking system, it will be helpful to start with the set of definitions used throughout the following description. These definitions help indicate that the tracking method applies not only to sequences of colored video frames, but also to other temporal sequences of multi-dimensional image data. In this context, "multi-dimensional" refers to the spatial coordinates of each discrete image point, as well as the value of the image at that point. A temporal sequence of image data can be referred to as a "vector image sequence" because it consists of consecutive frames of a multidimensional data array. As examples of vector image sequences, consider the examples listed in Table 1 below.

[0037]

[Table 1]

The dimension n refers to the number of dimensions in the spatial coordinates of the image sample. The dimension m refers to the number of dimensions of the value of the image located at the spatial coordinates of the image sample. For example, the spatial coordinates of a color volume image sequence include three spatial coordinates that define the position of the image sample in three-dimensional space, and therefore n = 3. Each sample of the color volume image has three color values R, G, and B, and therefore m = 3.

The following definitions provide a basis for describing a tracking system in the context of a vector image, using theoretical notation for sets and graphs.

Definition 1 Connection Point: S is an n-dimensional set. The point p∈ S ⇒p = (p1,..., _Pn ). ∀p, q
∈ S , p and q are connected only when their distance D _{p, q} is equal to one.

[0041]

(Equation 1)

[0042] Definition 2 connection path: P (P ⊆ S) is, m number of points p1,. . . a route consisting of a p _m. The route P is
p _k and _{p k + 1 (k∈ {1} , ..., m-1} is connected only when a connection point.

[0043] Definition 3 near point: R (R ⊆ S) is a region. point

[0044]

(Equation 2)

[0045] is proximate only area R if ∃ another point q (q∈ R) p and q are connected points.

[0046] Definition 4 connection region: R (R ⊆ S) is a region. R is, ∀x, y∈ R, ∃ connection path _{P (P = {p 1,} ... p m,}) if a _p 1 = x and _P n = y is only connected area.

Definition 5 Segmented Image: A segmented image P is a mapping P:S→ T, where T is fully ordered
FIG.R _p(X) is the point x:R _p(X) = ∪ _y∈S An area including {y | P (x) = P (y)}. Segmented images must meet the following conditions
Must meet. {X, y}S,R _p(X) =R _p(Y) orR _p(
x) ∩R _p(Y) = φ; ∪_x∈S R _p(X) =S.

Definition 6 Connection Division Image: The connection division image is the division image P, and {x} S , R _p (x) are always connected.

Definition 7 Fine Division The division image P isSIf it is finer than the other segmented image P 'above, this is {x} S ,R _p(X) ⊇R _{p '}Means (x).

[0050] Definition 8 crude Indicator: If partition image P is another segment crude from the image P 'on S, which, ∀X∈ S
Means _{R p (x) ⊆ R p} '(x).

There are two ultimate cases for a segmented image. One is the "coarse segment"
This is all the S: ∀x, y∈ S, up to _{R p (x) = R p} (y). The other is the "finest classification", each point S, the individual regions: ∀x, a _{y∈ S, x ≠ y⇒ R p} (x) ≠ R p (y).

[0052] Definition 9 adjacent regions: two areas R ₁ and R _2, ∃x, to y (x ∈ R ₁ and y∈ R _2), adjacent only when x and y are connected points.

Definition 10: Graph adjacent to area: P is a multidimensional setSIt is an upper division image. P has k (R ₁,. . . ,R _k)of
There is an area,S= ∪R _i, And i ≠ j⇒R _i∩R _j= Φ. Area adjacent graph
RAG is the vertexVSet and edge setLConsists ofV= ｛V₁,. . .
, V_k｝ And each v_iIs the corresponding areaR _iAssociated with Set of edges L Is ｛e₁,. . . , E_t｝,

[0054]

(Equation 3)

[0055] a, wherein each e _i has two corresponding regions may be contiguous region, is built between two vertices.

3A to 3C show examples of different types of segmented images, and FIG. 3D shows an example of an area adjacency graph based on these segmented images. In these examples, S is a set of two-dimensional images. White areas 300 to 308, hatched areas 310 to 314, and point areas 316
Represents different regions in the two-dimensional image frame. FIG. 3A shows a segmented image having two fragmentary regions (white areas 300 and 302). FIG. 3B shows a connection section image having two connection areas (white area 304 and hatched area 312). FIG. 3C
3A shows a finer segmented image as compared to FIG. 3A in that the shaded area of FIG. 3A comprises two regions, a shaded area 314 and a point area 316. FIG. 3D shows a corresponding region adjacency graph of the segmented image of FIG. 3C. Vertices 320, 322, 324 of the graph,
326 corresponds to the regions 306, 314, 316, and 308, respectively. Rim 33
0, 332, 334, 336, and 338 connect the vertices of adjacent regions.

Definition 11 Vector Image Sequence: Product

[0058]

(Equation 4)

M (m ≧ 1) fully ordered complete lattices L₁,. . . , L_mGiven
The vector image sequence is mappedI _t:S→LIs the sequence of S Is an n-dimensional set, and t is in the time domain.

Some types of vector image sequences are shown in Table 1. These vector image sequences can be obtained from a series of sensors, such as color images, or from a calculated parameter space, such as a field of dense motion. Although the physical meaning of the input signals varies from case to case, they are all without exception considered vector image sequences.

Definition 12 Semantic Video Object: Let I be a vector image on an n-dimensional set S. P is a semantic division image of I. S = ∪i _{= 1,. . . , M} O _i , where each O _i indicates the location of a semantic video object.

Definition 13 Meaning Video Object Division: Let I be a vector image on an n-dimensional set S. Means video object division shall locate the number of objects m and each object O _i.

For i = 1,. . . , M, S = ∪i _{= 1,. . . , M} O _i .

Definition 14 Semantic Video Object Tracking:I _t-1Is an n-dimensional setSVector image above_t-1Is the time t-
1, and the corresponding semantic division image.S= ∪_{i = 1,. . . , M} O _{t-1, i}so
is there. eachO _{t-1, i}(I = 1,..., M) is the semantic video pair at time t−1
It is an elephant.I _tThe video object tracking at time t, i = 1,. . . , M means
Defined when finding a taste video object. ∀x∈O _t-1,_iAnd {y} O _{t, i} : P_t-1(X) = P_t(Y).

Example Implementation The following sections describe in more detail certain semantic video object tracking methods. FIG. 4 is a block diagram showing the main components of the implementation described below. Each block in FIG. 4 represents a program module that implements part of the object tracking method outlined above. Depending on various considerations, such as cost, performance, and design complexity, each of these modules can also be implemented in digital logic.

Using the notation defined above, the tracking method shown in FIG.
Dividing the result of the preceding frame at 1 and incorporating the present vector image I _t. The current vector image consists of m (m ≧ 1) fully ordered complete grids L ₁ ,... Of the product L (see Definition 11) on the n-dimensional set S. . . , Defined in _{L m.}

[0067] _{∀p, q∈ S, I t (} p) = {L 1 (p), L 2 (p) ,. . . , L _m (p)
｝

Using this information, the tracking method calculates a segmented image for each frame of the sequence. The result of the division is a mask that identifies the position of each semantic object in each frame. Each mask has, in each frame, an object number that identifies which object it corresponds to.

For example, consider the color image sequence defined in Table 1. Each point p represents a pixel of the two-dimensional image. The number of points in the set S corresponds to the number of pixels in each image frame. The grid at each pixel comprises three sample values corresponding to the red, green, and blue intensity values. The result of the tracking method is a series of two-dimensional masks that identify the location of all pixels that form part of the corresponding semantic video object for each frame.

Region Pre-Processing The implementation shown in FIG. 4 starts processing on frames by simplifying the input vector image. In particular, the simplified filter 420 cleans the entire input vector image and then performs further processing. In this pre-processing stage design, it is preferable to select a simplified method that does not introduce false data. For example, a low-pass filter may clean and smooth the image, but may also distort the boundaries of the video object. Therefore, it is preferable to select a method that simplifies the input vector image while retaining the position of the boundary of the semantic video object.

Many non-linear filters, such as median filters or morphological filters, are candidates for this task. In the current implementation, to simplify the input vector image,
Use a vector median filter, Median (•).

The vector median filter calculates the value of the median image of a point adjacent to each point of the input image, and replaces the value of the image at that point with the median. For every point p in the n-dimensional set S , a structuring element E is defined around it, which includes all connection points (see definition 1 for connection points).

E = { _q } _S {D _{p, q} = 1}

The vector median of the point p is defined as the median of each component in the structuring element E.

The median (I _t (p)) = {median _q∈E ｛L ₁ (q),. . . Median _{q∈ E {L m (q)} }}

By using such a vector median filter, the vector imageI _t Small fluctuations can be eliminated, while the boundaries of the video object are
ElementaryEWell maintained under the spatial design. As a result, the tracking process is more effective
Consequently, semantic video object boundaries can be identified.

Region Extraction After filtering the vector input image, the tracking process extracts regions from the current image. To achieve this, the tracking process takes a current image and identifies a space segmentation method 42 that identifies regions of connection points having "same" image values.
Use 2. These connection regions are regions of points used in region-based motion estimation 424 and region-based classification 426.

There are three main issues that need to be addressed in performing the region extraction stage. First, the concept of “identical” needs to be strengthened. Second, the total number of regions should be found. Third, the position of each region must be fixed. Documents relating to the division of vector image data describe various spatial division methods.
Most common spatial partitioning methods use the following.

A polynomial function defining the identity of the regions; a deterministic method of finding the number of regions; and / or a boundary adjustment to finalize the position of all regions.

Although these methods can produce satisfactory results in some applications, non-rigid motion, fragmented regions, and a wide variety of semantic video objects with multiple colors Does not guarantee accurate results. Since the accuracy with which semantic objects can be classified depends on the accuracy of the region, the accuracy required for the space division method is very high. After the division step, it is preferred that no region of the semantic object is missing, and that no region contains a zone that does not belong to it.
Since the boundaries of the semantic video objects in the current frame are defined as a subset of the total boundaries of these connected regions, their accuracy directly affects the accuracy of the results of the tracking process. If the boundaries are incorrect, the resulting boundaries of the semantic video object will also be incorrect. Therefore, the spatial partitioning method should provide an accurate spatial segmentation image for the current frame.

The current implementation of the tracking method uses a new and fast space division method called LabelMinMax . This particular approach generates one region at a time, in a sequential fashion. This approach is different from other concurrent region generation processes, that is, the need to identify all seeds before region generation begins with any seed. The sequential area generation method extracts areas one after another. As a result, each area can be handled more flexibly, and the complexity of the overall calculation is reduced.

The identity of an area is controlled by the difference between the maximum and minimum values of the area. Input vector image I _t is completely lattice _L 1 of m which are ordered full of product L (m ≧ 1),.
. . , It assumed to be defined in _{L m} (see Definition 11).

[0083] _{∀p, q∈ S, I t (} p) = {L 1 (p), L 2 (p) ,. . . , L _m (p)
｝

The maximum value and the minimum value ( MaxL and MinL ) of the region R are defined as in the following expression.

[0085]

(Equation 5)

If the difference between MaxL and MinL is smaller than the threshold value ( H = {h ₁ , h ₂ ,..., H _m }), the regions are the same.

[0087] Identity; ∀i, 1 ≦ i ≦ m , (max p∈R {L i (p)} - min p∈R {L i (p)} ≦ h i

[0088]LabelMinMaxThe method names each region in turn. n-dimensional setS
Starting from point p. regionRIsLabelMinMaxIs working on it
Suppose the current area is At the start, the point p:R= {P} only. Next
ToLabelMinMaxIs the areaRInspect all adjacent points (see Definition 3)
, When the proximity point q is inserted therein,RAre still the same
Find out. If insertion does not change the identity of the region, point q is the regionRAdded to
It is. Point q is the areaR, The point q is the setSShould be erased from
. regionRGradually up to the same territory where no further points can be added
Expanding. ThenSA new region is constructed in terms of the points remaining in. S This process continues until there are no more points remaining. The whole process is
This can be clearly explained by the following pseudo code.

LabelMinMax:

[0090]

(Equation 6)

LabelMinMax has many advantages, including:

• MaxL and MinL provide a more precise description of the identity of the regions compared to other criteria. • The definition of identity gives more control over the identity of the regions that yields the exact region. LabelMinMax gives reliable spatial segmentation results. -LabelMinMax is much less complicated to calculate than many other methods.

These advantages make LabelMinMax a good choice for spatial analysis, and also allow alternative partitioning methods to be used to identify connected regions. For example, other region generation methods use different same criteria and models of "same" regions to determine whether additional points are added to the same region. For example, these criteria include an intensity threshold, and points are added to the region as long as the difference in intensity between each new point and the neighboring points of the region does not exceed the threshold. Also, the same criterion can be defined in terms of a mathematical function that describes how the intensity values of points in the region can vary and still be considered part of the connected region.

Region-Based Motion Estimation The process of region-based motion estimation 424 matches the image values of the region identified by the segmentation process with the corresponding image values of the previous frame, and moves the region from the previous frame. Estimate the method used. To illustrate this process, consider the following example. I _t-1 is the previous vector image on the time t-1 of the n-dimensional set S, _{I t} is the current vector image on the same set S of time t. Region extraction procedure, N pieces in the current frame I _t in the same region _{R i (i = 1,2, ...} , N
) To extract.

S = { _{i−1,. . . , N} R _i

Here, the tracking process proceeds, classifying each region as belonging to exactly one of the semantic video objects of the preceding frame. The tracking process solves this region-based classification problem using region-based motion estimation and compensation. For each extracted region R _i of the current frame I _t, running motion estimation procedure, these regions, find a place that occurred in the previous frame I _t-1. Although many motion models can be used, current implementations use a translational motion model as the motion estimation procedure. In this model, the motion estimation procedure computes a motion vector V _i for region R _i to minimize the expected error (PE) for that region.

[0097]

(Equation 7)

In the above equation, ‖ · ‖ represents the sum of absolute differences between two vectors,V _i≤V _max (V _maxIs the maximum search range). This motion vectorV _iIs the preceding frameI _t-1 Area indicating the position of the trajectory atR _iAssigned to.

[0099] Other motion models can be used as well. For example, an affine or perspective motion model can be used to model motion between a region of the current vector image and a corresponding region of the preceding vector image. Affine and perspective motion models use geometric transformations (eg, affine or perspective transformations) to define the motion of a region between one frame and another. This transformation is represented by motion coefficients that can be calculated by finding the motion vectors for some points in the region and then solving the system of equations using the motion vectors at the selected points to calculate the coefficients. You. Other schemes select an initial set of motion coefficients and then repeat until the error (eg, the sum of absolute differences or the sum of squared differences) is less than a threshold.

Region-Based Classification The region-based classification process 426 uses the motion information to change the position of each region and determine the estimated position of the region in the previous frame. Then, as compared with the boundary of the meanings video object of this estimated position prior frame (S _t), to determine the most easily form a portion of which means the video object.

To illustrate, consider the following example. I _t-1 and I _t are the preceding and current vector images on n-dimensional set _{S, P t-1} is the corresponding mean division image at time t-1.

S = ∪i _{= 1,. . . , M} O _{t−1, i}

Each O _{t−1, i} (i = 1,..., M) indicates the position of the semantic video object at time t−1. There are N extracted total regions R _i (i = 1, 2,..., N), and each region has an associated motion vector V _i (i = 1, 2,...) In the current frame.
. . . , N). Here, tracking method, it is necessary to construct the current meaning segmented image P _t at time t.

The tracking process performs this task by finding a semantic video object O _{t −1, j} ( _j {1,2, ..., m}) for each region R _i in the current frame. Is implemented.

Since the motion information for each region R _i is already available at this stage, the region classifier 426 uses reverse motion compensation to convert each region R _i in the current frame to
Warp to the previous frame. Warping a region by applying motion information for the region to points in that region. The area was warped in the previous area R
It is assumed that the _'i.

R ′ _i = {p} _{R i} {p + V _i }

Ideally, the warped region R ′ _i is one of the semantic video objects of the previous frame.
That should be the case.

{I, j} 1, 2,. . . , M} and _{R 'i ⊆ O t-1} , j

If this is the case, the tracking method assigns a semantic video object O _{t−1, j} to this region R _i . However, in practice, due to potential ambiguity from the motion estimation process, R ′ _i can overlap with multiple semantic video objects in the previous frame. That is,

[0110]

(Equation 8)

Is as follows.

The current implementation uses a majority criterion M for region-based classification. For each region R _i of the current frame, the warped region R _'i majority part of the meaning of the previous frame video object _{O t-1, j (J∈1,2} , ..., m) from If you, this region is assigned to the meaning video object O _{t-1, j.}

{P} R _i , and {q} O _{t−1, j} , P _t (p) = P _t−1 (q)

More specifically, a semantic video object O _{t−1, j} having a majority area (MOA) overlapping R ′ _i is found as follows:

[0115]

(Equation 9)

The full semantic video object O _{t, j} of the current frame uses this region-based classification procedure for all regions R _i (i = 1, 2,..., M ) of the current frame. By use, they are built one by one. Point q∈ O _{t-1, j,}

[0117] _{O t, j = ∪ p∈S {} p | P t (p) = P t-1 (q)}, j = 1,2 ,. .
. , M

It is assumed that With the design of this region-based classification process, there will be no holes / gaps or overlaps between different semantic video objects in the current frame.

∪ _{i = 1,. . . ,} MO _{t, i} = ∪ _{i = 1,. . . ,} NR _i = ∪ _{i = 1,. . .} , MO _{t−1, i} = S ∀i, j∈ ｛1,. . . , M｝, i ≠ j⇒Ot, i∩Ot, i = φ

This is an advantage of this tracking system as compared to a tracking system that tracks objects into frames where the boundaries of semantic video objects cannot be determined. For example, in a forward tracking system, object tracking proceeds to subsequent frames where the precise boundaries are unknown. The boundary is then adjusted to fit the unknown boundary based on some predetermined criteria that models the boundary conditions.

[0121] region aftertreatment tracking results of the current frame is assumed as meaning segmented image P _t. For various reasons, there may be some errors in the region-based classification procedure. The purpose of the region post-processing process is to eliminate these errors while smoothing the boundaries of each semantic video object in the current frame. Interestingly, segmented images are spatial images that are different from conventional images. The value at each point in this segmented image only indicates the location of the semantic video object. Therefore, in general,
All conventional linear or non-linear filters for signal processing are not suitable for this spatial post-processing.

This implementation uses a majority operator M (•) to perform this task.
For each point p of the n-dimensional set S , the structuring element E is defined around it, including all connection points (see 1 for connection points).

E = { _p } _S {D _{p, q} = 1}

[0124] First, the majority operator M (·) is meant video object O _t having an area (MOA) that maximize the structural element E _overlap, find _j.

[0125]

(Equation 10)

[0126] Second, the majority operator M (·) is the meaning video object O _t, the value of _j, assigned to the point p.

[0127] q∈ O _{t, j,} and P t (p) = M ( p) = P t (q).

Due to the adoption of the majority criterion, it is possible to eliminate very small areas (most likely to be errors), while at the same time smoothing the boundaries of each semantic video object.

Brief Overview of Computer System FIG. 5 and the following discussion are intended to provide a brief, general description of a suitable computer environment in which the invention may be implemented. Although the present invention or aspects thereof may be implemented in hardware devices, the above-described tracking systems are implemented with computer-executable instructions organized in program modules. Program modules are routines, programs,
Includes objects, components, and data structures that perform tasks and perform the data types described above.

While FIG. 5 shows a general configuration of a desktop computer, the present invention includes hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. It is possible to implement the present invention with other computer system configurations. Also, the present invention can be used in a distributed computing environment where tasks are performed by remote processing devices that are linked through a computer network. In a distributed computer environment, program modules may be located in both local and remote memory storage devices.

FIG. 5 shows an example of a computer system serving as the operating environment of the present invention. The computer system includes a personal computer 520 that includes a processing unit 521, a system memory 522, and a system bus 523 that interconnects various system components including the system memory to the processing unit 521. The system bus includes a memory bus or memory controller, a peripheral bus, a PCI,
Any of a number of types of bus structures can be provided, including VESA, Micro Channel (MCA), ISA and EISA, and local buses using bus architectures as examples. The system memory includes a read-only memory (ROM) 524 and a random access memory (RAM) 525. The basic input / output system 526 (BIOS) contains basic routines that help to transfer information between elements within the personal computer 520, such as during startup, and is stored in the ROM 524. Further, the personal computer 520 includes a hard disk drive 527 and, for example, a removable disk 52.
9, a magnetic disk drive 528 that reads from or writes to
An optical disk drive 530 for reading a D-ROM disk 531 or reading or writing to other optical media. Hard disk drive 5
27, the magnetic disk drive 528, and the optical disk drive 530 are connected to the system bus 523 by a hard disk drive interface 532, a magnetic disk drive interface 533, and an optical drive interface 534, respectively. The drives and their associated computer-readable media provide nonvolatile storage of data, data structures, computer-executable instructions (
Program code such as a dynamic link library and an executable file) are provided to the personal computer 520. The above description of computer-readable media refers to hard disks, removable magnetic disks, and CDs, but other types of computers that can be read by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, and the like. A medium can be included.

[0132] A number of program modules are stored in an operating system 535, one or more application programs 536, and other program modules 5
37 and the program data 538 can be stored in the drive and RAM 525. The user can input commands and information to the personal computer 520 through a position display device such as a keyboard 540 and a mouse 542. Other input devices (not shown) include a microphone,
It may include a joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 521 via a serial port interface 546 that is coupled to the system bus, but other ports such as a parallel port, game port, or universal serial bus (USB). It is also possible to connect by the interface of. A monitor 547 or other type of display device may also be connected to the system bus 52 via an interface such as a display controller or video adapter 548.
3 is connected. Typically, in addition to a monitor, a personal computer includes other peripheral output devices (not shown) such as speakers and a printer.

The personal computer 520 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 549. Remote computer 549 can be a server, router, peer device, or other general network node, and typically includes many or all of the elements described with respect to personal computer 520; Only the memory storage device 550 is shown. The logical connections shown in FIG. 5 include a local area network (LAN) 551 and a wide area network (WAN). Such a networking environment can be a company,
It is common in enterprise-wide computer networks, intranets, and the Internet.

When used in a LAN networking environment, the personal computer 520
Is connected to the local network 551 via a network interface or an adapter 553. When used in a WAN networking environment, the personal computer 520 typically includes a modem 554 or other means for establishing communication over a wide area network 552, such as the Internet.
The modem 554 can be internal or external, and is connected to the system bus 523 via a serial port interface 546. In a networked environment, program modules described relative to the personal computer 520, or portions thereof, may be stored in the remote memory storage device. The network connections shown are merely examples, and other means of establishing a communication link between the computers can be used.

Conclusion Although the invention has been described in the context of particular implementation details, the invention is not limited to these particular details. The present invention provides a method and system for semantic object tracking that identifies identical regions in a vector image frame and then classifies these regions as part of semantic objects. The classification method of the implementation described above is called "backward tracking" because the semantic region is projected onto the previous frame where the boundaries of the semantic object were previously calculated.

The tracking system is also generally applied to applications where the boundaries of semantic video objects are known, even if these frames are not the preceding frames in an ordered sequence, the application of a segmented region. Note that Therefore, the "reverse" tracking scheme described above applies to applications where the classification is not necessarily limited to the previous frame, but instead is limited to frames where the boundaries of the semantic object are known or previously calculated. Is done. Frames where semantic video objects have already been identified are more generally referred to as reference frames. Tracking of semantic objects relative to the current frame is calculated by classifying the regions divided by the current frame with respect to the semantic object boundaries of the reference frame.

As described above, the object tracking method is generally applied to a sequence of vector images. Therefore, it is not limited to a 2D video sequence, or a sequence in which the values of the images represent intensity values.

The description of the segmentation stage has identified criteria that are particularly useful, but are not required for all implementations of semantic video object tracking. As already mentioned, other segmentation techniques can be used to identify the connected areas of the points. The definition of region identity is
It can vary depending on the type of image values (eg, motion vector, color intensity) and application.

The motion model used to perform motion estimation and compensation can be varied as well. Although the calculation is more complicated, it is possible to calculate a motion vector for each individual point in the region. Alternatively, one motion vector can be calculated for each region, as in the transformation model described above. Preferably, a matching region should be found in the frame of interest using a region-based matching method. Region-based matching uses the current frame boundary or mask to exclude points located outside the region from the process of minimizing errors between the predicted point and the corresponding region in the reference frame I do. This type of approach is called Mi
Polygon Block Match by ng-Chie Lee
No. 5,796,855 to ing Method and is incorporated herein by reference.

Given the many possible implementations of the present invention, the implementations described above are only examples of the present invention and should not be considered as limitations on the scope of the present invention. Rather, the scope of the present invention is defined by the appended claims. We therefore claim that all our invention comes from within the scope and spirit of these claims.

[Brief description of the drawings]

FIG. 1A is an example representing different types of semantic objects to show the difficulty of tracking a general semantic object.

FIG. 1B is an example representing different types of semantic objects to show the difficulty of tracking a general semantic object.

FIG. 1C is an example representing different types of semantic objects to show the difficulty of tracking a general semantic object.

FIG. 2 is a block diagram showing a semantic object tracking system.

FIG. 3A is a diagram illustrating an example of a segmented image and a method of representing the segmented image in an area proximity graph.

FIG. 3B is a diagram illustrating an example of a segmented image and a method of representing the segmented image in an area proximity graph.

FIG. 3C is a diagram showing an example of a segmented image and a method of representing the segmented image in the area proximity graph.

FIG. 3D is a diagram illustrating an example of a segmented image and a method of representing the segmented image in the area proximity graph.

FIG. 4 is a flowchart illustrating an implementation of a semantic object tracking system.

FIG. 5 is a block diagram of a computer system that serves as an operating environment for an implementation of the present invention.

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Min-Cherry USA 98006 Bellevue, Washington Southeast 5558-166 Place (No address) F-term (reference) 5C054 EA05 FB03 FC13 FE24 HA31 5C059 MA01 MB02 MB03 NN24 NN36 PP04 PP26 PP28 RF11 5L096 FA02 FA36 HA04 JA11 [continued from summary] Does not propagate any or no errors.

Claims (20)

[Claims] 1. A method for tracking a video object in a video frame, comprising: performing a spatial division on the video frame to identify regions of pixels having the same intensity value; Perform motion estimation between frames, warp the position of pixels in each region to the position of the previous frame using motion estimation for each region, and position the warped pixels in the divided video object of the previous frame. To determine the set of regions that may be part of the video object, and as a combination of each region in the set of video frames, the boundary of the video object in the video frame Forming a method. 2. The method according to claim 1, wherein the steps of claim 1 are repeated for subsequent frames, using the boundaries of the video object as reference boundaries for the next frame. 3. The method of claim 1, wherein the video frames are filtered to remove noise from the video frames, and then performing spatial partitioning. 4. Each region is a connected group of pixels, and a difference between an intensity value of a pixel having a maximum intensity value of the region and an intensity value of another pixel having a minimum intensity value region is smaller than a threshold value. The method of claim 1, wherein each area is determined to be the same by ensuring that it is small. 5. The method of claim 1, wherein the segmentation is a sequential region generation method, wherein starting from a first pixel position of a video image frame, a first region of connected pixels is identified by adding a pixel to the region. Is generated around the first pixel so as to satisfy the following condition. When there is no boundary pixel that satisfies the same criterion, the generation step occurring at the position of the pixel outside the first region is repeated, and each pixel of the frame is 5. The method according to claim 4, wherein the steps of occurring occur until the identification is made. 6. Performing region-based motion estimation, wherein for each region identified through spatial partitioning in a video frame, matching only pixels in the region with pixels of a previous frame. The method of claim 1, further comprising: finding a corresponding position for each pixel in a previous frame, and applying a motion model that approximates the motion of the pixel in the region to the corresponding position in the previous frame. 7. Use a motion model to find a motion vector for each region that minimizes prediction errors between pixel values warped from a video frame and pixel values at corresponding pixel locations in a previous video frame. 7. The method according to claim 6, wherein
The method described in. 8. The step of determining: finding the number of warped pixels inside the boundary of the segmented video object of the previous frame; and when the majority of the warped pixels is inside the boundary of the segmented video object, The method of claim 1, wherein the region is classified as a portion of a video object in a video frame. 9. A computer readable medium having instructions for performing the steps of claim 1. 10. A computer readable medium having instructions for tracking semantic objects in a vector image sequence of image frames, a spatial division module for dividing a vector image frame of the image sequence into regions. A spatial partitioning module having a connected group of image points having image values satisfying the same criterion; estimating motion between each region of the input image frame and the reference frame; A motion estimation module that determines a motion parameter that approximates the motion of each region between frames; a motion estimation module that applies a motion parameter of each region to the region to calculate a predicted region in the reference frame; , At least in part, assessing whether it is within the boundaries of the semantic object in the reference frame, and Classifying each area as a part of the semantic object in the reference frame based on a certain degree within the boundary of the semantic object boundary of the reference frame. A computer-readable medium formed from each area classified as a part of a meaning object. 11. The same criterion of the space division module is that the point of the first image of the connected group of pixels having the largest image value and the second image of the connected group having the smallest image value. 9. The method according to claim 8, wherein the segmentation module selectively adds a point of the image to the connection region having a maximum difference from the point to generate a new connection region while the new connection region satisfies the same criterion. 11. The method according to 10. 12. The motion parameter of each region is a motion vector, and when the motion vector is used to project a point of each image in the region to a target frame, an image value of the projected point; 11. The method of claim 10, wherein the sum of differences between corresponding image points of the target frame and image values is minimized. 13. The target frame includes two or more semantic objects, each object occupying a non-overlapping area of the target frame, and by region analysis, for each prediction region, an overlapping image of the prediction region. Identifying a semantic object of the target frame having the maximum number of points of the overlapped image and classifying each region as being associated with the semantic video object of the target frame having the maximum number of points of the duplicate image; Calculating a boundary of each semantic object in the image frame as a combination of regions classified as being associated with a corresponding semantic object in the target frame. Method. 14. Define the structure of points around each point of the image frame, determine the semantic object in the image frame having the largest overlapping structure area, and convert the value of the semantic video object to a point in the image. 14. The medium of claim 13, further comprising a majority operator to assign. 15. A method for tracking semantic objects in a vector image sequence, comprising performing a spatial division on an image frame to identify regions of points of a discrete image with values of the same image, Performing motion estimation between each region and a target image frame whose boundary of the semantic object is known, using the motion estimation for each region to warp the image point of each region to the position of the target frame; Determine whether the location of the warped pixel in each region is within the boundary of the semantic object in the target frame, and when at least the threshold amount of the region overlaps the semantic object in the target frame, Classify the area as the source of the object, and set the boundary of the semantic object of the image frame as the combination of the areas of the image frame classified as the source of the semantic object of the target frame. Wherein the forming as a cause. 16. The step of claim 15, wherein the calculated boundary of the semantic object of the previous frame is used to repeat the steps of claim 15 and the region divided in the current frame from one of the semantic objects of the previous frame as a source. The method of claim 15, further comprising classifying: 17. Each region is a connected group of points of the image, and each region is determined to be the same only by adding a point of the proximity image to the region, wherein each region is determined by each proximity image. 16. The method of claim 15, wherein after adding the point, the difference between the intensity value of the largest image value and the smallest image value of the region is less than a threshold. 18. The target frame is a preceding frame of the current frame, wherein each region divided from the current frame is one of the previously calculated frames for the preceding frame using the steps of claim 15. The boundary for the semantic object that is correctly classified as being from the semantic object is calculated by combining the boundaries of the regions classified as being from the same semantic object in the previous frame. The method of claim 15, wherein the steps of claim 15 are repeated for successive frames of the vector image sequence. 19. A computer readable medium having instructions for performing the steps of claim 15. 20. A method for tracking semantic objects of a vector image sequence, comprising performing spatial division on an image frame to identify regions of points of a discrete image with values of the same image, each region being a point of the image. Connected regions, where each region is determined to be identical only by adding points in the proximity image to that region, and after adding each proximity image point, the maximum and minimum values for that region Performing a region-based motion estimation between each region of the image frame and the image frame immediately before the vector image sequence, and using the motion estimated for each region. Then, warping the image point of each area to the position of the immediately preceding frame, determining whether the position of the warped pixel of each area is within the boundary of the semantic object of the target frame, When at least the amount of the threshold value overlaps with the semantic object of the target frame, the area is classified as being derived from the semantic object of the target frame, and the boundary for each semantic object of the image frame is set to A method characterized in that it is formed as a combination of regions of an image frame classified as originating from semantic objects.