JP2002074375A - Method for estimating desired contour of image object - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve object dividing and tracking. SOLUTION: Dividing (94) and tracking (86) of an object are improved by leading out the location of a dynamic contour (112) (namely, of a curved line or 'snake' to freely elongate/contract) while taking direction information into the boundary estimation of the object. In the object boundary estimation, the dynamic contour is deformed from the first form and matched to the features of an image while using an energy minimizing function (90). This function minimizes all the energy of the dynamic contour while being guided by an external limitation and an image power. When minimizing the contour energy concerning a dynamic contour model, both a strength and a direction (λ) of the gradient of the image are analyzed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION The present invention relates to object tracking and segmentation within a sequence of image frames, and more particularly to tracking and segmenting objects using an active contour model to optimize the contour of the object being tracked. Related to estimating edges.

When tracking an object in multiple frames of a video sequence, the boundaries of the object are identified in each frame. An object is an area within this boundary. Identifying object boundaries in a given frame becomes more difficult as constraints on trackable objects are relaxed and objects that move, rotate or deform can be tracked. When an object is identified in one frame, the movement of the object can be detected using template matching in a subsequent frame. Typically, a template is an object identified in the previous frame.

Active contour models, also known as snakes, have been used to adjust image features within specific image object boundaries. Conceptually, an active contour model involves overlaying a stretchable curve over an image. The curve (ie, snake) itself deforms from its original shape to match the features of the image. An energy minimization function is used that fits the curve to image features such as lines and edges. This function is derived from external constraints and image forces. The best fit is obtained by minimizing the total energy calculation of the curve. In practice, continuity and smoothness constraints are imposed to control model deformation. This model is the object from the previous frame. A disadvantage of this active contour model is that if the position or shape of the object changes slightly from one frame to the next, the boundaries may not be discernable. Specifically, the estimated boundary captures a strong false edge in the background instead of following the object, and the contour of the object is distorted.

[0004] Yuille et al., "Feature extraction from face using deformable template"
es Using Deformable Templates) "(International Jo
urnal of Computer Vision, Vol. 8, 1992) discloses a process for identifying eyes and mouths in an image using a model and several parameters. For example, the eyes are 2
It is modeled using two parabolas and a circle radius. The eyes can be identified by changing the shape of the parabola and the radius of the circle. Yuille et al. And other deformation models typically include only highly constrained deformations. In particular, the object has a generally known shape that can be deformed in some generally known manner. Processes such as active contour models have relaxed constraints, but are only effective when the spatial range of motion is very narrow. Processes such as those disclosed by Yuille are effective for a wider spatial range of motion, but track highly constrained motion. Therefore, there is a need for a more flexible and effective object tracking device that can track more dynamic movements over a wider spatial range.

[0005]

SUMMARY OF THE INVENTION In accordance with the present invention, object segmentation and tracking incorporates directional information into object boundary estimation to guide the placement of active contours (ie, telescopic curves or "snake") by: Be improved.

A problem addressed by the present invention is that the active contour can converge to the wrong set of image points for objects that move rapidly from one frame to the next. In estimating the object boundaries, the active contour is deformed from its initial shape using an energy minimization function to match the features of the image. This function is guided by external constraint forces and image forces to minimize the total energy of the active contour. Conventionally, the deformation of the active contour is controlled by imposing continuity and smoothness constraints. According to one aspect of the invention, guidance and control are further included including direction information. Specifically, when minimizing the active contour energy function, both the gradient strength and the gradient direction of the image are analyzed.

One of the advantages of incorporating directional information is that
A directional “snake” (ie, active contour model based on gradient strength and gradient direction) allows for more accurate segmentation when there are multiple candidate edge points in different directions in a given neighborhood It is to be. Therefore, direction information is currently being incorporated into digital video segmentation.

[0008] The above and other aspects and advantages of the present invention will be better understood with reference to the following detailed description when taken in conjunction with the accompanying drawings.

[0009]

DESCRIPTION OF THE SPECIFIC EMBODIMENTS Overview FIG. 1 shows a block diagram of an example of a host interactive processing environment 10 for locating, tracking, and encoding video objects. Processing environment 10
Is a user interface 12, a shell environment 14, and a plurality of function software "plug-in" programs 1.
6 inclusive. The user interface receives and distributes operator input from various input sources, such as a pointing and clicking device 26 (eg, a mouse, touchpad, trackball), a key input device 24 (eg, a keyboard), or a pre-recorded scripted macro 13. I do. The user interface 12 also controls the formatting of the output to the display device 22. Shell environment 14 controls the interaction between plug-in 16 and user interface 12. Input video sequence 1
1 is input to the shell environment 14. The various plug-in programs 16a-16n can process all or part of the video sequence 11. One of the advantages of shell 14 is that it isolates the plug-in program from the various formats of possible video sequence input. Each plug-in program interfaces with the shell through an application program interface ("API") module 18.

In one embodiment, interactive processing environment 10 is implemented with a programmed digital computer of a type well known in the art, an example of which is shown in FIG. The computer system 20 includes a display device 22, a key input device 24, a pointing / click device 26, a processor 28, and a random access memory (RAM) 30. In addition, a communication or network interface 34 (eg, a modem, an Ethernet adapter), a non-volatile storage device such as a hard disk drive 32, and a portable storage medium drive 36 that reads a portable storage medium 38 are typically included. Provided. Various other storage devices 40 may include, for example, a floppy disk drive, CD-ROM drive, zip drive, bernoulli drive, or other magnetic, optical or other storage media. These various components interface and exchange data and commands over one or more buses 42. Computer system 20 receives information entered by key input device 24, pointing / click device 26, network interface 34, or another input device or input port. The computer system 20 includes a mainframe computer,
It may be any type of computer system known in the art, such as a minicomputer or a microcomputer, and may function as a network server computer, a networked client computer or a stand-alone computer. Computer system 20 may be configured as a workstation, personal computer, or reduction mechanism network terminal.

In another embodiment, interactive processing environment 10 is implemented in an embedded system. This embedded system includes digital processing units and peripherals similar to a programmed digital computer as described above. In addition, one or more input or output devices are provided for certain implementations, such as image capture.

Software codes for implementing user interface 12 and shell environment 14, including computer-executable instructions and computer-readable data, include embedded memory, RAM, ROM, hard disk, optical disk, floppy (R), and the like. disk,
It is stored on a digital processor readable storage medium, such as a magneto-optical disk, an electro-optical disk, or another known or later implemented portable or non-portable processor readable storage medium. The plug-ins 16 (along with the corresponding API 18) may be individually grouped on separate storage media or together on a common storage medium. Additionally, plug-ins 16 and corresponding APIs 18
May be combined with the user interface 12 and the shell environment 14, and the plug-ins 16 and corresponding APIs 18 may not be combined therewith. In addition, various software programs and plug-ins may be distributed or executed electrically over a network, such as a global computer network.

The processing environment 10 is based on various calculation models.
Is installed on an end-user computer or accessed remotely. For a standalone computing model, executable instructions and data are loaded into volatile or non-volatile memory accessible by the standalone computer. For the non-resident computer model, the executable instructions and data are processed locally or on a remote computer, the output of which is sent to the local computer,
Operator input is received from the local computer. Those skilled in the art will appreciate that numerous computational configurations can be implemented. For the non-resident computing model, the software program is stored locally or on a server computer in a public or private, local or wide area network, or global computer network. The executable instructions can be executed on either an end-user computer or a server computer,
The data is displayed on the end user's display.

Shell Environment and User Interface Shell environment 14 allows an operator to work in an interactive environment to test or use various video processing and common tools. In particular, video object segmentation, video object tracking and video encoding (eg, compression)
Plugins for are supported in the preferred embodiment. The interactive environment 10 includes a shell 14 and an MP
It provides a useful environment for creating video content, such as EG-4 video content or content for another video format. A pull-down menu or pop-up window is implemented, and the operator can select a plug-in to process one or more video frames.

In a specific embodiment, shell 14 includes a video object manager. A plug-in program 16, such as a split program, accesses one frame of video data with a set of user inputs through the shell environment 14. The split plug-in program identifies a video object within a video frame. The video object data is sent to the shell 14 and the shell 14
Stores this data in the Video Object Manager module. Thus, such video object data can be accessed by the same plug-in 16 or another plug-in 16 such as a tracking program. The tracking program identifies video objects in subsequent video frames. Data identifying the video object for each frame is sent to the video object manager module. In fact, video object data is extracted for each video frame in which the video object is tracked. Once the operator has finished extracting video objects, editing or filtering the video sequence, the encoder plug-in 16 can be activated to encode the final video sequence into the desired format. With such a plug-in architecture, the segmentation and tracking plug-in does not need to interface with the encoder plug-in. Further, such plug-ins do not need to support reading some video file formats or create a video output format. The shell handles video input compatibility issues, while the user interface handles display formatting issues. The encoder plug-in handles the creation of the runtime video sequence.

In the Microsoft Windows® operating system environment, plug-ins 16 are compiled as dynamically linked libraries. Processing environment 1
Upon execution of 0, the shell 14 scans a predefined directory for plug-in programs. If there is a plugin program name, it will be added to the list,
This list is displayed in a window or menu for user selection. When the operator chooses to execute plug-in 16, the corresponding dynamic link library is loaded into memory and the processor begins executing instructions from one of a set of predefined entry points for the plug-in. I do. To access video sequences and video object splits, the plug-in uses a set of callback functions. The plug-in interfaces to the shell program 14 through a corresponding application program interface module 18.

In addition, a split interface 44 of the user interface 12 supported by the split plug-in is provided. The split interface 14
Invoke the split plug-in to support the split command selected by the operator (eg, execute the split plug-in, configure the split plug-in, or perform boundary selection / editing).

Typically, an API allows a corresponding plug-in to access a particular data structure based only on whether linked access is required. For example, the API retrieves one frame of video data,
It is responsible for extracting video object data from the video object manager or storing video object data with the video object manager. By separating the plug-in and interfacing through the API, the plug-in can be written in a different programming language and in a different programming environment than used to create the user interface 12 and shell
Can be written. In one embodiment, user interface 12 and shell 14 are written in C ++.
Plug-ins can also be written in a language such as the C programming language.

In a particular embodiment, each plug-in 16 runs on a separate processing thread. As a result, the user interface 12 displays a dialog box that the plug-in can use to indicate the progress, from which the user can make a selection to halt or suspend execution of the plug-in.

Referring again to FIG. 1, the user interface 12 includes a split interface 44, various display windows 54-62, dialog boxes 64, menus 66, and button bars 68 for formatting and maintaining these displays. Include with software code. In a preferred embodiment, the user interface is defined by a main window in which the user selects one or more subordinate windows. Each window may be running simultaneously at a given time. Dependent windows may change size as they are open, closed, moved, or moved.

In the preferred embodiment, the video window 54, zoom window 56, timeline window 5
8, several dependent windows 52 are provided, including one or more encoder display windows 60 and one or more data windows 62. Video window 54 displays a single video frame or sequence of frames. Regarding the observation of the sequence of frames, the frames may be observed stepwise, in real time, in slow motion, or accelerated. This includes input controls that the operator can access by point or click or by a predefined key sequence. Controls for video display in the video window 54 include controls such as stop, pause, play, rewind, fast forward, step display, and other VCRs (video cassette recorders). In some embodiments, scaling and scrolling controls are provided for the video window 54.

The zoom window 56 enlarges a portion of the video window 54 at a substantially larger magnification than the video window. Timeline window 58 includes an incremental timeline of the video frame, with zero or more thumbnail views of the selected video frame. Timeline window 58 also includes a respective timeline for each video object defined for input video sequence 11. Video objects are defined by delineating the objects.

Data window 62 includes user input fields for using the object title, translucent mask color, encoding target bit rate, search range, and other parameters for defining and encoding the corresponding video object.

During encoding, one of the encoder windows 60 is displayed. For example, the encoder progress window indicates the encoding state of each defined video object in the input video sequence 11.

The first step in tracking video objects and tracking segmented objects is to determine which template to use for the object. FIG. 3 is a flowchart 70 for initially dividing a video object to obtain a first template according to an embodiment of the present invention. In one embodiment, the operator loads the input video sequence at step 72 and selects a point or line segment that approximates the boundaries of the object at step 74. Next, in step 76, division is performed to determine the boundaries more accurately. This division is performed using an active contour model as described in another section below.

The edge points that define the object boundaries are output at step 78. Using such edge points as control points for another plug-in, defining an object mask (ie, template) and superimposing it on the image frame, etc., so that the tracked object can be visually distinguished. . Also, the operator can adjust the points on the boundary to refine the boundary and re-execute the segmentation algorithm using the refined boundary points to obtain the desired accurate object. Such an object serves as an initial template for locating the object in another frame.

In the preferred embodiment, the object located in a given frame serves as an initial template when searching for that object in the next frame to be processed. The next frame is a subsequent image frame in the video sequence, or
The next frame to be sampled in the video sequence, or another frame within or outside the sequence that is the next frame to be processed. According to such a strategy, the initial template is constantly changing for each frame to be processed.

FIG. 4 is a flowchart 80 for tracking an object in a subsequent frame after identifying and splitting the object in the first frame. In step 81, the next image frame to be processed is input. At step 84, a test is performed to identify any changes in the scene. Although various strategies can be implemented, some embodiments implement a modified applied resonance theory M-ART2 method. This is assigned to the same assignee, 19
U.S. Patent Application No. 0 to Sun et al., Filed June 10, 1999
9/323, 501 “Video Object Segmentation Us
ing Active ContourModel with Global Relaxation ", which is incorporated herein by reference and forms a part thereof.

If a scene change is detected at step 84, process 80 ends. Alternatively, another image object is tracked by re-initialization. If the scene has not changed, then at step 86, image objects are roughly identified from the image frames using any of a variety of object tracking techniques. In one embodiment, a two-dimensional correlation automatic prediction search (2D-CAPS) process is performed. In another embodiment, a three-dimensional correlation automatic prediction search (3D
-CAPS) process. In step 88, 2
If no image object is found using the D-CAPS process, process 80 terminates or tracks another object by re-initialization.

Once the object is identified, step 9
At 0, the edge energy of the object boundary is derived. At step 94, an object segmentation process is performed based on the active contour model, as described in another section below. Active contour models are used to divide image boundaries and accurately model object boundaries. At step 96, the estimated image boundaries are output. As described above for the first image frame, in some embodiments, the output is a buffer, file, and / or
Or written on a video screen. Thus, the process 80 is repeated for another image frame.
As a result, the image object is split and tracked over many image frames.

Edge Energy Edge energy is a measure of the potential energy of a set of pixels that identifies edges based on the relative energy values of the pixels. Various measurements of potential energy can be realized. In one embodiment, the high-frequency components of the image are extracted using a multi-level wavelet decomposition algorithm. The high frequency details are analyzed to identify edges of the image object. For example, a Haar wavelet can be used.

The input processed to obtain edge potential energy is an image. In one embodiment, the image is an entire image frame. In another embodiment, the image is an image object. The resulting edge potential energy is an array of potential energies for each data point (pixel) of the image.

In one embodiment, the input image is a quadrature mirror filter (QM) that extracts the image from image details.
F) Decomposed by filtering using pairs and simultaneously smoothing the image. The QMF pair includes a high-pass filter for extracting image details and a low-pass filter for smoothing the image. Referring to FIG. 5, multi-level QMF decomposition 150 of image frame 152
Is shown. The image frame 152 is converted to the low-pass filter 15.
4 and the high-pass filter 156 to pass the low-pass component 158
And the high frequency component 160 is obtained. Now filter these components. The low-pass component 158 is converted to the low-pass filter 1
62 and a high pass filter 164. Low-pass filter 1
The output of 62 is a low-frequency remainder 166. High-pass filter 16
The output of 4 is the horizontal detail 165 of the image frame 152.
It is.

At the same time, the high-pass component 160 is passed through the low-pass filter 168 and the high-pass filter 170. The output of low pass filter 168 is vertical detail 169 of image frame 152. The output of high pass filter 170 is diagonal detail 171 of image frame 152. The low-frequency remainder 166 and the three image details 165, 169 and 171 are the first level QMF decomposition of the image frame 152. In some embodiments, the second level QMF decomposition 1
72, in which case the low-pass remainder 166 is also input through a two-stage low-pass and high-pass filter to provide a second-level low-pass remainder and three image details (horizontal detail, vertical detail and diagonal detail). Get. In some embodiments, the same filter used in the first level decomposition may be used in the second level decomposition. For example, low-frequency remainder 166 is input to filters 154 and 156 instead of simply image frame 152.

The high-pass filtering function is a wavelet transform (Ψ), and the low-pass filtering function is a scaling function (φ) that matches the wavelet. The scaling function causes smoothing, and the three wavelets extract details of the image. The scaling function and wavelet transform in one-dimensional space are given by the following equations.

[0036]

(Equation 1)

Although the scaling may be different for each level of decomposition, in one embodiment such scaling does not change.

Perform a first level QMF decomposition. For the second level decomposition, the lower residual 1 of the first level decomposition
66 is analyzed without further downsampling. In some embodiments, the lower level residue of the preceding level is 2
Additional levels of decomposition can be obtained by passing through a stage filtering process (similar to that of the preceding level).

For a given level of decomposition, there are four images: low-frequency remainder, vertical detail, horizontal detail and diagonal detail. The horizontal and vertical details are the gradients of the image along the x and y axes. The size of the image is captured at each level of decomposition. In some embodiments, diagonal details are omitted because they contribute less.

In one embodiment, up to five levels of decomposition are used for each color component of an image frame. The low-pass residue from the previous stage is input to filters 154 and 156 to generate image details and residue for the current stage. Preferably, only data from even levels (eg, levels 2, 4 and 6) are used to prevent a half pixel shift in edge energy. The integration of multi-level and multi-channel (color component) data is guided by the principal components. In one implementation, the ratio of the multi-level edge gradient is 1: 2: 4: 8:
16 is selected. For the color components (Y, Cr, Cb), an edge gradient ratio of 1: 1: 1 is used.

In one embodiment, for a given level of decomposition (i)
The horizontal and vertical details of the edge potential energy (EP
E) occurs as follows.

[0042]

(Equation 2)

In one embodiment that performs five levels of decomposition, the total edge potential energy (EPE) for a given color component is summed as follows:

[0044]

(Equation 3)

Where c is the color component being processed. The total edge potential energy of the entire frame, including all color components, is a weighted sum of the energy from the different color components. For a weighting factor of (1,1,1), the total potential energy is given as:

[0046]

(Equation 4)

In the formula, y, cr and cb are color components. In other embodiments, the R, G, and B color components or color components of another color component model may be used. The weighting factor can change depending on the color component model used.

The total edge potential energy is an array having an energy value for each pixel of the image being processed. Although the detailed description has been given of the derivation of the edge energy, an alternative method of deriving the edge energy may be used.

Object Using Modified Active Contour Model
When an object in a segmented image is identified, the boundaries (ie, edges) of the object are segmented to more accurately model the object edges. According to a preferred embodiment, the object segmentation process 76/94 is based on an active contour model. First split (eg, process 70)
For the image object boundaries to be further improved, the points were selected in step 74 (see FIG. 3).
Boundary input to the system. For subsequent image frames, the segmentation process 94 receives as input the derived total edge potential energy and the current image object boundary. At step 90, the total edge potential energy is derived. The current image object boundary is obtained in the object tracking process in step 86.

Active contour models are energy minimizing splines or curves, which are guided by internal constraints and are affected by external image forces that pull the curves toward features such as lines and edges. The alternative two active contour model embodiments described below perform object segmentation at steps 76 and 94. In one embodiment, the gradient descent search process is modified to account for direction information (see FIGS. 6 and 7). In another embodiment, a dynamic programming search is performed, which includes consideration of directional information (see FIGS. 8-11). Hereinafter, a modified gradient descent embodiment will be described. A dynamic programming embodiment is described in a subsequent section.

The conventional energy function in the gradient descent embodiment is defined in discrete form as follows:

[0052]

(Equation 5)

In the equation (1), N is the number of snaxels, and the order of the snaxels in the sequence is counterclockwise along the closed contour. The weighting factors α _i and β _i are defined to control the relative importance of the membrane and sheet terms for the i th snaxel, respectively,
_i is the two-dimensional coordinate of the i-th snaxel, x _i and y
_i . Since the external energy is set to be a negative value of the image gradient, snake (snake)
Are drawn to areas where the external energy is low, ie, strong edges.

The optimum position of snaxel is searched so that the snake energy E _snake can be minimized. In one embodiment, the gradient descent method seeks to minimize snake energy by iteratively finding new snaxel locations.

[0055]

(Equation 6)

In equation (4), I is an N × N identity matrix. Equation (4) is repeated to generate a local minimum for snake energy. As shown by these equations, the external energy is determined from the size of the gradient image.

The gradient image is represented by Sobel, Prewitt
It is obtained by using a gradient operator such as t or Gaussian derivative (DOG). The image is convolved with a gradient operator, which typically consists of horizontal and vertical gradient kernels.

[0058]

(Equation 7)

In the conventional contour defined by equations (1)-(3), only the magnitude of the gradient contributes to the contour energy. Thus, this contour can be attracted to strong edges that have a different gradient direction than the correct boundary.

FIGS. 6A to 6C show this limitation in a conventional size-only active contour model.
FIG. 6A shows a monochrome image 100. FIGS. 6B and 6C show a gradient magnitude image 102 of a black and white image. FIG. 6B also shows an initial handwritten outline 104 overlaid on the gradient magnitude image 102. FIG.
(C) is superimposed on the gradient magnitude image 102,
Active contour 1 derived based on a conventional active contour model
06 is also shown. In this example, the gradient magnitude image 102 is 5
Obtained by using a x5 DOG kernel. As shown in FIG. 6C, the conventional active contour 106 does not converge on the outer boundary 108 or the inner boundary 110 of the gradient magnitude image 103. Referring to FIGS. 7A and 7B, a gradient magnitude image 102 is shown with an active contour 112 obtained in accordance with one embodiment of the present invention. Specifically, the active contour 112 is based not only on the magnitude of the gradient, but also on the direction of the gradient. By including information about the direction of the gradient, the active contour 112 is partially transformed into both boundaries 108 and 110
, But converge to either the inner boundary 110 (FIG. 7A) or the outer boundary 108 (FIG. 7B). The specific result depends on the direction information.

In order to consider the direction information of the gradient,
Redefine the external energy of the image as follows:

[0062]

(Equation 8)

When the gradient direction at s _i is inward, λι is 1. Otherwise, λι is -1. When the normal direction of the contour is opposite to the direction of the gradient, the external energy is maximum or zero according to equation (7). In such a case,
The directional contours thus obtained are not attracted to edges in the direction opposite to the gradient direction, so that only edges having the correct gradient direction participate in the active contour minimization process of equation (4).

When applying the directional contour, the directional parameter {λ _i } is determined from the contour of the preceding frame as follows.

[0065]

(Equation 9)

Therefore, the direction parameter for the (k + 1) -th frame is calculated from the contour direction θ _C and the gradient direction θ _G of the k-th frame as in equation (10). The calculated direction parameter {λι} and the k-th frame contour (C _k ) are used as initial direction contours for the (k + 1) -th frame division. Since the directional parameters are determined from the divided contours of the preceding frame, the result of the division will be degraded if the directional parameters change frequently along the frame sequence. This means that the direction parameter of the preceding frame may be different from the direction parameter of the current frame.

The result of the division depends on the predefined direction parameters. FIGS. 7A and 7B respectively show the results of the division as 1 for all snaxels.
And -1 direction parameters. When the direction parameter is 1 (FIG. 7A), the direction contour is
The contour is converged to a boundary edge where the gradient direction is inward, ie, the gradient direction is to the left of the contour in a counterclockwise direction.
However, when the direction parameter is −1 (FIG. 7B), the contour moves toward the edge of the boundary in the outward gradient direction. In this example, all s in the contour
The direction parameters for the taxel are the same. However, according to the directional gradient analysis, different snaxels along the contour are assigned different directional parameters.
In such a case, snax having one direction parameter
el converges on the inner edge and snaxel with a directional parameter of -1 converges on the outer edge. Therefore,
It is desirable to derive direction parameters for all snaxels of the first contour. More specifically, in the video segmentation and tracking, the direction parameter from the segmentation result of the preceding frame is used.

Object Using Directional Active Contour Model
Referring to preparative division 8, for implementing the object segmentation step 76 and 94 (see FIG. 4), the flowchart 192 of an alternative active contour model examples are set forth. In this embodiment, a dynamic programming search is performed. First
In step 194, an edge point is input. The number of input edge points can vary. In step 196, edge points that are too close (i.e., the spacing is less than a first threshold distance) are deleted. In one embodiment, these points are considered too close when the spacing between the points is less than 2.5 pixels. In other embodiments, the spacing is smaller or larger. At step 198, if the adjacent points are too far apart (ie, the spacing is greater than a second threshold distance), additional points are added by interpolation. In one embodiment, these points are considered too far apart when the spacing between the points is greater than 6.0 pixels. In other embodiments, this interval is less than or greater than 6.0,
It is larger than the first threshold distance.

At this stage in the process, there is a given number of current edge points modified from the input edge points. Although the number of edge points may vary with the contour, the process for the current N edge points will be described below.

In step 199, the directional parameter {λ _l } for the edge point is derived using equations (7) through (10). In some embodiments, at step 200,
The active contour modeling process performs global relaxation on the current N edge points. To that end, for each current edge point, candidate points from the box around the current edge point to M are selected. In some embodiments, M is equal to four, but in various embodiments, the number of candidate points can vary. In one embodiment, a 5 × 5 box is used. However, the size of this box can also vary. The larger the box, the more flexible the contour but the longer the computation time. The shape of the box may be square, rectangular or another shape.

In the preferred embodiment, less than M potential candidate points are derived based on direction information. Specifically, for the edge point, a candidate point matching the direction gradient information is selected instead of identifying a candidate point in each potential direction. Therefore, candidate points that do not satisfy the criterion of Expression (10) are deleted.

Referring to FIG. 9, a 5 × 5 box 174 of pixels surrounding the current edge point 176 is divided into four regions 17.
8, 180, 182 and 184 and consider candidates in all directions. One or two of these regions are deleted when considering direction information.

There are six pixels in each of the remaining regions. One of these six pixels is selected as a candidate pixel ("point") in each region, which potentially replaces the current edge point 176 as an object boundary edge point. Thus, one of 186-189, 2
One or three candidate points are selected for each of the current edge points 176. In alternative embodiments, the number of candidate points may be different, or another candidate point selection method that matches the direction information may be used.

For a given region, a candidate point is the pixel with the highest edge potential energy of the six potential points. For an image object boundary that currently has N edge points, if there are M (eg, 4) alternative candidate points for each of the N points, there are (M + 1) ^N (eg, 5 ^N ) possible contours from which to model. Select image object boundaries. In step 202, the optimal contour path is selected by applying the travel algorithm to the current edge point together with the alternative candidate points. FIG. 13 shows a travel path diagram for a possible contour. There are (M + 1 (eg, 5)) points in each column. The five points are the current edge point 176
And four candidate edge points 186 and 189 for this current edge point 176. The number of points in each row (also equal to the number of columns) corresponds to N. The directional information is used to simplify the travel problem when some of the path permutations do not match the directional gradient information (see Equations 6-6 above).

To select the optimal image object boundary, a start location 190 on the current contour is selected. Such locations 190 correspond to a given current edge point 176 and respective candidate edge points (eg, a subset of points 186-189). Thus, the number of points at each determination point on the travel route is less than M + 1 = 5. The optimal route is obtained from the latent points. The best path among the potential paths is selected as the modeled object boundary. The process for deriving the optimal path is the same for each path to be derived.

Referring to FIG. 11, a path starting from edge point 176s will be considered. The segments of this path are
It is constructed by going to one of the points in the adjacent column s + 1. Thus, one option is point 176 (s + 1)
It is to proceed to. Another option is a candidate point 186 (s +
It is to go to 1). Another option is 187 (s +
1) may include 188 (s + 1) and 189 (s + 1). In this case, option 189 (s + 1) does not follow the directional gradient information and is not taken into account. Select only one. This selection is made by determining which of the points in column (s + 1) has the smallest energy difference in the resulting path (eg, the most energy savings). The selected point is the column s
With the distance of this point at +1 from the current point,
Will be saved. Consider an example in which point 186 (s + 1) is selected. This selected point is saved, along with a distance value (eg, expressed in pixels) of how many pixels there are with point 176 (s + 1).

Similarly, one of the potential points in column s + 2 is selected to construct the next segment of this path.
For each segment along the path, save only one of the potential segments, along with the distance such points are away from the current point 176 in the same column.

The same process is performed to derive a path starting at point 186s. The first segment of this path is
It is constructed by going to one of the points in the adjacent column s + 1. One option is to go to point 176 (s + 1). Another option is to go to candidate point 186 (s + 1). Other options are 187 (s + 1) and 188
(S + 1). Only one is selected. This selection is made by determining which of the points in column (s + 1) has the largest energy difference in the resulting path for current contour 173.
The selected point is saved, along with the distance that this point is from the current point in column s + 1. The paths starting from each of points 187s and 188s are configured in the same manner. Point 189s was not considered here because it was not selected as a candidate point based on directional gradient analysis.
The resulting paths (176s, 186s, 18 respectively)
7s and 188s) to determine which is the best path (e.g., the energy difference is greatest for the current contour 173).

[0079] Examples of implementations of the object segmentation method include, but are not limited to, video encoding, video editing, and computer vision. For example, segmentation and modeling may be performed in the context of MPEG-4 video encoding and content-based video editing, where video objects from different video clips are grouped to form a new video sequence. As an example of a computer vision application, segmentation and modeling may be performed with limited accessibility to track targets (eg, military or surveillance situations). When a target is locked with user assistance, the tracking and splitting method automatically provides information for tracking the target.

Worth and Advantageous Effects According to one advantage of the present invention, for objects that deform or move rapidly, the exact boundaries of the object are tracked. The ability to track a wide range of object shapes and different object deformation patterns is particularly advantageous for use with MPEG-4 image processing systems.

While the preferred embodiment of the invention has been illustrated and described, various alternatives, modifications and equivalents may be used. Therefore, the above description should not be taken as limiting the scope of the invention which is defined by the following claims.

[Brief description of the drawings]

FIG. 1 is a block diagram of an interactive processing environment for tracking video objects in a sequence of video frames.

FIG. 2 is a block diagram of an example of a host computing system of the interactive processing environment of FIG. 1;

FIG. 3 is a flow diagram of a splitting process for initially selecting and splitting an object to be tracked.

FIG. 4 is a flow diagram of an object tracking and splitting method according to an embodiment of the present invention.

FIG. 5 is a diagram of an orthogonal modeling filter for dividing an image to obtain a detailed image and a low-frequency remainder.

FIG. 6A is a diagram of a black and white sample image,
(B) and (C) are diagrams of the gradient size image of the black-and-white image of (A), in which a handwritten outline and a dynamic outline without direction information are also shown together.

FIGS. 7A and 7B are diagrams of the gradient magnitude image of the black-and-white image of FIG. 6A, which are given using direction information of a certain value and direction information of another value, respectively. The active contours are also shown together.

FIG. 8 is a flow diagram of an active contour modeling process.

FIG. 9: Current edge point (pixel) used to select other candidate points that can be used in place of the current edge point
FIG. 4 is a diagram of a 5 × 5 pixel domain with respect to FIG.

FIG. 10 is a diagram of potential edge points that are processed to preserve one optimal path for image object boundaries.

11 is a diagram of a portion of a travel path in a process in which a contour is derived from the set of points of FIG.

[Explanation of symbols]

 28 processors, 30 memories, 112 active contours.

 ──────────────────────────────────────────────────の Continued on the front page (72) Inventor Todd Sheplin, United States of America, 98105 Washington, Seattle, Eleventh Avenue, NE, 4131, Number 205 (72) Inventor of Sijung Sun United States of America, 98683 Washington Vancouver, S.E. Twenty-Sixth Drive, 16900, Number 47 (72) Inventor Yongmin Kim United States of America, 98155 Washington, Seattle, NE One Hundred and Einsteins Place, 4431 F-term ( Reference) 5L096 FA05 GA01 HA01

Claims (12)

[Claims] A method for estimating a desired contour (112) of an image object tracked over a plurality of image frames, the method comprising receiving a first estimate (104) of a desired contour for a current image frame. Step, the current image frame includes image data, the first estimate includes a plurality of data points, the method further comprises: convolving the image data with a gradient operator in a first manner. Deriving (102); and deriving a convolution gradient direction (λ) of the image data by the gradient operator in the second mode (199).
Identifying a plurality of candidate data points (186-189) for each of the plurality of data points (2).
00), and re-estimating the desired contour (112) by deriving an energy minimizing spline advancing from a first point (190), wherein the first point comprises:
(I) one of the plurality of data points (1
76) or (ii) one of the plurality of candidate points (186-189) for the one of the plurality of data points including a first estimate. The energy minimization spline is based on an internal constraint force on the image object and an external image force on the image object, wherein the external image force is derived as a function of the magnitude of the gradient and the direction of the gradient. A method for estimating a desired contour of an image object. 2. Convolving the image data with the gradient operator in the second aspect based on a gradient direction derived from a preceding image frame with the gradient operator in the second aspect; The method of claim 1, comprising deriving the gradient direction (λ) for an image frame. 3. The step of receiving, convolving in a first manner, convolving in a second manner, identifying and re-estimating are performed on a sequence of image frames and are performed on one image frame. A first estimate of a desired contour is a re-estimated desired contour from a previous image frame, and the derived gradient direction for the one image frame was derived for the previous image frame. The method of claim 1, wherein the method is derived based in part on a gradient direction. 4. The step of convolving in the first mode comprises, for each selected one of a plurality of image points of a current image frame (100), converting the selected one of the plurality of image points to the first one of the plurality of image points. Convolved with the gradient operator in a manner,
The method of claim 1, further comprising deriving a gradient magnitude corresponding to the selected one point, wherein the gradient operator is based on a kernel of image data near the selected one point. . 5. The step of convolving in the second aspect comprises, for each selected one of a plurality of image points of a current image frame (100), the selected one of the plurality of image points Convolving with the gradient operator in an aspect to derive a gradient direction corresponding to the selected one point, the gradient operator comprising: a kernel of image data near the selected one point; The method of claim 1, based on a corresponding gradient operator derived for a previous image frame. 6. An apparatus (10) for estimating a desired contour (112) of an image object tracked over a plurality of image frames, the apparatus comprising a first estimation (104) of a desired contour for a current image frame. ), The processor re-estimating the desired contour and further storing a memory (30) for storing the re-estimated desired contour.
Wherein the current image frame (100) includes image data, the first estimate includes a plurality of data points, the processor performs a plurality of tasks, the task comprising: (a) the image data; Deriving the magnitude (102) of the convolution gradient by the gradient operator in the first mode; and (b) deriving the convolution gradient direction (λ) from the image data by the gradient operator in the second mode (199). (C) identifying a plurality of candidate data points (186-189) for each data point of the plurality of data points; and (d) an energy minimizing spline advancing from a first point (190). Re-estimating the desired contour (202) by deriving (i) the first point.
Or (ii) a plurality of candidate points (186-1) for the one of the plurality of data points including the first estimate.
89), wherein the energy minimizing spline is based on an internal constraint force on the image object and an external image force on the image object, wherein the external image force is Derived as a function of the magnitude of the gradient and the direction of the gradient,
An apparatus for estimating a desired contour of an image object. 7. An apparatus (10) for estimating a desired contour (112) of an image object tracked over a plurality of image frames, the apparatus comprising a first estimation (104) of a desired contour for a current image frame. A) a first processor (28) for receiving
The current image frame includes image data, and the first
Comprises a plurality of data points, the apparatus further comprising: a second processor (28) for deriving the magnitude of the convolution gradient (102) with the gradient operator in a first manner from the image data; A second processor (2) that derives a convolution gradient direction (λ) by the gradient operator in the second mode.
8); a fourth processor (28) for identifying a plurality of candidate data points (186-189) for each of the plurality of data points; and an energy minimizing spline advancing from the first point (190). And a fifth processor (28) for re-estimating the desired contour by deriving the first contour, wherein the first point comprises: (i) one of the plurality of data points including the first estimate (176). Or (ii) a plurality of candidate points (186-86) for said one of said plurality of data points including a first estimate.
189), wherein the energy minimizing spline is based on an internal constraint force on the image object and an external image force on the image object, wherein the external image force is Apparatus for estimating a desired contour of an image object, which is derived as a function of gradient magnitude and gradient direction. 8. The processor according to claim 7, wherein the first processor, the second processor, the third processor, the fourth processor and the fifth processor are the same processor (28).
An apparatus according to claim 1. 9. The third processor convolves the image data with the gradient operator in the second aspect based on a gradient direction derived from a previous image frame, and convolves the image data with the current image frame. The device according to claim 7, wherein the gradient direction (λ) is derived. 10. The first processor, the second processor, a third processor, a fourth processor, and a fifth processor.
The processor operates on a sequence of image frames and the first of the desired contours for one image frame.
Is the desired contour re-estimated from the previous image frame, and the derived gradient direction for the one image frame is based in part on the derived gradient direction for the previous image frame. The apparatus of claim 7, wherein the apparatus is derived by: 11. The gradient processor according to claim 1, wherein, for each selected one of a plurality of image points of the current image frame, the second processor converts the selected one of the plurality of image points in the first manner. Convolved with the selected 1
The apparatus of claim 7, wherein a gradient magnitude (104) corresponding to one point is derived, and wherein the gradient operator is based on a kernel of image data near the selected one point. 12. The gradient processor according to claim 2, wherein, for each of a selected one of a plurality of image points of the current image frame, the third processor converts the selected one of the plurality of image points in the second manner. Convolved with the selected 1
8. Deriving a gradient direction corresponding to one point, wherein the gradient operator is based on a kernel of image data near the selected one point and a corresponding gradient operator derived for a preceding image frame. The described device.