KR20010012261A - Algorithms and system for object-oriented content-based video search - Google Patents

Abstract

Object oriented methods and systems are disclosed that allow a user to place one or more video objects from one or more video clips via an interactive network. The system includes a server computer 110, a communications network 120, and a client computer 130 that include storage 111 for a database and video clips of video object attributes. The client computer includes a query interface that specifies video object attribute information, including motion trajectory information 134, a browser interface for browsing through video object attributes stored within the server computer, and an interactive video player.

Description

ALGORITHMS AND SYSTEM FOR OBJECT-ORIENTED CONTENT-BASED VIDEO SEARCH}

In the last few years, as the Internet matures and multimedia applications are widely used, the readily available digital video information has increased. In order to reduce bandwidth requirements to a manageable level, such video information can be stored in a compressed bitstream in a standard format, such as JPEG, Motion JPEG, MPEG-1, MPEG-2, MPEG-4, H.261 or H.263. Form is usually stored in a digital environment. At present, tens of thousands of other still and moving pictures representing both skiing and baseball from the ocean and mountains are available from the Internet.

With the increasing amount of video information available in digital form, there is an urgent need to explore and organize meaningfully this information. In particular, the content-based video enables the user to search for and retrieve a set of specific video information that meets any predetermined criteria, such as the shape or motion characteristics of the video object embedded in the stored video information in response to a user-defined query. There is an increasing demand for search engines.

In response to this demand, several attempts have been made to develop video search and search applications. Existing techniques fall into two distinct categories: query by example ("QBE") and visual sketching.

Regarding image retrieval, examples of QBE systems are described in T. Minka's "An Image Database Browser that Learns from User Interaction" {MIT Media Laboratory Perceptual Computing Section, TR # 365 (1996)}, QBIC, PhotoBook, VisualSEEk, Virage and FourEyes. This system works under the pretext that several satisfactory matches must be in the database. Under this pretext, the search begins with an element in the database itself, and the user is directed towards a given image through a series of queries. Unfortunately, this "guide" results in substantial time consumption because the user must continue to refine the search.

Although the spatial partitioning scheme for precomputing hierarchical groupings can accelerate database search, this grouping is static and requires recalculation when new video is inserted into the database. As such, although it can be extended in principle to QBE, video shots contain a large number of objects described by complex multidimensional feature vectors. Complexity arises due in part to the problem of describing shape and movement features.

A second category of search and retrieval systems, namely sketch based query systems, calculates correlations between user drawn sketches and edge maps of each image in the database to locate video information. . Sketch-based query systems, such as those described in Hirata et al., "Visual Example Content Based Images Retrieval, Advances in Database Technology-EDBT", 580 Lecture Notes on Computer Science (1992, A.Pirotte et al.eds). Compute the correlation between the edge maps of each image in. A.Del Bimbo et al., "Visual Image Retrieval by Elasic Matching of User Sketches" {19IEEE Trans. on PAMI, 121-131 (1997)}, requires a technique that minimizes the energy acting to achieve a match. In CEJacobs et al., "Fast Mitiresolution Image Querying" {Proc. Of SIGGRAPH, 277-286, Los-Angeles (Aug. 1995)}, the artist writes the wavelet signature of the sketch and the respective Calculate the distance between images.

There have been several attempts to form index video shots, but no attempt has been made to represent video shots as a dynamic set of video objects. Rather, the prior art utilizes an image search algorithm for simply indexing video by assuming that the video clip is a collection of image frames.

In particular, the techniques developed by Zhang and Smoliar and those developed by QBIC use an image retrieval method for video (eg, using a color histogram). The "key-frame" is selected from each shot, e. G. R-frame in the QBIC method. For Zhang and Smoliar, key frames are extracted from the video clip by selecting a single frame from the clip. The clip is selected by averaging all the frames in the shot and selecting the frame in the clip closest to the average. By using conventional image search, such as color histograms, key frames are used to index the video.

As such, in a QBIC project an r-frame is selected by taking any frame, ie a first frame, such as a representative frame. When the video clip is moving, the mosaic representation is used as the representative frame for the shot. QBIC again uses image retrieval techniques for these r-frames to index video clips in order.

To index video clips, the Informedia project creates a copy of the video by using a speech recognition algorithm on the audio stream. The recognized word is aligned with the video frame in which the word is spoken. The user navigates through the video clips by doing a keyword search. However, speech-to-text conversion has proved to be a major tumbling block because the accuracy of the conversion algorithm is low (approximately 20-30%), which significantly affects search quality.

The prior art device described above has not met the growing demand for an efficient content based video search engine capable of searching and retrieving a particular series of video information that meets any predetermined criteria. This technique cannot search for motion video information, or only search for this information about global parameters such as panning or zooming. As such, the prior art does not describe a technique for retrieving video information based on spatial and temporal characteristics. Thus, the existing techniques described above can search for and retrieve a set of specific video information that satisfies any predetermined criteria, such as the shape or motion characteristics of a video object embedded within stored video information in response to a user-defined query. none.

This application is related to US Provisional Application No. 60 / 045,637, filed May 5, 1997, in which priority is claimed.

TECHNICAL FIELD The present invention relates to techniques for searching and retrieving visual information, and more particularly, to the use of content-based search queries to search and retrieve mobile visual information.

1 is a diagram illustrating a system for searching and retrieving video information in accordance with an aspect of the present invention.

FIG. 2 illustrates a query interface useful for the system of FIG. 1. FIG.

3 illustrates a video object navigation method performed in the system of FIG.

4 is a flowchart illustrating a method of extracting a video object from a frame sequence of video information according to an aspect of the present invention.

FIG. 5 is a flow chart illustrating a preferred method of region projection and interframe labeling useful in the method illustrated in FIG. 4.

FIG. 6 is a flow chart illustrating a preferred method of interframe region coalescing useful in the method shown in FIG.

FIG. 7 illustrates another video object navigation method performed in the system of FIG. 1. FIG.

One object of the present invention is to provide a video search engine that is truly content based.

Another object of the present invention is to provide a search engine that can search and search for video objects embedded in video information.

Another object of the present invention is to provide a mechanism for filtering identified video objects such that only objects that best match a user's search query are retrieved.

Another object of the present invention is to provide a video search engine capable of searching and retrieving a series of specific video information satisfying any predetermined criteria in response to a user-defined query.

It is another object of the present invention to provide a search engine capable of extracting video objects from video information based on integrated characteristic features of the video object including motion, color and edge information.

To satisfy these and other objects, which will become apparent with reference to the following, the present invention provides a system for allowing a user to search for and retrieve a video object from one or more sequences of video data frames on an interactive network. The system is configured to transmit one or more storages for a database of video object attributes, storage for one or more frame sequences of video data corresponding to the video object attributes, and one or more frame sequences of video data from a server computer. One or more server computers, including a communication network, and a client computer. The client computer includes a query interface for receiving selected video object property information including motion trajectory information; Receives selected video object property information, and via the stored video object property in the server computer via the communication network to determine one or more video objects having properties that match the selected video object property within a predetermined threshold. A browser interface for browsing; And an interactive video player for receiving one or more transmitted sequences of video data from a server computer corresponding to the determined one or more video objects.

In a preferred apparatus, the database stored on the server computer includes a motion trajectory database, a construction database, a shape database, a color database, and a texture database. One or more frame sequences of video data are stored on the server computer in a compressed form, such as MPEG-1 or MPEG-2.

The system may include a mechanism for comparing each selected video object property with a corresponding stored video object property in the server computer to generate a list of candidate video sequences, one for each video object property. As such, a mechanism is advantageously provided for determining one or more video objects having a collective attribute that matches a selected video object attribute based on a candidate list within a predetermined threshold. The system also includes a mechanism for matching the spatial and temporal relationships among the multiple objects in the query and the group of video object projects in the video clip.

According to a second aspect of the invention, a method is provided for extracting a video object from a frame sequence of video data comprising at least one recognizable property. The method includes quantizing a current frame of video data by determining and assigning a value to another deviation of at least one attribute represented by the video data to generate quantized frame information; Performing edge detection on a frame of video data based on the attribute to determine an edge point in the frame, thereby generating edge information; Receiving one or more segment regions of video information from previous frames; And extracting a region of video information sharing the attribute by comparing the received segment region with the quantized frame information and the generated edge information.

Preferably, the extracting step is adapted to project an area in the current frame of video data by projecting one of the received areas onto a current quantized edge detected frame to temporarily track any movement of the area. Performing interframe projection to extract; And performing intraframe division to coalesce the region extracted adjacent to the current frame under an arbitrary condition. The extracting step also includes labeling all edges in the current frame remaining in the neighboring region after intra-frame division such that each labeled edge defines a boundary of the video object in the current frame.

In a particularly preferred technique, a future frame of video information is also received, wherein an optical flow of the current frame of video information is a hierarchical block between the block of video information in the current frame and the block of video information in the future frame. Determined by performing matching, motion estimation on the extracted region of video information is performed by an affine matrix based on the optical flow. The extracted regions of the video information can be grouped based on the size and time duration as well as the affine model of each region.

In another aspect of the present invention, a method is provided for locating a video clip that best matches a user-input search query from a frame sequence of video data comprising one or more video clips including a predetermined trajectory. The method includes receiving a search query that defines at least one video object; Determining an overall distance between the received query and at least some of the one or more predefined video object trajectories; And selecting one or more of the defined video object trajectories having a minimum overall length from the received query to locate the best matched video clip or clips.

Both the search query and the predefined video object trajectories are normalized. The query normalization step maps the received query to each normalized video clip; It is desirable to scale the received and mapped query to each video object trajectory defined by the normalized video clip. The determining step is realized by either spatial distance comparison or construction distance comparison.

In another aspect of the present invention, a method is provided for positioning a video clip that best matches a user-entered search query from one or more video clips, each consisting of one or more video objects having predetermined characteristics. The method includes receiving a search query that defines one or more characteristics for one or more other video objects in the video clip; Searching for a video clip to locate a video object that matches at least one of the defined characteristics with a predetermined threshold; Determining from the positioned video object a video clip comprising one or more other video objects; And calculating a distance between the located video object and the one or more video objects defined by the search query to determine the best matched video clip from the determined video clip. Characteristics can include color, texture, motion, size or shape.

In a highly preferred apparatus, the video clip includes associated text information, the search query further includes a definition of a text property corresponding to one or more other video objects, and the method relates to positioning the text that matches the text property. Searching for text information. The best matched video clip is then determined from the determined video clip and the placed text.

Referring to FIG. 1, a system for searching and retrieving a series of specific video information that satisfies any predetermined criteria, such as the shape or motion characteristics of a video object embedded in stored video information in response to a user-defined query. An exemplary embodiment is provided. The structure of the system 100 is broadly classified into three components, the server computer 11, the communication network 120, and the client computer 130.

The server computer 110 includes a database 111 that stores metadata for video objects and visible features, and a storage subsystem that stores any textual information and raw audiovisual information associated with the extracted video objects and visible features. 112). The communication network 120 may be based on the Internet or a broadband network. Thus, although shown in FIG. 1 as one computer, server computer 110 may be any number of computers scattered around the world that can communicate with client computer 130 all over communication network 120. .

The client computer 130 includes a browser interface that allows a user to enter a search query into the computer 130 and browses the audiovisual information network 100, a keyboard 131, a mouse 132, which together form a query interface. And a monitor 133. Although not shown in FIG. 1, other query input devices, such as light pens and touch screens, can also be easily integrated into the client computer 130. The monitor 133 not only displays the visible information retrieved from the server computer 110 via the network 120, but is also used to describe the search query entered by the user of the computer 11. Such information is preferably retrieved in a compressed form, such as an MPEG-2 bitstream, and the computer 130 includes suitable commercially available hardware or software, such as an MPEG-2 decoder, to decompress the retrieved information into a displayable format. .

Using keyboard 131, mouse 132, and the like, a user writes a search query on computer 130 that specifies one or more searchable attributes of one or more video objects embedded in a clip of video information. can do. Thus, for example, if a user wants to navigate a video clip containing a baseball moving on a trajectory, the user sketches the motion 134 of the object to be included in the query, and adds additional navigable properties such as size, shape, color, and texture. Choose. An example query interface is depicted in FIG.

As used herein, a "video clip" is one or more video objects with identifiable properties such as, but not limited to, a baseball player swing bet, a surfboard moving across the ocean, or a horse running across the plains. It refers to the frame sequence of video information having a. A "video object" is a continuous set of pixels that is homogeneous with one or more important features, such as texture, color, motion, and shape. Thus, a video object is formed by one or more video regions that represent at least one characteristic and coincidence. For example, a walking person's shot (where a person is an “object”) is divided into a set of adjacent areas that differ in criteria such as shape, color, and texture, but all areas show correspondence to their motion properties.

Referring to FIG. 3, the search query 300 may include global parameters such as color 301, texture 302, motion 303, shape 304, size 305, and pan and zoom of a given video object. It can contain other attributes, such as Various weights indicating the relative importance of each attribute may also be incorporated into the search query 306. Upon receiving a search query, the browser at computer 301 searches for similar attributes stored in database 111 of server computer 110 via network 120. The server 110 includes several feature databases, one of each individual feature being indexed by the system, for example, a color database 311, a texture database 312, a motion database 313, a shape database 314, and Size database 315. Each database is stored as a compressed MPEG bitstream in storage 112. Of course, other compression formats or compressed data may be used.

At the server, each queried attribute is compared to a stored attribute, a detailed description of which will be given later. Thus, the queried color 301 is matched against the color database 311, and so is the matching of the texture 322, motion 323, shape 324, size 325, and any other attribute. For example, the list of candidate video shots, which are the color object list 331, texture object list 332, motion object list 333, shape object list 334, and size object list 335, is the respective object specified in the query. Occurs for In server computer 110, each list is combined with a preselected rank threshold or with a characteristic distance threshold, so that only the most likely candidate shots are regenerated.

Next, the candidate list for each object at the predetermined minimum threshold is combined 350 to form a single video shot list. The merging process involves a comparison of each of the generated candidate lists 331, 332, 333, 334 and 335, so that video objects that do not appear on all candidate lists are screened out. The candidate video objects remaining after this screening are sorted based on their global weighted distance from the queried attribute. In turn, the global threshold based on a predetermined individual threshold and preferably modified by the user-defined-weighted recorded in the query 306 is used to truncate the object list to the best matched candidate or candidate. Preferred global threshold is 0.4.

For these video shots in the merged list, key-frames are dynamically extracted from the video shot database and returned to the client 130 via the network 120. Once the user is satisfied with the result, the video shot corresponding to the key frame can be extracted from the video database in real time by "cutting" the video shot from the database. The video shots are extracted from the video database using video editing schemes in a compressed domain, such as Chang et al., Patent application number PCT / US97 / 08266, filed May 16, 1997, incorporated herein by reference.

Those skilled in the art understand that the matching technique of FIG. 3 can be performed at the object level or region level.

Various techniques used in the systems incorporated herein in connection with FIG. 1 are described below. To generate a meaningful search query, client computer 130 may limit or quantize the attributes to be searched. Thus, for color, the acceptable color set can uniformly quantize the HSV color space, and it is preferable that the color already be quantized in that any color is acceptable in the modem computer, even if true color is used.

For texture, known MIT texture databases are used to assign texture attributes to various objects. Thus, the user must be selected from the 56 available textures in the database to form a search query. Naturally, other texture sets can be used easily.

The shape of the video object may be any polygon depending on an ellipsoid of any shape and size. The user can thus sketch any polygon with the aid of the cursor, and other known shapes such as round, oval and rectangular are predefined, easily inserted and adjusted. The query interface converts the shape into a set of numbers that accurately represent the shape. For example, a circle is represented by a center point and a radius and an ellipse is represented by a focal point and a distance.

For motion, two different models are applied. First, the search may be based on the perceived motion of the video object as derived from the optical flow of the pixels in the video object. Optical flow is a combined effect of both global motion (ie camera motion) and local motion (ie object motion). For example, the camera tracks the car's motion, and the car appears static in video sequence.

Secondly, the search is based on "true motion of the video object". True motion refers to the local motion of an object after global motion is compensated for. In a moving car, the true motion of the car is the actual motion of the car driving.

The global motion of the main background scene is estimated using a known 6 parameter affine model, while the hierarchical pixel-domain motion estimation method is used to extract the optical flow. The affine model of global motion is used to compensate for the global compensation of all objects in the same scene. A six-parameter model is described below.

Where a _i is an affine parameter, x, y are axes, and dx, dy are displacement or optical flow in each pixel.

The classification of global camera motion, such as zoom, pan or tilt, is based on global affine estimation. For the detection of panning, the histogram of the global motion velocity field should be calculated in eight directions as will be appreciated by those skilled in the art. If there is one direction in the main number of moving pixels, camera panning in that direction is declared. Camera zooming is detected by examining the average amplitude of the global motion velocity field and the 2 scaling parameters a ₁ and a ₂ in the affine model. When there is sufficient motion (ie, when the average amplitude is above a predetermined threshold), a ₁ and a ₂ are both positive and above a certain threshold, and the camera zooming in is declared. Otherwise, when a ₁ and a ₂ are both negative and less than or equal to a random value, a camera zooming out is declared. This information may be included in the search query to indicate the presence or absence of camera panning or zooming.

The search may also include time information associated with one or more objects. This information can define the overall duration of the object in either relative periods, either long or short, or absolute periods, ie seconds. In the case of a multi-object query, the user is given the flexibility to specify the overall scene time sequence by specifying the "arrival" order of various objects in the scene, or the order of destruction, i.e., the order in which video objects disappear from the video clip. Another useful property related to time is the scale factor, or rate at which the size of the object changes over the duration of the object's existence. As such, acceleration may be a suitable attribute for searching.

Before forming the actual query for the browsing browser, various attributes can be weighted to reflect their relative importance in the query. Feature weights are global to the entire animated sketch, and the attribute color has the same weight for all objects. The final ranking of video shots returned by the system is influenced by the weights that the user has assigned to various attributes.

With reference to FIG. 4, a technique for extracting video objects from video clips is described below. The video clip formed by the frame sequence of compressed video information 400 including the current frame 401 is illustratively analyzed in FIG. 4.

Prior to any video object extraction, the raw video is preferably separated into a video clip such as video clip 400. Video clip separation can be accomplished by a scene change detection algorithm as disclosed in Chang et al., Patent Application No. PCT / US97 / 08266, filed May 16, 1997, above. Chang et al. Described a sudden and instantaneous (eg, dissolve) in a compressed MPEG-1 or MPEG-2 bitstream using motion vector and discrete cosine transform coefficients from an MPEG bitstream to compute a static measure. Techniques for detecting fade in / out, wipe scene changes are described. This measure is used to verify a heuristic model of abrupt or abrupt scene changes.

In order to segment and track the video object, the concept of an "image area" is utilized. An image area is typically an adjacent area of pixels with matching characteristics such as color, texture, or motion that correspond to a portion of a real object such as a car, person, or house. The video object consists of the sequence of an example of an image region tracked in successive frames.

The technique illustrated in FIG. 4 splits and tracks a video object by taking into account static attributes, edges, and motion information in the video shot. The current frame n 401 is preferably used for both projection and segmentation technique 430 and motion estimation technique 440, which will be described later.

Prior to projection and segmentation, the information is preprocessed in two different ways to obtain a consistent result. In parallel, the current frame n is used for both quantization 410 and edge map generation 420 based on one or more recognizable attributes for the information. In the preferred embodiment as will be described below, the color is selected as the attribute because of its matching under changing conditions. However, other attributes of information such as textures are formed as above based on the projection and segmentation process, as will be appreciated by those skilled in the art.

As illustrated in FIG. 4, the current frame (ie, frame n) is transformed 411 in a perceptually uniform color space, such as CIE L ^* u ^* v ^* space. Nonuniform color spaces, such as RGB, are not suitable for color segmentation because distance measurements in these spaces are not proportional to perceptual differences. The CIE L ^* u ^* v ^* color space divides the color into one luminance channel and two chrominance channels, allowing for deviations in the weight given by the luminance and chrominance. This is a very important option that allows the user the ability to assign different weights depending on the characteristics of the given video shot. Rather, it is generally better to assign more weighting to, for example, two or more color difference channels.

L ^* u ^* v ^* color space transform information is adaptively quantized (412). Preferably, a clustering based quantization technique, such as the known K-Means or Self Organization Map clustering algorithm, is used to quantize pallets from real video data in L ^* u ^* v ^* space. Is used to generate More general fixed level quantization techniques may also be used.

After adaptive quantization 412, nonlinear median filtering 413 is preferably used to remove minor details and outliers in the image while preserving edge information. Quantization and median filtering thus simplify the image by removing possible noise and tiny details.

Simultaneously with quantization 410, an edge map of frame n is generated 420 using an edge detection algorithm. An edge map is a binary mask in which the edge pixel is set to 1 and the non-edge pixel is set to zero. It is generated through a known Canny edge detection algorithm that performs 2-D Gaussian pre-smoothing on the image and takes directional differential coefficients in the horizontal and vertical directions. The differential coefficients are used in order to calculate the gradients and the maximum of the local gradients taken as candidate edge pixels. This output comes through a two-level threshold synthesis process to produce the final edge map. A simple algorithm can be used to automatically select two threshold levels in the synthesis process based on the gradient histogram.

Both the quantized attribute information and the edge map are utilized in the projection and segmentation step 430, where regions with matched attributes, such as color, are combined. Projection and segmentation consists of four substeps, including interframe projection 431, intraframe projection 432, edge point labeling 432, and simplification 433.

The interframe projection step 431 projects and tracks the previous frame, i.e., the segment region previously determined from frame n-1 in FIG. Referring to FIG. 5, in the affine projecting step 510, the region existing from the frame n-1 is first projected into the frame n according to the affine parameter to be described below. If the current frame is the first frame in sequence, this step is simply skipped. Next, a modified pixel labeling process 520 is applied. For every non-edge pixel in frame n, if they are covered by the projected area and the weighted Euclidean distance, WL = 1, Wu = 2, and Wv = 2 are the default values, and the color and area of the pixel The average color is between a given threshold, eg 256 or less, and the pixels are labeled consistent with the sphere area. If the pixel is covered by one or more projected areas below a given threshold, it is labeled as the area with the narrowest distance. However, if no area meets this condition, a new label is assigned to the pixel. Note that the edge pixels are not processed and are not labeled at the same time. As a result, the connection graph 530 is embedded among all labelings, i.e., regions, and the two regions are linked as neighbors if pixels in one region have neighboring pixels (four connection modes) in the other region.

In an in-frame projection step 432, the tracked new label (area) is incorporated into a large area. Referring to FIG. 6, an interactive spatial constraint clustering algorithm 610 is utilized, where two adjacent domains with a given threshold, preferably a color distance of less than 225, have a color distance between any two adjacent regions that is greater than the threshold. Until it merges into one new area 620. If a new region is to be generated from two adjacent regions, this average color is calculated by taking a weighted average of the average colors of the two sphere regions (630), and the sizes of the two sphere regions are used as weights. The region concatenation is updated 640 for all neighbors of the two old regions. The new region is assigned one label from the labels of the two sphere regions (650), and if all of the sphere labels are tracked from the previous frame, the larger region labels are selected, and if one sphere label is tracked and the other is not tracked Select the label, otherwise the label of the larger area is selected. The two sphere regions are dropped 660 and the process is repeated 670 until no new regions have been determined.

Returning to FIG. 4, edge points can be assigned 433 to their neighboring areas according to color measurements to ensure accuracy of area boundaries. In both the interframe and intraframe division processes described above, only non-edge pixels are processed and labeled. Edge pixels do not merge into any area. This ensures that the areas clearly separated by the long edges are not spatially connected and do not merge with each other. After labeling all non-edge pixels, the edge pixels are assigned to their neighboring regions according to the same color distance measurement. The aforementioned connection graph can be updated during the labeling process.

Finally, a simplification process 434 is applied to remove small areas, that is, areas less than a given number of pixels. The threshold parameter depends on the frame size of the image. For QCIF size (176x120) images, the preferred default value is 50. If the small area is close to one of its neighboring areas, that is, the color distance is below the color threshold, the small area is merged with the neighboring area. Otherwise, small areas are dropped.

In conjunction with the projection and segmentation process 430, the optical flow of the current frame n is described by M. Bierling's "Displacement Estimation by Hierarchical Block Matching" [1001 SPIE Visual Comm. & Immage Processing (1988) is derived from frames n and n + 1 in the motion estimation step 440 using a hierarchical block matching method such as the technique light described in the following. Unlike conventional block matching techniques, where only the minimum average absolute luminance difference is searched using a fixed measurement window size, this method uses a compact displacement vector field (optical flow) using a clear size of the measurement window at different levels of strata. Estimate This yields a relatively feasible uniform result. It is desirable to use a three level hierarchy.

After color or other attribute regions are extracted and a measurement of the optical flow in the frame occurs, a standard linear regression algorithm is used to estimate the affine motion for each region 450. For each region, linear regression is used to determine the six parameters of the affine motion equation, ie, the equation that best fits the dense motion field inside the area.

The affine motion parameters are preferably further refined 460 using a three step region matching method in six-dimensional affine space, which is an extension of the common three step block matching technique used in estimation / MPEG compression. A description of this known technique is described in "Digital Pictures: Representation, Compression and Standards, Second Edition" by Arun N. Netravali et al. (Pp. 340-344; Plenum Press, New York and London, 1995). You can find it at For each region, the original affine model is used to search for a new model that projects the region with the minimum mean absolute luminance error. The search along each dimension is defined as 10% of the initial parameters of that dimension.

Through affine motion estimation 450 and subdivision 460, a uniform color region with affine motion parameters is generated for frame N. Likewise, these regions will be tracked in the segmentation process of frame n + 1.

Finally, region grouping 470 can be applied in the process at the end, avoiding oversegmentation and obtaining a high level video object. Some criteria will be appropriate for grouping or identifying most major areas.

Firstly, the size of the determined regions, i.e. the average number of pixels, and the duration, i. Areas with both small size and / or small duration may be dropped.

Secondly, adjacent regions with similar motion can be grouped into one moving object. This applies to video sequences with moving objects to detect their objects. To realize this grouping, a spatial-constrained clustering process can be used to group adjacent regions based on their affine motion parameters in each praim. The temporary search process can then be used to link the area groups together as different video frames together as one video object if their area groups include at least one common area. For each region group in the starting frame, this search begins in the region with the long duration inside the group. If the area group has been continuously tracked for more than a predetermined amount of time, for example 1/3 of a second, a new object label is assigned to this area group. Finally, a temporary alignment process can be applied to ensure the consistency of the regions contained within the video object. If an area is briefly present, for example only for less than 10% of the duration of its own video object, it is regarded as an error of the area grouping process and is dropped from the video object.

As described in connection with FIG. 3, the server computer 110 may include a plurality of feature databases, such as a color database 311, a texture database 312, a motion database 313, a shape database 314, and a size. Database 315, each database being associated with original video information. For each video object extracted from the described video clips, for example the video object extracted by the method described with reference to FIG. 4, an additional feature is advantageously stored in the database of the server computer 110.

For the color database 311, the color represented for the video object is a quantized CIE-LUV space. Quantization is not a static process, but in the quantization palette, each video shot changes with color variations. Although the preferred configuration uses representative colors, the color database may also include a single color, average color, color histogram, and / or color pairs of video objects.

For texture database 312, three so-called Tamura texture measurements, namely coarseness, contrast and orientation, are calculated as a measure of the texture content of the object. Optionally, waveform domain textures, texture histograms, and / or Raws Filter-based textures can be used to enhance the database 312.

For motion database 313, the motion of each video object is stored as a list of N-1 vectors, where the number of frames in the video clip is N. Each vector is the average translation of the center of the object between successive frames after global motion compensation. In accordance with this information, we also store the frame rate of the video shot sequence so that both the "velocity" of the object and its duration are established.

For shape database 314, the principal component of the shape of each video object is E. Saber et al., "Region-based affine shape matching for automatic image annotation and query-by-example" [8 Visual Comm. and Image Representation 3-20 (1997)] by known eigenvalue analysis. Two other new properties are also calculated: normalized area and percentage area. The normalized area is the area of the object divided by the area of the circumscribed circle. If the area can be fairly approximated by a circle, then this approximation is made. For example, if the axis ratio of an object is greater than 0.9 and the normalization area is also greater than 0.9, the shape is classified as a circle. Optionally, geometric invariants, different order of moments in each dimension, polynomial approximations, spline approximations, and / or algebraic invariants can be used.

Finally, for size database 315, the size for the pixels is stored.

The evolution of the spatial relationship over time can be indexed as an edit or as a continuation of the original spatial graph. When the spatial relationship between objects in a frame is indexed by a spatial graph or a 2-D strip, other databases, such as a spatio-temporal database, can be used.

Next, a technique for comparing the search query with the information stored in the feature database 111 of the server computer 110 will be described. As described with reference to FIG. 3, server 110 may match matching operations 321, 322, 323, 324, 325, query color 301, texture 322, motion 323, shape 324, Perform other attributes on information stored in size 325 and database 311, 312, 313, 314, 315, etc. to generate lists of candidate video shots 331, 332, 333, 334, 335 Let's do it.

For the matching motion trajectory 323, the three-dimensional trajectory of the video object is optimally used. A three-dimensional dimension consisting of two spatial dimensions x, y and a temporal dimension t that normalizes the number of frames is expressed by the sequence {x [i], y [i] where i = 1, N}. The frame rate provides real time information.

In client computer 130, the user can sketch the object trajectory as the sequence of the highest point in the x-y plane, and also specify the duration of the object in the video clip. The duration is quantized to three levels with respect to the frame rate: long level, middle level, and short level. The overall trajectory can be easily computed by uniformly sampling the motion trajectory, for example based on a frame rate of 30 frames per second.

According to a preferred form of the invention, two main modes of the matching trail, namely the spatial mode and the construction mode, will be described. In spatial mode, the motion trail is projected on the x-y plane, resulting in an ordered contour. By measuring the distance between the query contour and the corresponding contour for each object in the database, the candidate trajectory is determined. This kind of matching provides a "time-scale invariance" and allows the user to execute the trajectory. This is useful when you are not sure of the time taken by an object.

In construction mode, the entire motion trail is used to calculate distance according to the following metric system.

Here, the subscripts q and t refer to the query and target trajectories, respectively, and the index i progresses over the number of frames. Optionally, the index may proceed over a set of subsamples.

In general, there is an advantage because the duration of the query object is different from that in the database. First, if the durations are different, the two trajectories will only match for two short durations, i.e., index i will proceed over the minimum query duration and database duration.

Optionally, the query and stored trajectory durations may each be normalized to a canonical duration before performing matching. For example, if each video clip is normalized such that the playback frame rate is time-scaled on a predetermined time scale, the search query maps the query and then the query mapped to the video object trajectory defined by the normalized video clip. By scaling, the video clip must be normalized on the same predetermined time scale.

As in the case of motion, matching the query color 201, texture 222, shape 224, size 225 and other attributes to information stored in the database includes an optimal comparison process. For color, the color of the query object matches the average color of candidate tracking objects in the database according to equation 4:

Where C _d is the weighted Euclidean color distance of the CIE-LUV space, and the subscripts q and t refer to the query and the target, respectively.

For the texture, three Tamura texture parameters for each tracking object are compared with the parameters stored in the database. The distance metric is the Euclidean distance weighted for each texture characteristic with a deviation along each channel, as shown in Equation 5:

Here, α, β, and Φ represent the contrast, contrast and orientation, respectively, and the variables σ (α, β, Φ) represent the deviations of the corresponding attributes.

For shapes, the metric can simply include only the main components of the shape, as shown in equation 6:

Where and are the eigenvalues along the principal axis of the object, ie their ratio is the aspect ratio. Other more complex algorithms, such as geometric invariant, can be used.

The size is satisfied as the distance to the area ratio as shown in equation 7:

Here, Aq and At represent percentage areas of the query and the target, respectively.

The total distance is normalized so that the diameter range of each metric lies at [0,1], according to Equation 8, and then simplifies the weighted sum of these distances:

With reference to FIG. 7, a combination video and text based on a search technique for searching for a location of a video clip based on both interposed video object information and associated audio or text information will be described. This technology allows not only the descriptive power of natural language, but also the use of visible content such as properties of objects such as motion, color and texture.

In addition to entering one or more visible attributes, such as color 701, texture 702, motion 703, and shape 704, when a search query 700 is entered, the user enters a string of text information 710. Will be entered. The information may be entered directly via the keyboard 131, through a microphone connected with commercially available speech recognition software, or through any other person with computer interfacing techniques.

The visible information will be matched 730 against the storage library 720 of visible attribute information as described in connection with FIG. 3 to generate an optimal video clip at a predetermined threshold. However, the structure of FIG. 7 is further illustrated in FIG. 3 by performing text matching 750 with the extracted keyword 740 associated with the same video clip used to generate the visible library 720. The result of text matching 750 is one or more optimal matching video clips based solely on text. Finally, the results of the visible match 730 and text match 750 are combined 760 to determine with high precision the video clip found by the original search query 700.

In the case of MPEG extruded audiovisual information, the library of extracted keywords 740 is either manually annotated or first extracts the audio information from the extruded bitstream to transfer the audio, and then the amount of text transferred by the keyword spotting technique. By reducing.

The foregoing description merely illustrates the principles included in the present invention. Other variations of the invention will be apparent to those skilled in the art, and the scope of the invention is limited only to that set forth in the appended claims.

Claims (35)

In an object-oriented system that allows a user to navigate the location of one or more video objects from one or more video clips via an interactive network, a. One or more server computers including storage for the one or more video clips and storage for one or more databases of video object attributes corresponding to the video clips; b. A communication network connected to the one or more server computers to perform transmission of the one or more video clips from the server computer; And c. Is connected to the communication network, Iii. A query interface for specifying video object attribute information including motion trajectory information; Ii. Access the query interface, receive the selected video object property information, and browse within the server computer via the stored video object property by the communication network to obtain an attribute that optimally matches the specified video object property. A browser interface for determining one or more video objects having; And Iii. An interactive video player receiving one or more transmission sequences of frames of video data from the server computer corresponding to the determined one or more video objects. Client computer Object-oriented system comprising a. The system of claim 1, wherein one of the one or more databases stored on the server computer comprises a motion trajectory database. The system of claim 1, wherein one of the one or more databases stored on the server computer comprises a spatio-temporal database. The system of claim 1, wherein one of the one or more databases stored on the server computer comprises a shape database. The system of claim 1, wherein one of the one or more databases stored on the server computer comprises a color database. The system of claim 1, wherein one of the one or more databases stored on the server computer comprises a texture database. The system of claim 1, wherein one of the one or more databases stored on the server computer comprises a pan database. The system of claim 1, wherein one of the one or more databases stored on the server computer comprises a zoom database. The system of claim 1, wherein the one or more sequences of frames of video data are stored on the server computer in a compressed format. The method of claim 1, Means for comparing at least one of said one or more specific video object properties with a stored video object property corresponding in said server computer to at least one of said server computers to generate lists of candidate video sequences, one for each video object property. Including system. 11. The apparatus of claim 10, wherein the server computer is connected to the comparison means to receive one or more video objects having an aggregate property that optimally matches the selected video object property based on the candidate list. Further comprising a comparison means for determining. The method of claim 11, wherein the query video object property information includes properties for one or more video objects. The comparing means compares each of the one or more specific video object properties for each video object with a corresponding stored video object property in the server computer to generate a list of candidate video sequences, one for each video object property for each video object. Let's And the determining means determines one or more video objects having an aggregate attribute that best matches the selected video object attribute based on the candidate list for each query video object. A method for extracting a video object from a video clip that includes at least one recognizable property, the method comprising: a. Quantizing the current frame of the video data by determining and assigning values for different deviations of the at least one attribute represented by the video data to generate quantized frame information; b. Generating edge information by performing edge detection on the frame of video data based on the at least one attribute to determine an edge point within the frame; c. Receiving information defining one or more segment areas from a previous frame; And d. Extracting an area of video information sharing the at least one attribute from the current frame by comparing the quantized frame information and the generated edge information with the received system region Video object extraction method comprising a. The method of claim 13, wherein the attribute is color, and the quantization step is Converting the current frame into uniform color space information, adaptively quantizing the color space information into a palette, and filtering the palette to remove noise therefrom. 15. The method of claim 14, wherein said adaptive quantization step comprises quantization by a clustering algorithm. 15. The method of claim 13, wherein the edge detection step includes performing canny edge detection on the current frame to generate the edge information as an edge map. The method of claim 13, wherein the extracting step, a. Performing inter-frame projection to extract an area within the current frame of video data by projecting one of the received areas onto a currently quantized edge detection frame to temporarily track any movement of the area; And b. Performing intra-frame division to coalesce adjacent extracted regions within the current frame; Video object extraction method comprising a. 18. The method of claim 17, wherein the attribute is color and the interframe projection step is a. Pitching the received area from the previous frame to the current frame to temporarily track an area; b. Labeling each non-edge pixel in the current frame that matches the received or new area; And c. Linking neighboring regions by generating a connection graph from the labeling Video object extraction method comprising a. The method of claim 18, wherein the intra-frame division step, a. Merging all adjacent areas having a color distance less than a predetermined threshold into a new area; b. Determining an average color for the new area; c. Updating the connection graph; d. Assigning a new label to the new region from a label previously assigned to the coalesced region; And e. Dropping the coalesced region Video object extraction method comprising a. 18. The method of claim 17, wherein the extracting step further comprises labeling all edges in the current frame remaining after in-frame division for neighboring regions such that each labeled edge defines a boundary of a video object in the current frame. Video object extraction method. 21. The method of claim 20, wherein said extracting step further comprises simplifying said extracted region by removing any region having a size below a predetermined threshold. The method of claim 13, e. Receiving a future frame of video information; f. Determining an optical flow of the current frame of video information by performing hierarchical block matching between the block of video information in the current frame and the block of video information in the future frame; And g. Performing motion estimation on the extracted region of the video information based on the optical flow Video object extraction method further comprising. 23. The method of claim 22, further comprising grouping the determined regions within the current frame by size and duration. 23. The method of claim 22, further comprising grouping the determined regions within the current frame by determining an internal moving object. 12. A method for searching for a location of a video clip that best matches a user input search query from one or more video clips, wherein each video clip includes one or more video objects that temporarily move in a predetermined trajectory. a. Receiving a search query defining at least one video object trajectory; b. Determining a total distance between the received query and at least a portion of one or more predetermined video object trajectories; And c. Searching for the location of the optimal matching video clip or clips by selecting one or more of the defined video object trajectories with the smallest distance from the received query. How to include. The method of claim 25, wherein the stored video clips are normalized such that a playback frame rate is scaled on a predetermined time scale, Normalizing the received search query by mapping the received query to each normalized video clip, and scaling the received matching query with each video object trajectory defined by the normalized video clips. Including, And the determining step determines a total distance between the normalized receive query and the normalized video object trajectory. 27. The method of claim 25, wherein the determining step comprises comparing a spatial distance between the received video object trajectory and at least a portion of the one or more predetermined video object trajectories. 27. The method of claim 25, wherein the determining comprises comparing a construction distance between the received video object trajectory and at least a portion of the one or more predetermined video object trajectories. 12. A method for searching for a location of a video clip that best matches a user input search query from one or more video clips, wherein the video clip includes one or more video objects each having a predetermined property. a. Receiving a search query that defines one or more attributes for one or more different video objects in the video clip; b. Searching for the video clip to search for a location of one or more video objects that match at least one of the defined attributes to a predetermined threshold; c. Determining, from the located video object, one or more video clips that include the one or more different video objects; And d. Determining an optimal matching video clip from the determined video clip by calculating a distance between the one or more video objects defined by the search query and the located video object. How to include. The method of claim 29, wherein the one or more attributes include color; The matching step includes determining an average color for each of the query video objects, and comparing the average color with color information stored in a database. The method of claim 29, wherein the one or more attributes include a texture, The matching step includes determining a coarseness, contrast, and orientation for each of the query video objects, and comparing the joy, contrast and orientation information to the joy, contrast, and orientation information stored in a database. . The method of claim 29, wherein the one or more attributes comprise a shape, The matching step includes determining an eigenvalue along a major axis for each of the query video objects, and comparing the eigenvalue with shape information stored in a database. The method of claim 29, wherein the one or more attributes include a size, The matching step includes determining a percentage region for each of the query video objects, and comparing the region with region information stored in a database. 30. The method of claim 29, wherein the video clip includes associated text information, The search query further includes a definition of a text characteristic corresponding to the one or more different video objects, and further comprising searching for the associated text information to place text that optimally matches the text characteristic. 31. The method of claim 30, wherein the optimal matching video clip is determined from the determined video clip and the placed text.