US20110064136A1 - Methods and architecture for indexing and editing compressed video over the world wide web - Google Patents

Methods and architecture for indexing and editing compressed video over the world wide web Download PDF

Info

Publication number
US20110064136A1
US20110064136A1 US12/874,337 US87433710A US2011064136A1 US 20110064136 A1 US20110064136 A1 US 20110064136A1 US 87433710 A US87433710 A US 87433710A US 2011064136 A1 US2011064136 A1 US 2011064136A1
Authority
US
United States
Prior art keywords
frame
coded
video information
video
compressed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US12/874,337
Inventor
Shih-Fu Chang
Horace J. Meng
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Columbia University of New York
Original Assignee
Columbia University of New York
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from PCT/US1997/008266 external-priority patent/WO1998052356A1/en
Application filed by Columbia University of New York filed Critical Columbia University of New York
Priority to US12/874,337 priority Critical patent/US20110064136A1/en
Assigned to THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK reassignment THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MENG, HORACE J., CHANG, SHIH-FU
Assigned to NATIONAL SCIENCE FOUNDATION reassignment NATIONAL SCIENCE FOUNDATION CONFIRMATORY LICENSE (SEE DOCUMENT FOR DETAILS). Assignors: COLUMBIA UNIVERSITY NEW YORK MORNINGSIDE
Publication of US20110064136A1 publication Critical patent/US20110064136A1/en
Priority to US13/078,626 priority patent/US9330722B2/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/19Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
    • G11B27/28Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/02Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
    • G11B27/031Electronic editing of digitised analogue information signals, e.g. audio or video signals
    • G11B27/034Electronic editing of digitised analogue information signals, e.g. audio or video signals on discs

Definitions

  • the present invention relates to techniques for editing and parsing compressed digital information, and more specifically, to editing and parsing visual information in the compressed domain.
  • video information is generally transmitted between systems in the digital environment the form of compressed bitstreams that are in a standard format, e.g., Motion JPEG, MPEG-1, MPEG-2, H.261 or H.263.
  • DCT Discrete Cosine Transform
  • Run-length encoding and entropy coding i.e., Huffman coding or arithmetic coding
  • Huffman coding or arithmetic coding are applied to the quantized bitstream to produce a compressed bitstream which has a significantly reduced bit rate than the original uncompressed source signal.
  • the process is assisted by additional side information, in the form of motion vectors, which are used to construct frame or field-based predictions from neighboring frames or fields by taking into account the inter-frame or inter-field motion that is typically present.
  • bitstreams In order to be usable by a receiving system, such coded bitstreams must be both parsed and decoded. For example, in the case of an MPEG-2 encoded bitstream, the bitstream must be parsed into slices and macroblocks before the information contained in the bitstream is usable by an MPEG-2 decoder. Parsed bitstream information may be used directly by an MPEG-2 decoder to reconstruct the original visual information, or may be subjected to further processing.
  • Rivl uses group of pictures (GOPs) level direct copying whenever possible for “cut and paste” operations on MPEG video, it does not use operations in the compressed domain at frame and macroblock levels for special effects editing. Instead, most video effects in Rivl are done by decoding each frame into the pixel domain and then applying image library routines.
  • GOPs group of pictures
  • a truly comprehensive technique for indexing and manipulating digital video must meet two requirements.
  • the technique must provide for key content browsing and searching, in order to permit users to efficiently browse through or search for key content of the video without full decoding and viewing the entire video stream.
  • key content refers to key frames in video sequences, prominent video objects and their associated visual features (motion, shape, color, and trajectory), or special reconstructed video models for representing video content in a video scene.
  • the technique must allow for video editing directly in the compressed domain to allow users to manipulate an specific object of interest in the video stream without having to fully decode the video. For example, the technique should permit a user to cut and paste any arbitrary segment from an existing video stream to produce a new video stream which conforms to the valid compression format.
  • An object of the present invention is to provide comprehensive techniques for indexing and manipulating digital video in the compressed domain.
  • Another object of the present invention is to provide techniques for key content browsing and searching of compressed digital video without decoding and viewing the entire video stream.
  • a further object of the present invention is to provide techniques which allow for video editing directly in the compressed domain
  • a still further object of the present invention is to provide tools that permit users who want to manipulate compressed digital video information to extract a rich set of visual features associated with visual scenes and individual objects directly from compressed video.
  • Yet another object of this invention is to provide an architecture which permits users to manipulate compressed video information over a distributed network, such as the Internet.
  • the present invention provides a method for detecting moving video objects in a compressed digital bitstream which represents a sequence of fields or frames of video information for one or more previously captured scenes of video.
  • the described method advantageously provides for analyzing a compressed bistream to locate scene cuts so that at least one sequence of fields or frames of video information which represents a single video scene is determined.
  • the method also provides for estimating one or more operating parameters of a camera which initially captured the video scene, by analyzing a portion of the compressed bitstream which corresponds to the video scene, and for detecting one or more moving video objects represented in the compressed bitstream by applying global motion compensation with the estimated camera operating parameters.
  • the compressed bitstream is a bitstream compressed in accordance with the MPEG-1, MPEG-2, H261, or H263 video standard.
  • analyzing can beneficially be accomplished by parsing the compressed bitstream into blocks of video information and associated motion vector information for each field or frame of video information which comprises the determined sequence of fields or frames of video information representative of said single scene, performing inverse motion compensation on each of the parsed blocks of video information to derive discrete cosign transform coefficients for each of the parsed blocks of video information, counting the motion vector information associated with each of the parsed blocks of video information, and determining from the counted motion vector information and discrete cosign transform coefficient information whether one of the scene cuts has occurred.
  • analyzing is performed by parsing the compressed bitstream into blocks of video information and associated motion vector information for each field or frame of video information which comprises the determined sequence of fields or frames of video information representative of the scene, and estimating is executed by approximating any zoom and any pan of the camera by determining a multi-parameter transform model applied to the parsed motion vector information.
  • the frame difference due to camera pan and zoom motion may be modeled by a six-parameter amine transform which represents the global motion information representative of the zoom and pan of the camera.
  • the detecting step advantageously provides for computing local object motion for one or more moving video objects based on the global motion information and on one or more motion vectors which correspond to the one or more moving video objects.
  • thresholding and morphological operations are preferably applied to the determined local object motion values to eliminate any erroneously sensed moving objects.
  • border points of the detected moving objects are determined to generate a bounding box for the detected moving object.
  • the present invention also provides for an apparatus for detecting moving video objects in a compressed digital bitstream which represents a sequence of fields or frames of video information for one or more previously captured scenes of video.
  • the apparatus includes means for analyzing the compressed bistream to locate scene cuts therein and to determine at least one sequence of fields or frames of video information which represents a single video scene, means for estimating one or more operating parameters for a camera which initially viewed the video scene by analyzing a portion of the compressed bitstream which corresponds to the video scene, and means for detecting one or more moving video objects represented in the compressed bitstream by applying global motion compensation to the estimated operating parameters.
  • a different aspect of the present invention provides techniques for dissolving an incoming scene of video information which comprises a sequence of fields or frame of compressed video information to an outgoing scene of video information which comprises a sequence of fields or frame of compressed video information.
  • This technique advantageously provides for applying DCT domain motion compensation to obtain DCT coefficients for all blocks of video information which make up a last frame of the outgoing video scene and the first frame of the incoming video scene, and for creating a frame in the dissolve region frame from the DCT coefficients of the last outgoing frame and the first incoming frame.
  • an initial value for a weighting function is selected prior to the creation of a first frame in the dissolve region and is used in the creation of the first frame in the dissolve region.
  • the weighting value is then incremented, and a second dissolve frame from the DCT coefficients is generated.
  • a technique for masking a compressed frame of digital video information is provided.
  • the technique first determines whether the frame to be masked is intra-coded, predictive-coded or bi-directionally predictive-coded. If the frame is intra-coded, the technique provides for extracting DCT coefficients for all blocks within the frame, examining a block, to determine where in the frame the block is located, setting DCT coefficients for the block to zero if the block is outside the mask region, and applying a DCT cropping algorithm to the DCT coefficients if the block is on the boundary of the mask region.
  • the technique provides for examining motion vectors associated with block, to determine whether they point to blocks outside or on the mask region, and reencoding the block if a motion vector points to blocks outside the boundary, or on, the mask region.
  • a technique for generating a frozen frame of video information from a sequence of frames of compressed video information is provided.
  • the technique attractively provides for selecting a frame of compressed video information to be frozen, determining whether the frame to be frozen is intra-coded, predictive-coded or bi-directionally predictive-coded, and if the frame is not intra-coded, converting it to become intra-coded, creating duplicate predictive-coded frames, and arranging the intra-coded frame and the duplicate predictive-coded frames into a sequence of compressed frames of video information.
  • a system for editing compressed video information over a distributed network includes a client computer, a network link for permitting said client computer to search for and locate compressed video information on said distributed network, and tools for editing a compressed bitstream of video information over the distributed network.
  • FIG. 1 illustrates a system in accordance with one aspect of the present invention
  • FIG. 2 is a flowchart which illustrates how a scene cut is detected in accordance with one aspect of the present invention
  • FIG. 3 is a flowchart which illustrates how camera parameters are estimated in accordance with one aspect of the present invention
  • FIG. 4 is a vector diagram which serves to explain global and local motion
  • FIG. 5 depicts an exemplary frame of compressed video information and motion vectors for the frame
  • FIG. 6 is a flowchart which illustrates global motion compensation in accordance with one aspect of the present invention.
  • FIG. 7 depicts prior art editing of compressed video bistreams
  • FIG. 8 a depicts the dissolve effect
  • FIG. 8( b ) is a flowchart which illustrates the dissolve effect
  • FIG. 9 a depicts masking
  • FIG. 9( b ) is a flowchart which illustrates masking
  • FIG. 10 is a flowchart which illustrates the freezeframe effect
  • FIG. 11 depicts two alternative techniques for the slow motion effect
  • FIG. 12 is a system diagram of a distributed network in accordance with one aspect of the present invention.
  • FIG. 13 depicts exemplarily techniques which may be executed in the distributed network illustrated in FIG. 12 .
  • FIG. 1 an exemplary embodiment of our invention which permits a user to edit and parse visual information in the compressed domain is provided.
  • the architecture of the system 100 is broadly arranged into three functional modules, a parsing module 110 , a visualization module 120 , and an authoring and editing module 130 .
  • an incoming bitstream of compressed video information 111 which may be, for example, an MPEG-2 compressed bitstream is examined for scene cuts 112 and broken into shot segments, where each segment includes one or more fields or frames of compressed video information.
  • the shot segments will be made of three types of fields or frames, i.e., Intra-coded (“I”) fields which are coded independently and entirely without reference to other fields, Predictive-coded (“P”) fields which are coded with reference to temporally preceding I or P fields in the sequence, and Bi-directionally predictive-coded (“B”) fields which are coded with reference to the nearest preceding and/or future I or P fields in the sequence.
  • I Intra-coded
  • P Predictive-coded
  • B Bi-directionally predictive-coded
  • the bistream will also include associated motion vector information for the P and B fields which “point” to blocks of video information in a preceding or succeeding field or frame which are needed to reconstruct the compressed field or frame of video information.
  • the parsing module compiles a list of scene cuts which are useful for indexing the compressed video information.
  • the individual shot segments are next analyzed 113 in order to derive camera operation parameters.
  • the vector field 114 is representative of the pan of the camera which originally captured the video information which is being analyzed.
  • Histogram 115 is used to detect the pan of the camera. Based on the derived operating parameters, moving objects 117 within the compressed video information 116 are detected and shape and trajectory features for such moving objects are extracted.
  • the complied list of scene cuts and the derived camera zoom and pan information are used to extract key frames 121 which represent each video shot.
  • the key frames 121 are placed in a hierarchical arrangement 125 so that they may be readily browsed with a hierarchical video scene browser, such as the browser described in D. Zhong et al., “Clustering Methods for Video Browsing and Annotation,” Storage and Retrieval for Still Image and Video Databases IV, IS&T/SPIE's Electronic Images: Science & Tech. 96, Vol 2670 (1996).
  • a content-based image query system 126 may then be used to index and retrieve key frames or video objects based on their visual features and spatial layout.
  • the parsing module 110 is explained in further detail with reference to the FIGS. 2 and 3 .
  • the preferred steps executed e.g. on a computer 131 to locate scene cuts in a compressed bitstream are fully disclosed in the above referenced article by Meng et al., the disclosure of which is incorporated by reference herein, that technique is now described with reference to FIG. 2 .
  • An MPEG-1 or MPEG-2 compressed bitstream 201 that is received by the parsing module 110 is first subjected to parsing 210 .
  • Such a bitstream represents an arrangement of N ⁇ N blocks of video information that are broadly organized into macroblocks, where each macroblock is defined by four blocks of luminance information and one block of chrominance information, and further organized into slices which represent contigous sequences of macroblocks of video information in raster scan order.
  • the N ⁇ N blocks of video information are preferably 8 ⁇ 8 blocks.
  • the bitstream also represents associated motion vector information and prediction error difference information which are needed to reconstruct original blocks of video information.
  • the bitstream is parsed down to the fundamental block level by parsing techniques known to those skilled in the art 211 .
  • the parsed blocks of video information are still in compressed format and are thus represented by Direct Cosign Trasform (“DCT”) coefficients which have been quantized, Zig-Zag run-length encoded and variable length coded, as those skilled in the art will appreciate.
  • DCT Direct Cosign Trasform
  • the parsed blocks of video information which belong to P and B frames are then subjected to inverse motion compensation 220 by using the associated motion vector information to locate reference blocks of video information and reconstruct the DCT coefficients of blocks of video information in the B and P frames.
  • this step 220 only the first (the “DC”) DCT coefficients are used.
  • Motion vectors associated with the B and P frames are counted 222 for each frame in the sequence.
  • three ratios i.e., the number of intra-coded macroblocks to the number of forward motion vectors, the number of backward motion vectors to the number of forward motion vectors, and the number of forward motion vectors to the number of backward motion vectors, are calculated 231 in order to detecting direct scene cuts in P, B, and I frames, respectively.
  • the fact that most video shots of compressed MPEG video are formed by consecutive P, I and B frames that have a high degree of temporal correlation is taken advantage of.
  • this correlation is characterized by the ratio of the number of backward motion vectors, or intracoded macroblocks, to the number of forward motion vectors. For example, when a direct scene cut occurs on a P-frame, most macroblocks will be intracoded (i.e., no interframe prediction).
  • the variance of the DCT DC coefficients of Luminance in the I, P and B frames is determined 232 . As those skilled in the art will appreciate.
  • the ratios calculated in 231 are compared to local adaptive thresholds in order to detect the peak values 241 .
  • P frames the ratio of the number of intra-coded macroblocks to the number of forward motion vectors is examined.
  • B frames the ratio of the number of backward motion vectors to the number of forward motion vectors is examined.
  • the variance of DCT DC coefficients calculated for I and P frames in 232 is used in two ways.
  • this variance information is used together with the ratio of the number of forward motion vectors to the number of backward motion vectors determined in 231 in order to detect candidate scene changes 242 . If the ratio is above a predetermined threshold, the frame is marked as containing a suspected scene cut. If the variable information is much different than was the variance information for the immediately preceding I frame, a suspected scene cut is likewise detectable.
  • the variance information is also used directly to detect dissolve regions, i.e. regions where one scene is fading out and a second is fading in 243 by examining the parabolic curve of the variance information.
  • bitstream is broken into scenes or “shots” which represent different sets of visual information that may be of interest to a user.
  • the next block of our parsing module 110 camera operating estimation 113 , examines the individual shots for recognizable information which is highly useful in indexing the different shots which have been parsed.
  • Camera zoom and pan which give the global motion of the shot being analyzed, can be estimated with a 6-parameter affine transform model by using the actual motion vectors from the MPEG compressed stream.
  • the motion vectors in MPEG are usually generated by block matching: finding a block in the reference frame so that the mean square error of prediction is minimized. Although the motion vectors do not represent the true optical flow, it is still good in most cases to estimate the camera parameters in sequences that do not contain large dark or uniform regions.
  • a global parameter Given the motion vector for each macroblock, a global parameter can be determined using Least Squares (“LS”) estimation by finding a set of parameters to minimize the error between the motion vectors estimated in (1) and the actual motion vectors obtained from the MPEG stream:
  • LS Least Squares
  • each motion vector associated with the B and P frames contained in the shot are decoded.
  • the intermediate variables N, A, B, C, D, and E are interactively calculated using the x and y coordinates for each macroblock in the current frame of video being analyzed 320 .
  • intermediate transform parameters u 1 , u 2 , u 3 , v 1 , v 2 , v 3 are calculated using the decoded motion vectors and the x and y coordinates for such macroblocks 330 .
  • the vector is solved for by solving for the matrix inverse operation 340 .
  • GMC global motion compensation
  • the moving objects themselves can be detected by comparing the magnitude of the local motion to a predetermined threshold value and by performing simple morphological operations to delete small false objects and to fill noisy spots. For example, in FIG. 5( c ) there is shown an extracted moving object.
  • the DCT coefficients of the moving object are extracted from the compressed video information to provide for later querying of the object. Extraction of the DCT coefficient is done by the DCT domain motion compensation algorithm, as disclosed in U.S. Pat. No. 5,408,274 to Chang et al., the disclosure of which is incorporated by reference herein.
  • the outermost points of the extracted object are used to form a bounding box, as illustrated in FIG. 5( d ).
  • the location and size of each extracted bounding box is saved in a database 126 for later browsing and indexing by a user.
  • visual features of extracted objects such as color, textures, and shape, are used to provide content-based visual querying 127 of these and associated video scenes.
  • each globally motion compensated motion vector is compared to a predetermined threshold value 620 in order to eliminate non-moving parts of the frame from further consideration.
  • the number of contiguous blocks are counted 630 .
  • the number of associated blocks are compared to a predetermined minimum threshold value 640 in order to eliminate false small objects from being detected.
  • the border points for the remaining objects are identified and saved in a database, e.g. database 126 , for later use 650 , together with corresponding DCT coefficients for all blocks within the border. In this way, the important moving video object can be extracted and indexed for later viewing by a user.
  • editing of compressed video is directed to permitting a user to cut a first segment of video 710 from a first video sequence 720 and a second segment of video 730 from a second sequence of video 740 to form a new bitstream of video information 750 .
  • Such techniques have been described in the art, including in the article by the present inventors, J. Meng et al., “Tools for Compressed-Domain Vide Indexing and Editing,” SPIE Conf. on Storage and Retrieval for Image and Video Database, Vol. 2670 (1996), the disclosure of which is incorporated by reference herein.
  • DCT Discrete Cosine Transform
  • the DCT of the output video Y can be obtained by linear matrix operations of the input DCT, P i , as follows:
  • H i and W i are special filter coefficient matrices in the DCT domain.
  • the compressed-domain manipulation functions can be implemented in two ways. First, transform-domain techniques can be used to convert B and P frames to intraframe DCT coefficients, on which the above techniques can be readily applied. An alternative approach is to keep the B or P structure (i.e., the DCT coefficients of residual errors and motion vectors) and develop algorithms directly utilizing these data.
  • dissolve refers to the technique where an outgoing video scene 801 is faded out while an incoming video scene 802 is faded in.
  • the actual DCT coefficients for each block of compressed video in the last frame of the outgoing video scene, and the DCT coefficients for each block of compressed video in the first frame of the incoming video scend must be extracted.
  • One technique for extracting such DCT coefficients is described in the above-mentioned Chang et al. patent.
  • the Chang et al. patent describes a technique which uses DCT domain inverse motion compensation to obtain the DCT coefficients for all blocks of video information which make up the needed frames of video.
  • v and v are coordinates within a frame
  • t is the frame index value which may range from 1 to N
  • n being the total number of frames in the dissolved region
  • ⁇ (t) is a weighing function that is variable from 100% to 0%
  • F 1 is the composite of the derived DCT coefficients for all blocks which make up the last frame of the outgoing video scene
  • F 2 is the composite of the derived DCT coefficients for all blocks which make up the first frame of the incoming video scene.
  • the resulting effect is a dissolve transition from a particular frozen frame of the outgoing video scene to another frozen frame of the incoming video scene.
  • step 810 DCT domain motion compensation is used to obtain the DCT coefficients for all blocks of video information which make up the last frame of an outgoing video scene F 1 .
  • step 820 DCT domain motion compensation is used to obtain the DCT coefficients for all blocks of video information which make up the first frame of an incoming video scene F 2 .
  • step 830 the initial value for the weighing function, ⁇ (t), is chosen.
  • equation 9 is applied to create a first frame in the dissolve region. The value of t is then incremented until a final value n is obtained 850 .
  • a second important tool used in film editing is masking.
  • the film effect of masking video refers to transforming an original video scene having e.g. a 4:3 aspect ratio to different aspect ratios such as 1:1.66, 1:1.85, 1:2.35, or 16:9.
  • Masking can also be used to crop part of the frame region to a different frame size.
  • the DCT coefficients for blocks outside of the desired region are set to 0, and the coefficients for blocks that lie on the masking boundaries can recalculated using a simplified DCT cropping algorithm:
  • A is an original block located on the boundary
  • B is the new masked block
  • I h is the identity matrix with size h ⁇ h, as shown in FIG. 9 a.
  • step 910 the frame to be masked is examined in order to determine whether it is an I, P or B frame. If the frame is an I frame 911 , all DCT coefficients for all blocks within the frame are extracted 920 . Block n is examined 930 to determine where in the frame it is located. If the block is inside the mask region 931 , the DCT coefficients for the block are unchanged 941 . If the block is outside the make region 932 , the DCT coefficients for the block are set to zero 942 . If the block is on the boundary of the masked region 933 , the DCT cropping algorithm, i.e., equation 10, is applied 943 . The block index n is incremented 950 and the process repeated 930 - 950 until all blocks have been examined.
  • the DCT cropping algorithm i.e., equation 10
  • motion vectors associated with block n are examined 960 to determine whether they point to blocks outside or on the mask region 970 . If a motion vector points to blocks outside or on the mask region 972 , the macroblock is reencoded 980 . The block index n is incremented 990 and the process repeated 960 - 990 until all blocks have been examined.
  • freeze effect A third important tool used in film editing is the freeze effect. Since the freeze effect is usually longer than 1 second, simply repeating duplicate frames is not desirable if interactive playback (e.g. random search) is desired. Instead, the film effect of freeze frame requires the use of periodic I frames.
  • the frozen frame in order to ensure interactive playback capabilities, if the frozen frame is a P or B frame, it should be converted to an I frame.
  • every block of video information for the frame that has an associated motion vector is converted into a pure array of DCT coefficients, without motion vector information.
  • the above-mentioned DCT domain motion compensation algorithm described in the Chang et al. patent is advantageously used to effect such conversion.
  • the converted blocks can be referred to as intracoded blocks.
  • the group of pictures which represent the frozen frame are filled with duplicated P frames, with all macroblocks in the duplicated P frames being to Motion Compensation Not Coded (i.e., 0 motion vector, and 0 residue error).
  • Duplicate P frames can easily be created independently and inserted after I or P frames, as those familiar with digital compression techniques will easily understand.
  • step 1010 a user-defined frame of compressed video information is selected for the freeze-frame effect.
  • step 1020 the frame is examined in order to determine whether it is an I, P or B frame. If the frame is not an I frame 1021 , it is converted into an I frame 1030 . Finally, original 1022 or converted 1030 I frames are then used as to create duplicate P frames 1040 . Periodical I frames can be inserted to increase interactivity, and to maintain a compatible bitrate.
  • variable speed playback A fourth important tool used in film editing is variable speed playback. Faster than normal playback, e.g., fast forward, is simply realized by dropping B, P, and then I frames according to the variable speed desired.
  • one approach is to simply insert duplicate B frames (B′) whenever an original B frame appears.
  • Duplicate B frames are frames which copy all DCT coefficient, and associated motion vectors from previous B frame.
  • I/P frame delay is increased by a factor equal to the inverse of the motion rate.
  • the reference frame I 0 must be transmitted 4 frames earlier. This limitation makes this approach suitable for slow playback up to about 1 ⁇ 2 normal frame rate. This will require an increased decoder buffer size.
  • Strobe motion is effected by dropping original B and/or P frames and inserting duplicated P frames.
  • a master server 1210 or a group of distributed servers is linked to distant servers 1220 , 1230 , 1240 and several clients 1250 , 1260 . From their client workstations, users are empowered to browse and edit compressed vide images in the manner described above.
  • the master server 1210 acts as the content aggregator 1310 to collect video content, and as a content analyzer 1320 to analyze the visual content for efficient indexing 1330 .
  • the server 1210 may accept the functions performed by the parsing module 110 described above.
  • the server 1210 also provides an editing engine 1340 with basic editing tools, such as the tools described above, for rendering basic editing functions and various special effects.
  • Distributed clients 1250 , 1260 access the video archives of the server through heterogeneous networks with interoperable interfaces, such as JAVA or MPEG MSDL manipulation tools.
  • the video server 1210 can be linked with other distributed servers 1220 , 1230 , 1240 that have video search engines which search for image and video over a network, e.g., the World Wide Web.
  • a video file Once a video file is found on any other hosts or online content providers, it will be downloaded and preprocessed by the video server 1210 to extract “keyframes,” i.e., frames 121 stored in the above described visualization module 120 , and associated visual features such as camera motion, moving objects, color, texture, and temporal features.
  • the Universal Resource Location (“URL”) of the video and the extracted features will be stored on the video server 1210 .
  • This client-server model empowers clients 1250 , 1260 with much richer resources beyond that provided by the client's local system.
  • the client 1260 may open any video shot stored at the server 1210 and browse 1330 the keyframes hierarchically using any one of numerous different viewing models, including sequential view, feature-based view, and story-based view.
  • a sequential view arranges the video shots according to the sequential time order;
  • a feature-based view clusters video shots with similar visual features to the same group;
  • a story view presents video shots according to the story structures in the semantic level.
  • the story model may use the anchorperson shot as the criterion for automatic segmentation of video into semantic-level stories.
  • the client can also perform shot-level editing or frame level editing 1390 using the above discussed editing tools.
  • the clients may also submit their own videos which may be analyzed and indexed by the server 1210 .
  • the client 1260 may choose different levels of resolution for the video shots sent by the server 1210 .
  • the client 1260 can render only straight cuts at low resolution without any special effects.
  • the client may send information to the server defining low-resolution video with desired special effects to be generated.
  • the master server 1210 will generate a full-resolution video with all the effects requested by the client from the highest quality source video which is located at either the master server 1210 or the distributed remote content servers 1220 , 1230 , 1240 . In this way, the user is given the flexibility to tradeoff between quality and speed.
  • a low resolution icon stream video In order to obtain a reduced resolution video sequence, for each video sequence stored on a server, a low resolution icon stream video must be extracted from the full resolution video.
  • the icon stream is an 8:1 down sampled version of the original video.
  • DCT DC coefficients are obtained directly.
  • B and P frames DCT domain inverse motion compensation is applied to the DCT coefficients.
  • the extracted coefficients are converted into a smaller sized, e.g., 8:2:1, I frame as those skilled in the art will appreciate.

Abstract

A system and method is provided for editing and parsing compressed digital information. The compressed digital information may include visual information which is edited and parsed in the compressed domain. In a preferred embodiment, the present invention provides a method for detecting moving objects in a compressed digital bitstream which represents a sequence of fields or frames of video information for one or more captured scenes of video.

Description

    NOTICE OF GOVERNMENT RIGHTS
  • The U.S. Government has certain rights in this invention pursuant to the terms of the National Science Foundation CAREER award IRI-9501266.
  • BACKGROUND OF THE INVENTION
  • I. Field of the Invention
  • The present invention relates to techniques for editing and parsing compressed digital information, and more specifically, to editing and parsing visual information in the compressed domain.
  • II. Description of the Related Art
  • With the increasing use of local area, wide area and global networks to spread information, digital video has become an essential component of many new media applications. The inclusion of video in an application often gives the application not only increased functional utility, but also an aesthetic appeal that cannot be obtained by text or audio information alone. However, while digital video greatly increases our ability to share information, it demands special technical support in processing, communication, and storage.
  • In order to reduce bandwidth requirements to manageable levels, video information is generally transmitted between systems in the digital environment the form of compressed bitstreams that are in a standard format, e.g., Motion JPEG, MPEG-1, MPEG-2, H.261 or H.263. In these compressed formats, the Discrete Cosine Transform (“DCT”) is utilized in order to transform N×N blocks of pixel data, where n typically is set to eight, into the DCT domain where quantization is more readily performed. Run-length encoding and entropy coding (i.e., Huffman coding or arithmetic coding) are applied to the quantized bitstream to produce a compressed bitstream which has a significantly reduced bit rate than the original uncompressed source signal. The process is assisted by additional side information, in the form of motion vectors, which are used to construct frame or field-based predictions from neighboring frames or fields by taking into account the inter-frame or inter-field motion that is typically present.
  • In order to be usable by a receiving system, such coded bitstreams must be both parsed and decoded. For example, in the case of an MPEG-2 encoded bitstream, the bitstream must be parsed into slices and macroblocks before the information contained in the bitstream is usable by an MPEG-2 decoder. Parsed bitstream information may be used directly by an MPEG-2 decoder to reconstruct the original visual information, or may be subjected to further processing.
  • In the case of compressed digital video, further processing of video information can occur either in the normal, uncompressed domain or in the compressed domain. Indeed, there have been numerous attempts by others in the field to realize useful techniques for indexing and manipulating digital video information in both the uncompressed and compressed domains.
  • For example, in the article by S. W. Smoliar et al., “Content-Based Video Indexing and Retrieval,” IEEE Multimedia, summer 1994, pp. 62-72, a color histogram comparison technique is proposed to detect scene cuts in the spatial (uncompressed) domain. In the article by B. Shahraray, “Scene Change Detection and Content-Based Sampling of Video Sequences,” SPIE Conf. Digital Image Compression: Algorithms and Technologies 1995, Vol. 2419, a block-based match and motion estimation algorithm is presented.
  • For compressed video information, the article by F. Arman et al., “Image Processing on Compressed Data for Large Video Databases,” Proceedings of ACM Multimedia '93, June 1993, pp. 267-272, proposes a technique for detecting scene cuts in JPEG compressed images by comparing the DCT coefficients of selected blocks from each frame. Likewise, the article by J. Meng et al., “Scene Change Detection in a MPEG Compressed Video Sequence,” IS&T/SPIE Symposium Proceedings, Vol. 2419, February 1995, San Jose, Calif., provides a methodology for the detection of direct scene cuts based on the distribution of motion vectors, and a technique for the location of transitional scene cuts based on DCT DC coefficients. Algorithms disclosed in the article by M. M. Yeung, et al. “Video Browsing using Clustering and Scene Transitions on Compressed Sequences,” IS&T/SPIE Symposium Proceedings, February 1995, San Jose, Calif. Vol. 2417, pp. 399-413, enable the browsing of video shots after scene cuts are located. However, the Smoliar et al., Shahraray, and Arman et al. references are limited to scene change detection, and the Meng et al. and Yeung et al. references do not provide any functions for editing compressed video.
  • Others in the field have attempted to address problems associated with camera operation and moving objects in a video sequence. For example, in the spatial domain, H. S. Sawhney, et al., “Model-Based 2D & 3D Dominant Motion Estimation for Mosaicking and Video Representation,” Proc. Fifth Int'l conf. Computer Vision, Los Alamitos, Calif., 1995, pp. 583-390, proposes to find parameters of an affine matrix and to construct a mosaic image from a sequence of video images. In similar vain, the work by A. Nagasaka et al., “Automatic Video Indexing and Full-Video Search for Object Appearances,” in E. Knuth and L. M. Wegner, editors, Video Database Systems, II, Elsevier Science Publishers B. V., North-Holland, 1992, pp. 113-127, proposes searching for object appearances and using them in a video indexing technique.
  • In the compressed domain, the detection of certain camera operations, e.g., zoom and pan, based on motion vectors have been proposed in both A. Akutsu et al., “Video Indexing Using Motion Vectors,” SPIE Visual Communications and Image Processing 1992, Vol. 1818, pp. 1522-1530, and Y. T. Tse et al., “Global Zoom/Pan Estimation and Compensation For Video Compression” Proceedings of ICASSP 1991, pp. 2725-2728. In these proposed techniques, simple three parameter models are employed which require two assumptions, i.e., that camera panning is slow and focal length is long. However, such restrictions make the algorithms not suitable for general video processing.
  • There have also been attempts to develop techniques aimed specifically at digital video indexing. For example, in the aforementioned Smoliar et al. article, the authors propose using finite state models in order to parse and retrieve specific domain video, such as news video. Likewise, in A. Hampapur, et al., “Feature Based Digital Video Indexing,” IFIP2.6 Visual Database Systems, III, Switzerland, March, 95, a feature based video indexing scheme using low level machine derivable indices to map into the set of application specific video indices is presented.
  • One attempt to enable users to manilupate image and video information was proposed by J. Swartz, et al., “A Resolution Independent Video Language,” Proceedings of ACM Multimedia '95, pp. 179-188, as a resolution independent video language (Rivl). However, although Rivl uses group of pictures (GOPs) level direct copying whenever possible for “cut and paste” operations on MPEG video, it does not use operations in the compressed domain at frame and macroblock levels for special effects editing. Instead, most video effects in Rivl are done by decoding each frame into the pixel domain and then applying image library routines.
  • The techniques proposed by Swartz et al. and others which rely on performing some or all video data manipulation functions in the uncompressed domain do not provide a useful, truly comprehensive technique for indexing and manipulating digital video. As explained in S.-F. Chang, “Compressed-Domain Techniques for Image/Video Indexing and Manipulation,” IEEE Intern. Conf. on Image Processing, ICIP 95, Special Session on Digital Image/Video Libraries and Video-on-demand, October 1995, Washington D.C., the disclosure of which is incorporated by reference herein, the compressed-domain approach offers several powerful benefits.
  • First, implementation of the same manipulation algorithms in the compressed domain is much cheaper than that in the uncompressed domain because the data rate is highly reduced in the compressed domain (e.g., a typical 20:1 to 50:1 compression ratio for MPEG). Second, given most existing images and videos stored in the compressed form, specific manipulation algorithms can be applied to the compressed streams without full decoding of the compressed images/videos. In addition, because that full decoding and re-encoding of video are not necessary, manipulating video in the compressed domain avoids the extra quality degradation inherent in the reencoding process. Thus, as further explained in the article by the present inventors, J. Meng and S.-F. Chang, “Tools for Compressed-Domain Video Indexing and Editing,” SPIE Conference on Storage and Retrieval for Image and Video Database, Vol. 2670, San Jose, Calif., February 1996, the disclosure of which is incorporated by reference herein, for MPEG compressed video editing, speed performance can be improved by more than 60 times and the video quality can be improved by about 3-4 dB if a compressed-domain approach is used rather than a traditional decode-edit-reencode approach.
  • A truly comprehensive technique for indexing and manipulating digital video must meet two requirements. First, the technique must provide for key content browsing and searching, in order to permit users to efficiently browse through or search for key content of the video without full decoding and viewing the entire video stream. In this connection, “key content” refers to key frames in video sequences, prominent video objects and their associated visual features (motion, shape, color, and trajectory), or special reconstructed video models for representing video content in a video scene. Second, the technique must allow for video editing directly in the compressed domain to allow users to manipulate an specific object of interest in the video stream without having to fully decode the video. For example, the technique should permit a user to cut and paste any arbitrary segment from an existing video stream to produce a new video stream which conforms to the valid compression format.
  • Unfortunately, none of the prior art techniques available at present are able to meet these requirements. Thus, the prior art techniques fail to permit users who want to manipulate compressed digital video information with the necessary tools to extract a rich set of visual features associated with visual scenes and individual objects directly from compressed video so as not only to enable content based query searches, but also to allow for integration with domain knowledge for derivation of higher-level semantics.
  • SUMMARY OF THE INVENTION
  • An object of the present invention is to provide comprehensive techniques for indexing and manipulating digital video in the compressed domain.
  • Another object of the present invention is to provide techniques for key content browsing and searching of compressed digital video without decoding and viewing the entire video stream.
  • A further object of the present invention is to provide techniques which allow for video editing directly in the compressed domain
  • A still further object of the present invention is to provide tools that permit users who want to manipulate compressed digital video information to extract a rich set of visual features associated with visual scenes and individual objects directly from compressed video.
  • Yet another object of this invention is to provide an architecture which permits users to manipulate compressed video information over a distributed network, such as the Internet.
  • In order to meet these and other objects which will become apparent with reference to further disclosure set forth below, the present invention provides a method for detecting moving video objects in a compressed digital bitstream which represents a sequence of fields or frames of video information for one or more previously captured scenes of video. The described method advantageously provides for analyzing a compressed bistream to locate scene cuts so that at least one sequence of fields or frames of video information which represents a single video scene is determined. The method also provides for estimating one or more operating parameters of a camera which initially captured the video scene, by analyzing a portion of the compressed bitstream which corresponds to the video scene, and for detecting one or more moving video objects represented in the compressed bitstream by applying global motion compensation with the estimated camera operating parameters.
  • In a preferred process, the compressed bitstream is a bitstream compressed in accordance with the MPEG-1, MPEG-2, H261, or H263 video standard. In this preferred embodiment, analyzing can beneficially be accomplished by parsing the compressed bitstream into blocks of video information and associated motion vector information for each field or frame of video information which comprises the determined sequence of fields or frames of video information representative of said single scene, performing inverse motion compensation on each of the parsed blocks of video information to derive discrete cosign transform coefficients for each of the parsed blocks of video information, counting the motion vector information associated with each of the parsed blocks of video information, and determining from the counted motion vector information and discrete cosign transform coefficient information whether one of the scene cuts has occurred.
  • In an alternative embodiment, analyzing is performed by parsing the compressed bitstream into blocks of video information and associated motion vector information for each field or frame of video information which comprises the determined sequence of fields or frames of video information representative of the scene, and estimating is executed by approximating any zoom and any pan of the camera by determining a multi-parameter transform model applied to the parsed motion vector information. In an especially preferred process, the frame difference due to camera pan and zoom motion may be modeled by a six-parameter amine transform which represents the global motion information representative of the zoom and pan of the camera.
  • The detecting step advantageously provides for computing local object motion for one or more moving video objects based on the global motion information and on one or more motion vectors which correspond to the one or more moving video objects. In addition, thresholding and morphological operations are preferably applied to the determined local object motion values to eliminate any erroneously sensed moving objects. Further, border points of the detected moving objects are determined to generate a bounding box for the detected moving object.
  • The present invention also provides for an apparatus for detecting moving video objects in a compressed digital bitstream which represents a sequence of fields or frames of video information for one or more previously captured scenes of video. Usefully, the apparatus includes means for analyzing the compressed bistream to locate scene cuts therein and to determine at least one sequence of fields or frames of video information which represents a single video scene, means for estimating one or more operating parameters for a camera which initially viewed the video scene by analyzing a portion of the compressed bitstream which corresponds to the video scene, and means for detecting one or more moving video objects represented in the compressed bitstream by applying global motion compensation to the estimated operating parameters.
  • A different aspect of the present invention provides techniques for dissolving an incoming scene of video information which comprises a sequence of fields or frame of compressed video information to an outgoing scene of video information which comprises a sequence of fields or frame of compressed video information. This technique advantageously provides for applying DCT domain motion compensation to obtain DCT coefficients for all blocks of video information which make up a last frame of the outgoing video scene and the first frame of the incoming video scene, and for creating a frame in the dissolve region frame from the DCT coefficients of the last outgoing frame and the first incoming frame.
  • In an especially preferred arrangement, an initial value for a weighting function is selected prior to the creation of a first frame in the dissolve region and is used in the creation of the first frame in the dissolve region. The weighting value is then incremented, and a second dissolve frame from the DCT coefficients is generated.
  • In yet another aspect of the present invention, a technique for masking a compressed frame of digital video information is provided. The technique first determines whether the frame to be masked is intra-coded, predictive-coded or bi-directionally predictive-coded. If the frame is intra-coded, the technique provides for extracting DCT coefficients for all blocks within the frame, examining a block, to determine where in the frame the block is located, setting DCT coefficients for the block to zero if the block is outside the mask region, and applying a DCT cropping algorithm to the DCT coefficients if the block is on the boundary of the mask region.
  • If the frame is predictive-coded or bi-directionally predictive-coded, the technique provides for examining motion vectors associated with block, to determine whether they point to blocks outside or on the mask region, and reencoding the block if a motion vector points to blocks outside the boundary, or on, the mask region.
  • In still another aspect of the present invention, a technique for generating a frozen frame of video information from a sequence of frames of compressed video information is provided. The technique attractively provides for selecting a frame of compressed video information to be frozen, determining whether the frame to be frozen is intra-coded, predictive-coded or bi-directionally predictive-coded, and if the frame is not intra-coded, converting it to become intra-coded, creating duplicate predictive-coded frames, and arranging the intra-coded frame and the duplicate predictive-coded frames into a sequence of compressed frames of video information.
  • In yet a further aspect of the present invention, a system for editing compressed video information over a distributed network is provided. The system includes a client computer, a network link for permitting said client computer to search for and locate compressed video information on said distributed network, and tools for editing a compressed bitstream of video information over the distributed network.
  • The accompanying drawings, which are incorporated and constitute part of this disclosure, illustrate a preferred embodiment of the invention and serve to explain the principles of the invention.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 illustrates a system in accordance with one aspect of the present invention;
  • FIG. 2. is a flowchart which illustrates how a scene cut is detected in accordance with one aspect of the present invention;
  • FIG. 3. is a flowchart which illustrates how camera parameters are estimated in accordance with one aspect of the present invention;
  • FIG. 4 is a vector diagram which serves to explain global and local motion;
  • FIG. 5 depicts an exemplary frame of compressed video information and motion vectors for the frame;
  • FIG. 6 is a flowchart which illustrates global motion compensation in accordance with one aspect of the present invention;
  • FIG. 7 depicts prior art editing of compressed video bistreams;
  • FIG. 8 a depicts the dissolve effect; FIG. 8( b) is a flowchart which illustrates the dissolve effect;
  • FIG. 9 a depicts masking; FIG. 9( b) is a flowchart which illustrates masking;
  • FIG. 10 is a flowchart which illustrates the freezeframe effect;
  • FIG. 11 depicts two alternative techniques for the slow motion effect;
  • FIG. 12 is a system diagram of a distributed network in accordance with one aspect of the present invention; and
  • FIG. 13 depicts exemplarily techniques which may be executed in the distributed network illustrated in FIG. 12.
  • DESCRIPTION OF THE PREFERRED EMBODIMENTS
  • Referring to FIG. 1, an exemplary embodiment of our invention which permits a user to edit and parse visual information in the compressed domain is provided. The architecture of the system 100 is broadly arranged into three functional modules, a parsing module 110, a visualization module 120, and an authoring and editing module 130.
  • In the parsing module, an incoming bitstream of compressed video information 111 which may be, for example, an MPEG-2 compressed bitstream is examined for scene cuts 112 and broken into shot segments, where each segment includes one or more fields or frames of compressed video information. In an MPEG-2 bitstream, the shot segments will be made of three types of fields or frames, i.e., Intra-coded (“I”) fields which are coded independently and entirely without reference to other fields, Predictive-coded (“P”) fields which are coded with reference to temporally preceding I or P fields in the sequence, and Bi-directionally predictive-coded (“B”) fields which are coded with reference to the nearest preceding and/or future I or P fields in the sequence. The bistream will also include associated motion vector information for the P and B fields which “point” to blocks of video information in a preceding or succeeding field or frame which are needed to reconstruct the compressed field or frame of video information.
  • The parsing module compiles a list of scene cuts which are useful for indexing the compressed video information. The individual shot segments are next analyzed 113 in order to derive camera operation parameters. In FIG. 1, the vector field 114 is representative of the pan of the camera which originally captured the video information which is being analyzed. Histogram 115 is used to detect the pan of the camera. Based on the derived operating parameters, moving objects 117 within the compressed video information 116 are detected and shape and trajectory features for such moving objects are extracted.
  • In the visualization module 120, the complied list of scene cuts and the derived camera zoom and pan information are used to extract key frames 121 which represent each video shot. The key frames 121 are placed in a hierarchical arrangement 125 so that they may be readily browsed with a hierarchical video scene browser, such as the browser described in D. Zhong et al., “Clustering Methods for Video Browsing and Annotation,” Storage and Retrieval for Still Image and Video Databases IV, IS&T/SPIE's Electronic Images: Science & Tech. 96, Vol 2670 (1996). A content-based image query system 126 may then be used to index and retrieve key frames or video objects based on their visual features and spatial layout.
  • In the authoring and editing module 130, software tools are provided to not only enable a user to cut and past arbitrary compressed video segments to form new video segments, but also to add special effects to such video segments, such as dissolve, key, masking and motion effects.
  • The parsing module 110 is explained in further detail with reference to the FIGS. 2 and 3. Although the preferred steps executed e.g. on a computer 131 to locate scene cuts in a compressed bitstream are fully disclosed in the above referenced article by Meng et al., the disclosure of which is incorporated by reference herein, that technique is now described with reference to FIG. 2.
  • An MPEG-1 or MPEG-2 compressed bitstream 201 that is received by the parsing module 110 is first subjected to parsing 210. Such a bitstream represents an arrangement of N×N blocks of video information that are broadly organized into macroblocks, where each macroblock is defined by four blocks of luminance information and one block of chrominance information, and further organized into slices which represent contigous sequences of macroblocks of video information in raster scan order. The N×N blocks of video information are preferably 8×8 blocks. The bitstream also represents associated motion vector information and prediction error difference information which are needed to reconstruct original blocks of video information.
  • In the parsing stage 210, the bitstream, is parsed down to the fundamental block level by parsing techniques known to those skilled in the art 211. The parsed blocks of video information are still in compressed format and are thus represented by Direct Cosign Trasform (“DCT”) coefficients which have been quantized, Zig-Zag run-length encoded and variable length coded, as those skilled in the art will appreciate.
  • In the inverse motion compensation stage 220, the parsed blocks of video information which belong to P and B frames are then subjected to inverse motion compensation 220 by using the associated motion vector information to locate reference blocks of video information and reconstruct the DCT coefficients of blocks of video information in the B and P frames. In this step 220, only the first (the “DC”) DCT coefficients are used. Motion vectors associated with the B and P frames are counted 222 for each frame in the sequence.
  • In the Statistical Stage 230, three ratios, i.e., the number of intra-coded macroblocks to the number of forward motion vectors, the number of backward motion vectors to the number of forward motion vectors, and the number of forward motion vectors to the number of backward motion vectors, are calculated 231 in order to detecting direct scene cuts in P, B, and I frames, respectively. In this stage, the fact that most video shots of compressed MPEG video are formed by consecutive P, I and B frames that have a high degree of temporal correlation is taken advantage of. For P and B frames, this correlation is characterized by the ratio of the number of backward motion vectors, or intracoded macroblocks, to the number of forward motion vectors. For example, when a direct scene cut occurs on a P-frame, most macroblocks will be intracoded (i.e., no interframe prediction).
  • Also in the statistical stage 230 the variance of the DCT DC coefficients of Luminance in the I, P and B frames is determined 232. As those skilled in the art will appreciate.
  • Next, in the detection stage 240, the ratios calculated in 231 are compared to local adaptive thresholds in order to detect the peak values 241. For P frames, the ratio of the number of intra-coded macroblocks to the number of forward motion vectors is examined. For B frames, the ratio of the number of backward motion vectors to the number of forward motion vectors is examined.
  • Also in the detection stage 240, the variance of DCT DC coefficients calculated for I and P frames in 232 is used in two ways. For I frames, this variance information is used together with the ratio of the number of forward motion vectors to the number of backward motion vectors determined in 231 in order to detect candidate scene changes 242. If the ratio is above a predetermined threshold, the frame is marked as containing a suspected scene cut. If the variable information is much different than was the variance information for the immediately preceding I frame, a suspected scene cut is likewise detectable. The variance information is also used directly to detect dissolve regions, i.e. regions where one scene is fading out and a second is fading in 243 by examining the parabolic curve of the variance information.
  • In the decision stage 250, duplicate detections of scene cuts are eliminated 251 before list of scenes 252 is determined. In addition, if a suspected scene change occurs within a time threshold T rejection of a previously detected scene change, no scene change is recorded.
  • With scene cuts detected, the bitstream is broken into scenes or “shots” which represent different sets of visual information that may be of interest to a user. The next block of our parsing module 110, camera operating estimation 113, examines the individual shots for recognizable information which is highly useful in indexing the different shots which have been parsed.
  • In particular, certain low level visual features such as camera zoom and pan, and the presence of moving visual objects are useful information for video indexing. Camera zoom and pan, which give the global motion of the shot being analyzed, can be estimated with a 6-parameter affine transform model by using the actual motion vectors from the MPEG compressed stream.
  • The motion vectors in MPEG are usually generated by block matching: finding a block in the reference frame so that the mean square error of prediction is minimized. Although the motion vectors do not represent the true optical flow, it is still good in most cases to estimate the camera parameters in sequences that do not contain large dark or uniform regions.
  • When the distance between the object or background and the camera is large, a 6 parameter affine transform can be used to describe the global motion of the current frame:
  • [ u v ] = [ 1 x y 0 0 0 0 0 0 1 x y ] · [ a 1 a 2 a 3 a 4 a 5 a 6 ] T ( 1 )
  • where (x, y) is the coordinate of a macroblock in the current frame, [u v]T is the motion vector associated with that macroblock, and [a1 a2 a3 a4 a5 a6]T is the affine transform vector. In order to simplify the mathematics, the following variables can be defined:
  • U for [u, v]T
  • X for
  • [ 1 x y 0 0 0 0 0 0 1 x y ] ,
  • and
    Figure US20110064136A1-20110317-P00001
    for [a1 a2 a3 a4 a5 a6]T.
  • Given the motion vector for each macroblock, a global parameter can be determined using Least Squares (“LS”) estimation by finding a set of parameters to
    Figure US20110064136A1-20110317-P00001
    minimize the error between the motion vectors estimated in (1) and the actual motion vectors obtained from the MPEG stream:
  • S ( a ) = X Y [ ( u ^ xy - u xy ) 2 + ( v ^ xy - v xy ) 2 ] ( 2 )
  • In equation (2), [û, {circumflex over (v)}]T is the estimated motion vector.
    Figure US20110064136A1-20110317-P00001
    is then solved for by setting the first derivative of S(
    Figure US20110064136A1-20110317-P00001
    ) to 0 to get:
  • [ N A B A C E B E D ] · [ a 1 a 2 a 3 ] = [ U 1 U 2 U 3 ] and [ N A B A C E B E D ] · [ a 4 a 5 a 6 ] = [ V 1 V 2 V 3 ] where , N = x y 1 , A = x y x , B = x y y , C = x y x 2 , D = x y y 2 , E = x y x · y , U 1 = x y u xy , U 2 = x y u xy · x , U 3 = x y u xy · y , V 1 = x y v xy , V 2 = x y v xy · x , V 3 = x y v xy · y . ( 3 )
  • All summations are computed over all valid macroblocks whose motion vectors survive after the nonlinear noise reduction process. (For example, median filter.) This process is used to eliminate obvious noise in the motion vector field. After the first LS estimation, motion vectors that have large distance from the estimated ones are filtered out before a second LS estimation. The estimation process is preferably iterated several times to refine the accuracy.
  • Referring to FIG. 3, the foregoing process to determine a global motion parameter is illustrated in flow diagram 300. In step 310, each motion vector associated with the B and P frames contained in the shot are decoded. The intermediate variables N, A, B, C, D, and E are interactively calculated using the x and y coordinates for each macroblock in the current frame of video being analyzed 320. Next, intermediate transform parameters u1, u2, u3, v1, v2, v3 are calculated using the decoded motion vectors and the x and y coordinates for such macroblocks 330. Finally, the vector
    Figure US20110064136A1-20110317-P00001
    is solved for by solving for the matrix inverse operation 340.
  • After the global camera parameters
    Figure US20110064136A1-20110317-P00001
    are found, object motion can be extracted by applying global motion compensation. Referring to FIG. 4, if an object located at (x, y) in the current frame moves from (x0, y0) to (x1, y1) in the reference frame with motion vector U, then U+M=X·
    Figure US20110064136A1-20110317-P00001
    . Thus, the motion of the local object can be recovered from its associated motion vectors as follows:

  • M=X·
    Figure US20110064136A1-20110317-P00001
    −U  (4)
  • This is referred to as the global motion compensation (“GMC”). With reference to FIG. 5, a moving ballerina is shown against a largely stagnant background. In FIG. 5( a), the original motion vectors for an exemplary frame of compressed video is illustrated. In FIG. 5( b), the vector map for the exemplary frame is shown after GMC has been applied. As illustrated in FIG. 5( b), for motion vectors of the background, GMC gives mostly zero values, while for motion vectors of the foreground moving objects, GMC reveals the local motion of objects.
  • The moving objects themselves can be detected by comparing the magnitude of the local motion to a predetermined threshold value and by performing simple morphological operations to delete small false objects and to fill noisy spots. For example, in FIG. 5( c) there is shown an extracted moving object.
  • The DCT coefficients of the moving object are extracted from the compressed video information to provide for later querying of the object. Extraction of the DCT coefficient is done by the DCT domain motion compensation algorithm, as disclosed in U.S. Pat. No. 5,408,274 to Chang et al., the disclosure of which is incorporated by reference herein. The outermost points of the extracted object are used to form a bounding box, as illustrated in FIG. 5( d).
  • Again referring to FIG. 1, in the visualization module 120, the location and size of each extracted bounding box is saved in a database 126 for later browsing and indexing by a user. Likewise, visual features of extracted objects, such as color, textures, and shape, are used to provide content-based visual querying 127 of these and associated video scenes.
  • The extraction of moving video objects from a compressed bitstream will now be described with reference to the flow diagram 600 of FIG. 6. First, GMC is applied to the present frame of video 610. Second, each globally motion compensated motion vector is compared to a predetermined threshold value 620 in order to eliminate non-moving parts of the frame from further consideration. Next, for blocks of information which still have non-zero motion vectors, the number of contiguous blocks are counted 630. For each contiguous area, the number of associated blocks are compared to a predetermined minimum threshold value 640 in order to eliminate false small objects from being detected. Finally, the border points for the remaining objects are identified and saved in a database, e.g. database 126, for later use 650, together with corresponding DCT coefficients for all blocks within the border. In this way, the important moving video object can be extracted and indexed for later viewing by a user.
  • With reference to FIG. 7, useful techniques for directly editing compressed digital video will now be described. In general, editing of compressed video is directed to permitting a user to cut a first segment of video 710 from a first video sequence 720 and a second segment of video 730 from a second sequence of video 740 to form a new bitstream of video information 750. Such techniques have been described in the art, including in the article by the present inventors, J. Meng et al., “Tools for Compressed-Domain Vide Indexing and Editing,” SPIE Conf. on Storage and Retrieval for Image and Video Database, Vol. 2670 (1996), the disclosure of which is incorporated by reference herein.
  • In addition to the basic editing function “cut and paste”, several more advanced visual effects can be created in the compressed domain. For I frames, the basic compression component is the Discrete Cosine Transform (DCT), which can be written in the following form:

  • F(u,v)=DCT(f(x,y))  (5)
  • Given the DCT, linear operations such as intensity addition and scaling can performed in accordance with equations (6) and (7):

  • DCT(f 1(x,y)+f 2(x,y))=F 1(u,v)+F 2(u,v)  (6)

  • DCT(α·f(x,y))=α·F(u,v)  (7)
  • Algorithms for other operations such as spatial scaling, translation, and filtering in DCT domain are well known in the art. Generally, the DCT of the output video Y can be obtained by linear matrix operations of the input DCT, Pi, as follows:
  • Y = i W i · P i · H i ( 8 )
  • where Hi and Wi are special filter coefficient matrices in the DCT domain. For motion compensated B and P frames, the compressed-domain manipulation functions can be implemented in two ways. First, transform-domain techniques can be used to convert B and P frames to intraframe DCT coefficients, on which the above techniques can be readily applied. An alternative approach is to keep the B or P structure (i.e., the DCT coefficients of residual errors and motion vectors) and develop algorithms directly utilizing these data. Several advanced visual effects which can be created in the compressed domain—dissolve, masking, freeze frame, variable speed, and strobe motion—are now particularly described.
  • One of the most important tools used in film editing is dissolve. As illustrated in FIG. 8( a), dissolve refers to the technique where an outgoing video scene 801 is faded out while an incoming video scene 802 is faded in. In order to perform a dissolve on two different scenes of video, the actual DCT coefficients for each block of compressed video in the last frame of the outgoing video scene, and the DCT coefficients for each block of compressed video in the first frame of the incoming video scend, must be extracted. One technique for extracting such DCT coefficients is described in the above-mentioned Chang et al. patent. The Chang et al. patent describes a technique which uses DCT domain inverse motion compensation to obtain the DCT coefficients for all blocks of video information which make up the needed frames of video.
  • When there is no or little motion in the two videos scenes, the dissolve effects can be approximated by the linear combination of the two video scenes F1 and F2:

  • F(u,v,t)=α(tF 1(u,v,t 1)+(1−α(t))·F 2(u,v,t 2)  (9)
  • v and v are coordinates within a frame, t is the frame index value which may range from 1 to N, n being the total number of frames in the dissolved region, and α(t) is a weighing function that is variable from 100% to 0%, F1 is the composite of the derived DCT coefficients for all blocks which make up the last frame of the outgoing video scene and F2 is the composite of the derived DCT coefficients for all blocks which make up the first frame of the incoming video scene. The resulting effect is a dissolve transition from a particular frozen frame of the outgoing video scene to another frozen frame of the incoming video scene.
  • Thus, to smooth out the dissolve where one or both of the dissolving scenes contain moving video objects, it is desirable to re-encode several dissolving frames over a transitional period t1-t2. The reader is referred to the Appendix of this patent document for our preferred source code for implementing a dissolve.
  • Referring to FIG. 8( b), a method for dissolving an incoming scene and an outgoing scene of video is now described by way of a flowchart 800. In step 810, DCT domain motion compensation is used to obtain the DCT coefficients for all blocks of video information which make up the last frame of an outgoing video scene F1. In step 820, DCT domain motion compensation is used to obtain the DCT coefficients for all blocks of video information which make up the first frame of an incoming video scene F2. In step 830, the initial value for the weighing function, α(t), is chosen. In step 840, equation 9 is applied to create a first frame in the dissolve region. The value of t is then incremented until a final value n is obtained 850. The process 830-850 is repeated to create all dissolve frames for the duration t=1 to n.
  • A second important tool used in film editing is masking. As illustrated in FIG. 9( a), the film effect of masking video refers to transforming an original video scene having e.g. a 4:3 aspect ratio to different aspect ratios such as 1:1.66, 1:1.85, 1:2.35, or 16:9. Masking can also be used to crop part of the frame region to a different frame size. For I frames, the DCT coefficients for blocks outside of the desired region are set to 0, and the coefficients for blocks that lie on the masking boundaries can recalculated using a simplified DCT cropping algorithm:
  • DCT ( B ) = DCT ( H ) · DCT ( A ) , where H = [ 0 0 0 I h ] ( 10 )
  • where A is an original block located on the boundary, B is the new masked block, and Ih is the identity matrix with size h×h, as shown in FIG. 9 a.
  • For P and B frames, only macroblocks with motion vectors pointing outside of the masking region need to be re-encoded. Macroblocks with motion vectors pointing inside do not need any modification. Efficient algorithms for reencoding macroblocks are well known in the art and are described in, for example, the above-referenced Chang et al. patent.
  • Referring to FIG. 9( b), a method for performing masking is now described. In step 910, the frame to be masked is examined in order to determine whether it is an I, P or B frame. If the frame is an I frame 911, all DCT coefficients for all blocks within the frame are extracted 920. Blockn is examined 930 to determine where in the frame it is located. If the block is inside the mask region 931, the DCT coefficients for the block are unchanged 941. If the block is outside the make region 932, the DCT coefficients for the block are set to zero 942. If the block is on the boundary of the masked region 933, the DCT cropping algorithm, i.e., equation 10, is applied 943. The block index n is incremented 950 and the process repeated 930-950 until all blocks have been examined.
  • If, however, the frame to be masked is either a P or B frame 920, motion vectors associated with blockn are examined 960 to determine whether they point to blocks outside or on the mask region 970. If a motion vector points to blocks outside or on the mask region 972, the macroblock is reencoded 980. The block index n is incremented 990 and the process repeated 960-990 until all blocks have been examined.
  • A third important tool used in film editing is the freeze effect. Since the freeze effect is usually longer than 1 second, simply repeating duplicate frames is not desirable if interactive playback (e.g. random search) is desired. Instead, the film effect of freeze frame requires the use of periodic I frames.
  • Thus, in order to ensure interactive playback capabilities, if the frozen frame is a P or B frame, it should be converted to an I frame. In order to convert a P or B frame into an I frame, every block of video information for the frame that has an associated motion vector is converted into a pure array of DCT coefficients, without motion vector information. The above-mentioned DCT domain motion compensation algorithm described in the Chang et al. patent is advantageously used to effect such conversion. The converted blocks can be referred to as intracoded blocks. Thereafter, the group of pictures which represent the frozen frame are filled with duplicated P frames, with all macroblocks in the duplicated P frames being to Motion Compensation Not Coded (i.e., 0 motion vector, and 0 residue error). Duplicate P frames can easily be created independently and inserted after I or P frames, as those familiar with digital compression techniques will easily understand.
  • A method for generating a frozen frame of video information is now described with reference to flowchart 1000 in FIG. 10. In step 1010, a user-defined frame of compressed video information is selected for the freeze-frame effect. In step 1020, the frame is examined in order to determine whether it is an I, P or B frame. If the frame is not an I frame 1021, it is converted into an I frame 1030. Finally, original 1022 or converted 1030 I frames are then used as to create duplicate P frames 1040. Periodical I frames can be inserted to increase interactivity, and to maintain a compatible bitrate.
  • A fourth important tool used in film editing is variable speed playback. Faster than normal playback, e.g., fast forward, is simply realized by dropping B, P, and then I frames according to the variable speed desired.
  • With respect to slow motion, depending on the slow motion rate, there are two possible approaches. With reference to FIG. 11, one approach is to simply insert duplicate B frames (B′) whenever an original B frame appears. Duplicate B frames are frames which copy all DCT coefficient, and associated motion vectors from previous B frame. There is no decoding or re-encoding involved in this approach. However, this approach has a drawback since the I/P frame delay is increased by a factor equal to the inverse of the motion rate. For example, in the illustrative sequence of frames shown in FIG. 11, the reference frame I0 must be transmitted 4 frames earlier. This limitation makes this approach suitable for slow playback up to about ½ normal frame rate. This will require an increased decoder buffer size.
  • In approach 2, original NB frames are converted to I frames using the DCT domain techniques described above, and duplicated P frames are inserted between I frames, as in the case with freeze frame. This approach reduces frame delay, although requiring extra DCT domain manipulations.
  • Another interesting tool used in film editing is the special effect of strobe motion, which is simply a combination of freeze frame and variable speed playback. Strobe motion is effected by dropping original B and/or P frames and inserting duplicated P frames.
  • Referring to FIG. 12, a system for enabling distributed clients to browse and manipulate digital video by way of a distant server or host computer is now described. A master server 1210 or a group of distributed servers is linked to distant servers 1220, 1230, 1240 and several clients 1250, 1260. From their client workstations, users are empowered to browse and edit compressed vide images in the manner described above.
  • Moving to FIG. 13, the master server 1210 acts as the content aggregator 1310 to collect video content, and as a content analyzer 1320 to analyze the visual content for efficient indexing 1330. Thus, the server 1210 may accept the functions performed by the parsing module 110 described above. The server 1210 also provides an editing engine 1340 with basic editing tools, such as the tools described above, for rendering basic editing functions and various special effects. Distributed clients 1250, 1260 access the video archives of the server through heterogeneous networks with interoperable interfaces, such as JAVA or MPEG MSDL manipulation tools.
  • The video server 1210 can be linked with other distributed servers 1220, 1230, 1240 that have video search engines which search for image and video over a network, e.g., the World Wide Web. Once a video file is found on any other hosts or online content providers, it will be downloaded and preprocessed by the video server 1210 to extract “keyframes,” i.e., frames 121 stored in the above described visualization module 120, and associated visual features such as camera motion, moving objects, color, texture, and temporal features. The Universal Resource Location (“URL”) of the video and the extracted features will be stored on the video server 1210. This client-server model empowers clients 1250, 1260 with much richer resources beyond that provided by the client's local system.
  • The client 1260 may open any video shot stored at the server 1210 and browse 1330 the keyframes hierarchically using any one of numerous different viewing models, including sequential view, feature-based view, and story-based view. A sequential view arranges the video shots according to the sequential time order; a feature-based view clusters video shots with similar visual features to the same group; a story view presents video shots according to the story structures in the semantic level. For example, for news applications, the story model may use the anchorperson shot as the criterion for automatic segmentation of video into semantic-level stories. The client can also perform shot-level editing or frame level editing 1390 using the above discussed editing tools. The clients may also submit their own videos which may be analyzed and indexed by the server 1210.
  • To view or to further refine retrieved video shots, the client 1260 may choose different levels of resolution for the video shots sent by the server 1210. There are three broad levels of video rendering. At the first level, the client 1260 can render only straight cuts at low resolution without any special effects. At the second level, the client may send information to the server defining low-resolution video with desired special effects to be generated. Finally, when the client 1260 no longer wishes to perform editing, the master server 1210 will generate a full-resolution video with all the effects requested by the client from the highest quality source video which is located at either the master server 1210 or the distributed remote content servers 1220, 1230, 1240. In this way, the user is given the flexibility to tradeoff between quality and speed.
  • In order to obtain a reduced resolution video sequence, for each video sequence stored on a server, a low resolution icon stream video must be extracted from the full resolution video. The icon stream is an 8:1 down sampled version of the original video. For I frames, DCT DC coefficients are obtained directly. For B and P frames, DCT domain inverse motion compensation is applied to the DCT coefficients. Finally, the extracted coefficients are converted into a smaller sized, e.g., 8:2:1, I frame as those skilled in the art will appreciate.
  • The foregoing merely illustrates the principles of the invention. Various modifications and alterations to the described embodiments will be apparent to those skilled in the art in view of the inventors teachings herein. For example, while the above description was directed to indexing and editing digital vide which has been compressed in accordance with the MPEG format, the foregoing is equally applicable to other compressed bitstreams, e.g., Motion JPEG, H.261 or H.263, or any other algorithm based on transform coding and motion compensation. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods which, although not explicitly shown or described herein, embody the principles of the invention and are thus within the spirit and scope of the invention.
  • APPENDIX
       int do_dissolve(int duration)
       {
        int i, j, k, s, cc;
        int blks;
        double alpha;
        short (*tmp[3])[64];
        Id = &S1;
        initbits( );
        Id->end = roundtoP(ld->end);
        do_cut(ld->begin, ld->end);
        getDCTframe(ld->end, GETDCT);
        /* copy the first DCT to tmpDCT */
        for (cc=0; cc<3; cc++) {
         if (cc==0)
          blks = (coded_picture_width*coded_picture_height)>>6;
         else
          blks = (chrom_width*chrom_height)>>6;
         memcpy(tmpDCT[cc], refDCT[cc], blks * sizeof(short [64]));
       }
       ld = &S2;
       initbits( );
       ld->begin = roundtoP(ld->begin);
       getDCTframe(ld->begin, GETDCT);
       /* first prediction is based on tmpDCT */
       for (cc=0; cc<3; cc++) {
        if (cc==0)
         blks = (coded_picture_width*coded_picture_height)>>6;
        else
         blks = (chrom_width*chrom_height)>>6;
        memcpy(auxDCT[cc], tmpDCT[cc], blks * sizeof(short [64]));
         }
         for(s=1;s<=duration;s++) {
          alpha=s/(double)duration;
          for (i=0; i<3; i++) {
           tmp[i] = oldrefDCT[i];
           oldrefDCT[i] = auxDCT[i];
           auxDCT[i] = tmp[i];
           newDCT[i]= auxDCT[i];
          }
          for (cc=0; cc<3; cc++) {
          if (cc==0)
            blks = (coded_picture_width*coded_picture_height)>>6;
           else
            blks = (chrom _width*chrom_height)>>6;
           for (j=0; j<blks; j++) {
            for (k=0; k<64; k++)
            newDCT[cc][j][k] = (short) CLAMP (-2048,(1-alpha)*tmpDCT[cc][j][k]
             +alpha*refDCT[cc][j][k],2047);
           }
          }
          /* use refoldDCT as reference frame for DPCM of P */
          putpictDCT(0);
         }
       }

Claims (10)

1-18. (canceled)
19. A method for masking a region of a compressed frame of digital video information, comprising the steps of:
a. determining whether said frame to be masked is intra-coded, predictive-coded or bi-directionally predictive-coded;
b. if said frame is intra-coded:
i. extracting DCT coefficients for all blocks within said frame;
ii. examining blockn to determine where in said frame said block is located;
iii. setting said DCT coefficients for said block to zero if said block is outside said mask region;
iv. applying a DCT cropping algorithm to said DCT coefficients if said block is on the boundary of said mask region; and
v. repeating steps (b)(ii)-(b)(iv) for each block in said frame;
c. If said frame is predictive-coded or bi-directionally predictive-coded:
i. examining motion vectors associated with blockn to determine whether they point to blocks outside or on said mask region;
ii. reencoding said block if a motion vector points to blocks outside or on said mask region; and
iii. repeating steps (c)(i)-(c)(ii) for all blocks in said frame.
20. A method for generating a reduced speed sequence of frames of video information from a sequence of frames of compressed video information, comprising the steps of:
a. selecting a frame of compressed video information to be repeated;
b. determining whether said frame to be repeated is intra-coded, predictive-coded or bi-directionally predictive-coded;
c. converting said frame into an intra-coded frame if said frame is a predictive-coded or bi-directionally predictive-coded frame;
d. creating duplicate predictive-coded frames; and
e. arranging said determined frame and said duplicate predictive-coded frames into a sequence of compressed frames of video information.
21. The method of claim 20, wherein said reduced speed sequence of frames generates a frozen frame effect.
22. The method of claim 20, further comprising the step of converting said frame into an intra-coded frame if said frame is a predictive-coded or bi-directionally predictive-coded frame.
23. A system for editing compressed video information over a distributed network, comprising:
a. a client computer;
b. a network link, coupled to said client computer, for permitting said client computer to search for and locate compressed video information on said distributed network; and
c. means for editing a compressed bitstream of video information over said distributed network.
24. The system of claim 21, wherein said editing means includes means for dissolving an incoming scene of video information which includes a sequence of fields or frame of compressed video information and an outgoing scene of video information which includes a sequence of fields or frame of compressed video information, said dissolving means comprising:
a. outgoing motion compensation means for applying DCT domain motion compensation to a last frame of said outgoing video scene to obtain DCT coefficients for all blocks of video information which make up said last frame of said outgoing video scene;
b. incoming motion compensation means, for applying DCT domain motion compensation to a first frame of said incoming video scene to obtain the DCT coefficients for all blocks of video information which make up said first frame of said incoming video scene; and
c. dissolve region creating means, coupled to said incoming motion compensation means and to said outgoing motion compensation means, for creating a first frame in a dissolve region from said DCT coefficients of said last outgoing frame and said first incoming frame.
25. The system of claim 21, wherein said editing means includes means for masking a region of a compressed frame of digital video information, said masking means comprising:
a. means for determining whether said frame to be masked is intra-coded, predictive-coded or bi-directionally predictive-coded;
b. means for processing intra-coded frames including:
i. means for receiving DCT coefficients and for extracting DCT coefficients for all blocks within said frame;
ii. means, coupled to said receiving means, for examining blocks of compressed video information in said frame to determine where in said frame said block is located;
iii. means, coupled to said examining means, for setting said DCT coefficients for said block to zero if said block is outside said mask region; and
iv. means, coupled to said setting means, for applying a DCT translation algorithm to said DCT coefficients if said block is on the boundary of said mask region; and
c. means for processing predictive-coded and bi-directionally predictive-coded frames, including:
i. means for receiving and examining motion vectors associated with blocks of video information to determine whether they point to blocks outside or on said mask region; and
ii. means, coupled to said receiving means, for reencoding said block if a motion vector points to blocks outside or on said mask region.
26. The system of claim 2, wherein said editing means includes means generating a reduced speed sequence of frames of video information from a sequence of frames of compressed video information, comprising:
a. selection means for selecting a frame of compressed video information to be repeated;
b. computational means, coupled to said selection means, for determining whether said frame to be repeated is intra-coded, predictive-coded or bi-directionally predictive-coded;
c. frame generating means, coupled to said computational means and to said converting means, for creating duplicate predictive-coded frames; and
d. frame arranging means, coupled to said frame generating means, for arranging said determined frame and said duplicate predictive-coded frames into a sequence of compressed frames of video information.
27. A method for converting a full resolution compressed domain bitstream into a reduced resolution compressed bitstream, comprising the steps of:
a. examining a frame of compressed video information from said full resolution bitstream;
b. determining whether said examined frame is intra-coded, predictive-coded or bi-directionally predictive-coded;
c. extracting DCT DC coefficients for said determined frame if said determined frame is intra-coded
d. applying DCT domain inverse motion compensation to said frame to extract DCT DC coefficients if said frame is predictive-coded or bi-directionally predictive-coded; and
e. converting said extracted DCT DC coefficients into DCT DC coefficients for a reduced size intra-coded frame of video.
US12/874,337 1997-05-16 2010-09-02 Methods and architecture for indexing and editing compressed video over the world wide web Abandoned US20110064136A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US12/874,337 US20110064136A1 (en) 1997-05-16 2010-09-02 Methods and architecture for indexing and editing compressed video over the world wide web
US13/078,626 US9330722B2 (en) 1997-05-16 2011-04-01 Methods and architecture for indexing and editing compressed video over the world wide web

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US09/423,769 US6735253B1 (en) 1997-05-16 1997-05-16 Methods and architecture for indexing and editing compressed video over the world wide web
PCT/US1997/008266 WO1998052356A1 (en) 1997-05-16 1997-05-16 Methods and architecture for indexing and editing compressed video over the world wide web
US10/728,345 US7817722B2 (en) 1997-05-16 2003-12-04 Methods and architecture for indexing and editing compressed video over the world wide web
US12/874,337 US20110064136A1 (en) 1997-05-16 2010-09-02 Methods and architecture for indexing and editing compressed video over the world wide web

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US10/728,345 Division US7817722B2 (en) 1997-05-16 2003-12-04 Methods and architecture for indexing and editing compressed video over the world wide web

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US13/078,626 Continuation US9330722B2 (en) 1997-05-16 2011-04-01 Methods and architecture for indexing and editing compressed video over the world wide web

Publications (1)

Publication Number Publication Date
US20110064136A1 true US20110064136A1 (en) 2011-03-17

Family

ID=32230418

Family Applications (4)

Application Number Title Priority Date Filing Date
US09/423,769 Expired - Lifetime US6735253B1 (en) 1997-05-16 1997-05-16 Methods and architecture for indexing and editing compressed video over the world wide web
US10/728,345 Expired - Fee Related US7817722B2 (en) 1997-05-16 2003-12-04 Methods and architecture for indexing and editing compressed video over the world wide web
US12/874,337 Abandoned US20110064136A1 (en) 1997-05-16 2010-09-02 Methods and architecture for indexing and editing compressed video over the world wide web
US13/078,626 Expired - Lifetime US9330722B2 (en) 1997-05-16 2011-04-01 Methods and architecture for indexing and editing compressed video over the world wide web

Family Applications Before (2)

Application Number Title Priority Date Filing Date
US09/423,769 Expired - Lifetime US6735253B1 (en) 1997-05-16 1997-05-16 Methods and architecture for indexing and editing compressed video over the world wide web
US10/728,345 Expired - Fee Related US7817722B2 (en) 1997-05-16 2003-12-04 Methods and architecture for indexing and editing compressed video over the world wide web

Family Applications After (1)

Application Number Title Priority Date Filing Date
US13/078,626 Expired - Lifetime US9330722B2 (en) 1997-05-16 2011-04-01 Methods and architecture for indexing and editing compressed video over the world wide web

Country Status (1)

Country Link
US (4) US6735253B1 (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070245400A1 (en) * 1998-11-06 2007-10-18 Seungyup Paek Video description system and method
US20080181308A1 (en) * 2005-03-04 2008-07-31 Yong Wang System and method for motion estimation and mode decision for low-complexity h.264 decoder
US20080303942A1 (en) * 2001-12-06 2008-12-11 Shih-Fu Chang System and method for extracting text captions from video and generating video summaries
US20090316778A1 (en) * 2002-04-26 2009-12-24 Jae-Gon Kim Method And System For Optimal Video Transcoding Based On Utility Function Descriptors
US20110025710A1 (en) * 2008-04-10 2011-02-03 The Trustees Of Columbia University In The City Of New York Systems and methods for image archeology
US20110145232A1 (en) * 2008-06-17 2011-06-16 The Trustees Of Columbia University In The City Of New York System and method for dynamically and interactively searching media data
US8671069B2 (en) 2008-12-22 2014-03-11 The Trustees Of Columbia University, In The City Of New York Rapid image annotation via brain state decoding and visual pattern mining
US9330722B2 (en) 1997-05-16 2016-05-03 The Trustees Of Columbia University In The City Of New York Methods and architecture for indexing and editing compressed video over the world wide web

Families Citing this family (143)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3774954B2 (en) * 1996-10-30 2006-05-17 株式会社日立製作所 Video encoding method
US20050028194A1 (en) * 1998-01-13 2005-02-03 Elenbaas Jan Hermanus Personalized news retrieval system
US7006569B1 (en) * 1999-02-05 2006-02-28 Samsung Electronics Co., Ltd. Digital video processing method and apparatus thereof
FR2790852B1 (en) * 1999-03-08 2001-04-20 Thomson Multimedia Sa METHOD FOR EVALUATING ENCODED IMAGES, DEVICE IMPLEMENTING THE METHOD AND USE OF THE DEVICE AND METHOD
US7085319B2 (en) * 1999-04-17 2006-08-01 Pts Corporation Segment-based encoding system using segment hierarchies
US7050503B2 (en) * 1999-04-17 2006-05-23 Pts Corporation Segment-based encoding system using residue coding by basis function coefficients
US7082162B2 (en) * 1999-04-17 2006-07-25 Pts Corporation Segment-based encoding system including segment-specific metadata
WO2001015459A1 (en) * 1999-08-24 2001-03-01 Fujitsu Limited Time-varying image processor, its method, and recorded medium
WO2001045425A1 (en) * 1999-12-14 2001-06-21 Scientific-Atlanta, Inc. System and method for adaptive decoding of a video signal with coordinated resource allocation
KR100698106B1 (en) * 2000-03-07 2007-03-26 엘지전자 주식회사 A hierarchical hybrid shot change detection method for mpeg-compressed video
CN101035277A (en) * 2000-03-13 2007-09-12 索尼公司 Method and apparatus for generating compact code-switching hints metadata
US7222163B1 (en) * 2000-04-07 2007-05-22 Virage, Inc. System and method for hosting of video content over a network
US8171509B1 (en) * 2000-04-07 2012-05-01 Virage, Inc. System and method for applying a database to video multimedia
US7260564B1 (en) * 2000-04-07 2007-08-21 Virage, Inc. Network video guide and spidering
US7962948B1 (en) 2000-04-07 2011-06-14 Virage, Inc. Video-enabled community building
US7103103B2 (en) * 2000-04-17 2006-09-05 Pts Corporation Segmenting encoding system encoding video data using segment-by-segment kinetic data including change information in addition to translation information
US7082166B2 (en) * 2000-04-17 2006-07-25 Pts Corporation Decoder for decoding segment-based encoding of video data using segmentation performed at a decoder
US20040125877A1 (en) * 2000-07-17 2004-07-01 Shin-Fu Chang Method and system for indexing and content-based adaptive streaming of digital video content
US6934334B2 (en) * 2000-10-02 2005-08-23 Kabushiki Kaisha Toshiba Method of transcoding encoded video data and apparatus which transcodes encoded video data
JP4107544B2 (en) * 2000-10-23 2008-06-25 三菱電機株式会社 Image signal re-encoding device
JP4524908B2 (en) * 2000-11-08 2010-08-18 日本電気株式会社 Moving image editing method, moving image editing apparatus, and storage medium storing moving image editing program
AUPR133700A0 (en) * 2000-11-09 2000-11-30 Mediaware Solutions Pty Ltd Transition templates for compressed digital video and method of generating same
KR100386583B1 (en) * 2000-11-30 2003-06-02 엘지전자 주식회사 Apparatus and method for transcoding video
US7058130B2 (en) * 2000-12-11 2006-06-06 Sony Corporation Scene change detection
JP4366023B2 (en) * 2001-03-16 2009-11-18 インターナショナル・ビジネス・マシーンズ・コーポレーション Partial image region extraction method of video image, partial image region extraction system, program for extracting partial image region, distribution method of extracted video image, and content creation method
DE10120395A1 (en) * 2001-04-25 2002-10-31 Bosch Gmbh Robert Device for the interpolation of samples as well as image encoder and image decoder
US6996183B2 (en) * 2001-09-26 2006-02-07 Thomson Licensing Scene cut detection in a video bitstream
US7274857B2 (en) * 2001-12-31 2007-09-25 Scientific-Atlanta, Inc. Trick modes for compressed video streams
US7319720B2 (en) * 2002-01-28 2008-01-15 Microsoft Corporation Stereoscopic video
US20030156649A1 (en) * 2002-01-28 2003-08-21 Abrams Thomas Algie Video and/or audio processing
US7463684B2 (en) * 2002-05-03 2008-12-09 Microsoft Corporation Fading estimation/compensation
US7277486B2 (en) * 2002-05-03 2007-10-02 Microsoft Corporation Parameterization for fading compensation
US7609767B2 (en) * 2002-05-03 2009-10-27 Microsoft Corporation Signaling for fading compensation
JP4222869B2 (en) * 2002-12-10 2009-02-12 株式会社ソニー・コンピュータエンタテインメント Image playback device
JP3795018B2 (en) * 2003-01-20 2006-07-12 富士通株式会社 Streaming transmission method, streaming transmission system, data processing apparatus, and computer program
CN103001923B (en) * 2003-06-05 2016-03-30 英特特拉斯特技术公司 For controlling the method and system of the access to digital content fragment on the computer systems
US7646437B1 (en) 2003-09-03 2010-01-12 Apple Inc. Look-ahead system and method for pan and zoom detection in video sequences
US7609762B2 (en) * 2003-09-07 2009-10-27 Microsoft Corporation Signaling for entry point frames with predicted first field
US7924921B2 (en) * 2003-09-07 2011-04-12 Microsoft Corporation Signaling coding and display options in entry point headers
US7852919B2 (en) * 2003-09-07 2010-12-14 Microsoft Corporation Field start code for entry point frames with predicted first field
US8009739B2 (en) * 2003-09-07 2011-08-30 Microsoft Corporation Intensity estimation/compensation for interlaced forward-predicted fields
US7839930B2 (en) * 2003-11-13 2010-11-23 Microsoft Corporation Signaling valid entry points in a video stream
US8213779B2 (en) * 2003-09-07 2012-07-03 Microsoft Corporation Trick mode elementary stream and receiver system
US7966642B2 (en) * 2003-09-15 2011-06-21 Nair Ajith N Resource-adaptive management of video storage
US9715898B2 (en) * 2003-12-16 2017-07-25 Core Wireless Licensing S.A.R.L. Method and device for compressed-domain video editing
WO2005096625A1 (en) * 2004-04-02 2005-10-13 Nds Limited System for providing visible messages during pvr trick mode playback
US8600217B2 (en) * 2004-07-14 2013-12-03 Arturo A. Rodriguez System and method for improving quality of displayed picture during trick modes
JP4491295B2 (en) * 2004-07-22 2010-06-30 キヤノン株式会社 Image reproduction method and image reproduction apparatus
US7792373B2 (en) * 2004-09-10 2010-09-07 Pioneer Corporation Image processing apparatus, image processing method, and image processing program
US7624302B2 (en) * 2004-10-25 2009-11-24 Hewlett-Packard Development Company, L.P. System and method for switching the role of boot processor to a spare processor responsive to detection of loss of lockstep in a boot processor
US7447337B2 (en) * 2004-10-25 2008-11-04 Hewlett-Packard Development Company, L.P. Video content understanding through real time video motion analysis
US7818614B2 (en) * 2004-10-25 2010-10-19 Hewlett-Packard Development Company, L.P. System and method for reintroducing a processor module to an operating system after lockstep recovery
US7627781B2 (en) * 2004-10-25 2009-12-01 Hewlett-Packard Development Company, L.P. System and method for establishing a spare processor for recovering from loss of lockstep in a boot processor
US20060187228A1 (en) * 2005-01-31 2006-08-24 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Sharing including peripheral shared image device
US8902320B2 (en) 2005-01-31 2014-12-02 The Invention Science Fund I, Llc Shared image device synchronization or designation
US9124729B2 (en) 2005-01-31 2015-09-01 The Invention Science Fund I, Llc Shared image device synchronization or designation
US20060174203A1 (en) 2005-01-31 2006-08-03 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Viewfinder for shared image device
US7876357B2 (en) * 2005-01-31 2011-01-25 The Invention Science Fund I, Llc Estimating shared image device operational capabilities or resources
US20060221197A1 (en) * 2005-03-30 2006-10-05 Jung Edward K Image transformation estimator of an imaging device
US20060171603A1 (en) * 2005-01-31 2006-08-03 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Resampling of transformed shared image techniques
US20060173972A1 (en) * 2005-01-31 2006-08-03 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Audio sharing
US7920169B2 (en) 2005-01-31 2011-04-05 Invention Science Fund I, Llc Proximity of shared image devices
US8606383B2 (en) 2005-01-31 2013-12-10 The Invention Science Fund I, Llc Audio sharing
US9489717B2 (en) 2005-01-31 2016-11-08 Invention Science Fund I, Llc Shared image device
US20070236505A1 (en) * 2005-01-31 2007-10-11 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Resampling of transformed shared image techniques
US9082456B2 (en) 2005-01-31 2015-07-14 The Invention Science Fund I Llc Shared image device designation
US9325781B2 (en) 2005-01-31 2016-04-26 Invention Science Fund I, Llc Audio sharing
US20060170956A1 (en) 2005-01-31 2006-08-03 Jung Edward K Shared image devices
US9910341B2 (en) 2005-01-31 2018-03-06 The Invention Science Fund I, Llc Shared image device designation
KR20060088303A (en) * 2005-02-01 2006-08-04 엘지전자 주식회사 Apparatus and method recording/playing moving picture in digital broadcasting receiver
US8705614B2 (en) * 2005-04-04 2014-04-22 Broadcom Corporation Motion estimation using camera tracking movements
US9001215B2 (en) 2005-06-02 2015-04-07 The Invention Science Fund I, Llc Estimating shared image device operational capabilities or resources
US9967424B2 (en) 2005-06-02 2018-05-08 Invention Science Fund I, Llc Data storage usage protocol
US20070222865A1 (en) 2006-03-15 2007-09-27 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Enhanced video/still image correlation
US9942511B2 (en) 2005-10-31 2018-04-10 Invention Science Fund I, Llc Preservation/degradation of video/audio aspects of a data stream
US8964054B2 (en) 2006-08-18 2015-02-24 The Invention Science Fund I, Llc Capturing selected image objects
US7872675B2 (en) * 2005-06-02 2011-01-18 The Invention Science Fund I, Llc Saved-image management
US9076208B2 (en) * 2006-02-28 2015-07-07 The Invention Science Fund I, Llc Imagery processing
US7782365B2 (en) * 2005-06-02 2010-08-24 Searete Llc Enhanced video/still image correlation
US9167195B2 (en) * 2005-10-31 2015-10-20 Invention Science Fund I, Llc Preservation/degradation of video/audio aspects of a data stream
US9819490B2 (en) 2005-05-04 2017-11-14 Invention Science Fund I, Llc Regional proximity for shared image device(s)
US8253821B2 (en) * 2005-10-31 2012-08-28 The Invention Science Fund I, Llc Degradation/preservation management of captured data
US9191611B2 (en) * 2005-06-02 2015-11-17 Invention Science Fund I, Llc Conditional alteration of a saved image
US8681225B2 (en) 2005-06-02 2014-03-25 Royce A. Levien Storage access technique for captured data
US9621749B2 (en) * 2005-06-02 2017-04-11 Invention Science Fund I, Llc Capturing selected image objects
US9451200B2 (en) 2005-06-02 2016-09-20 Invention Science Fund I, Llc Storage access technique for captured data
US8233042B2 (en) * 2005-10-31 2012-07-31 The Invention Science Fund I, Llc Preservation and/or degradation of a video/audio data stream
US10003762B2 (en) 2005-04-26 2018-06-19 Invention Science Fund I, Llc Shared image devices
US9093121B2 (en) 2006-02-28 2015-07-28 The Invention Science Fund I, Llc Data management of an audio data stream
US8072501B2 (en) * 2005-10-31 2011-12-06 The Invention Science Fund I, Llc Preservation and/or degradation of a video/audio data stream
US20060274153A1 (en) * 2005-06-02 2006-12-07 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Third party storage of captured data
US8060548B2 (en) * 2005-06-03 2011-11-15 The Commonwealth Of Australia Messaging method
WO2007015047A2 (en) * 2005-08-04 2007-02-08 Nds Limited Advanced digital tv system
KR101285946B1 (en) * 2005-10-18 2013-08-23 인터트러스트 테크놀로지즈 코포레이션 Methods for digital rights management
US20070204078A1 (en) * 2006-02-09 2007-08-30 Intertrust Technologies Corporation Digital rights management engine systems and methods
US20070120980A1 (en) 2005-10-31 2007-05-31 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Preservation/degradation of video/audio aspects of a data stream
US20070203595A1 (en) * 2006-02-28 2007-08-30 Searete Llc, A Limited Liability Corporation Data management of an audio data stream
US8949235B2 (en) * 2005-11-15 2015-02-03 Yissum Research Development Company Of The Hebrew University Of Jerusalem Ltd. Methods and systems for producing a video synopsis using clustering
US8300701B2 (en) * 2005-12-09 2012-10-30 Avid Technology, Inc. Offspeed playback in a video editing system of video data compressed using long groups of pictures
US8130841B2 (en) * 2005-12-29 2012-03-06 Harris Corporation Method and apparatus for compression of a video signal
GB2435140B (en) * 2006-02-13 2011-04-06 Snell & Wilcox Ltd Sport action coding
US20070268406A1 (en) * 2006-05-22 2007-11-22 Broadcom Corporation, A California Corporation Video processing system that generates sub-frame metadata
US8577204B2 (en) * 2006-11-13 2013-11-05 Cyberlink Corp. System and methods for remote manipulation of video over a network
JP2008147838A (en) * 2006-12-07 2008-06-26 Sony Corp Image processor, image processing method, and program
CN101689394B (en) 2007-02-01 2014-03-26 耶路撒冷希伯来大学伊森姆研究发展有限公司 Method and system for video indexing and video synopsis
US7465241B2 (en) * 2007-03-23 2008-12-16 Acushnet Company Functionalized, crosslinked, rubber nanoparticles for use in golf ball castable thermoset layers
JP4900811B2 (en) * 2007-03-30 2012-03-21 株式会社日立製作所 Storage system and storage control method
US8204955B2 (en) 2007-04-25 2012-06-19 Miovision Technologies Incorporated Method and system for analyzing multimedia content
US8103150B2 (en) * 2007-06-07 2012-01-24 Cyberlink Corp. System and method for video editing based on semantic data
US20090033791A1 (en) * 2007-07-31 2009-02-05 Scientific-Atlanta, Inc. Video processing systems and methods
JP5286732B2 (en) * 2007-10-01 2013-09-11 ソニー株式会社 Information processing apparatus and method, program, and recording medium
JP5400062B2 (en) * 2008-01-07 2014-01-29 トムソン ライセンシング Video encoding and decoding method and apparatus using parametric filtering
WO2009087641A2 (en) * 2008-01-10 2009-07-16 Ramot At Tel-Aviv University Ltd. System and method for real-time super-resolution
US8300696B2 (en) 2008-07-25 2012-10-30 Cisco Technology, Inc. Transcoding for systems operating under plural video coding specifications
KR101271461B1 (en) * 2009-10-09 2013-06-05 한국전자통신연구원 Apparatus and method for protecting privacy information of surveillance image
US8891831B2 (en) * 2010-12-14 2014-11-18 The United States Of America, As Represented By The Secretary Of The Navy Method and apparatus for conservative motion estimation from multi-image sequences
US9547911B2 (en) 2010-12-14 2017-01-17 The United States Of America, As Represented By The Secretary Of The Navy Velocity estimation from imagery using symmetric displaced frame difference equation
KR20120096317A (en) * 2011-02-22 2012-08-30 한국전자통신연구원 Method and apparatus of coding stereoscopic video
CN103597488B (en) 2011-04-11 2016-08-24 英特托拉斯技术公司 Information safety system and method
US9406303B2 (en) 2011-11-18 2016-08-02 Sirius Xm Radio Inc. Systems and methods for implementing cross-fading, interstitials and other effects downstream
TWI519147B (en) * 2011-12-28 2016-01-21 財團法人工業技術研究院 System and method for providing and transmitting condensed streaming content
KR20130099515A (en) * 2012-02-29 2013-09-06 삼성전자주식회사 Apparatas and method of displaying a contents using for key frame in a terminal
CA2866585C (en) 2012-03-06 2021-02-23 Sirius Xm Radio Inc. Systems and methods for audio attribute mapping
MX356952B (en) 2012-04-17 2018-06-21 Sirius Xm Radio Inc Systems and methods for implementing efficient cross-fading between compressed audio streams.
JP2015530034A (en) 2012-08-23 2015-10-08 トムソン ライセンシングThomson Licensing Method and apparatus for detecting pictures that show gradually changing transitions in a video bitstream
US9454208B2 (en) 2013-03-14 2016-09-27 Google Inc. Preventing sleep mode for devices based on sensor inputs
US9998750B2 (en) 2013-03-15 2018-06-12 Cisco Technology, Inc. Systems and methods for guided conversion of video from a first to a second compression format
US9532088B2 (en) * 2014-06-18 2016-12-27 Arris Enterprises, Inc. Trick-play streams for adaptive bitrate streaming
US9712828B2 (en) * 2015-05-27 2017-07-18 Indian Statistical Institute Foreground motion detection in compressed video data
US10248864B2 (en) * 2015-09-14 2019-04-02 Disney Enterprises, Inc. Systems and methods for contextual video shot aggregation
KR20170090868A (en) * 2016-01-29 2017-08-08 주식회사 마크애니 Scene cut frame detecting apparatus and method
US10963504B2 (en) * 2016-02-12 2021-03-30 Sri International Zero-shot event detection using semantic embedding
US10200572B1 (en) * 2016-03-03 2019-02-05 Amazon Technologies, Inc Motion detection
US10515115B2 (en) 2017-03-03 2019-12-24 Proactive Cctv System and method for closed-circuit television file archival and compression
KR102082815B1 (en) * 2018-04-24 2020-02-28 주식회사 지디에프랩 Artificial intelligence based resolution improvement system
WO2019209008A1 (en) * 2018-04-24 2019-10-31 주식회사 지디에프랩 System for improving video quality by using changed macroblock extraction technique
US11651016B2 (en) * 2018-08-09 2023-05-16 Walmart Apollo, Llc System and method for electronic text classification
US10937196B1 (en) 2018-08-21 2021-03-02 Perceive Corporation Compressive sensing based image capture device
JP6604413B1 (en) * 2018-10-09 2019-11-13 日本電気株式会社 Image search device, image search system, image search method and program
US10834458B2 (en) * 2019-03-29 2020-11-10 International Business Machines Corporation Automated video detection and correction
US11227160B2 (en) 2019-11-15 2022-01-18 International Business Machines Corporation Detecting scene transitions in video footage
CN113127679A (en) 2019-12-30 2021-07-16 阿里巴巴集团控股有限公司 Video searching method and device and index construction method and device
US11195024B1 (en) * 2020-07-10 2021-12-07 International Business Machines Corporation Context-aware action recognition by dual attention networks

Citations (70)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5694334A (en) * 1994-09-08 1997-12-02 Starguide Digital Networks, Inc. Method and apparatus for electronic distribution of digital multi-media information
US5835163A (en) * 1995-12-21 1998-11-10 Siemens Corporate Research, Inc. Apparatus for detecting a cut in a video
US6028635A (en) * 1996-12-03 2000-02-22 Stmicroelectronics, Inc. Reducing the memory required for decompression by storing compressed information using DCT based techniques
US6058205A (en) * 1997-01-09 2000-05-02 International Business Machines Corporation System and method for partitioning the feature space of a classifier in a pattern classification system
US6070167A (en) * 1997-09-29 2000-05-30 Sharp Laboratories Of America, Inc. Hierarchical method and system for object-based audiovisual descriptive tagging of images for information retrieval, editing, and manipulation
US6072542A (en) * 1997-11-25 2000-06-06 Fuji Xerox Co., Ltd. Automatic video segmentation using hidden markov model
US6157745A (en) * 1995-09-06 2000-12-05 U.S. Philips Corporation Method and system for coding and decoding a sequence of segmented images
US6275599B1 (en) * 1998-08-28 2001-08-14 International Business Machines Corporation Compressed image authentication and verification
US6332030B1 (en) * 1998-01-15 2001-12-18 The Regents Of The University Of California Method for embedding and extracting digital data in images and video
US6339450B1 (en) * 1999-09-21 2002-01-15 At&T Corp Error resilient transcoding for video over wireless channels
US6356309B1 (en) * 1995-08-02 2002-03-12 Matsushita Electric Industrial Co., Ltd. Video coding device and video transmission system using the same, quantization control method and average throughput calculation method used therein
US6366314B1 (en) * 1997-12-17 2002-04-02 Telediffusion De France Method and system for measuring the quality of digital television signals
US6385329B1 (en) * 2000-02-14 2002-05-07 Digimarc Corporation Wavelet domain watermarks
US6404925B1 (en) * 1999-03-11 2002-06-11 Fuji Xerox Co., Ltd. Methods and apparatuses for segmenting an audio-visual recording using image similarity searching and audio speaker recognition
US6442538B1 (en) * 1998-05-27 2002-08-27 Hitachi, Ltd. Video information retrieval method and apparatus
US20020118748A1 (en) * 2000-06-27 2002-08-29 Hideki Inomata Picture coding apparatus, and picture coding method
US6526099B1 (en) * 1996-10-25 2003-02-25 Telefonaktiebolaget Lm Ericsson (Publ) Transcoder
US6606393B1 (en) * 1999-12-02 2003-08-12 Verizon Laboratories Inc. Message authentication code using image histograms
US6700935B2 (en) * 2002-02-08 2004-03-02 Sony Electronics, Inc. Stream based bitrate transcoder for MPEG coded video
US6792434B2 (en) * 2001-04-20 2004-09-14 Mitsubishi Electric Research Laboratories, Inc. Content-based visualization and user-modeling for interactive browsing and retrieval in multimedia databases
US20040210819A1 (en) * 2001-06-15 2004-10-21 Alonso Antonio Used Dynamic browser interface
US6816836B2 (en) * 1999-08-06 2004-11-09 International Business Machines Corporation Method and apparatus for audio-visual speech detection and recognition
US20050238238A1 (en) * 2002-07-19 2005-10-27 Li-Qun Xu Method and system for classification of semantic content of audio/video data
US7093028B1 (en) * 1999-12-15 2006-08-15 Microsoft Corporation User and content aware object-based data stream transmission methods and arrangements
US20060224532A1 (en) * 2005-03-09 2006-10-05 Case Western Reserve University Iterative feature weighting with neural networks
US7145946B2 (en) * 2001-07-27 2006-12-05 Sony Corporation MPEG video drift reduction
US20070033170A1 (en) * 2000-07-24 2007-02-08 Sanghoon Sull Method For Searching For Relevant Multimedia Content
US20070245400A1 (en) * 1998-11-06 2007-10-18 Seungyup Paek Video description system and method
US7308443B1 (en) * 2004-12-23 2007-12-11 Ricoh Company, Ltd. Techniques for video retrieval based on HMM similarity
US7313269B2 (en) * 2003-12-12 2007-12-25 Mitsubishi Electric Research Laboratories, Inc. Unsupervised learning of video structures in videos using hierarchical statistical models to detect events
US7327885B2 (en) * 2003-06-30 2008-02-05 Mitsubishi Electric Research Laboratories, Inc. Method for detecting short term unusual events in videos
US20080055479A1 (en) * 2006-09-01 2008-03-06 Texas Instruments Incorporated Color Space Appearance Model Video Processor
US7386806B2 (en) * 2005-01-05 2008-06-10 Hillcrest Laboratories, Inc. Scaling and layout methods and systems for handling one-to-many objects
US7406409B2 (en) * 2004-01-14 2008-07-29 Mitsubishi Electric Research Laboratories, Inc. System and method for recording and reproducing multimedia based on an audio signal
US7409144B2 (en) * 2000-12-07 2008-08-05 Sony United Kingdom Limited Video and audio information processing
US20080193016A1 (en) * 2004-02-06 2008-08-14 Agency For Science, Technology And Research Automatic Video Event Detection and Indexing
US20080298464A1 (en) * 2003-09-03 2008-12-04 Thompson Licensing S.A. Process and Arrangement for Encoding Video Pictures
US20080304743A1 (en) * 2007-06-11 2008-12-11 Microsoft Corporation Active segmentation for groups of images
US20080303942A1 (en) * 2001-12-06 2008-12-11 Shih-Fu Chang System and method for extracting text captions from video and generating video summaries
US7496830B2 (en) * 1999-12-07 2009-02-24 Microsoft Corporation Computer user interface architecture that saves a user's non-linear navigation history and intelligently maintains that history
US20090055094A1 (en) * 2007-06-07 2009-02-26 Sony Corporation Navigation device and nearest point search method
US7519217B2 (en) * 2004-11-23 2009-04-14 Microsoft Corporation Method and system for generating a classifier using inter-sample relationships
US20090132561A1 (en) * 2007-11-21 2009-05-21 At&T Labs, Inc. Link-based classification of graph nodes
US7574409B2 (en) * 2004-11-04 2009-08-11 Vericept Corporation Method, apparatus, and system for clustering and classification
US7636662B2 (en) * 2003-09-30 2009-12-22 Koninklijke Philips Electronics N.V. System and method for audio-visual content synthesis
US20090316778A1 (en) * 2002-04-26 2009-12-24 Jae-Gon Kim Method And System For Optimal Video Transcoding Based On Utility Function Descriptors
US7653264B2 (en) * 2005-03-04 2010-01-26 The Regents Of The University Of Michigan Method of determining alignment of images in high dimensional feature space
US7676820B2 (en) * 2003-01-06 2010-03-09 Koninklijke Philips Electronics N.V. Method and apparatus for similar video content hopping
US20100082614A1 (en) * 2008-09-22 2010-04-01 Microsoft Corporation Bayesian video search reranking
US7720851B2 (en) * 2006-05-16 2010-05-18 Eastman Kodak Company Active context-based concept fusion
US7733956B1 (en) * 1996-12-17 2010-06-08 Oracle International Corporation Method and apparatus for storing base and additive streams of video
US7738550B2 (en) * 2000-03-13 2010-06-15 Sony Corporation Method and apparatus for generating compact transcoding hints metadata
US20100172591A1 (en) * 2007-05-25 2010-07-08 Masumi Ishikawa Image-sound segment corresponding apparatus, method and program
US7756338B2 (en) * 2007-02-14 2010-07-13 Mitsubishi Electric Research Laboratories, Inc. Method for detecting scene boundaries in genre independent videos
US7773813B2 (en) * 2005-10-31 2010-08-10 Microsoft Corporation Capture-intention detection for video content analysis
US7809192B2 (en) * 2005-05-09 2010-10-05 Like.Com System and method for recognizing objects from images and identifying relevancy amongst images and information
US7817855B2 (en) * 2005-09-02 2010-10-19 The Blindsight Corporation System and method for detecting text in real-world color images
US7860317B2 (en) * 2006-04-04 2010-12-28 Microsoft Corporation Generating search results based on duplicate image detection
US20110025710A1 (en) * 2008-04-10 2011-02-03 The Trustees Of Columbia University In The City Of New York Systems and methods for image archeology
US7884567B2 (en) * 2006-11-16 2011-02-08 Samsung Sdi Co., Ltd. Fuel cell system and method for controlling operation of the fuel cell system
US20110145232A1 (en) * 2008-06-17 2011-06-16 The Trustees Of Columbia University In The City Of New York System and method for dynamically and interactively searching media data
US7996762B2 (en) * 2007-09-21 2011-08-09 Microsoft Corporation Correlative multi-label image annotation
US8010296B2 (en) * 2002-12-19 2011-08-30 Drexel University Apparatus and method for removing non-discriminatory indices of an indexed dataset
US8019763B2 (en) * 2006-02-27 2011-09-13 Microsoft Corporation Propagating relevance from labeled documents to unlabeled documents
US20110255605A1 (en) * 1997-05-16 2011-10-20 Shih-Fu Chang Methods and architecture for indexing and editing compressed video over the world wide web
US20110314367A1 (en) * 2008-12-22 2011-12-22 The Trustees Of Columbia University In The City Of New York System And Method For Annotating And Searching Media
US8135221B2 (en) * 2009-10-07 2012-03-13 Eastman Kodak Company Video concept classification using audio-visual atoms
US8145677B2 (en) * 2007-03-27 2012-03-27 Faleh Jassem Al-Shameri Automated generation of metadata for mining image and text data
US20120089552A1 (en) * 2008-12-22 2012-04-12 Shih-Fu Chang Rapid image annotation via brain state decoding and visual pattern mining
US8332333B2 (en) * 2006-10-19 2012-12-11 Massachusetts Institute Of Technology Learning algorithm for ranking on graph data

Family Cites Families (223)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2141607A (en) 1983-06-15 1984-12-19 Philips Electronic Associated Video display system with index pages
JPS60204086A (en) 1984-03-28 1985-10-15 Fuji Electric Co Ltd Object discriminating device
US4649482A (en) 1984-08-31 1987-03-10 Bio-Logic Systems Corp. Brain electrical activity topographical mapping
US5579444A (en) 1987-08-28 1996-11-26 Axiom Bildverarbeitungssysteme Gmbh Adaptive vision-based controller
US5664177A (en) 1988-04-13 1997-09-02 Digital Equipment Corporation Data processing system having a data structure with a single, simple primitive
CA2001390C (en) 1988-12-19 1997-12-30 Ming-Chien Shan View composition in a data-base management system
US5144685A (en) 1989-03-31 1992-09-01 Honeywell Inc. Landmark recognition for autonomous mobile robots
US5682425A (en) 1990-04-23 1997-10-28 Canon Kabushiki Kaisha Information signal transmission system
FR2661585B1 (en) 1990-04-25 1992-08-07 Telediffusion Fse METHOD AND DEVICE FOR SCRAMBLING-SCRATCHING DIGITAL IMAGE DATA.
JPH04207878A (en) 1990-11-30 1992-07-29 Toshiba Corp Moving image management device
US5191645A (en) 1991-02-28 1993-03-02 Sony Corporation Of America Digital signal processing system employing icon displays
JP3177746B2 (en) 1991-03-20 2001-06-18 株式会社日立製作所 Data processing system and method
US5353391A (en) 1991-05-06 1994-10-04 Apple Computer, Inc. Method apparatus for transitioning between sequences of images
US5212742A (en) 1991-05-24 1993-05-18 Apple Computer, Inc. Method and apparatus for encoding/decoding image data
EP0528084B1 (en) 1991-08-15 1999-05-19 International Business Machines Corporation System and method for processing data representing stored images
US5546572A (en) 1991-08-28 1996-08-13 Hitachi, Ltd. Method for retrieving database of image information
US5701510A (en) 1991-11-14 1997-12-23 International Business Machines Corporation Method and system for efficient designation and retrieval of particular segments within a multimedia presentation utilizing a data processing system
US7006881B1 (en) 1991-12-23 2006-02-28 Steven Hoffberg Media recording device with remote graphic user interface
JPH0756652B2 (en) 1992-03-24 1995-06-14 インターナショナル・ビジネス・マシーンズ・コーポレイション Search for video frame sequence
US5623690A (en) 1992-06-03 1997-04-22 Digital Equipment Corporation Audio/video storage and retrieval for multimedia workstations by interleaving audio and video data in data file
US5262856A (en) 1992-06-04 1993-11-16 Massachusetts Institute Of Technology Video image compositing techniques
GB9215102D0 (en) 1992-07-16 1992-08-26 Philips Electronics Uk Ltd Tracking moving objects
US5721788A (en) 1992-07-31 1998-02-24 Corbis Corporation Method and system for digital image signatures
GB2270436A (en) 1992-09-05 1994-03-09 Ibm Target tracking system
JP2903904B2 (en) 1992-10-09 1999-06-14 松下電器産業株式会社 Image retrieval device
US5579471A (en) 1992-11-09 1996-11-26 International Business Machines Corporation Image query system and method
US5615112A (en) 1993-01-29 1997-03-25 Arizona Board Of Regents Synthesized object-oriented entity-relationship (SOOER) model for coupled knowledge-base/database of image retrieval expert system (IRES)
US5630121A (en) 1993-02-02 1997-05-13 International Business Machines Corporation Archiving and retrieving multimedia objects using structured indexes
US5408274A (en) 1993-03-11 1995-04-18 The Regents Of The University Of California Method and apparatus for compositing compressed video data
US5555354A (en) 1993-03-23 1996-09-10 Silicon Graphics Inc. Method and apparatus for navigation within three-dimensional information landscape
US5585852A (en) 1993-06-16 1996-12-17 Intel Corporation Processing video signals for scalable video playback using independently encoded component-plane bands
US5391199A (en) 1993-07-20 1995-02-21 Biosense, Inc. Apparatus and method for treating cardiac arrhythmias
US5619709A (en) 1993-09-20 1997-04-08 Hnc, Inc. System and method of context vector generation and retrieval
JP3303543B2 (en) 1993-09-27 2002-07-22 インターナショナル・ビジネス・マシーンズ・コーポレーション How to organize and play multimedia segments, and how to organize and play two or more multimedia stories as hyperstory
US6614914B1 (en) 1995-05-08 2003-09-02 Digimarc Corporation Watermark embedder and reader
US6122403A (en) 1995-07-27 2000-09-19 Digimarc Corporation Computer system linked by using information in data objects
US5600831A (en) 1994-02-28 1997-02-04 Lucent Technologies Inc. Apparatus and methods for retrieving information by modifying query plan based on description of information sources
US5555378A (en) 1994-03-02 1996-09-10 Bell Communications Research, Inc. Scheduling transmission multimedia information in broadband networks using a token passing scheme
US5668897A (en) 1994-03-15 1997-09-16 Stolfo; Salvatore J. Method and apparatus for imaging, image processing and data compression merge/purge techniques for document image databases
US5606655A (en) * 1994-03-31 1997-02-25 Siemens Corporate Research, Inc. Method for representing contents of a single video shot using frames
US5465353A (en) 1994-04-01 1995-11-07 Ricoh Company, Ltd. Image matching and retrieval by multi-access redundant hashing
US5488664A (en) 1994-04-22 1996-01-30 Yeda Research And Development Co., Ltd. Method and apparatus for protecting visual information with printed cryptographic watermarks
US5493677A (en) 1994-06-08 1996-02-20 Systems Research & Applications Corporation Generation, archiving, and retrieval of digital images with evoked suggestion-set captions and natural language interface
US5613032A (en) 1994-09-02 1997-03-18 Bell Communications Research, Inc. System and method for recording, playing back and searching multimedia events wherein video, audio and text can be searched and retrieved
US5642477A (en) 1994-09-22 1997-06-24 International Business Machines Corporation Method and apparatus for selectably retrieving and outputting digitally stored multimedia presentations with real-time non-interrupting, dynamically selectable introduction of output processing
US5802361A (en) 1994-09-30 1998-09-01 Apple Computer, Inc. Method and system for searching graphic images and videos
US5566089A (en) 1994-10-26 1996-10-15 General Instrument Corporation Of Delaware Syntax parser for a video decompression processor
DE69535098T2 (en) 1994-11-15 2007-04-19 Canon K.K. Method and apparatus for searching images in a database
WO1996017313A1 (en) 1994-11-18 1996-06-06 Oracle Corporation Method and apparatus for indexing multimedia information streams
WO1996017306A2 (en) 1994-11-21 1996-06-06 Oracle Corporation Media server
US5805733A (en) 1994-12-12 1998-09-08 Apple Computer, Inc. Method and system for detecting scenes and summarizing video sequences
US5646997A (en) 1994-12-14 1997-07-08 Barton; James M. Method and apparatus for embedding authentication information within digital data
US5530759A (en) 1995-02-01 1996-06-25 International Business Machines Corporation Color correct digital watermarking of images
US5821945A (en) 1995-02-03 1998-10-13 The Trustees Of Princeton University Method and apparatus for video browsing based on content and structure
US5794242A (en) 1995-02-07 1998-08-11 Digital Equipment Corporation Temporally and spatially organized database
US5943422A (en) 1996-08-12 1999-08-24 Intertrust Technologies Corp. Steganographic techniques for securely delivering electronic digital rights management control information over insecure communication channels
JPH08262965A (en) 1995-03-20 1996-10-11 Mitsubishi Electric Corp Closed caption decoder with pause function for language learning
JP3609488B2 (en) 1995-05-17 2005-01-12 株式会社日立製作所 Information processing system
US5721815A (en) 1995-06-07 1998-02-24 International Business Machines Corporation Media-on-demand communication system and method employing direct access storage device
US5684715A (en) 1995-06-07 1997-11-04 Canon Information Systems, Inc. Interactive video system with dynamic video object descriptors
JPH0916457A (en) 1995-06-28 1997-01-17 Fujitsu Ltd Multimedia data retrieval system
US5758076A (en) 1995-07-19 1998-05-26 International Business Machines Corporation Multimedia server system having rate adjustable data retrieval based on buffer capacity
US5822524A (en) 1995-07-21 1998-10-13 Infovalue Computing, Inc. System for just-in-time retrieval of multimedia files over computer networks by transmitting data packets at transmission rate determined by frame size
US5734893A (en) 1995-09-28 1998-03-31 Ibm Corporation Progressive content-based retrieval of image and video with adaptive and iterative refinement
US5969755A (en) 1996-02-05 1999-10-19 Texas Instruments Incorporated Motion based event detection system and method
US5949885A (en) 1996-03-12 1999-09-07 Leighton; F. Thomson Method for protecting content using watermarking
US5664018A (en) 1996-03-12 1997-09-02 Leighton; Frank Thomson Watermarking process resilient to collusion attacks
US5987459A (en) 1996-03-15 1999-11-16 Regents Of The University Of Minnesota Image and document management system for content-based retrieval
US5893095A (en) 1996-03-29 1999-04-06 Virage, Inc. Similarity engine for content-based retrieval of images
US5884298A (en) 1996-03-29 1999-03-16 Cygnet Storage Solutions, Inc. Method for accessing and updating a library of optical discs
US5767922A (en) * 1996-04-05 1998-06-16 Cornell Research Foundation, Inc. Apparatus and process for detecting scene breaks in a sequence of video frames
US5852435A (en) 1996-04-12 1998-12-22 Avid Technology, Inc. Digital multimedia editing and data management system
US5696964A (en) 1996-04-16 1997-12-09 Nec Research Institute, Inc. Multimedia database retrieval system which maintains a posterior probability distribution that each item in the database is a target of a search
US5870754A (en) * 1996-04-25 1999-02-09 Philips Electronics North America Corporation Video retrieval of MPEG compressed sequences using DC and motion signatures
JP3423835B2 (en) 1996-05-01 2003-07-07 沖電気工業株式会社 Compression encoding device with scramble and decompression reproduction device thereof
MY119560A (en) 1996-05-27 2005-06-30 Nippon Telegraph & Telephone Scheme for detecting captions in coded video data without decoding coded video data
US6078664A (en) 1996-12-20 2000-06-20 Moskowitz; Scott A. Z-transform implementation of digital watermarks
US6154755A (en) 1996-07-31 2000-11-28 Eastman Kodak Company Index imaging system
US6031914A (en) 1996-08-30 2000-02-29 Regents Of The University Of Minnesota Method and apparatus for embedding data, including watermarks, in human perceptible images
US6282299B1 (en) 1996-08-30 2001-08-28 Regents Of The University Of Minnesota Method and apparatus for video watermarking using perceptual masks
US5848155A (en) 1996-09-04 1998-12-08 Nec Research Institute, Inc. Spread spectrum watermark for embedded signalling
US5809139A (en) 1996-09-13 1998-09-15 Vivo Software, Inc. Watermarking method and apparatus for compressed digital video
US5873080A (en) 1996-09-20 1999-02-16 International Business Machines Corporation Using multiple search engines to search multimedia data
US5734752A (en) 1996-09-24 1998-03-31 Xerox Corporation Digital watermarking using stochastic screen patterns
US6075875A (en) * 1996-09-30 2000-06-13 Microsoft Corporation Segmentation of image features using hierarchical analysis of multi-valued image data and weighted averaging of segmentation results
US5852823A (en) 1996-10-16 1998-12-22 Microsoft Image classification and retrieval system using a query-by-example paradigm
US5825892A (en) 1996-10-28 1998-10-20 International Business Machines Corporation Protecting images with an image watermark
US5915027A (en) 1996-11-05 1999-06-22 Nec Research Institute Digital watermarking
US6141447A (en) 1996-11-21 2000-10-31 C-Cube Microsystems, Inc. Compressed video transcoder
US6172675B1 (en) 1996-12-05 2001-01-09 Interval Research Corporation Indirect manipulation of data using temporally related data, with particular application to manipulation of audio or audiovisual data
JP3106985B2 (en) 1996-12-25 2000-11-06 日本電気株式会社 Electronic watermark insertion device and detection device
US5963203A (en) 1997-07-03 1999-10-05 Obvious Technology, Inc. Interactive video icon with designated viewing position
US5790703A (en) 1997-01-21 1998-08-04 Xerox Corporation Digital watermarking using conjugate halftone screens
US5901245A (en) 1997-01-23 1999-05-04 Eastman Kodak Company Method and system for detection and characterization of open space in digital images
TR199700058A3 (en) 1997-01-29 1998-08-21 Onural Levent Moving object segmentation based on rules.
US6157746A (en) 1997-02-12 2000-12-05 Sarnoff Corporation Apparatus and method for encoding wavelet trees generated by a wavelet-based coding method
AUPO525497A0 (en) 1997-02-21 1997-03-20 Mills, Dudley John Network-based classified information systems
US5930783A (en) 1997-02-21 1999-07-27 Nec Usa, Inc. Semantic and cognition based image retrieval
JP3736706B2 (en) 1997-04-06 2006-01-18 ソニー株式会社 Image display apparatus and method
WO1998046005A2 (en) 1997-04-07 1998-10-15 At & T Corp. System and method for processing object-based audiovisual information
US5937422A (en) 1997-04-15 1999-08-10 The United States Of America As Represented By The National Security Agency Automatically generating a topic description for text and searching and sorting text by topic using the same
JP3485438B2 (en) 1997-04-16 2004-01-13 シャープ株式会社 Electronic calculator with graph display function
US6741655B1 (en) 1997-05-05 2004-05-25 The Trustees Of Columbia University In The City Of New York Algorithms and system for object-oriented content-based video search
US6125229A (en) * 1997-06-02 2000-09-26 Philips Electronics North America Corporation Visual indexing system
US6131162A (en) 1997-06-05 2000-10-10 Hitachi Ltd. Digital data authentication method
US5960081A (en) 1997-06-05 1999-09-28 Cray Research, Inc. Embedding a digital signature in a video sequence
US6222932B1 (en) 1997-06-27 2001-04-24 International Business Machines Corporation Automatic adjustment of image watermark strength based on computed image texture
US5983218A (en) 1997-06-30 1999-11-09 Xerox Corporation Multimedia database for use over networks
US6195458B1 (en) 1997-07-29 2001-02-27 Eastman Kodak Company Method for content-based temporal segmentation of video
US6360234B2 (en) 1997-08-14 2002-03-19 Virage, Inc. Video cataloger system with synchronized encoders
US6208735B1 (en) 1997-09-10 2001-03-27 Nec Research Institute, Inc. Secure spread spectrum watermarking for multimedia data
US6886013B1 (en) 1997-09-11 2005-04-26 International Business Machines Corporation HTTP caching proxy to filter and control display of data in a web browser
US6249318B1 (en) 1997-09-12 2001-06-19 8×8, Inc. Video coding/decoding arrangement and method therefor
US6108434A (en) 1997-09-12 2000-08-22 Signafy, Inc. Counteracting geometric distortions for DCT based watermarking
US6807231B1 (en) 1997-09-12 2004-10-19 8×8, Inc. Multi-hypothesis motion-compensated video image predictor
US5995978A (en) 1997-09-24 1999-11-30 Ricoh Company, Ltd. Navigation system for document image database
US6070228A (en) 1997-09-30 2000-05-30 International Business Machines Corp. Multimedia data storage system and method for operating a media server as a cache device and controlling a volume of data in the media server based on user-defined parameters
US7154560B1 (en) 1997-10-27 2006-12-26 Shih-Fu Chang Watermarking of digital image data
WO1999022318A1 (en) 1997-10-27 1999-05-06 Massachusetts Institute Of Technology Image search and retrieval system
JPH11196386A (en) 1997-10-30 1999-07-21 Toshiba Corp Computer system and closed caption display method
US6100930A (en) * 1997-12-19 2000-08-08 Thomson Consumer Electronics, Inc. Process and apparatus for performing wipes on compressed MPEG video bitstreams
US5956026A (en) * 1997-12-19 1999-09-21 Sharp Laboratories Of America, Inc. Method for hierarchical summarization and browsing of digital video
US6037984A (en) 1997-12-24 2000-03-14 Sarnoff Corporation Method and apparatus for embedding a watermark into a digital image or image sequence
US6208745B1 (en) 1997-12-30 2001-03-27 Sarnoff Corporation Method and apparatus for imbedding a watermark into a bitstream representation of a digital image sequence
US6654931B1 (en) 1998-01-27 2003-11-25 At&T Corp. Systems and methods for playing, browsing and interacting with MPEG-4 coded audio-visual objects
JP3738939B2 (en) 1998-03-05 2006-01-25 Kddi株式会社 Moving image cut point detection device
US6628824B1 (en) 1998-03-20 2003-09-30 Ken Belanger Method and apparatus for image identification and comparison
US6064764A (en) 1998-03-30 2000-05-16 Seiko Epson Corporation Fragile watermarks for detecting tampering in images
US6092072A (en) 1998-04-07 2000-07-18 Lucent Technologies, Inc. Programmed medium for clustering large databases
US6240424B1 (en) 1998-04-22 2001-05-29 Nbc Usa, Inc. Method and system for similarity-based image classification
US6104441A (en) * 1998-04-29 2000-08-15 Hewlett Packard Company System for editing compressed image sequences
US6504941B2 (en) 1998-04-30 2003-01-07 Hewlett-Packard Company Method and apparatus for digital watermarking of images
US6487301B1 (en) 1998-04-30 2002-11-26 Mediasec Technologies Llc Digital authentication with digital and analog documents
US7444308B2 (en) 2001-06-15 2008-10-28 Health Discovery Corporation Data mining platform for bioinformatics and other knowledge discovery
CA2269651A1 (en) 1998-05-12 1999-11-12 Lucent Technologies, Inc. Transform domain image watermarking method and system
US6581058B1 (en) 1998-05-22 2003-06-17 Microsoft Corporation Scalable system for clustering of large databases having mixed data attributes
US6081278A (en) 1998-06-11 2000-06-27 Chen; Shenchang Eric Animation object having multiple resolution format
US6178416B1 (en) 1998-06-15 2001-01-23 James U. Parker Method and apparatus for knowledgebase searching
US6285995B1 (en) 1998-06-22 2001-09-04 U.S. Philips Corporation Image retrieval system using a query image
US6476814B1 (en) 1998-06-25 2002-11-05 Wordgraph, Inc. Display structure for representation of complex systems
KR20010034920A (en) 1998-06-26 2001-04-25 매클린토크 샤운 엘 Terminal for composing and presenting mpeg-4 video programs
US6041079A (en) * 1998-06-30 2000-03-21 Thomson Consumer Electronics, Inc, Field/frame conversion of DCT domain mixed field/frame mode macroblocks using 1-dimensional DCT/IDCT
US6480538B1 (en) 1998-07-08 2002-11-12 Koninklijke Philips Electronics N.V. Low bandwidth encoding scheme for video transmission
CN1147079C (en) 1998-07-17 2004-04-21 皇家菲利浦电子有限公司 Device for demultiplexing coded data
US6714909B1 (en) 1998-08-13 2004-03-30 At&T Corp. System and method for automated multimedia content indexing and retrieval
US6226618B1 (en) 1998-08-13 2001-05-01 International Business Machines Corporation Electronic content delivery system
US6346124B1 (en) 1998-08-25 2002-02-12 University Of Florida Autonomous boundary detection system for echocardiographic images
US6708055B2 (en) 1998-08-25 2004-03-16 University Of Florida Method for automated analysis of apical four-chamber images of the heart
US6167084A (en) 1998-08-27 2000-12-26 Motorola, Inc. Dynamic bit allocation for statistical multiplexing of compressed and uncompressed digital video signals
JP3563270B2 (en) 1998-08-28 2004-09-08 株式会社日立製作所 Authentication method of digital watermark image
US6970602B1 (en) 1998-10-06 2005-11-29 International Business Machines Corporation Method and apparatus for transcoding multimedia using content analysis
US6185329B1 (en) 1998-10-13 2001-02-06 Hewlett-Packard Company Automatic caption text detection and processing for digital images
US6549911B2 (en) 1998-11-02 2003-04-15 Survivors Of The Shoah Visual History Foundation Method and apparatus for cataloguing multimedia data
US6385602B1 (en) 1998-11-03 2002-05-07 E-Centives, Inc. Presentation of search results using dynamic categorization
WO2000028440A1 (en) 1998-11-06 2000-05-18 The Trustees Of Columbia University In The City Ofnew York Systems and methods for interoperable multimedia content descriptions
US7254285B1 (en) 1998-11-06 2007-08-07 Seungup Paek Image description system and method
US6252975B1 (en) * 1998-12-17 2001-06-26 Xerox Corporation Method and system for real time feature based motion analysis for key frame selection from a video
US6678389B1 (en) 1998-12-29 2004-01-13 Kent Ridge Digital Labs Method and apparatus for embedding digital information in digital multimedia data
US6327390B1 (en) * 1999-01-14 2001-12-04 Mitsubishi Electric Research Laboratories, Inc. Methods of scene fade detection for indexing of video sequences
US6532541B1 (en) 1999-01-22 2003-03-11 The Trustees Of Columbia University In The City Of New York Method and apparatus for image authentication
US6643387B1 (en) 1999-01-28 2003-11-04 Sarnoff Corporation Apparatus and method for context-based indexing and retrieval of image sequences
US6366701B1 (en) 1999-01-28 2002-04-02 Sarnoff Corporation Apparatus and method for describing the motion parameters of an object in an image sequence
US6223183B1 (en) 1999-01-29 2001-04-24 International Business Machines Corporation System and method for describing views in space, time, frequency, and resolution
US7185049B1 (en) 1999-02-01 2007-02-27 At&T Corp. Multimedia integration description scheme, method and system for MPEG-7
US6941325B1 (en) 1999-02-01 2005-09-06 The Trustees Of Columbia University Multimedia archive description scheme
US6236395B1 (en) 1999-02-01 2001-05-22 Sharp Laboratories Of America, Inc. Audiovisual information management system
US6556695B1 (en) 1999-02-05 2003-04-29 Mayo Foundation For Medical Education And Research Method for producing high resolution real-time images, of structure and function during medical procedures
WO2000049797A1 (en) 1999-02-16 2000-08-24 Koninklijke Philips Electronics N.V. Authentication and verification within a digital camera architecture
US6556958B1 (en) 1999-04-23 2003-04-29 Microsoft Corporation Fast clustering with sparse data
US6847980B1 (en) 1999-07-03 2005-01-25 Ana B. Benitez Fundamental entity-relationship models for the generic audio visual data signal description
US6393394B1 (en) 1999-07-19 2002-05-21 Qualcomm Incorporated Method and apparatus for interleaving line spectral information quantization methods in a speech coder
US6546135B1 (en) 1999-08-30 2003-04-08 Mitsubishi Electric Research Laboratories, Inc Method for representing and comparing multimedia content
US7072398B2 (en) 2000-12-06 2006-07-04 Kai-Kuang Ma System and method for motion vector generation and analysis of digital video clips
US6725372B1 (en) 1999-12-02 2004-04-20 Verizon Laboratories Inc. Digital watermarking
DE19960372A1 (en) 1999-12-14 2001-06-21 Definiens Ag Process for processing data structures
US6282300B1 (en) 2000-01-21 2001-08-28 Signafy, Inc. Rotation, scale, and translation resilient public watermarking for images using a log-polar fourier transform
AU2001238354A1 (en) 2000-02-18 2001-08-27 The University Of Maryland Methods for the electronic annotation, retrieval, and use of electronic images
KR20010087552A (en) * 2000-03-07 2001-09-21 구자홍 Dissolve/fade detection method for mpeg-compressed video using spatio-temporal distribution of the macro blocks
US6701309B1 (en) 2000-04-21 2004-03-02 Lycos, Inc. Method and system for collecting related queries
US6567805B1 (en) 2000-05-15 2003-05-20 International Business Machines Corporation Interactive automated response system
US20020157116A1 (en) 2000-07-28 2002-10-24 Koninklijke Philips Electronics N.V. Context and content based information processing for multimedia segmentation and indexing
US8538770B2 (en) 2000-08-01 2013-09-17 Logical Images, Inc. System and method to aid diagnoses using cross-referenced knowledge and image databases
US6714683B1 (en) 2000-08-24 2004-03-30 Digimarc Corporation Wavelet based feature modulation watermarks and related applications
WO2002025588A2 (en) 2000-09-21 2002-03-28 Md Online Inc. Medical image processing systems
DE10048029A1 (en) 2000-09-26 2002-04-25 Philips Corp Intellectual Pty Procedure for calculating a transformation connecting two images
US6904408B1 (en) 2000-10-19 2005-06-07 Mccarthy John Bionet method, system and personalized web content manager responsive to browser viewers' psychological preferences, behavioral responses and physiological stress indicators
US7398275B2 (en) 2000-10-20 2008-07-08 Sony Corporation Efficient binary coding scheme for multimedia content descriptions
US6879703B2 (en) 2001-01-10 2005-04-12 Trustees Of Columbia University Of The City Of New York Method and apparatus for watermarking images
US20020124104A1 (en) 2001-03-01 2002-09-05 Yigal Rappaport Network element and a method for preventing a disorder of a sequence of data packets traversing the network
US6915241B2 (en) 2001-04-20 2005-07-05 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Method for segmentation and identification of nonstationary time series
US20020169771A1 (en) 2001-05-09 2002-11-14 Melmon Kenneth L. System & method for facilitating knowledge management
WO2003019416A1 (en) 2001-08-28 2003-03-06 Koninklijke Philips Electronics N.V. Automatic question formulation from a user selection in multimedia content
KR100959470B1 (en) 2002-03-22 2010-05-25 마이클 에프. 디어링 Scalable high performance 3d graphics
US7043474B2 (en) 2002-04-15 2006-05-09 International Business Machines Corporation System and method for measuring image similarity based on semantic meaning
US20030229278A1 (en) 2002-06-06 2003-12-11 Usha Sinha Method and system for knowledge extraction from image data
JP3944016B2 (en) 2002-07-18 2007-07-11 パロマ工業株式会社 rice cooker
JP3988599B2 (en) 2002-09-20 2007-10-10 富士ゼロックス株式会社 Image processing method, falsification detection method, image processing apparatus, falsification detection apparatus, image processing program, falsification detection program, and image forming medium
US7103225B2 (en) 2002-11-07 2006-09-05 Honda Motor Co., Ltd. Clustering appearances of objects under varying illumination conditions
JP4112968B2 (en) 2002-12-26 2008-07-02 富士通株式会社 Video text processing device
US7194035B2 (en) 2003-01-08 2007-03-20 Apple Computer, Inc. Method and apparatus for improved coding mode selection
US7403302B2 (en) 2003-08-06 2008-07-22 Hewlett-Packard Development Company, L.P. Method and a system for indexing and tracking digital images
US7603370B2 (en) 2004-03-22 2009-10-13 Microsoft Corporation Method for duplicate detection and suppression
CN101031882B (en) 2004-06-08 2010-09-08 达尔特设备互操作有限公司 Architecture, apparatus and method for device team recruitment and content renditioning for universal device interoperability platform
US7590589B2 (en) 2004-09-10 2009-09-15 Hoffberg Steven M Game theoretic prioritization scheme for mobile ad hoc networks permitting hierarchal deference
US7551780B2 (en) 2005-08-23 2009-06-23 Ricoh Co., Ltd. System and method for using individualized mixed document
US7812986B2 (en) 2005-08-23 2010-10-12 Ricoh Co. Ltd. System and methods for use of voice mail and email in a mixed media environment
US8156427B2 (en) 2005-08-23 2012-04-10 Ricoh Co. Ltd. User interface for mixed media reality
WO2006096612A2 (en) 2005-03-04 2006-09-14 The Trustees Of Columbia University In The City Of New York System and method for motion estimation and mode decision for low-complexity h.264 decoder
US8732025B2 (en) 2005-05-09 2014-05-20 Google Inc. System and method for enabling image recognition and searching of remote content on display
JP2008546232A (en) 2005-05-11 2008-12-18 プラネツトワイド・ゲームズ・インコーポレイテツド Creating publications that use game-based media content
US7801893B2 (en) 2005-09-30 2010-09-21 Iac Search & Media, Inc. Similarity detection and clustering of images
US8874477B2 (en) 2005-10-04 2014-10-28 Steven Mark Hoffberg Multifactorial optimization system and method
US7644373B2 (en) 2006-01-23 2010-01-05 Microsoft Corporation User interface for viewing clusters of images
US7439989B2 (en) 2006-02-17 2008-10-21 Microsoft Corporation Detecting doctored JPEG images
US7949186B2 (en) 2006-03-15 2011-05-24 Massachusetts Institute Of Technology Pyramid match kernel and related techniques
US7707481B2 (en) 2006-05-16 2010-04-27 Pitney Bowes Inc. System and method for efficient uncorrectable error detection in flash memory
US8230457B2 (en) 2007-03-07 2012-07-24 The Nielsen Company (Us), Llc. Method and system for using coherence of biological responses as a measure of performance of a media
US8520949B1 (en) 2008-06-20 2013-08-27 Google Inc. Self-similar descriptor filtering
EP2353251B1 (en) 2008-11-21 2012-10-17 Nero Ag Apparatus for verifying and for generating an encrypted token and methods for same
US8375032B2 (en) 2009-06-25 2013-02-12 University Of Tennessee Research Foundation Method and apparatus for predicting object properties and events using similarity-based information retrieval and modeling
US8725766B2 (en) 2010-03-25 2014-05-13 Rovi Technologies Corporation Searching text and other types of content by using a frequency domain
US8788434B2 (en) 2010-10-28 2014-07-22 Google Inc. Search with joint image-audio queries
US8854973B2 (en) 2012-08-29 2014-10-07 International Business Machines Corporation Sliced routing table management with replication

Patent Citations (74)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5694334A (en) * 1994-09-08 1997-12-02 Starguide Digital Networks, Inc. Method and apparatus for electronic distribution of digital multi-media information
US6356309B1 (en) * 1995-08-02 2002-03-12 Matsushita Electric Industrial Co., Ltd. Video coding device and video transmission system using the same, quantization control method and average throughput calculation method used therein
US6157745A (en) * 1995-09-06 2000-12-05 U.S. Philips Corporation Method and system for coding and decoding a sequence of segmented images
US5835163A (en) * 1995-12-21 1998-11-10 Siemens Corporate Research, Inc. Apparatus for detecting a cut in a video
US6526099B1 (en) * 1996-10-25 2003-02-25 Telefonaktiebolaget Lm Ericsson (Publ) Transcoder
US6028635A (en) * 1996-12-03 2000-02-22 Stmicroelectronics, Inc. Reducing the memory required for decompression by storing compressed information using DCT based techniques
US7733956B1 (en) * 1996-12-17 2010-06-08 Oracle International Corporation Method and apparatus for storing base and additive streams of video
US6058205A (en) * 1997-01-09 2000-05-02 International Business Machines Corporation System and method for partitioning the feature space of a classifier in a pattern classification system
US20110255605A1 (en) * 1997-05-16 2011-10-20 Shih-Fu Chang Methods and architecture for indexing and editing compressed video over the world wide web
US6070167A (en) * 1997-09-29 2000-05-30 Sharp Laboratories Of America, Inc. Hierarchical method and system for object-based audiovisual descriptive tagging of images for information retrieval, editing, and manipulation
US6072542A (en) * 1997-11-25 2000-06-06 Fuji Xerox Co., Ltd. Automatic video segmentation using hidden markov model
US6366314B1 (en) * 1997-12-17 2002-04-02 Telediffusion De France Method and system for measuring the quality of digital television signals
US6332030B1 (en) * 1998-01-15 2001-12-18 The Regents Of The University Of California Method for embedding and extracting digital data in images and video
US6442538B1 (en) * 1998-05-27 2002-08-27 Hitachi, Ltd. Video information retrieval method and apparatus
US6275599B1 (en) * 1998-08-28 2001-08-14 International Business Machines Corporation Compressed image authentication and verification
US20070245400A1 (en) * 1998-11-06 2007-10-18 Seungyup Paek Video description system and method
US6404925B1 (en) * 1999-03-11 2002-06-11 Fuji Xerox Co., Ltd. Methods and apparatuses for segmenting an audio-visual recording using image similarity searching and audio speaker recognition
US6816836B2 (en) * 1999-08-06 2004-11-09 International Business Machines Corporation Method and apparatus for audio-visual speech detection and recognition
US6339450B1 (en) * 1999-09-21 2002-01-15 At&T Corp Error resilient transcoding for video over wireless channels
US6606393B1 (en) * 1999-12-02 2003-08-12 Verizon Laboratories Inc. Message authentication code using image histograms
US7496830B2 (en) * 1999-12-07 2009-02-24 Microsoft Corporation Computer user interface architecture that saves a user's non-linear navigation history and intelligently maintains that history
US7093028B1 (en) * 1999-12-15 2006-08-15 Microsoft Corporation User and content aware object-based data stream transmission methods and arrangements
US6385329B1 (en) * 2000-02-14 2002-05-07 Digimarc Corporation Wavelet domain watermarks
US7738550B2 (en) * 2000-03-13 2010-06-15 Sony Corporation Method and apparatus for generating compact transcoding hints metadata
US20020118748A1 (en) * 2000-06-27 2002-08-29 Hideki Inomata Picture coding apparatus, and picture coding method
US20070038612A1 (en) * 2000-07-24 2007-02-15 Sanghoon Sull System and method for indexing, searching, identifying, and editing multimedia files
US20070033170A1 (en) * 2000-07-24 2007-02-08 Sanghoon Sull Method For Searching For Relevant Multimedia Content
US7624337B2 (en) * 2000-07-24 2009-11-24 Vmark, Inc. System and method for indexing, searching, identifying, and editing portions of electronic multimedia files
US20110093492A1 (en) * 2000-07-24 2011-04-21 Sanghoon Sull System and Method for Indexing, Searching, Identifying, and Editing Multimedia Files
US7409144B2 (en) * 2000-12-07 2008-08-05 Sony United Kingdom Limited Video and audio information processing
US6792434B2 (en) * 2001-04-20 2004-09-14 Mitsubishi Electric Research Laboratories, Inc. Content-based visualization and user-modeling for interactive browsing and retrieval in multimedia databases
US20040210819A1 (en) * 2001-06-15 2004-10-21 Alonso Antonio Used Dynamic browser interface
US7145946B2 (en) * 2001-07-27 2006-12-05 Sony Corporation MPEG video drift reduction
US20080303942A1 (en) * 2001-12-06 2008-12-11 Shih-Fu Chang System and method for extracting text captions from video and generating video summaries
US6700935B2 (en) * 2002-02-08 2004-03-02 Sony Electronics, Inc. Stream based bitrate transcoder for MPEG coded video
US8218617B2 (en) * 2002-04-26 2012-07-10 The Trustees Of Columbia University In The City Of New York Method and system for optimal video transcoding based on utility function descriptors
US20090316778A1 (en) * 2002-04-26 2009-12-24 Jae-Gon Kim Method And System For Optimal Video Transcoding Based On Utility Function Descriptors
US20050238238A1 (en) * 2002-07-19 2005-10-27 Li-Qun Xu Method and system for classification of semantic content of audio/video data
US8010296B2 (en) * 2002-12-19 2011-08-30 Drexel University Apparatus and method for removing non-discriminatory indices of an indexed dataset
US7676820B2 (en) * 2003-01-06 2010-03-09 Koninklijke Philips Electronics N.V. Method and apparatus for similar video content hopping
US7327885B2 (en) * 2003-06-30 2008-02-05 Mitsubishi Electric Research Laboratories, Inc. Method for detecting short term unusual events in videos
US20080298464A1 (en) * 2003-09-03 2008-12-04 Thompson Licensing S.A. Process and Arrangement for Encoding Video Pictures
US7636662B2 (en) * 2003-09-30 2009-12-22 Koninklijke Philips Electronics N.V. System and method for audio-visual content synthesis
US7313269B2 (en) * 2003-12-12 2007-12-25 Mitsubishi Electric Research Laboratories, Inc. Unsupervised learning of video structures in videos using hierarchical statistical models to detect events
US7406409B2 (en) * 2004-01-14 2008-07-29 Mitsubishi Electric Research Laboratories, Inc. System and method for recording and reproducing multimedia based on an audio signal
US20080193016A1 (en) * 2004-02-06 2008-08-14 Agency For Science, Technology And Research Automatic Video Event Detection and Indexing
US7574409B2 (en) * 2004-11-04 2009-08-11 Vericept Corporation Method, apparatus, and system for clustering and classification
US7519217B2 (en) * 2004-11-23 2009-04-14 Microsoft Corporation Method and system for generating a classifier using inter-sample relationships
US7308443B1 (en) * 2004-12-23 2007-12-11 Ricoh Company, Ltd. Techniques for video retrieval based on HMM similarity
US7386806B2 (en) * 2005-01-05 2008-06-10 Hillcrest Laboratories, Inc. Scaling and layout methods and systems for handling one-to-many objects
US7653264B2 (en) * 2005-03-04 2010-01-26 The Regents Of The University Of Michigan Method of determining alignment of images in high dimensional feature space
US20060224532A1 (en) * 2005-03-09 2006-10-05 Case Western Reserve University Iterative feature weighting with neural networks
US7809192B2 (en) * 2005-05-09 2010-10-05 Like.Com System and method for recognizing objects from images and identifying relevancy amongst images and information
US7817855B2 (en) * 2005-09-02 2010-10-19 The Blindsight Corporation System and method for detecting text in real-world color images
US7773813B2 (en) * 2005-10-31 2010-08-10 Microsoft Corporation Capture-intention detection for video content analysis
US8019763B2 (en) * 2006-02-27 2011-09-13 Microsoft Corporation Propagating relevance from labeled documents to unlabeled documents
US7860317B2 (en) * 2006-04-04 2010-12-28 Microsoft Corporation Generating search results based on duplicate image detection
US7720851B2 (en) * 2006-05-16 2010-05-18 Eastman Kodak Company Active context-based concept fusion
US20080055479A1 (en) * 2006-09-01 2008-03-06 Texas Instruments Incorporated Color Space Appearance Model Video Processor
US8332333B2 (en) * 2006-10-19 2012-12-11 Massachusetts Institute Of Technology Learning algorithm for ranking on graph data
US7884567B2 (en) * 2006-11-16 2011-02-08 Samsung Sdi Co., Ltd. Fuel cell system and method for controlling operation of the fuel cell system
US7756338B2 (en) * 2007-02-14 2010-07-13 Mitsubishi Electric Research Laboratories, Inc. Method for detecting scene boundaries in genre independent videos
US8145677B2 (en) * 2007-03-27 2012-03-27 Faleh Jassem Al-Shameri Automated generation of metadata for mining image and text data
US20100172591A1 (en) * 2007-05-25 2010-07-08 Masumi Ishikawa Image-sound segment corresponding apparatus, method and program
US20090055094A1 (en) * 2007-06-07 2009-02-26 Sony Corporation Navigation device and nearest point search method
US20080304743A1 (en) * 2007-06-11 2008-12-11 Microsoft Corporation Active segmentation for groups of images
US7996762B2 (en) * 2007-09-21 2011-08-09 Microsoft Corporation Correlative multi-label image annotation
US20090132561A1 (en) * 2007-11-21 2009-05-21 At&T Labs, Inc. Link-based classification of graph nodes
US20110025710A1 (en) * 2008-04-10 2011-02-03 The Trustees Of Columbia University In The City Of New York Systems and methods for image archeology
US20110145232A1 (en) * 2008-06-17 2011-06-16 The Trustees Of Columbia University In The City Of New York System and method for dynamically and interactively searching media data
US20100082614A1 (en) * 2008-09-22 2010-04-01 Microsoft Corporation Bayesian video search reranking
US20110314367A1 (en) * 2008-12-22 2011-12-22 The Trustees Of Columbia University In The City Of New York System And Method For Annotating And Searching Media
US20120089552A1 (en) * 2008-12-22 2012-04-12 Shih-Fu Chang Rapid image annotation via brain state decoding and visual pattern mining
US8135221B2 (en) * 2009-10-07 2012-03-13 Eastman Kodak Company Video concept classification using audio-visual atoms

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9330722B2 (en) 1997-05-16 2016-05-03 The Trustees Of Columbia University In The City Of New York Methods and architecture for indexing and editing compressed video over the world wide web
US20070245400A1 (en) * 1998-11-06 2007-10-18 Seungyup Paek Video description system and method
US8370869B2 (en) 1998-11-06 2013-02-05 The Trustees Of Columbia University In The City Of New York Video description system and method
US20080303942A1 (en) * 2001-12-06 2008-12-11 Shih-Fu Chang System and method for extracting text captions from video and generating video summaries
US8488682B2 (en) 2001-12-06 2013-07-16 The Trustees Of Columbia University In The City Of New York System and method for extracting text captions from video and generating video summaries
US20090316778A1 (en) * 2002-04-26 2009-12-24 Jae-Gon Kim Method And System For Optimal Video Transcoding Based On Utility Function Descriptors
US20080181308A1 (en) * 2005-03-04 2008-07-31 Yong Wang System and method for motion estimation and mode decision for low-complexity h.264 decoder
US9060175B2 (en) 2005-03-04 2015-06-16 The Trustees Of Columbia University In The City Of New York System and method for motion estimation and mode decision for low-complexity H.264 decoder
US8849058B2 (en) 2008-04-10 2014-09-30 The Trustees Of Columbia University In The City Of New York Systems and methods for image archaeology
US20110025710A1 (en) * 2008-04-10 2011-02-03 The Trustees Of Columbia University In The City Of New York Systems and methods for image archeology
US8364673B2 (en) 2008-06-17 2013-01-29 The Trustees Of Columbia University In The City Of New York System and method for dynamically and interactively searching media data
US20110145232A1 (en) * 2008-06-17 2011-06-16 The Trustees Of Columbia University In The City Of New York System and method for dynamically and interactively searching media data
US8671069B2 (en) 2008-12-22 2014-03-11 The Trustees Of Columbia University, In The City Of New York Rapid image annotation via brain state decoding and visual pattern mining
US9665824B2 (en) 2008-12-22 2017-05-30 The Trustees Of Columbia University In The City Of New York Rapid image annotation via brain state decoding and visual pattern mining

Also Published As

Publication number Publication date
US9330722B2 (en) 2016-05-03
US6735253B1 (en) 2004-05-11
US20080131072A1 (en) 2008-06-05
US20110255605A1 (en) 2011-10-20
US7817722B2 (en) 2010-10-19

Similar Documents

Publication Publication Date Title
US9330722B2 (en) Methods and architecture for indexing and editing compressed video over the world wide web
Meng et al. CVEPS-a compressed video editing and parsing system
WO1998052356A1 (en) Methods and architecture for indexing and editing compressed video over the world wide web
US7046731B2 (en) Extracting key frames from a video sequence
US7469010B2 (en) Extracting key frames from a video sequence
JP3719933B2 (en) Hierarchical digital video summary and browsing method and apparatus
Meng et al. Scene change detection in an MPEG-compressed video sequence
Meng et al. Tools for compressed-domain video indexing and editing
US7823055B2 (en) System and method for indexing, searching, identifying, and editing multimedia files
US6501794B1 (en) System and related methods for analyzing compressed media content
US20080043848A1 (en) Video/audio signal processing method and video/audio signal processing apparatus
JP2001313956A (en) Hierarchical mixed type shot conversion detecting method in mpeg compression video environment
Yeo Efficient processing of compressed images and video
Faernando et al. Scene change detection algorithms for content-based video indexing and retrieval
Nakajima A video browsing using fast scene cut detection for an efficient networked video database access
Schonfeld et al. Vortex: video retrieval and tracking from compressed multimedia databases—multiple object tracking from mpeg-2 bit stream
Yeo On fast microscopic browsing of MPEG-compressed video
Gu et al. Dissolve detection in MPEG compressed video
Meng et al. A distributed system for editing and browsing compressed video over the network
Kaup et al. Video analysis for universal multimedia messaging
Meng et al. Searching and editing MPEG-compressed video in a distributed online environment
AU762791B2 (en) Extracting key frames from a video sequence
Steiger et al. MPEG-7 description of generic video objects for scene reconstruction
Yi et al. A motion-based scene tree for compressed video content management
Mithlesh et al. Detection of Scene Change in Video

Legal Events

Date Code Title Description
AS Assignment

Owner name: THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHANG, SHIH-FU;MENG, HORACE J.;SIGNING DATES FROM 20101012 TO 20101013;REEL/FRAME:025399/0927

AS Assignment

Owner name: NATIONAL SCIENCE FOUNDATION, VIRGINIA

Free format text: CONFIRMATORY LICENSE;ASSIGNOR:COLUMBIA UNIVERSITY NEW YORK MORNINGSIDE;REEL/FRAME:025549/0546

Effective date: 20101210

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION