WO2003105489A1 - Procede et dispositif de compression video et d'indexage video semantique dynamique en ligne - Google Patents

Procede et dispositif de compression video et d'indexage video semantique dynamique en ligne Download PDF

Info

Publication number
WO2003105489A1
WO2003105489A1 PCT/US2002/018231 US0218231W WO03105489A1 WO 2003105489 A1 WO2003105489 A1 WO 2003105489A1 US 0218231 W US0218231 W US 0218231W WO 03105489 A1 WO03105489 A1 WO 03105489A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
portions
buffered
video portions
video data
Prior art date
Application number
PCT/US2002/018231
Other languages
English (en)
Inventor
Liu Tiecheng
John R. Kender
Original Assignee
The Trustees Of Columbia University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by The Trustees Of Columbia University filed Critical The Trustees Of Columbia University
Priority to PCT/US2002/018231 priority Critical patent/WO2003105489A1/fr
Publication of WO2003105489A1 publication Critical patent/WO2003105489A1/fr
Priority to US10/987,646 priority patent/US7489727B2/en
Priority to US12/349,786 priority patent/US8391355B2/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/20—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/71—Indexing; Data structures therefor; Storage structures

Definitions

  • the present invention relates to video data compression and indexing and, more particularly, to a semantic compression and indexing of video data.
  • Video compression and indexing are crucial in multimedia applications.
  • a number of video compression and indexing techniques have been developed.
  • One exemplary technique is a key frame selection technique, which selects key frames as indices for video data. The indices are then used for browsing, searching, retrieval and comparison of video data.
  • the key frame selection techniques are based on video segmentation, frame clustering, or some hybrid thereof.
  • an exemplary video segmentation technique is disclosed.
  • one or more representative key frames are selected for each segmented structural video unit and used as indices for video data.
  • video indexing and summarization methods based on video segmentation are tuned to highly structured and professionally edited commercial products.
  • these products have camera shots that are rather short (on the order of four seconds), scene changes that are well-defined and frequent (about every 90 seconds or less), and changes in content and cinematography ("montage") that are visually appealing.
  • montage changes in content and cinematography
  • An object of the present invention is to provide inexpensive semantic video compression and indexing techniques.
  • Another object of the present invention is to provide a technique for deriving semantically summarized video by extracting less important video segments.
  • Yet another object of the present invention is to provide a technique for the semantic compression of video data at dynamically changing rates in view to provide accessibility to a wide variety of platforms and connections, including some whose capacities are severely limited.
  • Still another object of the present invention is to provide video indexing and summarization techniques that are user-tunable, particularly in domains in which there exists little formal shot structure and a high amount of frame-to-frame redundancy.
  • An exemplary method for semantically compressing video data includes the steps of: (a) receiving uncompressed video data including a plurality of video segments; (b) organizing at least a portion of the received uncompressed video data into two or more buffer slots such that each of the buffer slots is filled with one or more of the received video data segments, thus forming two or more buffered video portions corresponding to the two or more buffer slots; (c) applying a leaking rule to the buffered video portions to extract buffered video portions therefrom; and (d) outputting buffered video portions which were not extracted in step (c) as compressed video data.
  • the applying step further includes the step of (i) evaluating each of the two or more buffered video portions to determine one or more significance values corresponding thereto; and (ii) using the determined significance values to extract one or more buffered video portions.
  • the applying step can further include the steps of (iii) saving the extracted video portions together with a corresponding determined significance value, and (iv) reorganizing buffered video portions which were not extracted to leave sequential empty buffer slots followed by filled buffer slots corresponding to the non-extracted buffered video portions.
  • the method further includes the steps of: (e) organizing at least a second portion of the received uncompressed video data into the sequential empty buffer slots such that each of the buffer slots is filled with one or more of the received video data segments, thus forming two or more buffered video portions corresponding to the two or more buffered slots; (f) applying the leaking rule to the buffered video portions to extract buffered video portions therefrom; (g) outputting buffered video portions which were not extracted in step (f) as compressed video data; and (h) repeating steps (e)-(g) a predetermined number of times.
  • the predetermined number of times is preferably a function of a compression ratio .
  • the extracted buffered video portions are recorded together with corresponding leaking rule data.
  • a method for indexing the semantically compressed data includes the steps of labeling the output buffered video portions as top level units; and organizing the extracted video portions as a set of secondary units.
  • the secondary units can include a set of video parameters corresponding to buffered video portions therein.
  • the video parameters can include temporal video parameters, difference video parameters, object video parameters, etc.
  • the secondary units may be organized by utilizing the corresponding leaking rule data.
  • the leaking rule data preferably includes a set of ranking parameters.
  • the ranking parameters may include temporal ranking parameters, difference ranking parameters, object ranking parameters, etc.
  • the leaking rule data may also include one or more significance values, each significance value corresponding to a particular buffered video portion.
  • the secondary organizing step includes associating secondary units to top level units.
  • the associating step preferably includes utilizing the leaking rule data corresponding to the extracted video portions to associate the top level units with the secondary units.
  • the leaking rule datum corresponding to each extracted video portion preferably includes a set of relationship indicators corresponding to one or more buffered video portions located in adjacent buffer slots.
  • a computer readable medium used for semantically compressing video data includes a program that causes a processor to: (a) receive uncompressed video data including a plurality of video segments; (b) organize at least a portion of the received uncompressed video data into two or more buffer slots such that each of the buffer slots is filled with one or more of the received video data segments, thus forming two or more buffered video portions corresponding to the two or more buffer slots; (c) apply a leaking rule to the buffered video portions to extract buffered video portions therefrom; and (d) output one or more buffered video portions which were not extracted as compressed video data.
  • a device for semantically compressing video data includes: (a) a buffer having two or more buffer slots, for receiving uncompressed video data including a plurality of video segments; (c) a processor, operationally coupled to the buffer, for (i) organizing at least a portion of the uncompressed video data received by the buffer into the two or more buffer slots such that each of the two or more buffer slots is filled with one or more of the received video data segments, thereby forming two or more buffered video portions corresponding to the two or more buffer slots; (ii) applying a leaking rule to the two or more buffered video portions to extract one or more buffered video portions therefrom; and (iii) outputting one or more buffered video portions which were not extracted from the buffer as compressed video data.
  • the processor-controlled buffer of the present invention advantageously helps determine which "non-key" portions may be extracted.
  • the video segments enter the video buffer, and one or more portions of them are extracted, i.e., "leaked” away, from the buffer according to a predefined leaking rule.
  • the leakage activities, as well as the extracted video data are recorded in a database using a kind of data structure that can recover the order of extraction ("leakage").
  • the video portions that are not extracted are the desired, semantically compressed, video portions, which are then outputted from the video buffer.
  • a device for video indexing and searching is based on a dynamic video compression according to the present invention.
  • the video portions left in the video buffer form the top level of the key frame hierarchy, and the other portions are ranked according to a set of predetermined rules, e.g., by their significance. Any level of significance or any number of these secondary units can be recovered from this predefined data structure, which provides an interactive searching method which is more psychologically accurate and functionally more efficient than the existing techniques.
  • this novel video indexing scheme can provide an arbitrary number of levels of key frames according to a user's requirement, which enables flexible interactive searching of content in videos.
  • this semantic compression does not use video-dependent thresholds.
  • the time-constrained semantic video buffer selects video portions according to their semantic importance, but nevertheless balances this with the need to provide a long-term steady output rate.
  • These semantically selected portions may be used in MPEG-4 as reference frames, or can be described in a script like that suggested by MPEG-7 together with other information.
  • FIG. 1 is a block diagram of a system for semantically compressing and indexing video data according to the present invention
  • FIG. 2 is a block diagram of a time-constrained video buffer of the system from FIG. 1
  • FIG. 3 is a flow diagram of a method for semantically compressing video data according to the present invention
  • FIG. 4 is a flow diagram of a method for semantically indexing compressed video data according to the present invention.
  • FIG. 5 is an illustrative diagram presenting an exemplary recordation of leakage activities in a video buffer with 8 buffer slots, as well as a relationship between a video compression process and a frame recovery process,
  • FIG. 6 is an illustrative diagram with a significance array and an inverted significance array of video data organized in FIG. 5,
  • FIG. 7a is a Cartesian diagram illustrating a 17-minute instructional video
  • FIG. 7b is a set of 13 semantically compressed frames of a 17-minute instructional video
  • FIG. 8 is a set of video frames resulting from a semantic compression of an exemplary video data, illustrating reconstruction of video frames and "click-drag" function.
  • a system 100 for semantically compressing and indexing video data includes a processing subsystem 110 and a database 170.
  • the processing subsystem 110 further includes a video buffer 120, a leaking rule applicator 130, a video portion extractor 140, an input/output (I/O) communication interface 150, a buffer reorganizer 160, a top level unit labeling module 180 and a recorded data section organizer 190.
  • I/O input/output
  • Uncompressed video data which includes a plurality of video data segments is received via the I/O communication interface 150 and the video buffer 120 is filled with one or more video segments.
  • the video buffer is a time-constrained video buffer.
  • An exemplary time-constrained video buffer is a C++ Application shown in Appendix 1.
  • the time constrained video buffer 120 has two or more buffer slots (See FIG. 2) each of which is filled with one or more video data segments, thereby forming two or more buffered video portions corresponding to the two or more buffer slots.
  • a leaking rule is then applied by the leaking rule applicator 130 to the two or more buffered video portions, and one or more buffered video portions are extracted by the video portion extractor 140.
  • An exemplary leaking rule applicator 130 is a C++ software application shown in Appendix 2.
  • An exemplary video portion extractor 140 is a C++ application shown in Appendix 3.
  • the extracted video portions may further be recorded together with corresponding leaking rule data in the database 170 and organized as secondary units by the recorded data section organizer 190.
  • An exemplary recorded data section organizer 190 is a C++ application shown in Appendix 4.
  • the buffered video data portions which were not extracted are then outputted from the buffer 120 and labeled as top level units by the top level unit labeling module 180.
  • An exemplary top level unit labeling module 180 is a C++ application shown in Appendix 5.
  • the top level units and the organized secondary units may then be presented for viewing via the I/O communication interface 150 or saved in the database 170.
  • FIG. 2 An exemplary semantic video buffer 120 is illustrated in FIG. 2.
  • the video buffer 120 receives uncompressed video data segments 210 and initially stores them in one or more buffer slots 220.
  • the buffer 120 has n buffer slots 220.
  • Each buffer slot 220 may hold one or more video segments 210.
  • a leaking rule is applied and one or more buffered video portions in the one or more buffer slots are evaluated and extracted, i.e., "leaked.”
  • One or more buffered video portions can "leak" at any slot position, based on, e.g., their significance values, semantic differences, etc.
  • the buffered video portions may be randomly evaluated and extracted or "leaked.”
  • Other leaking rules known in the art may also be used without departing from the spirit and scope of the invention.
  • Such examples include, but are not limited to, using a semantic difference between two adjacent video frames as a leaking criterion, which is applied to frames that are found to be semantically similar to each other. Once a pair of frames, which are found to be the most similar, is detected, one of those frames may be extracted either randomly or according to a predetermined leaking criterion.
  • the video stream having a plurality of video segments 210 comes into si, moves through the buffer slots s , s 3 , ... , s n in order, partially leaks video portions from the buffer 120 at any slot, and is finally outputted from the buffer 120 at slot s n .
  • the portions of the video stream output from the buffer 120 at slot s n form a semantically compressed version 230 of the original video stream.
  • a preferred number of buffer slots is five. '
  • the semantic compression process is initiated by receiving uncompressed video data, which includes a plurality of video segments (step 310). This is followed by a step 320 of organizing at least a portion of the received uncompressed video data into two or more buffer slots such that each of the two or more buffer slots is filled with one or more of the received video data segments, thus forming two or more buffered video portions corresponding to the two or more buffer slots.
  • a portion may be a frame, several frames, a field or other suitable unit of video.
  • a leaking rule is applied to the two or more buffered video portions to extract one or more buffered video portions therefrom.
  • the applying step 330 preferably includes evaluating each of the buffered video portions to determine corresponding significance values, using the determined significance values to extract one or more buffered video portions, and saving the one or more extracted video portions together with corresponding determined significance values.
  • the buffered video portions which were not extracted are reorganized, in order to leave one or more sequential empty buffer slots followed by one or more filled buffer slots corresponding to the non-extracted buffered video portions.
  • the compression method according to the present invention optionally includes a step 350 of organizing at least a second portion of the received uncompressed video data into one or more of the sequential empty buffer slots such that each of the two or more buffer slots is filled with one or more of the received video data segments, thereby forming two or more buffered video portions corresponding to the two or more buffered slots.
  • a step 360 of applying the leaking rule to the two or more buffered video portions may be included to extract one or more buffered video portions therefrom.
  • step 370 one or more buffered video portions which were not extracted in step 360 are outputted as compressed video data.
  • Steps 340, 350 and 360 may be repeated a predetermined number of times. The predetermined number of times is preferably a function of a compression ratio.
  • This difference could be defined in many ways: it could be the L 1 norm of their histogram differences, the amount of residual object motion after background registration, some measure of instructional content differences, etc.
  • the differences between all adjacent pairs of video portions in the buffer may be calculated. One of the two video portions in a pair whose semantic difference is the smallest is then extracted (leaked) according to a predetermined leaking criterion.
  • leaking criteria can be used, depending in part on their look-ahead effect on the measured semantic similarity of the frames remaining in the buffer 120. There are several leaking criteria, each of which quantifies the effect of dropping one or the other buffered video portion of a particular pair with a minimum semantic difference.
  • One exemplary leaking criterion is a "min-min" criterion described in more detail further below.
  • a semantic video compression technique in accordance with the present invention also provides a QoS-enabled dynamic compression.
  • a compression ratio can be changed dynamically by adjusting the output rate of a video buffer, and the dynamic compression delay is controlled by the size of the video buffer.
  • the method according to the present invention greatly reduces the chance of missing important video shots or semantically significant frames.
  • a compression factor of approximately 100 which is more severe than that of most general key frame selection techniques, this dynamic compression appears very useful in a client-server environment where some clients have severe bandwidth and display requirements, such as found with hand-held viewing devices.
  • a compression factor of approximately 1000 the output appears useful for semantic summarization of the video.
  • step 410 uncompressed video data with a plurality of video segments is received.
  • step 420 at least a portion of the received uncompressed video data is organized into two or more buffer slots such that each of the two or more buffer slots is filled with one or more of the received video data segments, thereby forming two or more buffered video portions corresponding to the two or more buffer slots.
  • step 430 a leaking rule is applied to the two or more buffered video portions to extract and record one or more buffered video portions.
  • step 440 the buffered video portions that were not extracted from the video buffer 120 are output.
  • step 450 these output buffered video portions are labeled as top level units in a key frame hierarchy, and in step 460, the recorded video portions are organized, e.g., via a Leakage History Directed Acyclic Graph (DAG) 240 (See FIG. 2), which indicates the relative significance of all the frames.
  • DAG Leakage History Directed Acyclic Graph
  • frames in the video buffer 120 are extracted at a leaking rate determined by the incoming frame rate (set by the video source or server) and the output frame rate (set by a desired amount of top-level frames, or the available bandwidth to the client); the output frame rate can be changed dynamically over time.
  • the leaking process selects which buffered video portion, e.g., one or more frames, in the video buffer to drop, and removes it from the queue.
  • the queue which may be implemented as a doubly-linked list, thus implicitly shifts all the frames to their successor buffer slots, freeing up the first slot again.
  • d ⁇ ij ⁇ be a semantic distance from frame i to frame j. If the minimum distance between any two adjacent frames in the video buffer 120 is d k,k+ ⁇ . then the frame k and the frame k+1 are evaluated as candidate frames for extraction. Which of the two frames to drop is determined by a leaking rule sensitive to the "video context" of the two frames, which is defined as the sequence of video frames that are within a predetermined neighborhood thereof.
  • a video sequence of the minimum video context of f 3 is the set ⁇ ⁇ , ⁇ , although ⁇ f l5 f 2 ,f 4 ,f5 ⁇ could also be so defined.
  • the leaking rules compute the effect that extracting either frame k or k+1 have on their context, and the frame that accentuates the semantic individuality of the neighboring frames of the evolving compressed video stream is then extracted.
  • the leaking criterion of extracting f k or f k+ i . may depend on a video context of the frame set ⁇ f k -i, f k+2 ⁇ , and may be defined by a "min-min" calculation, extracting the frame that has the minimum distance to its video context: min ⁇ min ⁇ d k .
  • This exemplary leaking criteria can maximize the minimum of all adjacent frame distances in the video buffer, thus allowing for a more efficient extraction of redundant frames and outputting of significant frames.
  • the leaking rule when the leaking rule is applied to the two or more buffered video portions, one or more buffered video portions are extracted from the video buffer 120.
  • the buffered video portions that were not extracted from the video buffer 120 are then outputted and labeled as top level units in a key frame hierarchy.
  • the extracted video portions, as well as the leakage activities may be recorded in the Leakage History Directed Acyclic Graph (DAG) 240 (See FIG. 2).
  • DAG Leakage History Directed Acyclic Graph
  • Each record may contain e.g., a current extracted video portion number, two associated pointers, and two associated semantic distances.
  • prev- pointer which points to the record of the most recently extracted video portion adjacent to and sequentially before the buffer slot of the current video portion
  • prev-distance is a semantic distance between these two video portions.
  • ext-pointer and “next-distance” refer to the most recently extracted video portion adjacent to and sequentially after the current video portion.
  • FIG. 5 an illustrative diagram presenting an exemplary recordation of leakage activities in a video buffer with 8 buffer slots, as well as a relationship between a video compression process and a frame recovery process is shown.
  • a first row in FIG. 5 represents a video buffer 510 having eight buffer slots 520, where each buffer slot is filled with one or more buffered video portions.
  • each buffer slot is filled with one or more buffered video portions.
  • one or more buffered video portions are extracted leaving a corresponding number of empty slots 530.
  • the buffered video portion from the slot 6 is extracted leaving slot 6 empty.
  • the records of the adjacent buffered video portions that have not been extracted change dynamically, so the "next pointer" of the buffered video portion 5 and the “previous pointer” of the buffered video portion 7 will be redirected to point to the extracted video portion 6. Similarly, their respective distances will be correspondingly updated. This is represented by the arrows pointing from the slots 5 and 7 towards the slot 6 in the second row.
  • the buffered video portion from the slot 2 is extracted, as illustrated in row 3.
  • the records of the buffered video portions 1 and 3 will be dynamically changed to point to the extracted video buffer from the slot 2.
  • the buffered video portion from the slot 5 is extracted and the records from the adjacent buffered video portions in slots 4 and 7 are updated to point to the extracted video portion 5. This is illustrated with the arrows in row 4. Note that the record of the buffered video portion in slot 7 has dynamically changed to point to the extracted video portion 5. As the buffered video portion from the slot 7 is extracted, as shown in row 5, the records of the adjacent buffered video portions 4 and 8 are dynamically changed to point to the extracted video portion 7. Similarly, in row 6, when the buffered video portion 3 is extracted, the records of the adjacent buffered video portions 1 and 4 dynamically change to point to the extracted video portion 3. It must be noted that the records of the extracted video portions remain fixed, as indicated by the dashed arrows in rows 4, 5 and 6.
  • the semantically compressed video data that exited the buffer, plus any frames left in the buffer are labeled as the top level units. All other video portions, organized as secondary units, can be recovered in hierarchical semantic order from these top level units, by following the pointers in their data records. This video portion recovery process is the inverse process of video compression.
  • the top-down procedure shows frame leakage activity
  • the bottom-up procedure shows the frame recovery process.
  • the recovery can be parameterized e.g., by tree depth, by a semantic difference, etc.
  • the video portion recovery process is complete, i.e., every video portion in a video sequence can be recovered from top level units following pointers in the video portion records.
  • the top level units can be recovered directly from the top level structure.
  • two other top level units, fii c - ! and f ⁇ k+l5 redirected to it their "next-pointer" and "prev-pointer", respectively. If both of these top level units remain in the top level, then fi k is still accessible via either pointer.
  • one or the other of these pointing video portions may have also subsequently leaked.
  • Every frame record has two pointers, so there may be multiple choices for selecting which frame to retrieve as the next key frame.
  • a largest semantic difference criterion is used: among all non-null and not yet followed pointers in records of the current key frame list, the frame that has the largest difference from the current key frames is selected.
  • the video portion recovery process provides a novel interactive approach for video indexing and interactive searching, since its semantic depth (or number of frames) is tunable by the user. It is more effective than searching either every r-th frame, or a fixed number of key frames, or a fixed key frame hierarchy.
  • the "significance array” in which the video data portions e.g., frames, are ranked in order of their significance.
  • a third data structure can also be provided, the "inverted significance array”, which is an inverted array of the significance array.
  • the significance array and the inverted significance array are further illustrated in FIG. 6.
  • certain buffered video portions are extracted from the buffer slots and organized as secondary units in the following order: 6, 2, 5, 7 and 3.
  • the remaining buffered video portions that were not extracted, 8, 4, and 1, are outputted from the buffer and labeled as the top level units.
  • a significance array 610 represents the video data compressed and organized in an 8-slot buffer of FIG. 5.
  • the elements of the significance array are 1, 4, 8, 3, 7, 5, 2 and 6.
  • the video data is semantically compressed and indexed, only the top level units are outputted and viewed by the user, while the extracted video portions are saved as secondary units in a database.
  • the user would view only the outputted buffered video portions 1, 4 and 8.
  • the user may change the criterion by moving a cut-off point 640 to include one or more extracted video portions.
  • the user can move the cut-off point to include the extracted video units 3 and 7.
  • an inverted significance array 630 may be used to quickly determine which video portions may be displayed.
  • the complete retrieval data structure has two arrays. Together, they provide efficient ratio-adjustable semantic compression and efficient video playback at variable compression ratios.
  • the significance array is ranked by frame significance.
  • the following procedure may be used to retrieve and/or play a specified subsequence of the video at a specified compression ratio. First, the number of key frames to be displayed is calculated. Then the "cut-off point" in the significance array is determined. Finally, the top level units are retrieved from the significance array, or the semantically compressed version of the video is played by checking if the values of corresponding elements in inverted significance array exceed the cut-off point.
  • the records are checked one by one to find the k-th record whose frame number is within [nl,n2]; this is the cutoff point.
  • the cut-off point can be estimated as N/r, where N is the total number of frames in the video.
  • the value from nl-th element until n2-th element is checked in the inverted significance array. If the value of the m-th (nl ⁇ m ⁇ n2) element is less than the cut-off point, the m-th frame is declared to be significant, and it is displayed.
  • the value of the first element in the inverted significance array 630 is 1, which refers to the first element of the significance array
  • the value of the second element of the inverted significance array is 7, which refers to the seventh element of the significance array etc.
  • the shaded elements 2, 5 and 6 of the inverted significance array having the values 7, 6 and 8, respectively, will not be displayed.
  • the methods for creating significance arrays and inverted significance arrays are known in the art.
  • An exemplary procedure for creating and using the significant arrays and inverted significance arrays in semantic compression is a C++ application shown in Appendix 6. These two arrays can be also be generated off-line. While only the significance array may be used, for compressed video searching and playback purposes, the inverted significance array is preferably also used for better performance.
  • This ratio-adjustable compression method is also useful in a client-server environment where the video server must provide a compressed video at different ratios to clients with different bandwidth requirements.
  • FIG. 7a an illustrative diagram of a 17- minute instructional video is illustrated. Focusing on one dominant scene type of instructional video, that of handwritten lecture notes, an appropriate measure of semantic content is defined as "ink pixels.” Low-level content extraction techniques recognize such "ink pixels.” The semantic content difference is defined between two frames as the number of different "ink” pixels. This difference is used in the time-constrained video buffer, with the "min-min” leaking rule used for semantic compression.
  • Results on a 17-minute 32,000 frame video sequence show that even at a very high compression rate of 4000, the compressed video frames still contain all the content of the original video, as compared to a set of frames hand-selected as "ground truth," which represents a minimum frame set covering the content of a particular video sequence.
  • Semantic compression using compression factors of 2000, 1000, 500, and 200 also captured the semantically important frames.
  • the 13 frames that resulted from the compression at 4000 are displayed in FIG. 7b.
  • the positions of these 13 frames are shown in the video sequence of FIG. 7a, with the seven triangle marks showing the positions of the seven frames hand-selected as ground truth. These seven frames form the minimum frame set covering the content of this video sequence.
  • FIG. 8 a result of indexing a sitcom video is shown in FIG. 8.
  • the user is provided with a set of top level units, in this case 11 frames, from which he or she may choose a particular segment to view. This may be done by clicking on a start key frame and an end key frame. For example, the user may select a segment between a key frame 656, and a key frame 2251. Then, any number of key frames of this video segment can be shown for user review.
  • indexing level 2 provides 7 key frames in this segment. The user can choose to display more key frames, by using a simple "click-drag" operation.
  • An exemplary "click-drag" is a C++ Application shown in Appendix 7.
  • This software tool also implements ratio-adjustable compressed video playback. By selecting start and end frames and a compression ratio, the semantically significant key frames will be displayed for interactive searching and fast viewing.

Abstract

L'invention concerne une technique de compression vidéo sémantique illustrée dans le bloc (120) de la figure 2. Des données vidéo non compressées (210), notamment plusieurs segments de données vidéo (S1, S2, ..., Sn), sont organisées en deux ou plusieurs intervalles tampon (220), de sorte que chacun des deux ou plusieurs intervalles tampon soient remplis d'un ou de plusieurs segments de données vidéo reçues, formant ainsi deux ou plusieurs parties vidéo en mémoire tampon correspondant aux deux ou plusieurs intervalles tampon. Les données vidéo en mémoire tampon sont ensuite traitées au moyen d'une règle de fuite, en vue d'extraire une ou plusieurs parties vidéo en mémoire tampon, tout en émettant une ou plusieurs parties vidéo en mémoire tampon non extraites, sous forme de données vidéo compressées (230). Les données de règle de fuite sont stockées dans un histogramme (240) puis utilisées en vue d'organiser et d'indexer les données selon une requête d'utilisateurs.
PCT/US2002/018231 2002-06-07 2002-06-07 Procede et dispositif de compression video et d'indexage video semantique dynamique en ligne WO2003105489A1 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/US2002/018231 WO2003105489A1 (fr) 2002-06-07 2002-06-07 Procede et dispositif de compression video et d'indexage video semantique dynamique en ligne
US10/987,646 US7489727B2 (en) 2002-06-07 2004-11-12 Method and device for online dynamic semantic video compression and video indexing
US12/349,786 US8391355B2 (en) 2002-06-07 2009-01-07 Method and device for online dynamic semantic video compression and video indexing

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2002/018231 WO2003105489A1 (fr) 2002-06-07 2002-06-07 Procede et dispositif de compression video et d'indexage video semantique dynamique en ligne

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US10/987,646 Continuation US7489727B2 (en) 2002-06-07 2004-11-12 Method and device for online dynamic semantic video compression and video indexing

Publications (1)

Publication Number Publication Date
WO2003105489A1 true WO2003105489A1 (fr) 2003-12-18

Family

ID=29731319

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2002/018231 WO2003105489A1 (fr) 2002-06-07 2002-06-07 Procede et dispositif de compression video et d'indexage video semantique dynamique en ligne

Country Status (1)

Country Link
WO (1) WO2003105489A1 (fr)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2007122541A2 (fr) * 2006-04-20 2007-11-01 Nxp B.V. Système de réduction de données et procédé de réduction d'un flux de données
WO2008034213A1 (fr) * 2006-09-18 2008-03-27 Infobright Inc. Procédé et système pour une compression de données dans une base de données relationnelle
US8266147B2 (en) 2006-09-18 2012-09-11 Infobright, Inc. Methods and systems for database organization
US8417727B2 (en) 2010-06-14 2013-04-09 Infobright Inc. System and method for storing data in a relational database
US8521748B2 (en) 2010-06-14 2013-08-27 Infobright Inc. System and method for managing metadata in a relational database
US9493540B2 (en) 2001-02-20 2016-11-15 Intrexon Corporation Ecdysone receptor/invertebrate retinoid X receptor-based inducible gene expression system
CN113518094A (zh) * 2021-09-14 2021-10-19 深圳市普渡科技有限公司 数据处理方法、装置、机器人和存储介质

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6219837B1 (en) * 1997-10-23 2001-04-17 International Business Machines Corporation Summary frames in video

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6219837B1 (en) * 1997-10-23 2001-04-17 International Business Machines Corporation Summary frames in video

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9493540B2 (en) 2001-02-20 2016-11-15 Intrexon Corporation Ecdysone receptor/invertebrate retinoid X receptor-based inducible gene expression system
WO2007122541A2 (fr) * 2006-04-20 2007-11-01 Nxp B.V. Système de réduction de données et procédé de réduction d'un flux de données
WO2007122541A3 (fr) * 2006-04-20 2008-02-21 Koninkl Philips Electronics Nv Système de réduction de données et procédé de réduction d'un flux de données
US8798169B2 (en) 2006-04-20 2014-08-05 Nxp B.V. Data summarization system and method for summarizing a data stream
US8700579B2 (en) 2006-09-18 2014-04-15 Infobright Inc. Method and system for data compression in a relational database
US8266147B2 (en) 2006-09-18 2012-09-11 Infobright, Inc. Methods and systems for database organization
US8838593B2 (en) 2006-09-18 2014-09-16 Infobright Inc. Method and system for storing, organizing and processing data in a relational database
WO2008034213A1 (fr) * 2006-09-18 2008-03-27 Infobright Inc. Procédé et système pour une compression de données dans une base de données relationnelle
US8521748B2 (en) 2010-06-14 2013-08-27 Infobright Inc. System and method for managing metadata in a relational database
US8417727B2 (en) 2010-06-14 2013-04-09 Infobright Inc. System and method for storing data in a relational database
US8943100B2 (en) 2010-06-14 2015-01-27 Infobright Inc. System and method for storing data in a relational database
CN113518094A (zh) * 2021-09-14 2021-10-19 深圳市普渡科技有限公司 数据处理方法、装置、机器人和存储介质
CN113518094B (zh) * 2021-09-14 2021-12-28 深圳市普渡科技有限公司 数据处理方法、装置、机器人和存储介质

Similar Documents

Publication Publication Date Title
US8391355B2 (en) Method and device for online dynamic semantic video compression and video indexing
US8467610B2 (en) Video summarization using sparse basis function combination
Rui et al. Constructing table-of-content for videos
JP3568117B2 (ja) ビデオ画像の分割、分類、および要約のための方法およびシステム
Li et al. An overview of video abstraction techniques
US20120148149A1 (en) Video key frame extraction using sparse representation
US9355330B2 (en) In-video product annotation with web information mining
JP4351994B2 (ja) スケーラブルなビデオの要約化
US8467611B2 (en) Video key-frame extraction using bi-level sparsity
US7643597B2 (en) Methods for selecting a subsequence of video frames from a sequence of video frames
Sun et al. Video summarization using R-sequences
JP2006268866A (ja) ツリー構造の動映像索引記述子の生成方法
Yeo et al. Classification, simplification, and dynamic visualization of scene transition graphs for video browsing
WO2003105489A1 (fr) Procede et dispositif de compression video et d'indexage video semantique dynamique en ligne
Jin et al. Network video summarization based on key frame extraction via superpixel segmentation
Liu et al. Time-constrained dynamic semantic compression for video indexing and interactive searching
Chen et al. Multi-criteria video segmentation for TV news
Adams Where does computational media aesthetics fit?
Zhu et al. Automatic scene detection for advanced story retrieval
Kavitha et al. Static and multiresolution feature extraction for video summarization
Zhang et al. Hierarchical video organization based on compact representation of video units
EP2608059A1 (fr) Procédé et appareil pour prioriser des métadonnées
Zhang Video content analysis and retrieval
Zhu et al. A novel scheme for video scenes segmentation and semantic representation
Hanjalic et al. Recent advances in video content analysis: From visual features to semantic video segments

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): CA JP US

WWE Wipo information: entry into national phase

Ref document number: 10987646

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: JP

WWW Wipo information: withdrawn in national office

Country of ref document: JP