Classifications

• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/005—Reproducing at a different information rate from the information rate of recording
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B20/00—Signal processing not specific to the method of recording or reproducing; Circuits therefor
  • G11B20/00086—Circuits for prevention of unauthorised reproduction or copying, e.g. piracy
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B20/00—Signal processing not specific to the method of recording or reproducing; Circuits therefor
  • G11B20/00086—Circuits for prevention of unauthorised reproduction or copying, e.g. piracy
  • G11B20/00884—Circuits for prevention of unauthorised reproduction or copying, e.g. piracy involving a watermark, i.e. a barely perceptible transformation of the original data which can nevertheless be recognised by an algorithm
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
  • G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
  • G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
  • G11B27/34—Indicating arrangements 
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/76—Television signal recording
  • H04N5/91—Television signal processing therefor
  • H04N5/913—Television signal processing therefor for scrambling ; for copy protection
  • H04N2005/91307—Television signal processing therefor for scrambling ; for copy protection by adding a copy protection signal to the video signal
  • H04N2005/91335—Television signal processing therefor for scrambling ; for copy protection by adding a copy protection signal to the video signal the copy protection signal being a watermark

Definitions

• the present invention relates to apparatus and methods for manipulating sequences of images.
• Israel Patent Application No. 119504 describes a system and method for audio-visual content verification.
• “Intro” is a known function in audio applications in which a user of a CD player can "scan” a CD by hearing a small portion of each audio segment (e.g. song) on the CD.
• the present invention seeks to provide improved apparatus and methods for manipulating sequences of images. There is thus provided in accordance with a preferred embodiment of the present invention a system for capturing the signature of video frames, using only small amounts of data.
• the video signature technology typically captures a small amount of data characterizing each frame.
• the applicability of the invention includes all uses that require video identification, without the necessity of viewing.
• the system of the present invention has a PC-based platform and is operative in real-time to analyze motion pictures, video and broadcasting, inter alia.
• the system of the present invention typically uses small amounts of data, to capture a signature from a stream of video frames. The signature is then matched to a continuous stream of data.
• the system of the present invention includes a matcher which synchronizes various versions of a motion picture for diverse multi-language needs including but not limited to satellite TV broadcasts, on-board film projections and DVD authoring.
• a matcher which synchronizes various versions of a motion picture for diverse multi-language needs including but not limited to satellite TV broadcasts, on-board film projections and DVD authoring.
• Another application for the system of the present invention is simplification of the restoration of damaged films by using the best footage from different versions.
• Yet another application is rapid adaptation of sound tracks for colorized movies.
• the matcher subunit typically does not digitize video sources but rather fingerprints pictures.
• the matcher can process substantially any video source, such as a S-VHS video source or a 1" video source.
• a cassette is inserted, and a checklist is employed to choose the language to be used as a reference for matching.
• the user then presses PLAY and the matcher autonomously and typically without user intervention registers the fingerprint of each frame. This procedure is repeated for the next language version of the film to be checked (cassette insertion, language selection, play). After the various versions have been fingerprinted, the versions are automatically matched, showing the differences that were detected.
• the matcher preferably is operative to generate any of a variety of outputs. For example: if it is desired to broadcast multiple language versions of a film simultaneously on satellite TV, the versions must be synchronized, matcher can generate an EDL (editing list) based on the shots common to all the versions. In multi- language DVD applications, the matcher may be operative to automatically generate a branching instruction list, based on 'holes' caused by missing data in the various versions .
• the system of the present invention also preferably includes a synopter for efficient viewing of video sequences.
• Applications include stock footage, rushes and speed-viewing of selected (typically user-selected) items of interest.
• the system of the present invention also preferably includes a storyboard application which displays the first frame of every shot in an image sequence, thereby to facilitate fast-tracking of shots from rushes or stock footage.
• This application can operate as a search option for professional and home-use.
• the technology shown and described herein may be integrated into VCR's, thereby facilitating speed-searching.
• a user may press a first activating button and as a result, his VCR automatically adjusts search speed according to the amount of action in any given scene of a movie: slower for action-packed sequences and faster for less active moments. If the user presses a second activating button, the VCR automatically screens the first few seconds of every shot in a video, allowing the user to quickly preview the video's content.
• the system of the present invention preferably includes a spot shotter which monitors the off-air signal, detecting the exact moment when specific portions of any given transmission are broadcast, and automatically logging relevant information such as time of transmission and duration.
• the spot shotter may be "told" to detect every appearance of commercials belonging to a particular manufacturer.
• Another difficult, time-consuming function for which the system of the present invention preferably is suited is automatic checking of video dubs for uniformity of content .
• video sequence viewing apparatus including an image sequence display unit operative to display a sequence of images at a speed determined in accordance with a control signal, and an image sequence analyzer operative to perform an analysis of the sequence of images and to generate the control signal in accordance with a result of the analysis.
• the analysis of the sequence of images includes an analysis of the amount of motion in different images within the sequence and the control signal receives a value corresponding to relatively high speed for images in which there is a small amount of motion and a value corresponding to relatively low speed for images in which there is a large amount of motion.
• image sequence viewing apparatus including a shot identifier operative to perform an analysis of a sequence of images and to identify shots within the sequence of images, and an image sequence display unit operative to sequentially display at least one initial images of each identified shot.
• the image sequence display unit is operative to display the at least one initial images of each identified shot in response to a user request .
• the image sequence display unit is operative to display the at least one initial images of all shots sequentially until stopped by the user.
• a display system for displaying a first image sequence as aligned relative to a second, related image sequence
• the system including an image sequence analyzer operative to generate a representation of a first image sequence including at least one row of pixels of each image in the first image sequence, and an aligned image sequence display unit operative to display the rows generated by the analyzer, side by side, in a single screen, wherein gaps are provided between the rows, in order to denote images which are missing, relative to the second image sequence.
• the at least one row includes at least one horizontal row of pixels and at least one vertical row of pixels.
• the display unit is operative to display an isometric view of a stack of the images in at least one of the first and second image sequences .
• the stack includes a horizontal stack.
• the analyzer also includes an image sequence aligner operative to align the first and second image sequences to one another and to provide an output denoting images which are missing from the first image sequence, relative to the second image sequence .
• a copyright monitoring system including ar image sequence comparing unit operative to conduct a comparison between an original image sequence and a suspected pirate copy of the original image sequence and to generate copyright information describing infringement of copyright of the original image sequence by the suspected pirate copy, and a copyright infringement information generator operative to generate a display of the copyright information.
• At least a portion of the comparison is conducted at the shot level.
• At least a portion of the comparison is conducted at the frame level.
• the copyright information quantifies the infringement of copyright of the original image sequence by the suspected pirate copy.
• a watermarking method including providing an image sequence to be watermarked, and performing a predetermined alteration of the length of the image sequence.
• the performing step includes duplicating at least one predetermined image (e.g. frame or field) in the image sequence.
• predetermined image e.g. frame or field
• the performing step includes omitting at least one predetermined image (e.g. frame or field) from the image sequence.
• predetermined image e.g. frame or field
• the image sequence analyzer is operative to generate aligned representations of the first and second image sequences and the display unit is operative to display the aligned representations on a single screen.
• a video sequence viewing method including displaying a sequence of images at a speed determined in accordance with a control signal, and performing an analysis of the sequence of images and generating the control signal in accordance with a result of the analysis.
• an image sequence viewing method including performing an analysis of a sequence of images and to identify shots within the sequence of images, and sequentially displaying at least one initial images of each identified shot.
• a method for displaying a first image sequence as aligned relative to a second, related image sequence including generating a representation of a first image sequence including at least one row of pixels of each image in the first image sequence, and displaying the rows generated by the analyzer, side by side, in a single screen, wherein gaps are provided between the rows, in order to denote images which are missing, rela- tive to the second image sequence.
• a copyright monitoring method including conducting a comparison between an original image sequence and a suspected pirate copy of the original image sequence and to generate copyright information describing infringement of copyright of the original image sequence by the suspected pirate copy, and generating a display of the copyright information.
• a watermarking system including an image sequence input device operative to input an image sequence to be watermarked, and an image sequence length alteration device operative to perform a predetermined alteration of the length of the image sequence.
• Fig. 1 is a simplified block diagram illustration of a commercial verification system constructed and operative in accordance with a preferred embodiment of the present invention
• FIG. 2 is a simplified flowchart illustration of a preferred method of operation for the system of Fig. l;
• Fig. 3 is a simplified block diagram illustration of a system for viewing image sequences at variable speed, depending on temporally local characteristics of the image sequence such as the amount of action;
• Fig. 4 is a simplified flowchart illustration of a preferred method of operation for the system of Fig. 3;
• Fig. 5 is a simplified block diagram illustration of a system for finding and displaying shots in an image sequence
• Fig. 6 is a simplified flowchart illustration of a preferred method of operation for the system of Fig. 5;
• Fig. 7 is a simplified block diagram illustration of a system for displaying alignment of twc image sequences
• Fig. 8 is an isometric view of an image sequence
• Fig. 9 is an example of an isometric view of three different-language versions of the same motion picture, where gaps in the representation of a particular version indicate missing images, relative to other versions;
• Fig. 10 is a simplified block diagram illustration of a copyright monitoring system constructed and operative in accordance with a preferred embodiment of the present invention.
• Fig. 11 is a simplified block diagram of an electronic watermarking system constructed and operative in accordance with a preferred embodiment of the present invention.
• Appendix A is a copy of Israel Patent Application No. 119504;
• Fig. 1 is a simplified block diagram illustration of a commercial verification system constructed and operative in accordance with a preferred embodiment of the present invention.
• Fig. 2 is a simplified flowchart illustration of a preferred method of operation for the system of Fig. 1. It is appreciated that the system of Figs. 1 - 2 is also useful for applications other than commercial verification, such as searching for illicit use of copyrighted sequences of images.
• the apparatus of Fig. 1 includes a broadcasting system 10 which broadcasts commercials provided on a suitable receptacle 20 such as a CD or DVD or video cassette.
• a commercial verification workstation 30 is operative to receive broadcasts from the broadcasting system (either from the air or from a receptacle which was used to store broadcast material coming from the air) and to compare the broadcasts to an original commercial residing on the receptacle 20. The workstation attempts to identify some or all of the original commercial within the broadcasted material.
• any suitable method may be used to compare the broadcast with the original commercial.
• the comparison is on the frame-level, i.e. individual frames in the broadcast, or signatures thereof, are compared to individual frames in the original commercial, or signatures thereof.
• Shot level comparison in which entire shots in the broadcast are compared to entire shots in the original commercial, are typically not accurate enough.
• Preferred methods for comparing sequences of images, such as video images, including signature extraction and signature search (steps 60 and 70 of Fig. 2) are described in issued US Patent No. 5,790,236 and in Appendix A.
• the broadcast and the original commercial are compared based only on the content of the advertisement and without requiring any special additions, e.g. without external indices, special information in vertical blanks and other special additions.
• the output of the workstation 30 typically includes a recording of the commercial as broadcast and an indication of the time or times at which the commercial was broadcast, plus an indication of any incomplete- ness in the commercial as broadcast.
• the output may be provided on a screen, in electronic form, as hard copy or in any other suitable format.
• Figs. 1 - 2 illustrate a "cooperative" application in which the original commercial is available. It is appreciated that in some applications, in which the broadcaster and/or the advertiser are non-cooperative, the original commercial may not be available. For example, commercial monitoring of a competitor's commercials may be carried out, in which case the original commercial is, of course, not available. In these cases, a first appearance of a target commercial can be identified by a human being viewing the broadcast, and this appearance of the target commercial can then be treated as the original commercial. Alternatively, commercial monitoring can be carried out without having an original commercial, i.e. without having a model to which to compare the broadcast. For example, the system may monitor recurrence of short image sequence (i.e. image sequences which correspond in length to the known range of lengths which characterize a commercial ) at time intervals which correspond to known intervals between commercial breaks.
• short image sequence i.e. image sequences which correspond in length to the known range of lengths which characterize a commercial
• Fig. 3 is a simplified block diagram illustration of a system for viewing image sequences at variable speed, depending on temporally local characteristics of the image sequence such as the amount of action.
• Fig. 4 is a simplified flowchart illustration of a preferred method of operation for the system of Fig. 3.
• the apparatus of Fig. 3 includes a receptacle 90 storing an image sequence and an image sequence analyzer 100 which is typically operative to derive from each image in the image sequence a signature representing at least one characteristic of the image.
• a signature representing at least one characteristic of the image.
• a "span" signature may be employed, which represents the amount of action in the image.
• the amount of acticn in an image is typically defined as the rate of change between that image and adjacent images.
• Preferred methods for derivation of a "span" signature is described in issued US Patent No. 5,790,236 and in Appendix A.
• the analyzer typically thresholds the signature (step 140) in order to obtain a control signal having a small number of possible values, such as 3 or 4 possible values. More generally, the control signal need not be a simple thresholded version of the signature (e.g. of the span).
• the control signal can have only as many values as the image sequence display unit 110 has viewing speeds.
• any suitable function may be employed to assign values to the control signal as a function of the signature. For example, the values assigned to the control signal may depend in part on second or higher order derivatives of the signature variable.
• the control signal is fed to an image sequence display unit 110 such as a VCR which adjusts its speed accordingly.
• Different viewing speeds can be provided by mechanical display units having motors with adjustable speed.
• the display unit is electronic, different viewing speeds may be provided by varying the rate of display of images stored in the electronic unit.
• Fig. 5 is a simplified block diagram illustration of a system for finding and displaying shots in an image sequence.
• Fig. 6 is a simplified flowchart illustration of a preferred method of operation for the system of Fig. 5.
• the system of Fig. 5 includes a receptacle 160, such as a CD, DVD or video cassette, which stores an image sequence.
• An image sequence display unit such as a VCR, is operative to display the image sequence as stored on the receptacle.
• the image sequence is also accessed by a shot identifier 170 which is operative, preferably online, to identify shots in the image sequence. Any suitable method may be used to identify the shots (step 200). Preferred methods for identifying shots are described in issued US Patent No. 5,790,236 and in Appendix A.
• the shot identifier provides a control signal, based on the locations of the shots within the image sequence, to the display unit 180.
• the control signal typically instructs the image sequence to display a predetermined number of frames, such as one or a few frames, at each cut, i.e. at each interface between shots.
• the image sequence display unit typically displays the first one or few images in each shot.
• the receptacle storing the image sequence is a physical medium such as video cassette, there is typically a time-gap between the display of the frames representing the i ' th shot, and the display of frames representing the (i+l)'th shot. However, if the receptacle storing the image sequence is an electronic medium, there is typically no time-gap between the display of the frames representing subsequent shots.
• the image sequence display unit may display initial images for all of the shots in response to a single user command.
• the user may provide a "next shot” input each time s/he wishes to view the initial images of the next shot.
• Fig. 7 is a simplified block diagram illustration of a system for displaying alignment of two image sequences.
• the system of Fig. 7 includes two image sequence receptacles 220 and 230, such as CDs, DVDs or video cassettes, storing two respective image sequences, such as two versions of the same motion picture.
• the two image sequences are aligned by an image sequence aligner 240.
• Image sequence aligner 240 may use any suitable image sequence aligning method to align the two sequences to one another. Preferred image sequence aligning methods are described in issued US Patent No. 5,790,236 and in Appendix A.
• An isometric view generator 250 is operative to generate an isometric view of each of the image sequences. A simple isometric view of an image sequence, as illustrated in Fig.
• each image may comprise an isometric view of a stack of the images in the sequence, wherein each image is regarded as a one-pixel thick rectangle, wherein all visible faces of each pixel have the color value of the pixel. It is appreciated that in the isometric view of Fig. 8, the top row of each image is visible along the top of the horizontal stack and the rightmost column of each image is visible along the side of the horizontal stack.
• the isometric view generator 250 receives information regarding the alignment of the two sequences to one another from the image sequence aligner 240 and introduces gaps into the isometric view so as to illustrate the alignment.
• the output of the isometric view generator is typically an electronic representation 260 of an isometric view of the aligned image sequences.
• This representation 260 is provided to an image sequence display unit 270, such as a VCR, for display.
• image sequence display unit 270 such as a VCR, for display.
• both aligned sequences are displayed, in isometric view, on a single screen.
• Fig. 9 is an example of an isometric view of three different-language versions of the same motion picture, where gaps in the representation of a particular version indicate missing images, relative to other versions.
• the German version is most complete and includes no gaps
• the French version has one large gap (sequence of missing frames, relative to the German version) and two smaller subsequent gaps
• the English version has a total of four gaps which are not in the same locations as any of the 3 gaps of the French version.
• Fig. 10 is a simplified block diagram illustration of a copyright monitoring system constructed and operative in accordance with a preferred embodiment of the present invention.
• the apparatus of Fig. 10 typically includes receptacles 300 and 310, which may comprise video cassettes, DVDs, CDs and the like, which respectively store an original motion picture and a suspect pirate copy thereof .
• the image sequences stored in receptacles 300 and 310 are accessed by an image sequence comparison unit 320 which typically operates either at shot level or at frame level, to compare the two image sequences . Any suitable method may be employed for comparison of the two image sequences such as the methods described in issued US Patent No. 5,790,236 and in Appendix A.
• the output of the image sequence comparison unit 320 typically comprises copyright monitoring information such as two aligned isometric views of the original movie and the suspect pirate copy, in which gaps denote missing frames and identical frames are placed opposite one another.
• copyright monitoring information such as two aligned isometric views of the original movie and the suspect pirate copy, in which gaps denote missing frames and identical frames are placed opposite one another.
• quantitative copyright monitoring information may be provided such as the number of frames in the original movie which appear in the suspect pirate copy.
• Fig. 11 is a simplified block diagram of an electronic watermarking system constructed and operative in accordance with a preferred embodiment of the p ⁇ -esent invention.
• image sequences such as motion pictures, news clips, commercials etc. are watermarked not by tampering in any way with any particular frame, since this tampering may impair viewing quality, but rather by either removing or adding a small number of frames from or to the image sequence.
• the watermark of each version or each image sequence is typically stored in an electronic databank.
• original and pirate copies 350 and 360 respectively of a motion pic- ture are received by a frame-level image sequence aligner 370, in electronic form, from a video cassette (after digitization) or from a CD or DVD or other suitable image sequence receptacle.
• the frame-level image sequence aligner 370 is operative, according to a first embodiment of the present invention, to align the image sequence of the pirate copy to the image sequence of the original copy which preferably includes a "maximal", i.e. "union" version of the motion picture whose frames include the union of all frames in all versions of the motion picture.
• Any suitable method may be employed to align the two image sequences, preferably at frame level. Preferred methods for alignment of image sequences are described in issued US Patent No. 5,790,236 and in Appendix A.
• a watermark identifier 380 is operative to attempt to compare each of a plurality of watermarks to the aligned pirate copy.
• each version of a motion picture is watermarked, including the post-production version, and each subsequent version.
• the "post-production version” is the motion picture as originally produced, before subsequent versions are derived therefrom.
• Subsequent versions are typically characterized by at least one of the following: a. Intended distribution (airline, cable TV, cinema, etc. ) ; b. Language c. Censorship (X-rated, PG-rated, R-rated, etc.)
• the watermarks may be defined relative to the original copy 350. For example, "Frame #4974" is typically frame no. 4974 in image sequence 350. This is advantageous because then each suspected pirate copy need only be aligned once, to the original copy 350 (e.g. the post-production copy).
• the frame-level image sequence aligner 370 is operative, according to a second embodi- ment of the present invention, to align the image sequence, of the pirate copy to the image sequences of each watermarked version separately, rather than aligning the pirate copy image sequence only once, to the "maximal" or "union” version of the motion picture.
• the watermark of each version need not be defined relative to the original copy 350. For example, if every 500th field is duplicated in a PG-rated version of a motion picture, this easy rule is stored rather than computing the fields, in the maximal (complete) version, which correspond to each 500th field in the PG-rated (incomplete) version.
• three watermarks are stored in this system, for each of three versions of a motion picture: post-production version, airline version, and cinema version.
• the airline and cinema version are typically produced from the watermarked post-production version.
• the watermark of the post-production version is deleted when the airline, cinema, television versions, etc., are derived from the post-production version.
• the post-production watermark is replaced by the watermark of the version being generated. For example, if every 500th frame is duplicated in the post-production version, whereas the watermark of the airline version calls for deletion of every 1000th frame, then the airline version is generated from the post-production version as follows: a. the duplications of each 500th frame are removed; and b. each 1000th frame is deleted.
• the post- production watermark comprises a duplication of four specific frames.
• the cinema version watermark comprises removal of 3 specific frames.
• the watermark identifier 380 is operative to indicate the version from which the pirate copy is derived. For example, if the watermark identifier 380 finds that frames 17, 479 and 19,999 in the original copy 350 are missing in the pirate copy 360, the watermark identifier puts out a suitable output indication that the pirate copy was derived from the cinema version of a film.
• the software components of the present invention may, if desired, be implemented in ROM (read-only memory) form.
• the software components may, generally, be implemented in hardware, if desired, using conventional techniques.
• the present invention relates to audio-visual test and measurement systems and more particularly to a method and apparatus for comparing a given content stream with a reference content stream for verifying the correctness of a given data stream and for detecting various content-related problems, such as missing or distorted content, as well as badly synchronized content streams such as audio or sub-titles delayed with respect to the video stream.
• Audio- visual content is herein defined as a stream or sequence of video, audio, graphics (sub-pictures) and other data where the semantics of the data stream is of value.
• stream or “sequence'” is of particular importance, since it is assumed that the ordering of content elements along a time or space line constitutes pan of the content.
• Elementary content streams may be combined to a composite stream.
• an application which involves two video streams (for stereoscopic display), six or eight surround audio channels and several sub-picture channels can be formed.
• the relative alignment of these streams is highly significant and should be verified.
• an analysis is made of video signal for detecting disturbances of that signal, such as illegal colors.
• An "illegal color" is one that is outside the practical iimit set for a particular format.
• Other types of video measurement involve injecting known signals at the source and evaluating certain properties thereof at the receiving end.
• SDI serial digital interface
• the known video test and measurement systems are. however, generally not capable of detecting content-related problems, such as missing or surplus frames, program time shift, color or luminance disto ⁇ ions which are within the acceptable parameter range, mis-alignment of content streams such as audio or sub-pictures with respect to video, etc.
• an observer will look at the display to detect quality problems.
• An experienced operator may detect and interpret a varier/ of problems in recording and transmission. .An observer can do good rule-based or subjective evaluation of video content, however, human inspection of content is costly and unpredictable. Additionally, some content-related defects cannot be detected by an observer.
• the reference stream consists of the original' program material and the actual stream consists of the broadcast or played content.
• the designation of one stream as the reference stream is arbitrary, for example, comparing one content stream with a backup stream.
• the terms “reference content stream” and “actual content stream” will be used, without limiting the generality of the invention.
• a daily schedule may consist of hundreds of video segments, intended to play seamlessly.
• Such a schedule is usually implemented by an automation system.
• the schedule is logged into the system as some form of a table (a "play-list " ) describing the program ' s name, sta ⁇ time, duration and source, e.g.. storage media, unique identifier, time-code of first frame.
• the storage media can be a tape or a digital file.
• the program source material is organized in an hierarchical manner, with most of the content stored off-line.
• the forthcoming programs are loaded on a tape machine and sometimes, as in the case of a commercial or trailer, digitized to a disk-based server.
• the complex paths of the various elements of content may further increase the content mismatch probability.
• ADC- 100 from Louth Automation.
• ADC- 100 can run up to 16 lists simultaneously, and control multi ⁇ le devices includins disk servers, video servers, ta ⁇ e machines, can machines. VTRs. switchers, character generators and audio cans.
• the present invention can verify the identity and integriry of the broadcast content, providing imponant feedback for the automation system or facility manager.
• DVD is a new generation of the compact disc format which provides increased storage capacity and performance, especially for video and multimedia applications.
• DVD for video is capable of storing eight audio tracks and thirty- two "sub-picture " tracks, which are used for subtitles, menus, etc. Tnese can be used to put several selectable languages on each disc.
• the interactive capabilities of consumer ⁇ DVO players include menus with a small set of navigation and control commands, with some functions for dynamic video stream control, such as seamless branching, which can be used for playing different "cuts " of the same video material for dramatic purposes, censorship, etc. DVD-ROM. which will be used for multi-media applications, will exhibit a higher level of interactivity.
• DVD contains multiple content streams with many options tor bra ⁇ chinz from one stream to the other or combinins several streams, such as a menu or sub-titles overlaid on a video frame, one has to verify that a given set of initial settings, followed by a specific set of navigation commands, indeed produces the correct content.
• This step in DVD production is known as "emulation " , currently designed to be performed by an observer.
• the present invention also allows automation of DVD emulation.
• the audio-visual program comprises at least one video channel, or at least one audio channel, or at least one sub-picrure channel comprising sub-titles, closed-captions and any kind of auxiliary graphics information which is timed synchronously with the video or audio. While in c ⁇ nain applications sub-pictures are embedded in the video image sequence, in other applications they are carried by a separate stream/Tile.
• the present invention therefore provides a method of comparing the content obtained by broadcast or playback with a reference content, including the steps of extracting frame characteristic data streams from said reference content and from actual received or playback content, aligning said streams and comparing said streams on a frame-by-frame basis.
• U.S. Patent No. 5.339,166. entitled “Motion-Dependent Image Classification for Editing Purposes." describes a system for comparing two or more versions, typicaih of different dubbing languages, of the same feature fiim. By identifying camera shot boundaries in both versions and comparing sequences of shot lengm. a common video version, comprising camera shots which exist in all versions, can be automaticallv generated. While the embodiment described in this patent allows, in principle, the location of content differences between versions at camera shot level, frame-by-frame alignment for all frames in the respective version is not performed. Funher. the differences detected are in the existence or absence of video frames as a whole. In contrast, the present invention allows frame-by-frame inspection of color propenies. detection of compression anifacts. audio distonions. etc.
• the content of each frame is fixed and characteristic data are computed from the content.
• the present invention addresses the on-line composition of a content stream from basic content streams, such that characteristic data are pre-computed only for these basic streams. Given the branching'navigation/editing commands, a composite reference characteristic data stream is predicted from the component characteristic data stream and then compared with the actual content stream.
• the present invention does not depend on. the specific format/representation of the content sources and streams.
• one stream may be analog and the other digital.
• one stream may be compressed and the other may be of full bandwidth.
• the input will be CCIR-601 digital video and AES digital audio. Multiple audio streams may be due to different dubbing languages, as well as stereo and surround sound channels.
• the extraction of characteristic data will be done in real-time. tlius saving intermediate storage and also enabling real-time error detection in a broadcasting environment.
• this is not a limitation, since the present invention can be used off-line bv recordins both the reference and the acrual audio-visual program.
• processing can be slower than real-time or faster, depending on the computational resources.
• a faster than real-time performance may be needed, depending, of course, on the availability of a suitable analog to digital convener which can cope with fast-forward video signals.
• Fig. 1 is a block diagram of a top level flow of processing of an audio-visual content verification system:
• Fig. 2 is a block diagram of a circuit for storing detected content problems:
• Fig. 3 schematically illustrates an array of video sequence characteristic data:
• Fig. 4 schematically illustrates an array of video frame or still image spatial characteristic data:
• Fig. 5 schematically illustrates a set of regions in a video frame:
• Fig. 6 schematically illustrates relative location of graphics sub-pictures with respect to the video frame:
• Fig. 7 is a block diagram illustrating extraction of sub-title characteristic data:
• Fig. 8 is a block diagram illustrating sub-title image sequence processing;
• Fig. 9 schematically depicts a record of sub-pictures characteristic data: Fig.
• Fig. 10 is a block diagram illustrating derivation of audio characteristic data:
• Fig. 11 is a block diagram of a circuit for the selection of anchor frames for coarse alignment:
• Fig. 12 is a block diagram of a circuit for alignment of a composite stream with the component reference streams:
• Fig. 13 is a block diagram of a circuit for frame verification processing; and
• Fig. 14 is a block diagram of a characteristic data design workstation.
• Fig. 1 shows a top level flow of processing of an audio-visual content verification system according to the present invention.
• Reference sub-picture stream 11. video stream 12 and audio stream 13 are stored in their respective stores 14, 15 and 16. to be eventually processed by processors 17. 18 and 19. respectively.
• the combination of sub-pictures with video, as well as transition/branching between program segments, is applied at characteristic data level by predictor 20. driven by navigation/playback commands 21.
• Actual video stream 22 and audio stream 23 are stored in their respective stores 24 and 25. to be later processed by processors 26 and 27 respectively.
• the video stream 22 and the corresponding characteristic data are composed of video and sub-pictures.
• the data streams are input to the characteristic data alignment processor 30. resulting in frame-aligned characteristic data.
• the alignment process also results in a program time-shift value, as well as indices or time-codes of missing or surplus frames.
• characteristic data are compared on a frame-by-frame basis in comparator 32. yielding a frame quality report.
• Fig. 2 shows means for storing detected content problems.
• Recently played/received video from store 24 undergoes compression in engine 34 and is then stored in buffer 35.
• the recently played/received audio from store 25 is directly stored in buffer 36.
• Transfer controller 37 is activated by verification repons 38 to transfer the content into hard disk storage 39. where it can be later analyzed.
• Fig. 3 shows an array of video sequence characteristic data 40.
• the list comprises image difference measures, as well as image motion vectors. These measures may include properties of the histogram of the difference image. obtained by subtracting two adjacent images, as is known per se. In particular. the "span " ' characteristic data, bed as the difference in gray levels between a high (e.g.. 85) percentile and a low (e.g.. 15) percentile of said histogram, was found to be useful. Alternatively, a measure of difference of intensity histogram of two adjacent images, aiso by a known technique, may be used. Motion vector fields are computed at pre-determined locations while using a block-matching motion estimation algorithm. Alternatively, a more concise representation may consist of camera motion parameters, preferably estimated from image motion vector fields.
• Fig. 4 shows an array of video frame or still image spatial characteristic data.
• Tne list comprises color characteristic data 41.
• texture characteristic data 42 and statistics derived from image regions. Such statistics may include the mean, the variance and the median of luminance values.
• Useful color characteristic data include the first three moments: average, variance and skewness of color components:
• Color spaces of convenience may include the (R.G.B) representation or the (Y.U.V), which provide luminance characteristic data through the Y component.
• Texture provides measures to describe the structural composition, as well as the distribution, of image gray-levels.
• Useful texture characteristic data are derived from spatial gray-level dependence matrices. These include measures such as energy, entropy and correlation.
• characteristic data for a specific application of content verification is important. Texture and color data are important for matching still images. Video frame sequences with significant motion can be aligned by motion characteristic data. For more static sequences, color and texture data can facilitate the alignment process.
• the region of support that is. the image region on which these data are computed
• Using the entire image, or most of it. is preferred when robustness and reduced storage are required.
• deriving multiple characteristics at numerous, relatively small image regions has two important advantages:
• Fig. 5 shows a set of regions 42 in a video frame 43. such that color or texture characteristic data are computed for each such region.
• Fig. 6 illustrates the relative location of graphics sub-pictures with respect to the video frame.
• Number 44 represents a sub-title sub-picture and number 45 represents a menu- item sub-picture.
• Figs. 7 and 8 show the extraction of sub-title characteristic data.
• Sub-titles or closed captions in a movie are used to bring translated dialogues to the viewer. Generally, a sub-title will occupy several dozen frames.
• a suitable form for subtitle characteristic data is time-code-in. time-code-out of that specific sub-title, with additional data describing the sub-title bitmap.
• the sub-title image sequence processor 46 analyses every video frame of the sequence to detect specific frames at which sub-title information is changed. The result is a sequence of sub-title bitmaps, with the frame interval each such bitmap occupies in a time-code-in. time-code-out representation. Characteristic data are then extracted by unit 47 from the sub-title bitmap.
• Fig. 8 shows the sub-title image sequence processor 46.
• the video image passes through a character binarization processor 48. operative to identify' pixels belonging to sub-title characters and paint them white, for example, where the background pixels are painted black.
• the current frame bitmap 49 is compared, or matched, with the stored sub-title bitmap from the first instance of that bitmap.
• the sub-title bitmap is reported with the corresponding time-code interval, and a new matching cycle begins.
• the matching process can be implemented by a number of binary template-matching or correlation algorithms.
• the spatial search range of the template-matching should accommodate mis-registration of a sub-title and additionally the case of scrolling sub-titles.
• the characteristic data of a single sub-title should be concise and allow for efficient matching.
• the sub-title bitmap usually run-length coded, is a suitable representation.
• sub-pictures consist of graphics elements such as bullets, highlight or shadow rectangles, etc.
• Useful characteristic data are obtained by using circle and rectangle detectors.
• Fig. 9 shows a record 50 of sub- pictures characteristic data.
• Fig. 10 shows the derivation of audio characteristic data.
• the signal is digitized by the arrangement comprising an analog anti-aliasing filter 51 and an AT) converter 52 and then filtered by the pre-emphasis filter 53.
• Spectral analysis uses a digital filter bank 54. 54 ' . . .54".
• the filter output is squared and integrated by the power estimation unit 55. 55 ' . . .55 n .
• the set of characteristic data is computed for each video frame duration (40 msec for PAL, or 33.3 msec for NTSC) and stored in store 56. Window duration controls the amount of averaging or smoothing used in power computation. Typically, a 60 or 50 msec window, for an overlap of 33 . can be used.
• the filter bank is a series of linear phase FIR filters, so that the group delay for all filters is zero and the output signals from the filters are synchronized in time.
• Each filter is specified by its center frequency and its bandwidth.
• the reference characteristic data stream is not available explicitly, but has to be derived from said source characteristic data and from playback commands such as denoted in Fig. 1.
• a simple case is when a program consists of consecutive multiple content segments. Each such segment is specified by a source content identifier . a beginning time-code and a ending time-code.
• Said reference characteristic data stream can be constructed or predicted from the corresponding segments of source characteristic data by means of concatenation. If content verification involves computing the actual content segment insertion points, these source characteristic data segments will be padded by characteristic data margins to allow for inaccuracies in insertion.
• the transitions involve not only cuts, but also dissolves or fades.
• some characteristic data can be predicted based on the original source data as well as the blending values. These data include, for example, color moments computed over some region of support. In alignment and verification, the predicted values are compared against the actual values.
• An important step in the verification process is the frame-by-frame alignment of the characteristic data streams.
• the choice of the subset of characteristic data used for alignment is important to the success of that step.
• frame difference measures such as the span described above, are well suited to alignment.
• a coarse-fine strategy is employed, in which anchor frames are used to solve the major time-shift between the content streams. Once that shift is known, fine frame-by- frame alignment takes place.
• An anchor frame is one with an unique structure of characteristic data in its neighborhood.
• Fig. 1 1 shows the selection of anchor frames for coarse alignment.
• the frame difference data for example, the span sequence
• local variance estimation is effected in estimator 57 by means of a sliding window.
• Processors 58 and 59 produce a list of local variance maxima which are above a suitable threshold.
• A. consecutive processing stec in processor 60 estimates the auto-correlation of the candidate anchor frame with its frame difference data neighborhood.
• a further criterion may be used to increase the effectiveness of the alignment step.
• the anchor frames are graded by uniqueness, i.e.. dissimilarity with other anchor frames, to reduce the probability of false matches in the next alignment step.
• Uniqueness is computed by means of cross-correlation between the anchor frame and other anchor frames. By associating the number of anchor frames with a cross-correlation value lower than a specified threshold with the specific anchor frame, those frames with highest uniqueness are selected.
• the matching process can be described as a sequence of edit operators which transform the first interval of frame characteristic data to the second interval.
• the sequence consists of three such operators:
• the fine frame alignment problem has now been transformed to finding a minimum cost sequence of operators which implements the transformation. If m is the length of the first interval and n is the length of the second interval in frames, then the matching problem can be solved in space and time proportional to (m*n). All that remains is to set the respective costs. Deletion and insertion can be assigned a fixed cost each, based on a-prio information on the probability of dropped or surplus frames. Replacement is a distance measure on the characteristic data vector, such as weighted Euclidean distance.
• Fig. 12 shows the alignment of a composite stream with the component reference streams by means of a processor 61 and geometric filter 62.
• sub-title graphics of the language of choice are combined with the video frame sequence. Tne location of sub-titles in the video frame can be specified either manually, in the characteristic data design workstation as described below, or can be automaticallv comDuted. based on anah sis of the sub- title sub-picture stream. For that simple case, video frame verification is done in the image region free from sub-titles. Additionally, sub-title picture verification is done in the sub-title image region.
• a more difficult case is when graphics are overlaid on the video frame, such as in the case of displaying a menu in a DVD player.
• Tne location of menu bullets and text may be. for example, as illustrated in Fig. 6.
• the graphics stream has been pre-processed to extract the graphics regions of support, in the form of bounding rectangles for text lines and graphics primitives. These regions are stored as auxiliary characteristic data.
• the streams can be aligned. Once aligned, the composite frame graphics regions are known to be those of the corresponding graphics stream. Then, based on these regions, only color and texture actual frame characteristic data which are not occluded by overlay graphics [see Fig. 6] are compared with the respective reference data.
• Fig. 13 depicts the frame verification processes performed by the frame characteristic data comparator 32 (Fig. 1). which start from aligned characteristic data streams. It is important to note that the characteristic data alignment processor 30 detects a variety- of content problems. Failure in alignment may be due to the fact that a wrong content stream is playing, or the content stream is severely time-shifted, or the stream is distorted beyond recognition. A successful alignment yields the indices of missing or surplus frames. Once aligned, each actual content frame is compared with the corresponding reference frame, based on the characteristic data. Then for the remaining data, frame-by- frame comparison can take place in processors 63. 64 and 65 and comparators 66 and 67.
• the distance between characteristic data of corresponding frames detects qualit - problems such as luminance or color change, as well as audio distortions.
• qualit - problems such as luminance or color change, as well as audio distortions.
• graphics characteristic data errors in sub-picture content and overlay may be detected.
• characteristic data sensitive to compression artifacts such artifacts can be detected.
• the comparison process requires the notions of distance and threshold.
• vector characteristic data such as color, luminance and audio
• a vector distance measure is used, such as the Mahalonobis distance:
• D (. ⁇ -- N 0 ) r C- , (N' - N' , )
• X r , X a are the reference and actual characteristic data vectors.
• C is the co-variance matrix which models pairwise relationships among the individual characteristic data.
• the proper threshold may be computed at a training phase, using the characteristic data design workstation described hereinafter with reference to Fig. 14.
• Comparator 68 compares blockiness ⁇ characteristic data derived from the reference and actual video frames,_respectively.
• data may include power estimates of a filter designed to enhance an edge grid structure, such as. for example, the grid spacing equals the compression block size, which is usually 8 or 16.
• Bv com ⁇ arin ⁇ these estimates with the reference value, an increase in blockiness may be detected.
• absolute blockiness may be misleading, since it mav orisinate from the original frame texture.
• Comparison of sub-pictures can be done at bitmap level, at the exclusive OR of the corresponding bitmaps, by computing the distance between corresponding shape characteristic data vectors, or by comparing recognized subtitle text strings, where applicable.
• frame-by-frame which is used in conjunction with the comparison process, relates to the fact that once the content streams are aligned, inspection of every frame with the corresponding frame can be done.
• comparison may include all frames or a sub-set of the frames.
• Fig. 14 shows a characteristic data design workstation 69.
• the characteristic data acquisition part of the work-station replicates the reference content processing front-end of Fig. 1.
• workstation 69 has access, by network 70, to the actual content data and not just to the characteristic data, for display at 71 and further analysis at 72.
• the development of the specific content verification application is conducted using an arrangement of a combination of manual semi-automatic and automatic processes.
• the user may specify' the sub-titling type-face and its location in the video frame.
• the user may select several representative content segments and the system then extracts a full characteristic data set. possibly in multiple passes or slower than real-time, ranking their discriminating power over the sample reference content and retaining their best features.
• the method of the invention may further comprise the step of computing actual characteristic data from at least pan of the actual broadcast or playback content streams. It may also comprise the step of computing reference characteristic data from at least pan of said reference content streams.
• Said reference characteristic data may be derived from video frame sequences, still images, audio and graphics, and said actual characteristic data may be derived from a video sequence and an audio channel.
• said video image sequence characteristic data may include an image motion vector field, or data derived from an image difference signal
• said video frame or still image characteristic data may include luminance statistics in predefined regions of said frame or image.
• said video frame or still image characteristic data also include texture characteristic data and/or colour data
• said colour characteristic data include colour moments
• said video frame or still image characteristic data also include a low resolution or highly compressed version of the original image
• said audio characteristic data include audio signal parameters, estimated at a window size which is comparable with video frame duration
• said graphics characteristic data exhibit printed text
• said graphics characteristic data also exhibit common graphics elements, including bullets and hiahliahted rectanales
• said step of predicting may include generating a characteristic data stream from source streams and navigation commands or play-lists , branching from one source stream to another source stream.
• Said step of predicting mav also include generating a characteristic data stream from source streams and transition commands such as cut. dissolve, fade to/from black, or said step may include computing characteristic data of graphics sub-pictures overlay on a video image sequence or still.
• the evaluation of the information content of a cenain frame may be based on the temporal variation of characteristic data in said frame and in its adjacent frames.
• the method may funher comprise grading the information content of all frames in a sequence, denoting frames with locally maximal information content as anchor frames.
• the method may still further comprise evaluating the similarity between two anchor points, based on a measure of temporal correlation between the respective sets of neighbouring characteristic data.
• the method may funher comprise evaluating the similarity between all pairs of anchor frames, such that, for each pair, one frame is from the reference data and the other is from the actual data.
• the method may funher comprise reporting said alignment results, including the time shift between the designed and actual content broadcast-playback, as well as an indication of missing or surplus frames.
• the step of comparing may comprise first aligning the graphics of said composite frame sequence with said reference graphics streams, and the step of aligning may facilitate computing the location of all overlaid graphics in said composite frame sequence.
• the step of computing may facilitate filtering out colour and texture actual frame characteristic data which are occluded by said overlay graphics.
• the method may funher comprise comparing characteristic data of aligned frames to indicate quality or content problems, and said problems may be selected from the group comprising luminance or colour shifts, compression anifacts, audio anifacts. and audio or sub-pictures mismatch or mis-alignment.
• a method for video content verification operative to compare and venfy the content of a first audio-visual stream with the content of a second audio-visual stream, the method comprising the steps of extracting characteristic data from a first audio-visual stream, extracting characteristic data from a second audio-visual stream, and compa ⁇ ng the extracted characteristic data from said first and second audio-visual streams
• step of comparison comprises aligning said first and second audio-visual streams on a frame-by-frame basis, and performing a frame-by-frame comparison of said aligned streams of frames
• said first and second streams are selected from the group comprising the elementary content streams, including video image sequence, audio channel, and sub-picture streams
• a method for video content verification operative to compare and venfy the content of a first audio-visual stream with the content of a second audio-visual stream, wherein said second audio-visual content stream is embedded by at least one source content stream and a set of editing instructions, the method comprising the steps of extracting characteristic data from said first audio-visual stream. extracting characteristic data from said source content stream, and computing characteristic data of said second content-stream, based on characteristic data of said source content stream and on said editing instructions.
• a method as claimed in claim 7, funher comprising the step of predicting the reference characteristic data stream from said reference characteristic data and from playback instructions
• a method as claimed in claim 7, funher comprising aligning the reference characteristic data stream with the actual characteristic data stream, on a frame-by-frame basis, and evaluating the information content of a ce ⁇ ain frame.
• a method as claimed in claim 1 funher compnsing computing the frame-index offset between the reference and actual frames, based on the most likely offsets derived from evaluation of the similarity between all anchor frames
• a method as claimed in claim 1 funher comprising matching the reference frame sequence with the actual frame sequence, based cn an identified frame-index offset, and further comprising the step of designating an actual frame as a surplus frame, or assigning to it a unique reference frame
• a system for audio-visual content verification operative to compare and verify the content of a first audio-visual data stream with the content of a second audio-visual data stream, the system comprising. means for extracting characteristic data from a first audio-visual data stream; means for extracting characteristic data from a second audio-visual data stream; and means for comparing characteristic data of said first and second audio-visual data streams
• comparison means comprises: means for aligning said audio-visual data streams on a frame-by-frame basis; and means for frame-by-frame comparison of said aligned data streams.
• the invention provides a method for video content verification, operative to compare and verify the content of a first audio- visual stream with the content of a second audio-visual stream, comprising the steps of extracting characteristic data from a first audio-visual stream, extracting characteristic data from a second audio-visual stream, and comparing the extracted characteristic data from the first and second audio-visual streams.
• the invention also provides a system for carrvins out the method.
• a system for audio-visual content verification operative to compare and verify t he content of a first audio-visual data stream with the content of a second audio-visual data stream, wherein said second audio-visual data stream is defined by at least one source content data stream and a set of editing instructions, the system comprising: means for extracting characteristic data from said first audio-visual data stream: means for extracting characteristic data from said source content data stream: and means for computing characteristic data of said second content data stream, based on characteristic data of said source content data stream and said editing instructions.
• Fig. 2 40 image sequence characteristic data
• Fig. 1 actual frame reference frame characteristic data sub-picture stream characteristic dats stream
• Fig. 12 color luminance quality quality actual report report blockiness reference characte ⁇ stic blockiness characte ⁇ stic data data

Landscapes

• Engineering & Computer Science (AREA)
• Computer Security & Cryptography (AREA)
• Signal Processing (AREA)
• Television Signal Processing For Recording (AREA)
• Television Systems (AREA)

Applications Claiming Priority (3)

Publications (2)

Family

ID=11070933

Family Applications (1)

Country Status (5)

Families Citing this family (8)

Family Cites Families (9)

• 1997
  • 1997-12-07 IL IL12249897A patent/IL122498A0/xx unknown
• 1998
  • 1998-12-07 CA CA002312997A patent/CA2312997A1/en not_active Abandoned
  • 1998-12-07 WO PCT/IL1998/000596 patent/WO1999030488A1/en not_active Application Discontinuation
  • 1998-12-07 EP EP98959122A patent/EP1046283A4/de not_active Withdrawn
  • 1998-12-07 AU AU15035/99A patent/AU1503599A/en not_active Abandoned

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events